Fix handling when there is no NextRowKey header

[joelverhagen]

According to the Azure Table Storage documentation, the x-ms-continuation-NextRowKey header can be excluded when the x-ms-continuation-NextPartitionKey is present. From my own usage of Azure Table Storage, this seems like a rare case but I encountered it.

I believe it can happen when you have a very large/hot table that is split into multiple physical partitions. In such cases I have performed some range queries on PartitionKey that cause a bug in Azure.Data.Tables. There is improper handling of when the NextRowKey header is missing.

My repro code looks like this, but it only works sometimes (i.e. when the table is very large or active):

var client = new TableServiceClient(connectionString);
var table = client.GetTableClient(tableName);
await table
    .QueryAsync<Azure.Data.Tables.TableEntity>("PartitionKey gt '0'", maxPerPage: 2)
    .AsPages()
    .Take(2)
    .ToListAsync();

This will not repro on an empty or small table, from my tests. I even tried a repro where I put 100,000 entities each with an 8 KiB binary property. This didn't trigger the case. I am able to trigger it somewhat reliably with my ExplorePackages project.

But I don't think a repro is hugely important because the bug is clear from the code. When you split the continuation token:

{pk} {rk}

by space and {rk} is null or an empty string, string.Split returns two pieces, not one.

This causes an exception when the second page is queries. The exception is:

Unhandled exception. Azure.RequestFailedException: Service request failed.
Status: 400 (One of the request inputs is not valid.)

Content:
{"odata.error":{"code":"InvalidInput","message":{"lang":"en-US","value":"One of the request inputs is not valid.\nRequestId:64314db5-9002-0058-2421-24300d000000\nTime:2021-03-28T22:26:33.2582887Z"}}}

Headers:
Server: Windows-Azure-Table/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id: REDACTED
x-ms-client-request-id: 6dbce964-927b-4ad1-a906-2b2e9e89491c
x-ms-version: REDACTED
x-ms-error-code: REDACTED
Date: Sun, 28 Mar 2021 22:26:32 GMT
Content-Length: 199
Content-Type: application/json

   at Azure.Data.Tables.TableRestClient.QueryEntitiesAsync(String table, Nullable`1 timeout, String nextPartitionKey, String nextRowKey, QueryOptions queryOptions, CancellationToken cancellationToken)
   at Azure.Data.Tables.TableClient.<>c__DisplayClass33_0`1.<<QueryAsync>b__1>d.MoveNext()
--- End of stack trace from previous location ---
   at Azure.Core.PageableHelpers.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+MoveNext()
   at Azure.Core.PageableHelpers.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at System.Linq.AsyncEnumerablePartition`1.ToListAsync(CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerablePartition.cs:line 348
   at System.Linq.AsyncEnumerablePartition`1.ToListAsync(CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerablePartition.cs:line 353
   at ConsoleApp4.Program.NewSdk(String connectionString, String tableName) in C:\Users\jver\Desktop\ConsoleApp4\ConsoleApp4\Program.cs:line 36
   at ConsoleApp4.Program.Main(String[] args) in C:\Users\jver\Desktop\ConsoleApp4\ConsoleApp4\Program.cs:line 17
   at ConsoleApp4.Program.<Main>(String[] args)

The response from the first request looks like this (notice no row key response header):

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8
Server: Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 64314dad-9002-0058-1f21-24300d000000
x-ms-client-request-id: db153f50-ed2e-42f5-8f84-58492de4936e
x-ms-version: 2019-02-02
X-Content-Type-Options: nosniff
x-ms-continuation-NextPartitionKey: 1!84!MDg1ODU4NDY0NzE0MTI5ODM1ODUtM3V0eGkzNjZ6ZmJlZmNsbGNkd3h2N2dqMzQtYXBhbGxhLmJ1YnUudXdw
Date: Sun, 28 Mar 2021 22:26:32 GMT
Content-Length: 135

{"odata.metadata":"https://REDACTED.table.core.windows.net/$metadata#catalogleafscans3utxi366zfbefcllcdwxv7gj34","value":[]}

The second request looks like this (notice an empty string row key header):

GET https://REDACTED.table.core.windows.net/catalogleafscans3utxi366zfbefcllcdwxv7gj34()?$format=application%2Fjson%3Bodata%3Dminimalmetadata&$top=2&$filter=PartitionKey%20gt%20%270%27&NextPartitionKey=1%2184%21MDg1ODU4NDY0NzE0MTI5ODM1ODUtM3V0eGkzNjZ6ZmJlZmNsbGNkd3h2N2dqMzQtYXBhbGxhLmJ1YnUudXdw&NextRowKey= HTTP/1.1
Host: REDACTED.table.core.windows.net
x-ms-version: 2019-02-02
DataServiceVersion: 3.0
Accept: application/json;odata=minimalmetadata
x-ms-client-request-id: 6dbce964-927b-4ad1-a906-2b2e9e89491c
x-ms-return-client-request-id: true
User-Agent: azsdk-net-Data.Tables/12.0.0-beta.6 (.NET 5.0.4; Microsoft Windows 10.0.19042)
x-ms-date: Sun, 28 Mar 2021 22:26:32 GMT
Authorization: SharedKeyLite REDACTED

The error response is:

HTTP/1.1 400 One of the request inputs is not valid.
Content-Length: 199
Content-Type: application/json
Server: Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 64314db5-9002-0058-2421-24300d000000
x-ms-client-request-id: 6dbce964-927b-4ad1-a906-2b2e9e89491c
x-ms-version: 2019-02-02
x-ms-error-code: InvalidInput
Date: Sun, 28 Mar 2021 22:26:32 GMT

{"odata.error":{"code":"InvalidInput","message":{"lang":"en-US","value":"One of the request inputs is not valid.\nRequestId:64314db5-9002-0058-2421-24300d000000\nTime:2021-03-28T22:26:33.2582887Z"}}}

All SDK Contribution checklist:

This checklist is used to make sure that common guidelines for a pull request are followed.

• Please open PR in Draft mode if it is:
  • Work in progress or not intended to be merged.
  • Encountering multiple pipeline failures and working on fixes.
• If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
• I have read the contribution guidelines.
• The pull request does not introduce breaking changes.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

SDK Generation Guidelines

• The generate.cmd file for the SDK has been updated with the version of AutoRest, as well as the commitid of your swagger spec or link to the swagger spec, used to generate the code. (Track 2 only)
• The *.csproj and AssemblyInfo.cs files have been updated with the new version of the SDK. Please double check nuget.org current release version.

Additional management plane SDK specific contribution checklist:

Note: Only applies to Microsoft.Azure.Management.[RP] or Azure.ResourceManager.[RP]

• Include updated management metadata.
• Update AzureRP.props to add/remove version info to maintain up to date API versions.

Management plane SDK Troubleshooting

• If this is very first SDK for a services and you are adding new service folders directly under /SDK, please add new service label and/or contact assigned reviewer.
• If the check fails at the Verify Code Generation step, please ensure:
  • Do not modify any code in generated folders.
  • Do not selectively include/remove generated files in the PR.
  • Do use generate.ps1/cmd to generate this PR instead of calling autorest directly.
    Please pay attention to the @microsoft.csharp version output after running generate.ps1. If it is lower than current released version (2.3.82), please run it again as it should pull down the latest version,

Old outstanding PR cleanup

Please note:
If PRs (including draft) has been out for more than 60 days and there are no responses from our query or followups, they will be closed to maintain a concise list for our reviewers.

[ghost]

Thank you for your contribution @joelverhagen! We will review the pull request and get back to you soon.

[joelverhagen]

I was able to verify the fix by making a project reference to my cloned version of Azure.Data.Tables.csproj. I couldn't figure out how to run dotnet pack since it kept breaking on a CHANGELOG validation.

[jsquire]

Hi @joelverhagen. Thank you for your continued efforts to help improve the Tables experience. I've looped @christothes in for his feedback.

[christothes]

Thanks for this PR! This is a great find.

I just left a few minor comments.

[christothes]

👍