ChangeFeedClient.list_changes() causes InvalidQueryParameterValue error

[sylnsr]

• azure-storage-blob-changefeed:
• 12.10.0b3:
• OSX:
• 3.9:

I'm using the sample code to list all change feed events. My code is almost the same, verbatim, save the fact that I need to workaround for issue 12568. Following the sample code you provide, I get this error:

Exception: HttpResponseError: (InvalidQueryParameterValue) Value for one of the query parameters specified in the request URI is invalid.
RequestId:2bc2b517-f01f-0001-67cb-55e1c0000000
Code: InvalidQueryParameterValue

To Reproduce
Call this code to list all change events .. based on the sample code provided example, but using the workaround for issue 12568

    cf_client = ChangeFeedClient(
        account_url = "https://{}.dfs.core.windows.net".format(STORAGE_ACCOUNT),
        # bug work around - see: https://github.com/Azure/azure-sdk-for-python/issues/12568#issuecomment-661678790
        credential={"account_name": STORAGE_ACCOUNT, "account_key": KEY_STR}
    )

    change_feed = cf_client.list_changes()   # breaks here

Expected behavior
According to the documentation, we should be able to get a result without any error. I mean, its only 2 function calls at this point to get it to break .... what could a user be doing wrong at that point?

Both my STORAGE_ACCOUNT and KEY_STR values are correct since I use them with azure-storage-file-datalake==12.5.0 without any problems.

Additional context
My requirements. file: req.txt

[rakshith91]

@sylnsr Thanks for the issue. We will take a look asap

[jalauzon-msft]

Hi @sylnsr Mike, I believe your issue might be caused by the fact that you are using the dfs endpoint for account URL. I believe we only support ChangeFeed on the Blob endpoint so you should provide the blob endpoint for your account when constructing the client. This matches the URL used in our samples. Additionally, I don't think you should need the workaround with credential in this case since you are using a standard account URL?

So, can you try something like this?

cf_client = ChangeFeedClient(
    account_url = "https://{}.blob.core.windows.net".format(STORAGE_ACCOUNT),
    credential=KEY_STR
)

change_feed = cf_client.list_changes() 

Hopefully that helps, thanks!

[navba-MSFT]

@sylnsr Thanks for reaching out to us. I hope you had a chance to follow the above suggestion provided by @jalauzon-msft.

Only standard general-purpose v2, premium block blob, and Blob storage accounts can enable the change feed. Accounts with a hierarchical namespace enabled are not currently supported. More Info here and here.

# [START create_change_feed_client]
   cf_client = ChangeFeedClient("https://{}.blob.core.windows.net".format(self.ACCOUNT_NAME),
                                credential=self.ACCOUNT_KEY)
   # [END create_change_feed_client]

   change_feed = cf_client.list_changes(results_per_page=10).by_page()

   # print first page of events
   change_feed_page1 = next(change_feed)
   for event in change_feed_page1:
       print(event)

   # print second page of events
   change_feed_page2 = next(change_feed)
   for event in change_feed_page2:
       print(event)

Python Sample code here.

[sylnsr]

My "Account Kind" in the console currently shows StorageV2 (general purpose v2). I believe that's the same as what you're referring to as "standard general-purpose v2". I do see that in the URL, you're using blob where I am using dfs. Is that actually what you are referring to? Must be because that change prevents the error.

So this presents the core issue for devs that run into this: The end point URL is not supported as specified. Can we at least get a more intuitive error message then please. The current error is a red-herring. The current error message is:
Value for one of the query parameters specified in the request URI is invalid. when it should indicate what the real problem is. Maybe something like: "URL not supported in current format. Please see <url-to-supported-...>".

So other than ".blob.core.windows.net" .. what other URL formats work then? Where is the documentation to that? i.e. .. "blob" and what else? Thanks

[jalauzon-msft]

Hi @sylnsr Mike, you are correct that StorageV2 (general purpose v2) is the correct account type however what I believe @navba-MSFT was pointing out was that Change Feed is not supported for account with hierarchical namespace enabled. This is a property on a Storage account that can be seen on the home screen for an account:

This was mentioned since you were trying to use the dfs URL which is typically used for hierarchical namespace accounts. So, changing your URL to use blob will fix the error but you may not get full change feed support on your account if hierarchical namespace is enabled. This support is documented here. In either case, only the ".blob.core.windows.net" account URLs are supported since only the blob service is supported.

As for the error message, I agree that is fairly misleading, but it is also a tricky one for the service since the requests for ChangeFeed are just standard List/Get requests for the ChangeFeed blobs but the dfs endpoint interprets these requests a little differently and therefore throws the error. We can discuss improving this error message with the service team. Thanks for the feedback.

[sylnsr]

Say whaaaaaaaaat?. If that part of the URL is immutable, then its absurd that its part of a param value. The logical thing to do in that method, would be to take ONLY the storage account name as the param then simply do:

"https://{}.blob.core.windows.net".format(ACCOUNT_NAME_PARAM)

..inside the method in behalf of the caller.

The current design choice for that method is perplexingly illogical, un-intuitive and anti-pattern because it implies some variability in the URL.

I realize there is a whole disparate set of documentation that could technically be woven together over time to start making sense of this lunacy, but that's not productive for anyone. That method really needs to be changed as well.

[jalauzon-msft]

@sylnsr Mike, sorry, I was simplifying a bit. We do need to support other account URLs such as custom domains, CDNs, static websites, as well as some other upcoming features.. These would also work as long as they point to the blob endpoint. So we need to keep support for mutable URLs.

[sylnsr]

OK, that makes a lot more sense. So yeah, if we could just get that error message improved, that would be great.

[jalauzon-msft]

We will talk with the service team about trying to catch this specific case and put in a better error message, but it may be technically difficult. We're also reluctant to put in a client-side check here in case the service team adds support for this in the future.

Going to close this issue though, as it seems the primary issue has been resolved. Thanks