[Fix] Decouple OAuth functionality from Config

[mgyucht]

Changes

OAuth Refactoring

Currently, OAuthClient uses Config internally to resolve the OIDC endpoints by passing the client ID and host to an internal Config instance and calling its oidc_endpoints method. This has a few drawbacks:

1. There is nearly a cyclical dependency: Config depends on methods in oauth.py, and OAuthClient depends on Config. This currently doesn't break because the Config import is done at runtime in the OAuthClient constructor.
2. Databricks supports both in-house OAuth and Azure Entra ID OAuth. Currently, the choice between these options depends on whether a user specifies the azure_client_id or client_id parameter in the Config. Because Config is used within OAuthClient, this means that OAuthClient needs to expose a parameter to configure either client_id or azure_client_id.

Rather than having these classes deeply coupled to one another, we can allow users to fetch the OIDC endpoints for a given account/workspace as a top-level functionality and provide this to OAuthClient. This breaks the cyclic dependency and doesn't require OAuthClient to expose any unnecessary parameters.

Further, I've also tried to remove the coupling of the other classes in oauth.py to OAuthClient. Currently, OAuthClient serves both as the mechanism to initialize OAuth and as a kind of configuration object, capturing OAuth endpoint URLs, client ID/secret, redirect URL, and scopes. Now, the parameters for each of these classes are explicit, removing all unnecessarily coupling between them. One nice advantage is that the Consent can be serialized/deserialized without any reference to the OAuthClient anymore.

There is definitely more work to be done to simplify and clean up the OAuth implementation, but this should at least unblock users who need to use Azure Entra ID U2M OAuth in the SDK.

Tests

The new OIDC endpoint methods are tested, and those tests also verify that those endpoints are retried in case of rate limiting.

I ran the flask app example against an AWS workspace, and I ran the external-browser demo example against AWS, Azure and GCP workspaces with the default client ID and with a newly created OAuth app with and without credentials.

• make test run locally
• make fmt applied
• relevant integration tests applied

[github-actions]

This PR breaks backwards compatibility for databrickslabs/blueprint downstream. See build logs for more details.

_{Running from downstreams #551}

[github-actions]

This PR breaks backwards compatibility for databrickslabs/ucx downstream. See build logs for more details.

_{Running from downstreams #551}

[hectorcast-db]

This changes public classes/methods and introduces backward incompatible changes.
Any way we can minimize the impact for the customers?

[mgyucht]

I'll list the breaking changes here and we can discuss whether the mitigation is sufficient:

• OAuthClient constructor: I have to break the constructor definition because the OAuthClient doesn't need to know the workspace host but needs to know the OIDC endpoints. To mitigate this, I've added from_host() as a static method that preserves the original behavior.
• TokenCache constructor: Previously, the OAuthClient was used as a configuration class, but now users will have to pass in the parameters one at a time. Importantly, because OAuthClient doesn't depend on host anymore, the TokenCache needs a host parameter. In all honesty, this should really be a parameter to load.
• external_browser behavior: previously, it would use a multi-tenant app that we published in our account: 6128a518-99a9-425b-8333-4cc94f04cacd. Now that all customers have access to the databricks-cli app in their account by default, we can use that globally as a default, even in GCP.

Otherwise, it's a confusing pattern for the token to contain a reference to the client which refreshes it. That's part of the reason for the spaghetti architecture where parameters like client ID and token endpoint need to be passed through many layers of classes to be used. I would rather not try to fix these things in this PR (and probably should save them for the next SDK version).

We can include the following in the changelog for this PR in the release:

Breaking Changes

• external_browser now uses the databricks-cli app instead of the third-party "6128a518-99a9-425b-8333-4cc94f04cacd" application when performing the U2M login flow for Azure workspaces when a client ID is not otherwise specified. This matches the AWS behavior.
• The signatures of several OAuth-related constructors have changed to support U2M OAuth with Azure Entra ID application registrations. See https://github.com/databricks/databricks-sdk-py/blob/main/examples/flask_app_with_oauth.py for examples of how to use these classes.
  • OAuthClient(): renamed to OAuthClient.from_host()
  • SessionCredentials() and SessionCredentials.from_dict(): now accepts token_endpoint, client_id, client_secret, and refresh_url as parameters, rather than accepting the OAuthClient.
  • TokenCache(): now accepts host, token_endpoint, client_id, client_secret, and refresh_url as parameters, rather than accepting the OAuthClient.