feat: implement Ory Hydra v2.0

[aeneasr]

To Do

1. Update OAuth2 blog post https://github.com/ory/web/blob/master/src/markdown/blog/run-oauth2-server-open-source-api-security.mdx
2. In docs, search for hydra clients create commands and remove --id flag and rename --callback to --redirect-uris
3. Update --dangerous-force-http in docs
4. Benchmark script uses fixed --id flag for hydra clients create, needs to be updated and also checked whether it still works with PBKDF2
5. Update docs and add PBKDF2
6. Update all CLI commands
7. breaking changes note for CLI commands
8. is the flush endpoint documented anywhere?
9. Cookies need unqiue name per host
10. SDK and routing
11. Update node UI to new SDK
12. In conformity docker compose, remove admin suffix from the SDK config for the consent UI once point 11 is done
13. chore: regenerate pre-v2.x release hydra-client-go#13
14. Probably upgrade go module for hydra-client-go to v2?
15. e2e tests for hydra are exiting unexpectedly
16. remove ory/x mod replace

Changes Requiring Attention

Changes you must know about!

Cookies

New cookie configuration options have been introduced, allowing a higher degree of control:

serve:
  cookies:
    same_site_mode: Lax
    same_site_legacy_workaround: false
    domain: example.com
    names:
      login_csrf: ory_hydra_login_csrf
      consent_csrf: ory_hydra_consent_csrf
      session: ory_hydra_session

The default names of cookies have changed:

- oauth2_authentication_csrf
+ ory_hydra_login_csrf
- oauth2_consent_csrf
+ ory_hydra_consent_csrf
- oauth2_authentication_session
+ ory_hydra_session

Use the new configuration option to change the cookie names back to v1.x if required:

serve:
  cookies:
    names:
      login_csrf: oauth2_authentication_csrf
      consent_csrf: oauth2_consent_csrf
      session: oauth2_authentication_session

PBKDF2 and Bcrypt Client Secret Hashing

This release adds support for hashing OAuth2 Client Secrets using pbkdf2 instead of bcrypt, which might be a more appropriate algorithm in certain settings. As we assume that most environments fall in this category, we also changed the default to pbkdf2 with 25.000 rounds (roughly 1-3ms per hash on an Apple M1 Max core).

High hash costs are needed when hashing user-chosen passwords, as users often reuse passwords across sites. A high hash cost will make it much harder for the attacker to guess the user-chosen password and try using it on other sites (e.g. Google).

As most client secrets are auto-generated, using high hash costs is not useful. The password (OAuth2 Client Secret) is not user chosen and unlikely to be reused. As such, there is little point in using excessive hash costs to protect users. High hash costs in a system like Ory Hydra will cause high CPU costs from mostly automated traffic (OAuth2 Client interactions). It has also been a point of critizism from some who wish for better RPS on specific endpoints.

Other systems like Keycloak do not hash client secrets at all, referencing more secure authentication mechanisms such as assertion-based client authentication.

We and the IETF disagree though, as rfc6819#section-5.1.4.1.3 states:

The authorization server should not store credentials in clear text.
Typical approaches are to store hashes instead or to encrypt
credentials. If the credential lacks a reasonable entropy level
(because it is a user password), an additional salt will harden the
storage to make offline dictionary attacks more difficult.

For that reason, cleartext storage of client secrets is not going to be supported.

BREAKING CHANGE: This change is backwards compatible, but changes the default hashing algorithm to PBKDF2. To keep using BCrypt for hashing new OAuth2 Client Secrets set the following configuration option in your configuration file:

oauth2:
  hashers:
    algorithm: bcrypt

Token Prefixes

This release adds token prefixes to access tokens (ory_at_), refresh tokens (ory_rt_), and authorize codes (ory_ac_). Token prefixes are useful when scanning for secrets in e.g. git repositories. Token prefixes are only use for opaque tokens, not JSON Web Tokens as that would invalidate the JSON Web Tokens.

Tokens issued before this change will continue working!

OAuth2 Client ID Generated

It is no longer possible to set arbitrary OAuth 2.0 Client IDs. Instead, OAuth 2.0 Client IDs are generated by Ory Hydra. Clients created before the 2.0 release will continue to work with their legacy IDs.

This change was made to improve scalability of the number of clients in distributed databases such as CockroachDB. Ory Hydra v3.0 will most likely remove support for these legacy IDs. Please start migrating your users to new OAuth2 Clients.

New --dev mode

To better support local development, a new flag --dev has been added. With --dev it is possible to use to use http and localhost URLs. If --dev is not set, URLs (e.g. the consent url) must be https and non-localhost.

iss no longer has a forced trailing slash

The iss (issuer) value no longer appends a trailing slash but instead uses the raw value set in the config.

Setting

urls:
  self:
    issuer: https://auth.example.com

has changed

-  "iss": "https://auth.example.com/"
+  "iss": "https://auth.example.com"

To set a trailing slash make sure to set it in the config value:

urls:
  self:
    issuer: https://auth.example.com/

Database schema

This release comes with significant changes to the database schema, which affect most tables in Hydra. Some of the notable changes include:

• Replacing the four tables used to hold different states of an authentication flow with one unified table.
• Replacing the last remaining autoincrement primary keys with UUIDs for the hydra_client and hydra_jwk tables.
• Changing the way we encode string slices from pipe-separated values to JSON arrays.

We've also upgraded the databases used in tests and examples to the following versions:

• MySQL from 5.7 to 8.0
• postgres from 9.6 to 11.8
• CockroachDB from 20.2 to 22.1

Config hot reloading

Ory Hydra is now capable of hot reloading any OAuth2-related configuration setting and no longer needs to be restarted to change certain values (e.g. the access token strategy).

TLS

The CLI flag --dangerous-force-http has been removed. To control TLS, use the TLS configuration instead:

serve:
  tls:
    # Defaults to **false**!
+   enabled: true
    # ...
  public:
    tls:
      # Overrides serve.tls.enabled
+     enabled: true
      # ...
  admin:
    tls:
      # Overrides serve.tls.enabled
+     enabled: true
      # ...

Please keep in mind that TLS is disabled by default as we expect most deployments to run behind a proxy which does TLS termination.

TLS termination requirement

The TLS Termination Middleware Requirement has been removed. The TLS Termination configuration flags have no effect any more and can be removed.

JSON Web Key generation

Previously, JSON Web Keys were generated in public and private key pairs with public: and private: prefixes. This practice has been removed. Generating new JSON Web Key will from now on return one key in the JSON Web Key Set

{
  "keys": [
-   {
-     "kid": "private:some-id"
-   },
-   {
-     "kid": "public:some-id"
-   }
+   {
+     "kid": "some-id"
+     // Contains both private and public key, and exposes the correct one depending on context automatically.
+   }
  ]
}

Deployments with keys generated in Ory Hydra v1.x will continue to work!

Admin routes

To match compatibility with other Ory services, we have moved all admin APIs under the /admin prefix on the admin port. There are redirects set up, but we encourage to upgrade the SDK and paths in your system to match the new prefix.

Breaking changes

Changes which require code adjustments.

CLI

The CLI has been reworked and now follows the structure of Ory Keto, Ory Kratos, and the Ory CLI. Mostly, commands and flags have been renamed, and support for output formats and STDIN have been added. Please head over to the CLI documentation to get a sense of the new Ory CLI.

To highlight the changes, here are some examples:

- hydra clients create --grant-types implicit --grant-types authorize_code
+ hydra create client --grant-type implicit --grant-types authorize_code --format json-pretty --endpoint https://<project>.projects.oryapis.com

- hydra token user ...
+ hydra perform authorization-code --format json ...

- hydra token client ...
+ hydra perform client-credentials --format json ...

Please note that all environment variables named HYDRA_ADMIN_URL and HYDRA_URL have been renamed and unified under ORY_SDK_URL

HTTP Flush Endpoint Removed

The HTTP Endpoint /oauth/flush has been removed. The command was not useful and had a tendency to time out. Please use the hydra janitor command instead.

SDKs

Ory Hydra 2.0 moved to the OpenAPI spec for SDK generation. API methods and payload names also were updated to match other services in the ecosystem. Please check the respective SDK code documentation when upgrading.

[codecov]

Codecov Report

Merging #2796 (b260c8f) into master (ac61740) will decrease coverage by 2.54%.
The diff coverage is 62.63%.

@@            Coverage Diff             @@
##           master    #2796      +/-   ##
==========================================
- Coverage   79.26%   76.72%   -2.55%     
==========================================
  Files         111      123      +12     
  Lines        8107     8846     +739     
==========================================
+ Hits         6426     6787     +361     
- Misses       1263     1634     +371     
- Partials      418      425       +7     

Impacted Files	Coverage Δ
cmd/cli/error.go	0.00% <0.00%> (-17.65%)	⬇️
cmd/cli/handler_migrate.go	1.04% <0.00%> (-2.04%)	⬇️
cmd/migrate.go	100.00% <ø> (ø)
cmd/server/helper_cert.go	13.95% <0.00%> (-34.62%)	⬇️
consent/subject_identifier_algorithm_pairwise.go	37.50% <0.00%> (ø)
driver/config/helper.go	33.33% <ø> (-16.67%)	⬇️
driver/config/provider.go	82.97% <ø> (-2.37%)	⬇️
driver/config/provider_fosite.go	90.24% <ø> (ø)
driver/config/serve.go	92.59% <ø> (ø)
driver/config/tls.go	85.71% <ø> (-14.29%)	⬇️
... and 119 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[grantzvolsky]

@aeneasr We user swagger:ignore here, but sqlxx uses a different swagger definition. Can you think of any way to depend on sqlxx's implementation while keeping our swagger:ignore?