[23.6.0] vault-agent - http2: invalid Upgrade request header: ["h2c"]

[gg718]

Just upgraded to Web3Signer v23.6.0 and started seeing this error in my vault-agent proxy logs:

2023-06-19T10:30:06.722Z [ERROR] agent.apiproxy.client: request failed: error="Get \"https://vault.example.com/v1/secret/data/goerli/12345\": http2: invalid Upgrade request header: [\"h2c\"]" method=GET url=https://vault.example.com/v1/secret/data/goerli/12345

This error was not happening in v23.3.1 so seems to be a new bug.

Config example:

keyName: key
keyPath: /v1/secret/data/goerli/12345
keyType: BLS
serverHost: localhost
serverPort: 8200
timeout: 30000
tlsEnabled: false
token: unused_handled_by_vault_agent
type: hashicorp

Where localhost:8200 is the local port my Vault Agent is listening on.

[usmansaleem]

@gg718 Thank you for reporting this, our acceptance test run against Vault binary directly hence we didn't observe this issue. We have changed the underlying http library from vertx to standard Java. Can you provide further information in order for us to reproduce this issue:

1. Have you configured TLS in vault agent?
2. Any other special configuration for vault agent that would help us in reproduce this?
3. Can Vault-Agent be configured to disable auto h2c upgrade when in non-TLS mode?

[usmansaleem]

@gg718 I tried to reproduce this issue using docker compose but unable to reproduce it. I setup non-tls mode i.e. w3s -> agent -> vault (all non-tls) with api_proxy { use_auto_auth_token = "force" } so that w3s doesn't need to specify the actual auth token.

See https://github.com/usmansaleem/signers_docker_compose/tree/main/web3signer-hashicorp-agent. The vault agent configuration file is here.

The error you are observing is most likely happening because of some configuration in vault agent (or vault) which is resulting in failure of Upgrade: h2c header. The other problematic thing is the URL scheme as https://..... Is there any other API gateway/proxy in front of vault agent?

We have few solutions that we can implement on w3s side (assuming that it can't be solved in vault or vault-agent configuration):

1. Force w3s to always use HTTP/1.1 instead of HTTP/2 (which is now the default due to change in underlying http client library). This would mean we don't get additional performance benefits of HTTP/2 when talking to same host.
2. Force w3s to use HTTP/1.1 in non-tls mode while use default HTTP/2 in TLS mode.
3. Default to HTTP/2 (as it is more performant), but provide an option in w3s hashicorp configuration file to override the http version to HTTP/1.1. The main problem here is to bulk update of existing hashicorp files.

We are more inclined to implement the third option as a workaround.

Let me know your thoughts.

[gg718]

@usmansaleem Here are the relevant bits from the Vault Agent config file:

vault {
  address = "https://vault.example.com" // load balancer endpoint
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = true
}

cache {
  use_auto_auth_token = "force"
}

Note that I am using Vault Agent v1.13.4 as a proxy, not the new Vault Proxy feature released in Vault v1.14.0.

And yes, I agree - I think the third option makes the most sense. 👍

---

Also, would it be possible to make token optional? Right now I am having to specify a bogus value to appease Web3Signer.

[usmansaleem]

@gg718 What kind of load balancer you are using? nginx or some other software? Is it possible that there is a misconfigure HTTP/2 on load balancer?