[BUG] - Nebari Upgrade Keycloak Client Cannot Connect if initial_root_password changed

[kenafoster]

Describe the bug

When upgrading to Nebari 2024.11.1rc1, there is a step where the upgrade command will try to create a new client role and assign groups in Keycloak. It authenticates by accessing security.keycloak.initial_root_password from Nebari config

However, it is a best practice in a production environment to change that value in the config file after the initial deployment... otherwise a secret would be committed to CICD

The resulting error is:

Would you like Nebari to assign the corresponding role/scopes to all of your current groups automatically? [y/N] (N): y
...
ValueError: Failed to connect to Keycloak server: 401: b'{"error":"invalid_grant","error_description":"Invalid user credentials"}'

Expected behavior

If you choose "Y" to assign corresponding roles scopes, the upgrade step should succeed even if your Keycloak root PW isnt' saved in plain text

OS and architecture in which you are running Nebari

Mac Sequoia / Apple silicon

How to Reproduce the problem?

Start with Nebari 2024.7.1 and security.keycloak.initial_root_password should no longer be valid (i.e. you have manually reset it from within the Keycloak web app)

When you install Nebari 2024.11.1rc1 and run Nebari upgrade, choose 'Y' at the prompt to create new client role and assign groups for shared directory access

Keycloak client fails to authenticate

Command output

Would you like Nebari to assign the corresponding role/scopes to all of your current groups automatically? [y/N] (N): y
...
ValueError: Failed to connect to Keycloak server: 401: b'{"error":"invalid_grant","error_description":"Invalid user credentials"}'

Versions and dependencies used.

Nebari 2024.11.1rc1

Compute environment

AWS

Integrations

No response

Anything else?

Probably, prompting the user to enter valid Keycloak credentials in the upgrade CLI is a good solution

In one of our implementations, we keep the valid Keycloak root password in AWS Secret Manager, and from there it could be accessed programmatically with valid AWS credentials, and then used to generate an oAuth token to talk to Keycloak's REST API. This could be another option, but it would need cloud specific implementations and also a clear administrative workflow on when/where/how to update the Keycloak password secret (and be sure the AWS secret stays in sync with the actual password value). So probably user prompting is easier...

Here's some snippets from a script which uses this approach:

NEBARI_CONFIG=${DIR}/../nebari-config.yaml

PROJECT_NAME=$(yq '.project_name' ${NEBARI_CONFIG})
REGION=$(yq '.amazon_web_services.region' ${NEBARI_CONFIG})

KC_SECRET_JSON=$(aws secretsmanager get-secret-value --secret-id ${PROJECT_NAME}/keycloak/admin --region ${REGION} | jq -rc '.SecretString')
KC_USERNAME=$(echo ${KC_SECRET_JSON} | jq -rc '.username')
KC_PASSWORD=$(echo ${KC_SECRET_JSON} | jq -rc '.password')
KC_URL=https://$(yq '.domain' ${NEBARI_CONFIG})/auth

if [ "${1}" == "" ]; then
  echo "ERROR: yaml file path required"
  exit 1
fi

IMPORT_YAML=$(realpath ${1})

echo "Importing users from ${IMPORT_YAML}..."

for username in $( yq '.[] | key' ${IMPORT_YAML} ); do
  first=$( yq ".\"${username}\".first" ${IMPORT_YAML} )
  last=$( yq ".\"${username}\".last" ${IMPORT_YAML} )
  groups=$( yq -I0 -M -o=json ".\"${username}\".groups" ${IMPORT_YAML} )

  ACCESS_TOKEN=$(curl -s -X POST -H 'Content-Type: application/x-www-form-urlencoded' --data "username=${KC_USERNAME}&password=${KC_PASSWORD}&grant_type=password&client_id=admin-cli" ${KC_URL}/realms/master/protocol/openid-connect/token | jq -rc '.access_token')
  GET_USER_RESP=$(curl -s -H "Content-Type: application/json" -H "Authorization: Bearer ${ACCESS_TOKEN}" ${KC_URL}/admin/realms/nebari/users?username=${username}&exact=true)

[marcelovilla]

Thanks for bringing this up @kenafoster.

We discussed this issue yesterday and @dcmcand proposed that we add the option to read Keycloak's root password from an environment variable for this particular upgrade step. We think that's a reasonable approach and would solve the issue for all users that are not storing the password on their config, regardless of what cloud provider they're using.

@viniciusdc is this something you can take on?

[viniciusdc]

This will be addressed by #2843 by allowing the user to set the correct password via env vars, a better error message will let the user know about the problem and solution as well.

[kenafoster]

I manually tested the fix in #2843 with version 2024.11.1rc2. LGTM