RFD - Nebari Permissions Model

[aktech]

Status	Accepted ✅
Author(s)	@aktech
Date Created	04-04-2024
Date Last updated	10-04-2024
Decision deadline	30-04-2024

Summary

Nebari doesn't have a proper RBAC model yet. As a consequence, providing fine grained control of access to Nebari's services to users and groups is not possible. This poses a risk of the user might inadvertently accessing and modifying any data or service within Nebari that violates the principle of least privilege.

User benefit

Role Based Access Control (RBAC) in Nebari will provides fine grained control of access to Nebari’s services.

Design Proposal

Current Permissions model in Nebari

To understand the proposal for the new RBAC, its important to understand current permissions model, we will go through this very briefly here. Also see nebari-dev/nebari#2304 for more context.

JupyterHub

RBAC came in JupyterHub in 2.x and we upgraded from 1.5 in Aug, 2023:

• Upgrade to JupyterHub 3.1.1 nebari-docker-images#85
• K8s 1.25 + More Improvements nebari#1856

This means we never got to implement JupyterHub's RBAC and have our limited in-house baked permissions model.

Which is basically getting server options for the given user from keycloak

Only two levels of permissions available at this point:

• jupyterhub_developer
• jupyterhub_admin

If the user is not in any group, the user will get 403 on accessing Nebari.

Conda Store

At the moment we have the following roles:

• conda_store_superadmin
• conda_store_admin
• conda_store_developer
• conda_store_viewer

This is set in conda-store configuration:

c.CondaStoreServer.authentication_class = KeyCloakAuthentication

The KeyCloakAuthentication class fetches the user data from keycloak
via keycloak's conda-store client and finds the roles user have and based on that, it returns user's role binding such that user have corresponding permissions on conda-store.

This also fetches user's groups and creates conda-store namespaces (adding them to the conda-store db).

Grafana

Nebari has following Grafana roles, which in code maps to corresponding Grafana roles (1-to-1 mapping):

Nebari Roles	Grafana roles
grafana_admin	Admin
grafana_developer	Editor, Viewer
grafana_viewer	Viewer

Dask

• dask_gateway_admin
• dask_gateway_developer

This uses NebariAuthentication(JupyterHubAuthenticator) to define custom authentication, which checks if either of the above are present on user's roles

If none of these are present, user has no access to create dask clusters.
This makes calls to JupyterHub's API to get user roles and groups from the following keys in JupyterHub's /user endpoint.

• auth_state.oauth_user.roles
• auth_state.oauth_user.groups

Argo

Argo has following roles:

• argo-admin
• argo-developer
• argo-viewer

Three k8s service accounts are created with the above three levels of permissions and based on that user is assigned permissions. These roles are assigned in keycloak.

Problems

• Fine grained access cannot be provided to users/groups.
• Roles are static. New roles cannot be created and existing roles cannot be updated (In theory they can be updated and created, but it will not have the desired effect on Nebari services)

Idea

The idea is to not re-invent the wheel but rather try to use existing RBAC frameworks wherever possible with little or no modifications to support wide range of fine-grained control. We will use JupyterHub's RBAC as a motivation to implement RBAC in Nebari. We'll also try to use similar conventions to avoid confusion and reduce the learning curve of yet another rbac system.

JupyterHub RBAC: brief overview

Read more about it here: https://jupyterhub.readthedocs.io/en/latest/rbac/index.html

JupyterHub defines the following fundamental concepts:

• Group / User: single or a set of users
• Role: Roles are collections of scopes.
• Scope: Permission - specific permissions used to evaluate API requests.

Groups/Users are assigned Roles which are a collection of scopes.

Nebari RBAC: Proposal

The idea is to be able to manage roles and permissions from a central place, in this case keycloak. An admin or anyone who has permissions to create a role in keycloak will create role(s) with assigned scopes (permissions) to it and attach it to user(s) or group(s). We define the following concepts (some of them already exist):

Service

This represents the main services in Nebari. There is a keycloak client created for most nebari services. The idea here is those services will call keycloak API (they already do at the moment for authentication) to fetch roles from a particular client for a user and using the roles's attributes it will decide what permissions the user have on that service. Here are some of the main core services:

• Jupyterhub
• Grafana
• Conda Store
• Argo

Component

A service can have several components and each component can have the need for a user or group to have different levels of access. We call them as component. For example, the JupyterHub service can have the following components:

• JupyterHub itself (JupyterHub API).
• Jupyterhub Profiles (this is spawner dependent, which is KubeSpawner profiles in Nebari)
• shared-directory (e.g. creating a shared nfs for a particular group), it is categorized under JupyterHub service because it is created by jupyterhub hooks.
• Dask: dask is accessed from JupyterHub.

Roles and Scopes

This figure depicts how services, components and scopes are related. Note that the scopes for grafana and argo are only for demonstration purpose.

Role is a collection of scopes (permissions).

Scope is a permissions to resource in a component of a service. We're borrowing the syntax for defining scopes (or permissions) from JupyterHub's RBAC. See https://jupyterhub.readthedocs.io/en/latest/rbac/scopes.html#scope-conventions for reference.

In a nutshell, it looks something like this:

<access-level><resource>:<subresource>!<object>=<objectname>

• <resource>:<subresource> - vertical filtering
• <resource>!<object>=<objectname> - horizontal filtering.
• If <access-level> is not provided, we assume maximum permissions.
• <subresource> is optional
• !<object>=<objectname> is optional

This is an example of how users, groups and roles interact. In this the group gpu-access-conda-store-pycon-argo-viewer-group has 3 roles, group gpu-users-group has one role and the group pycon-tutorial-group has one role attached to it. User alice has one role attached to them and the user john has no roles attached to them.

Examples:

1. Create a role such that when attached to a group it will create shared nfs directory in /shared.

You'll go to keycloak client jupyterhub and create a role with a meaningful name and add the following attributes to the role:

Role: create-shared-directory

Key	Value
component	shared-directory
scopes	create:shared

When you'd attach this role to a group, then Nebari will make sure to create a shared directory for the group.

Next, we want to have two sets of permissions:

• group of people (pycon-shared-read-group) with read access to a shared directory.
• group of people (pycon-shared-write-group) with write access to a shared directory.

We will create two roles:

• Role: read-access-to-pycon-shared-directory-role
  This role will be attached to pycon-read-group

  Key	Value
  component	shared-directory
  scopes	read:shared!shared=pycon

• Role: write-access-to-pycon-shared-directory-role
  This role will be attached to pycon-read-group

  Key	Value
  component	shared-directory
  scopes	write:shared!shared=pycon

Note: The names of roles and groups are arbitrary and can be anything.

2. Roles to control access to conda store namespace

• Create a role such that when attached to a group name read-conda-pycon-group, the group members will have read access to conda environments in pycon namespace.
• Create a role such that when attached to a group name write-conda-pycon-group, the group members will have read access to conda environment in pycon namespace.

Role: read-access-conda-pycon-namespace-role
This role will be attached to read-conda-pycon-group

Key	Value
component	conda-store
scopes	read:conda-store!namespace=pycon

Role: write-access-conda-pycon-namespace-role
This role will be attached to write-conda-pycon-group

Key	Value
component	conda-store
scopes	write:conda-store!namespace=pycon

3. App sharing permission

Create a role to allow everyone to share apps in a particular group.

Since we're using JupyterHub's RBAC, we can use scopes convention directly here.

Role: allow-app-sharing-role
This role will be attached to allow-app-sharing-group

Key	Value
component	jupyterhub
scopes	shares!user,read:users:name,read:groups:name

See https://jupyterhub.readthedocs.io/en/latest/reference/sharing.html#enable-sharing

By default, it will be disabled for everyone.

Implementation steps:

Since this is a complex piece of functionality, we would need to implement this in small modular steps. The steps are mentioned below:

1. Fetch keycloak groups and roles from Keycloak and sync them in JupyterHub. [ENH] - Make JupyterHub use groups and roles from Keycloak nebari#2308. At the moment we do have these available from the /user endpoint in the Hub API, under following keys. These needs to to be synced with JupyterHub roles and groups so that the permissions are actually applied at JupyterHub level.
   • auth_state.oauth_user.roles
   • auth_state.oauth_user.groups
2. Update the render_profiles functionality in profiles in jupyterhub config, such that it creates shared directory only if the group has permissions. This would also require us to implement a way to parse role scopes, e.g: parsing read:shared!shared=pycon for shared directory, when the component is shared-directory.
3. Implement scopes parsing can evolve as we implement rbac for each service. I don't expect us to write perfect scope parsing in one go. We should start with a service. I would suggest JupyterHub and make scope parsing for shared-directory work for it. For jupyterhub component it would work without parsing, after they are synced from keycloak into jupyterhub.
4. Implement scope parsing for conda-store and based on that incorporate conda-store's new permissions model.
5. Follow the same for Argo and Grafana.

Notes

• The RBAC system described above gives us the flexibility to create fine-grained permissions to specific resources and groups. While this is useful, we also need to make sure we set sensible defaults. Every user of Nebari might not need this level of fine-grain ability so we need to create some roles and groups by default when deploying a Nebari to facilitate basic permissions.
• JupyterHub also has a concept of custom scopes, but it's not very flexible and it does not enforce that your services apply them, so might not be very useful, but still worth exploring.
• JupyterHub doesn't have a way to update roles without restarting Hub, we need a way to tackle this limitation, ideally contributing to JupyterHub to update roles dynamically. See: https://jupyterhub.readthedocs.io/en/stable/rbac/roles.html#removing-roles

Alternatives or approaches considered (if any)

Best practices

The goal of this proposal is to implement the following best practices:

• Principle of least privilege.
• Adopting from an existing RBAC system in the Jupyter Ecosystem.

User impact

The implementation would change the default permissions of a user so it would affect the user, but we can set up with sensisble defaults to reduce the impact.

Unresolved questions

[Adam-D-Lewis]

I don't think a conda_store_viewer role is defined in keycloak currently.

[Adam-D-Lewis]

Fine grained access cannot be provided to users/groups.

@aktech can you provide an example of fine grained access that isn't possible to grant currently?

Update: I think I thought of an example. An example could be that we can't currently give a user read only access to a particular conda namespace, and read write to other namespaces.

[Adam-D-Lewis]

So eventually our "scopes" (e.g. conda-store!namespace=pycon) will need to map to whatever permissions model is implemented by each service (e.g. conda-store, argo, etc.).

I believe conda-store is currently flexible enough to support create, read, update, delete scopes for individual namespaces, but Argo Workflows is not nearly so flexible from what I've seen. It may be worth listing anything that we expect the new permissions model to be able to do that the current one can't do if any for each service (e.g. We expect to give individual users permission to view Argo Workflows workflows in particular k8s namespaces, etc.). At least for Argo Workflows and Grafana it's not clear to me that this change would enable us to support more fine grained permissions than what we already have.

Update: I see you have a start to expectations in nebari-dev/nebari#2304 under "Problems / Concerns / Questions:". We should look into if we'll be able to address those concerns with this change.

[Adam-D-Lewis]

Rather than have a "create share" scope, I think we should just have read and write scopes for the shared directories and we'll create the shared directory if it doesn't already exist and the user has one of the read or write scopes.

[Adam-D-Lewis]

Implementation steps:

Since this is a complex piece of functionality, we would need to implement this in small modular steps. The steps are mentioned below:

1. Fetch keycloak groups and roles from Keycloak and sync them in JupyterHub. [ENH] - Make JupyterHub use groups and roles from Keycloak nebari#2308. At the moment we do have these available from the /user endpoint in the Hub API, under following keys. These needs to to be synced with JupyterHub roles and groups so that the permissions are actually applied at JupyterHub level.
   - auth_state.oauth_user.roles
   - auth_state.oauth_user.groups
2. Update the render_profiles functionality in profiles in jupyterhub config, such that it creates shared directory only if the group has permissions. This would also require us to implement a way to parse role scopes, e.g: parsing read:shared!shared=pycon for shared directory, when the component is shared-directory.
3. Implement scopes parsing can evolve as we implement rbac for each service. I don't expect us to write perfect scope parsing in one go. We should start with a service. I would suggest JupyterHub and make scope parsing for shared-directory work for it. For jupyterhub component it would work without parsing, after they are synced from keycloak into jupyterhub.
4. Implement scope parsing for conda-store and based on that incorporate conda-store's new permissions model.
5. Follow the same for Argo and Grafana.

These steps don't necessarily need to be completed in this order. Only jupyter is dependent on steps 1-2. Argo, Grafana, and conda store permissions could be switched over at any time.

[aktech]

I don't think a conda_store_viewer role is defined in keycloak currently.

Yep correct, it doesn't. I. noticed it here.

@aktech can you provide an example of fine grained access that isn't possible to grant currently?

The three examples mentioned in the proposal are not currently possible.

I believe conda-store is currently flexible enough to support create, read, update, delete scopes for individual namespaces, but Argo Workflows is not nearly so flexible from what I've seen. It may be worth listing anything that we expect the new permissions model to be able to do that the current one can't do if any for each service (e.g. We expect to give individual users permission to view Argo Workflows workflows in particular k8s namespaces, etc.). At least for Argo Workflows and Grafana it's not clear to me that this change would enable us to support more fine grained permissions than what we already have.

Yes, correct. The scope of this proposal is to create a framework to be able to assign fine-grained permissions for each service and component Nebari have right now and is limited by the service/component. If a particular service / component doesn't have the an ability to allow for fine-grained permissions, then solving that problem is beyond the scope of this proposal.

Rather than have a "create share" scope, I think we should just have read and write scopes for the shared directories and we'll create the shared directory if it doesn't already exist and the user has one of the read or write scopes.

This is open to discussion, and it is an implementation detail. The problem we're trying to solve is the ability to specify a role on a group, such that it tells jupyterhub to either create or NOT create a shared directory as in there maybe certain groups for which you may not want to create a shared directory at all. Chosing create or write is more of a cosmetics, whichever sounds more intuitive.

These steps don't necessarily need to be completed in this order. Only jupyter is dependent on steps 1-2. Argo, Grafana, and conda store permissions could be switched over at any time.

Agreed.

[Adam-D-Lewis]

I believe conda-store is currently flexible enough to support create, read, update, delete scopes for individual namespaces, but Argo Workflows is not nearly so flexible from what I've seen. It may be worth listing anything that we expect the new permissions model to be able to do that the current one can't do if any for each service (e.g. We expect to give individual users permission to view Argo Workflows workflows in particular k8s namespaces, etc.). At least for Argo Workflows and Grafana it's not clear to me that this change would enable us to support more fine grained permissions than what we already have.

Yes, correct. The scope of this proposal is to create a framework to be able to assign fine-grained permissions for each service and component Nebari have right now and is limited by the service/component. If a particular service / component doesn't have the an ability to allow for fine-grained permissions, then solving that problem is beyond the scope of this proposal.

A summary of my takeaways to make sure we're on the same page. No model can foresee all potential future needs, so we may need to not adhere to this in the future for some service, and this model likely won't provide any benefit for Argo or Grafana currently (it's unlikely to affect them negatively either), it'll just allow more fine grained Jupyterhub and conda-store permissions.

[aktech]

No model can foresee all potential future needs, so we may need to not adhere to this in the future for some service

Not sure about the future, but the proposal is that this model will handle all the current needs.

and this model likely won't provide any benefit for Argo or Grafana currently (it's unlikely to affect them negatively either)

I don’t think that’s correct. It will provide many benefits, Grafana has a well defined RBAC which can be translated into the above syntax in the form of keycloak roles, I have not explicitly created examples for them here, but they can definitely be created along the lines of the given examples.

For Argo, they also have an RBAC but I haven’t looked at it very closely yet and moreover we’re yet not sure if it will be part of core Nebari for long enough, so wasn’t sure if we should put too much effort yet.

it'll just allow more fine grained Jupyterhub and conda-store permissions.

Three things on this:

• “Jupyterhub” and “conda-store” combined are most of Nebari.
• JupyterHub permissions also includes App Sharing permissions as well, which is also one of the major use cases of the permissions model.
• This “just” is a lot. There are no fine grained permissions as of yet for either of these except for 2-3 predefined ones and also there is no way to create custom roles at the moment.

The point of these clarification is to make sure that this proposal is evaluated on these expectation.

[Adam-D-Lewis]

Thanks for your detailed repsonses @aktech.

I don’t think that’s correct. It will provide many benefits, Grafana has a well defined RBAC which can be translated into the above syntax in the form of keycloak roles, I have not explicitly created examples for them here, but they can definitely be created along the lines of the given examples.

I agree with you after looking at that. I hadn't seen the more detailed RBAC for Grafana previously.

it'll just allow more fine grained Jupyterhub and conda-store permissions.

Three things on this:
“Jupyterhub” and “conda-store” combined are most of Nebari.
JupyterHub permissions also includes App Sharing permissions as well, which is also one of the major use cases of the permissions model.
This “just” is a lot. There are no fine grained permissions as of yet for either of these except for 2-3 predefined ones and also there is no way to create custom roles at the moment.

I agree with you 100% that adding more fine-grained permissions for jupyterhub and conda-store is very useful by itself! :) My comment was more about clarifying the benefits and limitations of this RFD.

[viniciusdc]

Looking great, just finished reading and here are my takeaways on this:

JupyterHub doesn't have a way to update roles without restarting Hub, we need a way to tackle this limitation, ideally contributing to JupyterHub to update roles dynamically. See: https://jupyterhub.readthedocs.io/en/stable/rbac/roles.html#removing-roles

@aktech this is what was discussed today, and jupyterhub will now support is that right?

I don’t think that’s correct. It will provide many benefits, Grafana has a well defined RBAC which can be translated into the above syntax in the form of keycloak roles, I have not explicitly created examples for them here, but they can definitely be created along the lines of the given examples.

See the note at the beginning; I am not sure if we are using the enterprise or cloud versions, though I might be wrong.

[viniciusdc]

@aktech THe only one that I am not sure about is the gpu-access-role here:

based on the syntax proposed, what would be the access level for jupyterhub-profile!profile=...? Would it be read ?

[viniciusdc]

JupyterHub also has a concept of custom scopes, but it's not very flexible and it does not enforce that your services apply them, so might not be very useful, but still worth exploring.

I think we can use this for any extensions that require access to jupyterhub API for anything

[viniciusdc]

Overall, I am okay with moving forward with this structure. (as an RFD it needs to be accepted XD)

[Adam-D-Lewis]

@viniciusdc @aktech

See the note at the beginning; I am not sure if we are using the enterprise or cloud versions, though I might be wrong.

Good point, Vini. We're using Grafana OSS which doesn't seem to have RBAC abilites. https://grafana.com/oss-vs-cloud/

[aktech]

My comment was more about clarifying the benefits and limitations of this RFD.

Makes sense, we definitely should clarify those limitations.

See the note at the beginning; I am not sure if we are using the enterprise or cloud versions, though I might be wrong.

Good point, Vini. We're using Grafana OSS which doesn't seem to have RBAC abilites. grafana.com/oss-vs-cloud

@viniciusdc @Adam-D-Lewis Good catch, correct we're limited by the service, i.e. Grafana OSS's inability to provide RBAC.

@aktech this is what was discussed today, and jupyterhub will now support is that right?

Yes, thanks to Mike's merged PR in jupyterhub to manage roles. I'll update that note in the RFD.

based on the syntax proposed, what would be the access level for jupyterhub-profile!profile=...? Would it be read ?

I believe there is only one level of access for jupyterhub profiles, either you can use them or not. In general if nothing is specified before resource name, it's admin access.

[viniciusdc]

For the Grafana OSS, I was working on a possible workaround for this (more like a plausible scheme) by imposing oauth2proxy in front of Grafana and handling any scope permissions checks there... I am still not sure how reliable it would be, but at least it would work as a gateway for mapping Grafana basic permissions with our new RBAC structure -- this process could also be extended later to any extensions as a basic RBAC service

[viniciusdc]

Just to give more context on that regard, I was planning on using oauth2 proxy alpha settings with the Keycloak client to handle access using scopes. In general, it would only work as a way to restrict from read/view access to it, as any extra permissions would require grafana's RBAC to be in place -- but at least would serve as a compatibility gateway of sorts.

Or, in any case, it seems that the Generic Oauth2 authenticator is part of the public version, and might support this with some extra attention to their docs: https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/generic-oauth/#configure-role-mapping

However we are still limited by their main roles: Admin, Editor, Viewer or None

[dcmcand]

@aktech , when looking at parsing the roles, we will need to be sure we strike the right balance between being constrained in what we accept, but flexible enough to be useful. I would think we would need some concept of wildcards, but that can get very complex very quickly. We also may need some default variables. Like if we want to automatically grant each user permissions on their namespace being able to write something like write:conda-store!namespace={USER.NAME} would be useful.

[dcmcand]

High level though, I like the design and think it will serve well, I also agree with the iterative build strategy. Don't build things we don't need, but leave the flexibility for future needs is a good design principle.

[aktech]

@aktech , when looking at parsing the roles, we will need to be sure we strike the right balance between being constrained in what we accept, but flexible enough to be useful. I would think we would need some concept of wildcards, but that can get very complex very quickly.

Yeah, we might. I left that for more of an implementation detail.

We also may need some default variables. Like if we want to automatically grant each user permissions on their namespace being able to write something like write:conda-store!namespace={USER.NAME} would be useful.

Agree, this should be part of defaults, as mentioned in the notes:

The RBAC system described above gives us the flexibility to create fine-grained permissions to specific resources and groups. While this is useful, we also need to make sure we set sensible defaults. Every user of Nebari might not need this level of fine-grain ability so we need to create some roles and groups by default when deploying a Nebari to facilitate basic permissions.

[aktech]

Thanks everyone for the comments and approval, I have marked it as accepted now.

[Adam-D-Lewis]

@aktech I think this is done, right? Could you update the labels and close it if so?