Kerberos/GSSAPI backend feature request

[nrhall-deshaw]

This ER is to request GSSAPI / SPNEGO / Kerberos support in Vault:

Motivation:
Many corporate environments use Kerberos heavily (a variety of flavours) for authentication between users, services, etc. It would be useful for Vault to be able to handle this style of authentication to avoid needing to use yet another form of authentication for getting tokens (i.e. non-SSO) or some kind of gateway to convert between the two.

Requirements:

Authentication process should be possible via two methods
a. Vault client API in the usual manner, using SPNEGO (base64 encoded) strings in the metadata passed to the back-end
b. HTTP browser based auth (where the browser responds to a 401 / WWW-Authenticate: Negotiate request with an SPNEGO token automatically)
Auth backend should use the regular system GSSAPI libraries, via dynamic loading (static compilation doesn't work with glibc) and cgo
Client and service should mutually authenticate, and there should be some client-side verification of the service to ensure it's the service that it's expecting to talk to
Notes:

There's an implied need for a client-side plugin to handle this authentication style, as there are additional token handling requirements for SPNEGO, as well as doing the mutual authentication check.
Go does not have native support for the GSSAPI functions. An incomplete but working wrapper library is at https://github.com/apcera/gssapi; this is the library that was used for the PoC. This will need to be brought in as a dependency to make this functionality work.
POC:
Proof of concept code has been built which incorporates a number of items:

Kerberos auth backend
Patches to the header handling code so that selected headers as passed back and forward
Patches to the client side code to handle the backend communication

[jefferai]

This functionality needs to be in a separate patch if you want to propose adding it.

[nrhall-deshaw]

Apologies - I hadn't intended to merge this commit. I think it's a useful feature though, I'll disentangle it from the PR.

[nrhall-deshaw]

Now disentangled.

[nrhall-deshaw]

New PR raised at #3035

[jefferai]

This isn't the right mechanism for this. If we want to allow arbitrary headers to be returned in a response, it should be added as an object to logical.Response, with appropriate rules around HMACing in auditing and so on.

[nrhall-deshaw]

Fair enough - I saw the existing cases where this information was being passed around. The rest seem to have been added for the PKI backend. I think the cleanest way to do this might be to expose some limited information about the HTTP request/response in logical.Request and logical.Response, rather than just the headers (then all these other 'should be avoided unless absolutely necessary' comments can be tidied up...)

WRT auditing, I think that there's a case to be made for not logging SPNEGO tokens, or other forms of authorization header (or at least the secret part of them) but instead perhaps replacing that information with some audit logging around which user is making the request.

Perhaps the headers in need of hashing should be specified by the backend when making the request for the headers that it needs to be passed? (see my other comment below).

[jefferai]

Similarly to https://www.vaultproject.io/api/system/config-auditing.html there should be a way to specify which response headers created by a backend should be logged, and if so, hashed or not.

[jefferai]

If we're going to add support for sending headers to backends, we need a mechanism, like with auditing, for the administrator to control which headers are allowed to be sent.

[nrhall-deshaw]

I'm not sure an administrator view is that useful. The headers (and indeed the body) which backends need (this being one of two that need access to the underlying request) is really a function of the backend, rather than the administrator - e.g. it wouldn't seem sensible to allow an administrator to disable a header required by a backend.

So perhaps the list of headers (and whether their values should be hashed in the audit logs) should be requested by the backend at mount time?

[jefferai]

An administrator needs to have positive control over which headers a backend sees. Given that backend plugins will be available shortly, not all backends will have been reviewed by the Vault team to make sure they're doing reasonable things.

[jefferai]

We had an internal discussion and landed on some things:

• Plugins/backends should predeclare request headers they need; Vault will then only send the declared headers to the backend for each request
• Any headers Vault itself uses will be blacklisted
• All headers in a response will be audited and HMAC'd; a sys/config/auditing/response-headers section should be added to allow disabling HMACing of specific headers

Hopefully this sounds agreeable to you!

[nrhall-deshaw]

Yes - that seems good. I think that gives a decent flexibility while protecting sensitive headers that we wouldn't want backends to be able to divert to them.

[nrhall-deshaw]

One further quick query here; the GSSAPI plugin needs client-side support (much like there's dedicated client handling for radius and a number of other auth backends). This requires use of the GSSAPI, which as in the server cannot be statically linked (hence the vault client will not able to be static); how would you like this handled?

[jefferai]

@nrhall-deshaw I thought the whole point of the SPNEGO stuff was to not require client library calls?

[nrhall-deshaw]

It avoids the need to keep or track client tokens, instead allowing the kerberos authentication to be a substitute (which then fetches what can then be a shorter-lived token to use for subsequent activity).

However, the initial authentication requires client/server interaction for SPNEGO - much like the way a browser authenticates to a web server in a kerberos / MS Active Directory environment.

This is done by constructing an SPNEGO and exchanging it with the service - either directly via a header in the HTTP interaction if the client is a 'browser' (or a commandline tool like a browser) or using the vault client directly, by passing the token in the vault API interaction, in which case the vault client needs to understand how to build and evaluate the token.

The build and evaluation step is done simply using the GSSAPI in the client; the code is straightforward, but relies on pulling in the GSSAPI libraries on the client side, which makes it require to be dynamically linked.

The current code for this is in the pull request, and it uses a similar approach to the client side integration for other protocols that require it, such as radius.

[jefferai]

Probably the best thing to do would be to just build a tiny separate client, e.g. vault-kerberos-auth. It could re-use the Meta stuff from Vault and other bits to have similar methods for formatting and so but be able to be dynamically linked.

Alternately, if there is a way to shell out to binaries on the system to get the information needed, you could have the cli helper do that.

[nrhall-deshaw]

That would handle some of the more basic use-cases with the vault client, but does reduce its usefulness somewhat, not to mention making it different from other auth methods.

Shelling out to another command might work (e.g. to generate the token which gets passed to the server) but that does seem somewhat sub-optimal - and that command would have to be written (in C or dynamically linked Go for performance probably) and packaged; there isn't currently something that would do that out of the box.

What about an approach that supports a client-side plugin written in go (even if to use it involved creating a dynamic build)? Is there anything feasible there?

[jefferai]

I'm hesitant to support a plugin system for an auth helper that fundamentally just runs a couple of API calls that can be done by a user on the command line in the first place. Considering that this would be built as a plugin in the first place, it seems like a plugin repo could easily hold both binaries and a makefile to build both the plugin and the auth-helper.

[nrhall-deshaw]

Ok, yeah - I think that could probably work out ok.

Can you point me in the direction of some docs and/or examples of how to write an auth-plugin? (or even just a few pointers to the right classes to look in..)

[calvn]

You can take a look at vault-plugin-auth-gcp as an example of an auth plugin.

[nrhall-deshaw]

With apologies for the delay, I've returned to this work.

I now have a dynamic plugin that works against a statically compiled Vault (thanks @calvn for the pointer to the gcp plugin)! How would you like this code contributed - as a separate repo, or as a PR against the Vault core?

Also, I've been giving some thought to the headers problem (which is needed to do a proper job here so that SPNEGO can work 'vanilla' from a browser; somewhere up the PR, @jefferai noted:

[1] Plugins/backends should predeclare request headers they need; Vault will then only send the declared headers to the backend for each request
[2] Any headers Vault itself uses will be blacklisted
[3] All headers in a response will be audited and HMAC'd; a sys/config/auditing/response-headers section should be added to allow disabling HMACing of specific headers

The most obvious place to put [1] is in the 'Backend' structure itself; I've started with a 'Requirements' structure which has an 'HTTPInHeaders' list, and am using that to reduce the headers that the plugin gets (HTTPInHeaders could equally be in something called other than 'Requirements', but I was struggling for a more inspirational name for that place...)

For [2], is there a canonical list of those headers anywhere?

For [3], I see that in the AuditBroker, response headers are already being logged (or at least they seem to be), although it looks like headers are selectively HMAC'd - you're suggesting reversing that so that all headers are audited, and the configuration supports disabling rather than enabling?

As an alternative approach for [3], perhaps the backend should register those headers it needs to send back, and only those headers are permitted and HMAC'd - i.e. we'd have an HTTPInHeaders list and an HTTPOutHeaders list?

[jefferai]

Hi there,

We have some work ongoing to allow headers to plumb through to plugins. I thought this effort was dead so I didn't give a heads up. It won't really look like what you described above; at this point I'd say to just hold off dealing with the headers aspect of it and then when it's implemented it should be easy to make it work for you.

[nrhall-deshaw]

Excellent news - no, not dead - just slightly dormant due to other priorities getting in the way.

We’ve been using our Kerberos patches along with a dynamic build for a few months and all is looking good. The current code has had a few fixes from that experience so I think is better as a result. We plan to move to using the plug-in over the next few weeks, although we will maintain our headers patches locally until you release your changes. Is there another ticket I can follow or any early code?

What’s your preference from a plug-in perspective? Do we contribute that to you or release separately now that it’s a plug-in?

[ah-]

We had the same desire and have recently built a plugin that addresses many of the comments here.
It's available here: https://github.com/wintoncode/vault-plugin-auth-kerberos

So far it is a simple vault plugin that uses gokrb5 to authenticate and looks up policies with LDAP.

I am quite keen to merge our efforts, and move support as far upstream as possible.

As the plugin is quite basic so far there are still some bits that would be nice to add:

• Support browser auth via standard SPNEGO headers. It would be really nice to e.g. support login to goldfish with SSO. It sounds like that's already implemented in @nrhall-deshaw s version, would be interesting to see that.
• Add support to the vault cli and commonly used client libraries like hvac.
• De-duplication of the LDAP lookup logic.

The last two points are closely connected to choice of kerberos library. For the backend we've picked gokrb5, as it's pure Go and doesn't depend on any system configuration. It doesn't even need to do any socket communication at all, so it's very simple and lightweight. No cgo, works on all platforms. All it does is check a crypto token.

For support in the vault cli we need to get an SPNEGO token and post it to /v1/auth/kerberos/login. To get the token we might prefer to use a wrapper around a native kerberos implementation like kerby or gssapi, as those can use the system configuration to talk to the right kdc and use your existing login token.

And finally, LDAP. We really want to support looking up Vault Policies based on LDAP groups, as that greatly simplifies our internal user management. I imagine many other companies with existing LDAP/Kerberos setups will want the same.
And after writing the plugin, it turns out there is a lot of duplication with the ldap plugin. Basically the only difference is that the kerberos authenticates the user via kerberos, while the ldap plugin does it via a bind with username and password. Everything else is exactly the same. LDAP configuration, policy/group mapping, etc. To get things going we've duplicated some of the LDAP code, but it would be nice for longer term maintainability to get rid of this duplication.
As plugins can't share configuration that also means that if you want to support logging in both with username/password and SSO you need to configure everything ldap related twice.

I can see two ways to de-duplicate:

1. Make the vault ldap functionality re-usable. I'm not that experienced with Go yet, but maybe all the group lookup could be turned into a library that vault-plugin-auth-kerberos users. This does however not solve the config duplication.
2. Fold vault-plugin-auth-kerberos into the ldap plugin. All this requires is a branch in the login check to use either user/password bind to LDAP or otherwise validate the SPNEGO token, plus a bit of config boilerplate to store the keytab. Since we don't have any cgo dependency anymore this would be easy and clean to implement and would also help with config duplication.

[jefferai]

Hi there,

That's super neat about gokrb5 -- cgo was a large part of the problem! So that does remove a blocker to getting something in Vault proper.

We have active coding in process to allow sending request headers to backend plugins. (It was actually put on the roadmap specifically for this use case!) That should appear in 0.9.6 or 0.10, which should make it easy to do SPNEGO stuff.

The thing I'm less keen on is adding LDAP directly to a Kerberos plugin. Is there no way to glean user/group information directly from what Kerberos provides? If this is a super common setup I guess we could look into it, but I'd also want to understand what other methods there are that people might need to fall back on.

[ah-]

Great, looking forward to proper header support!

Just had a quick search around groups with Kerberos and it seems that at least for AD there's PAC, which is even supported by gokrb5: https://github.com/jcmturner/gokrb5/tree/master/pac
That said, my Kerberos knowledge is a bit limited, so if anyone here has experiences to share I'm super interested. I just remember that encoding too much in Kerberos tokens has caused issues for us in the past, with the tokens hitting size limits.
That's why I quite liked the LDAP approach, since then vault can look up as many groups as needed.

How are other people managing Vault Policies?

[nrhall-deshaw]

Agreed - when we originally looked at this, gokrb5 hadn't been released, so it would be interesting to see this integrated. Please make sure that it's flexible enough from a configuration standpoint - particularly in terms of the number of back-end keytabs that Vault can handle. :)

The plugin we have is dynamic now anyway so the cgo stuff doesn't get in the way, but it would be interesting to merge approaches. I'm just getting our code approved for release; we're doing a few other things in the plugin to set identities and a few other things properly. We also use an approach where we use a separate login step using the SPNEGO token inside the auth request to get a token, then use the token for interaction with the cluster (then revoke it afterwards).

One thing that would be good to fix is the principals in a cluster scenario; for example, when you authenticate in a cluster, the authentication request gets sent to the primary server; so you have to do one of a few things:

• use the same principal for all the servers and authenticate to that principal, rather than to a principal for the actual host you're connecting to - for example, we added the SRV record approach so that we could manage a cluster as a single thing, and we use it to determine the nodes in the cluster and pick one (without really caring if it's the primary) to send our request to - we use a 'hostname' derived from the SRV record in the server principal so that no matter which server the request goes to, it has a keytab that can authenticate the request - for example

• always authenticate on the local cluster node (not sure if it's possible to allocate tokens from multiple nodes in the cluster at the same time, but I suspect not)

APIs wise, I've added support in both the blessed Java and Python APIs for the approach we've taken, using their native GSSAPI support in each language, and hope to get these pushed upstream for review.

For the LDAP and policy stuff, we're considering pre-processing the data in LDAP and feeding Vault a static feed of users and groups and creating identities/groups in Vault with static policies attached to each that we can modify within Vault - partly because doing some of the group queries properly against AD can be quite intensive, partly for housekeeping reasons (e.g. cleaning up groups/users that get deleted as well as cleaning up any policies) and because we'd prefer to keep Vault operating isolated from AD (so we can use it in places where AD isn't easy to get to). It also makes things a bit easier to troubleshoot if it's not all quite so magic. Interested to hear how others are thinking of solving this...

[ah-]

Oh great to hear you're close to publishing your internal version!
It'll be interesting to see how your REST interface looks like, as that's something that we can't easily change once we have support in client libraries etc.

Ideally we can merge our implementations relatively soon. Our setup is still pretty flexible, and if it means we get hvac/java support for free it's a no-brainer.
Maybe we could even get this to live within the hashicorp GitHub org once it's somewhat stable?

I wonder how people manage Vault policies with other backends? Do they copy all the permissions into Vault or use dynamic lookups?

[jefferai]

@ah- The way we're moving towards is taking advantage of Vault's Identity system for policy assignment and management. Identity was designed to be scriptable so that you can sync it with whatever your source of truth is. This prevents each backend from having to implement logic to fetch groups from external sources; the backend can simply focus on validating an appropriate unique identifier (and optionally groups) and pass that information back, and have policies derive from individual Identity entities and the groups they belong to. We basically took the stance of "having Vault (or every Vault backend) sync with every other system is very hard to do from within Vault, but it's a pretty trivial 10-20 lines of scripting for a sysop that can be run as needed".

As for hosting in a hashicorp repo, we'd absolutely love to have a first-party kerberos plugin, but we'd have to do significant review. I think the best approach would be for you two to look at the common parts of authentication and figure out a merged approach, take into account what I said about Identity above (unless it is a solution that won't work for you for some reason), and then we can have a chat about the final proposed design!

[ah-]

@jefferai, thanks that helped my understanding quite a bit. I didn't realise that Identity was now part of Open Source Vault, that makes it quite a lot easier to use in a backend that is meant to work for many people.

If we focus the Kerberos support to just authentication it becomes a whole lot simpler.

I'll wait for @nrhall-deshaw s release to see their requirements, but I think we can then drop the LDAP hack and just have a really simple plugin that just validates Kerberos tokens.

Do you think the main LDAP plugin might move in a similar direction? Separating authentication from looking up groups?

[jefferai]

I don't think we'll change the LDAP plugin right now as it's working for many people, but for those that simply cannot make it work for group lookup for their particular setup it's a nice alternative to have.

[vinzent]

@jefferai

Is there no way to glean user/group information directly from what Kerberos provides?

Kerberos is mostly about authentication ("who am i") not authorization. The mentioned MS-PAC extension only exists in MS AD environments and I don't think they have plaintext names in it but only SID's which also would require lookups to LDAP.

What I think is that it should be possible to define 2 plugins: authentication and authorization. One that authenticates (kerberos or whatever) and another that that retrieves groups/policies for authorization (ldap or whatever).

[ah-]

That sounds very sensible, @jefferai, would that work via the identity mechanism? How do identity permissions get populated?

@nrhall-deshaw did you get anywhere yet re. open sourcing?

[jefferai]

Kerberos is mostly about authentication ("who am i") not authorization.

Right. What Vault cares about is who you are, and optionally, what groups you belong to. It doesn't care whether that identity information authorizes you to access X or Y via that third party; authorization on the Vault side is mapping authentication information from a third party source to internal Vault authentication information, and then selecting authorization policies based on that information. Users/unique IDs/groups all fit into that category.

If Kerberos systems require extra steps in order to get a complete authentication picture (it wouldn't be the only such system, look at e.g. OAuth/OIDC and UserInfo endpoints), then there are two possibilities. One is that all such systems are built into a plugin, which would make it rather unwieldy. But the likely better scenario is to just have an administrator do what I suggested before and map kerberos IDs (which will generate Identity Aliases) into Identity Groups and assign authorization information that way. That keeps the Kerberos plugin way simpler without having to pull in full LDAP client setups, NIS client setups, and what have you.

[megakid]

Very keen to hear thoughts on ways forward on allowing plugins to return custom HTTP headers (or HTTPOutHeaders as named above) to allow for WWW-Authenticate as per https://tools.ietf.org/html/rfc7235#section-4.1

This would open up good opportunity for Vault to be used in .NET shops that are migrating slowly to Linux (as we are now). We are currently seeking a slick way of using existing AD/LDAP permissions with Vault, especially from our own .NET applications where the 401 + WWW-Authenticate: Negotiate response is automagically handed (atleast to the best of its abilities) without any raw Kerberos code.

Client: GET someUrl
Server: 401 WWW-Authenticate Negotiate
Client: GET with Authorization headers
Server: 200 OK

[29x10]

any update on this?

[hashicorp-cla]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

_{Have you signed the CLA already but the status is still pending? Recheck it.}

[sambott]

Very keen to hear thoughts on ways forward on allowing plugins to return custom HTTP headers (or HTTPOutHeaders as named above) to allow for WWW-Authenticate as per https://tools.ietf.org/html/rfc7235#section-4.1

This would open up good opportunity for Vault to be used in .NET shops that are migrating slowly to Linux (as we are now). We are currently seeking a slick way of using existing AD/LDAP permissions with Vault, especially from our own .NET applications where the 401 + WWW-Authenticate: Negotiate response is automagically handed (atleast to the best of its abilities) without any raw Kerberos code.

Client: GET someUrl
Server: 401 WWW-Authenticate Negotiate
Client: GET with Authorization headers
Server: 200 OK

https://github.com/wintoncode/vault-plugin-auth-kerberos now supports these headers

[jmls]

may I ask what the current status on this plugin is ?

[tyrannosaurus-becks]

@jmls the plugin is located here. We've successfully upstreamed it to Hashicorp, but we're not yet at a place where we can merge it because we have yet to perform successful sunny-path integration tests on it.

I've created an environment for testing it, but I'm unclear on how to create a login request and am seeking an example in any language.

I do suggest that we close this PR in favor of bringing in the linked plugin. If any changes need to happen to support further Kerberos use cases, it would be best if they occurred there.

[chrishoffman]

Closing this in favor of the upstreamed repo.