Confidential Containers KBS as a secrets provider

[mythi]

Starting with a discussion first to get feedback (and so that I don't forget the idea).

Currently, Gramine provides a secrets provisioning library that is used by a few "CI examples" to demonstrate how it works. This uses the RA-TLS underneath.

The Confidential Containers (CoCo) project has worked on a generic Key Broker Service with an attestation protocol for TEEs . Its original use was to provide container image decryption keys and image pull policies in CoCo but we have also seen demos where unencrypted containers use it to pull data decryption keys.

The idea proposed here is to have a Gramine preload secret provisioning lib that provides secrets from the CoCo KBS. This would add a use-case for an "external" secret provider.

[kailun-qin]

The idea proposed here is to have a Gramine preload secret provisioning lib that provides secrets from the CoCo KBS. This would add a use-case for an "external" secret provider.

@mythi Thanks for initiating the discussion and pls correct me if I'm wrong.

If I understand correctly, you're proposing to add a new "secret provisioning" lib (following the KBS Attestation Protocol -- basically a protocol over HTTP(S), acting similarly to what KBC does or even at a higher level) in Gramine that can integrate w/ CoCo KBS?

I however see some potential gaps. And I'd rather call the proposed libs w/ a different name (pls see some reasons below).

• First of all, the current Gramine secret provisioning libs (serving as high-level attestation interface) rely heavily on Gramine RA-TLS libs. However, the proposal should be RA-TLS agnostic and may interact directly w/ Gramine low-level /dev/attestation interface (e.g., for the nonce specified in the protocol). I.e., the designs are quite different.
• Second, the proposal seems to only need provide client-side APIs, whereas Gramine secret prov libs provide APIs for both sides: client (attester) and server (verifer).
• Third, the only APIs that the secret prov libs currently expose are listed here:

  gramine/tools/sgx/ra-tls/secret_prov.h

  Lines 41 to 168 in 4b49ff2

  	/*!
  	* \brief Write arbitrary data in an established RA-TLS session.
  	*
  	* \param ctx Established RA-TLS session, obtained from secret_provision_start() or in
  	* secret_provision_cb_t() callback.
  	* \param buf Buffer with arbitrary data to write.
  	* \param size Size of buffer.
  	*
  	* \returns 0 on success, specific error code (negative int) otherwise.
  	*
  	* This function can be called after an RA-TLS session is established via client-side call to
  	* secret_provision_start() or in the server-side callback secret_provision_cb_t().
  	*
  	* This function always writes all \p size bytes on success (partial writes are not possible).
  	*/
  	int secret_provision_write(struct ra_tls_ctx* ctx, const uint8_t* buf, size_t size);
  	/*!
  	* \brief Read arbitrary data in an established RA-TLS session.
  	*
  	* \param ctx Established RA-TLS session, obtained from secret_provision_start() or in
  	* secret_provision_cb_t() callback.
  	* \param[out] buf Buffer with arbitrary data to read.
  	* \param size Size of buffer.
  	*
  	* \returns 0 on success, specific error code (negative int) otherwise.
  	*
  	* This function can be called after an RA-TLS session is established via client-side call to
  	* secret_provision_start() or in the server-side callback secret_provision_cb_t().
  	*
  	* This function always reads all \p size bytes on success (partial reads are not possible).
  	*/
  	int secret_provision_read(struct ra_tls_ctx* ctx, uint8_t* buf, size_t size);
  	/*!
  	* \brief Close an established RA-TLS session and its associated secret.
  	*
  	* \param ctx Established RA-TLS session.
  	*
  	* \returns 0 on success, specific error code (negative int) otherwise.
  	*
  	* This function can be called after an RA-TLS session is established via client-side call to
  	* secret_provision_start(). Typically, application-specific protocol to provision secrets is
  	* implemented via secret_provision_read() and secret_provision_write(), and this function is called
  	* to finish secret provisioning.
  	*
  	* This function zeroes out the memory where provisioned secret is stored and frees it.
  	*
  	* This function must not be called again even if it returned an error. \p ctx is always freed.
  	*/
  	int secret_provision_close(struct ra_tls_ctx* ctx);
  	/*!
  	* \brief Get a copy of the provisioned secret.
  	*
  	* \param ctx Established RA-TLS session.
  	* \param[out] out_secret Pointer to newly allocated buffer with secret.
  	* \param[out] out_secret_size Size of allocated buffer.
  	*
  	* \returns 0 on success, specific error code (negative int) otherwise.
  	*
  	* This function is relevant only for clients. Typically, the client would ask for secret
  	* provisioning via secret_provision_start() which will obtain the secret from the server and
  	* save it in enclave memory. After that, the client can call this function to retrieve the
  	* copy of the secret from memory. Content of \p out_secret is dynamically allocated and should be
  	* released using `free(*out_secret)`.
  	*/
  	int secret_provision_get(struct ra_tls_ctx* ctx, uint8_t** out_secret, size_t* out_secret_size);
  	/*!
  	* \brief Establish an RA-TLS session and retrieve first secret (client-side).
  	*
  	* \param in_servers List of servers (in format "server1:port1;server2:port2;..."). If
  	* not specified, environment variable `SECRET_PROVISION_SERVERS` is
  	* used. If the environment variable is also not specified, default
  	* value is used.
  	* \param in_ca_chain_path Path to the CA chain to verify the server. If not specified,
  	* environment variable `SECRET_PROVISION_CA_CHAIN_PATH` is used. If
  	* the environment variable is also not specified, function returns
  	* with error code EINVAL.
  	* \param[out] out_ctx Pointer to an established RA-TLS session. Cannot be NULL.
  	*
  	* \returns 0 on success, specific error code (negative int) otherwise.
  	*
  	* This function must be called before other functions. It establishes a secure RA-TLS session
  	* with the first available server from the \p in_servers list and retrieves the first secret.
  	*
  	* The user can continue this secure session with the server via secret_provision_read() and
  	* secret_provision_write(). The user must finish the session by calling secret_provision_close().
  	*
  	* The first secret can be retrieved via secret_provision_get(). The secret is destroyed together
  	* with the session during secret_provision_close().
  	*/
  	int secret_provision_start(const char* in_servers, const char* in_ca_chain_path,
  	struct ra_tls_ctx** out_ctx);
  	/*!
  	* \brief Start a secret provisioning service (server-side).
  	*
  	* \param secret First secret (arbitrary binary blob) to send to client after establishing
  	* RA-TLS session.
  	* \param secret_size Size of first secret.
  	* \param port Listening port of the server.
  	* \param cert_path Path to X.509 certificate of the server.
  	* \param key_path Path to private key of the server.
  	* \param m_cb Callback for user-specific verification of measurements in client's SGX
  	* quote. If user supplies NULL, then default logic of RA-TLS is invoked.
  	* \param f_cb Callback for user-specific communication with the client, e.g., to send
  	* more secrets. If user supplies NULL, then only the first secret is sent
  	* to the client and the RA-TLS session is closed.
  	*
  	* \returns 0 on success, specific error code (negative int) otherwise.
  	*
  	* This function starts a multi-threaded secret provisioning server. It listens to client
  	* connections on \p port. For each new client, it spawns a new thread in which the RA-TLS
  	* mutually-attested session is established. The server provides a normal X.509 certificate to the
  	* client (initialized with \p cert_path and \p key_path). The server expects a self-signed RA-TLS
  	* certificate from the client. During TLS handshake, the server invokes a user-supplied callback
  	* m_cb() for user-specific verification of measurements in client's SGX quote (if user supplied
  	* it). After successfuly establishing the RA-TLS session and sending the first secret \p secret,
  	* the server invokes a user-supplied callback f_cb() for user-specific communication with the
  	* client (if user supplied it).
  	*
  	* This function is thread-safe and requires pthread library.
  	*/
  	int secret_provision_start_server(uint8_t* secret, size_t secret_size, const char* port,
  	const char* cert_path, const char* key_path,
  	verify_measurements_cb_t m_cb, secret_provision_cb_t f_cb);

  . And we haven't (yet) considered/expected users to have their own impl. or APIs (see [tools/sgx] Install RA-TLS and SecretProv libs properly #1114 (review)). If we really want to have it as a Gramine "secret prov" lib, we'd have to evaluate whether these serve well the purpose of the proposal.

Some questions:

1. What APIs or which level of APIs do you expect the proposed libs to expose? (relative to the second and the third point above)
2. Once we have the proposed libs, do we still need a CoCo attestation-agent?
3. Is KBC supplied as libs already? If so, what APIs does it provide, and any plugin mechanisms for attestation evidence? (the reason for this is about whether we need to implement everything from scratch or just an integration w/ KBC)

[mythi]

The idea proposed here is to have a Gramine preload secret provisioning lib that provides secrets from the CoCo KBS. This would add a use-case for an "external" secret provider.

@mythi Thanks for initiating the discussion and pls correct me if I'm wrong.

If I understand correctly, you're proposing to add a new "secret provisioning" lib (following the KBS Attestation Protocol -- basically a protocol over HTTP(S), acting similarly to what KBC does or even at a higher level) in Gramine that can integrate w/ CoCo KBS?

That's the gist of it, yes.

I however see some potential gaps. And I'd rather call the proposed libs w/ a different name (pls see some reasons below).

* First of all, the current Gramine secret provisioning libs (serving as high-level attestation interface) rely heavily on Gramine RA-TLS libs. However, the proposal should be RA-TLS agnostic and may interact directly w/ Gramine low-level `/dev/attestation` interface (e.g., for the `nonce` specified in the protocol). I.e., the designs are quite different.

* Second, the proposal seems to only need provide client-side APIs, whereas Gramine secret prov libs provide APIs for both sides: client (attester) and server (verifer).

* Third, the only APIs that the secret prov libs currently expose are listed here: https://github.com/gramineproject/gramine/blob/4b49ff2ff053857a0e492e5a22aa6d6c107e66d9/tools/sgx/ra-tls/secret_prov.h#L41-L168
  . And we haven't (yet) considered/expected users to have their own impl. or APIs (see [[tools/sgx] Install RA-TLS and SecretProv libs properly #1114 (review)](https://github.com/gramineproject/gramine/pull/1114#pullrequestreview-1317146016)). If we really want to have it as a Gramine "secret prov" lib, we'd have to evaluate whether these serve well the purpose of the proposal.

I'm not able to follow what the gaps are in the first two items. My thinking is that this could be an independent "client" shared lib implementation following the KBS API. I do understand that it would have the server side missing from Gramine code base but is that a problem? From CI/validation perspective it probably is which would make it more suitable as a "third party solution", I guess?

Some questions:

1. What APIs or which level of APIs do you expect the proposed libs to expose? (relative to the second and the third point above)

it could piggyback env.SECRET_PROVISION_* functionality that I currently feel is sufficient.

2. Once we have the proposed libs, do we still need a CoCo attestation-agent?


3. Is KBC supplied as libs already? If so, what APIs does it provide, and any plugin mechanisms for attestation evidence? (the reason for this is about whether we need to implement everything from scratch or just an integration w/ KBC)

CoCo attestation-agent by default runs a daemon serving gRPC but some of its functions can also be consumed as library dependencies. This provides an abstraction to the underlying KBCes. It's a valid design consideration whether the "secret provider" should use the higher level attestation-agent APIs instead but haven't thought about it too much.