Epic: automate openapi spec generation and synchronization

[LizardWizzard]

Motivation

For components written in rust (mainly pageserver and safekeeper, but soon proxy as well) we have http apis which are documented by manually writing openapi specs for them.
Manual spec maintenance leads to several problems. Code and spec can diverge. E g #3669

Additionally these specs are used in different repo (cloud) and there is no checks that spec is up to date, so it can diverge from time to time.

DoD

• Spec file is generated automatically from annotations on request handlers
• CI integration to raise warnings when spec changes (similar to backwards compatibility test we have). This is useful for indication when spec needs to be updated in other repos

Implementation ideas

There are several libraries that fulfill the goal of the first step.

1. Poem

https://github.com/poem-web/poem

Is is a different web micro framework build on top of hyper. It has builtin support for openapi spec generation https://github.com/poem-web/poem/tree/master/poem-openapi

2. Utoipa

https://github.com/juhaku/utoipa

Utoipa follows different strategy, instead of being a web framework it is a library that has support for most popular web frameworks. And our current configuration of routerify + hyper is not in the list of supported integrations. Probably makes sense to shift to axum or warp then. The syntax for annotations is also a bit different from poem.

3. Paperclip

https://paperclip-rs.github.io/paperclip/

The project has wider scope, it includes an ability to generate type-safe clients for provided openapi spec. For framework integration it supports only actix.

Tasks

Preview Give feedback

1. Use library to automatically generate openapi spec from handler definition (lets start with pageserver, it has the biggest http api surface)
2. Do the same for safekeeper
3. Integrate consistency checks into CI pipeline
4. Continuously test pageserver http api with generated clients #1297
   a/ci a/test c/storage/pageserver +3
5. Integrate spec linting into

Other related tasks and Epics

• Continuously test pageserver http api with generated clients #1297

[SomeoneToIgnore]

Code and spec can diverge

To note, we have a few HTTP methods that have the working Rust code, but are not deliberately added into the openapi spec file, e.g. eviction API: #3486 (comment)

Either the tool has to allow such exceptions, or we have to think of a better way to gate our new, unfinished management API methods.

[LizardWizzard]

I think partially the reason they're not added is the hassle to properly document them. To me its ok to have them documented, but we can keep them under some unstable prefix/suffix so it is clear that these methods do not provide backwards compatibility guarantees.

I the the following use case: Clients we use for tests can be autogenerated from the spec, and in those clients its reasonable to have access to this kind of testing/unstable endpoints

[LizardWizzard]

Tried poem, it uses its own set of traits to define types of parameters, even request/response bodies are not using serde types, so I had to manually implement a bunch of traits for our types like tenant ids.

Next step is to try utoipa + axum. Combination of axum + tower_http is quite promising, tower_http has many useful middleware for observability so we can avoid our own hacks to have per handler spans

[LizardWizzard]

Discovered that poem has tower compatibility: https://github.com/poem-web/poem/blob/10de9d11daceed3def451999ea8496b636f7d1b2/examples/poem/tower-layer/src/main.rs#L2 So may work with tower_http

[LizardWizzard]

Here is my analysis:

https://github.com/LizardWizzard/rust_openapi_playground

So to me poem's strict typing gives it a huge advantage over utoipa

[LizardWizzard]

related #1297

Last time we discussed it the conclusion was that some time in the future we'd rather switch to grpc