Generate client code from OpenAPI specification

[Xtansia]

Description

As I mentioned in #284 (comment) this is a PoC creating a code generator that uses OpenAPI specs from scratch (ie. not based off openapi-generator). The code is not feature complete and not production ready.

Please direct general comments/discussion about the topic to the issue to keep things consolidated.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[reta]

@Xtansia do you have an examples of request / response models the generator produces? thank you!

[Xtansia]

@Xtansia do you have an examples of request / response models the generator produces? thank you!

@reta Yes, they're included in this PR, the remote store ones. Under the java-client/src/generated/java directory

[reta]

@Xtansia do you have an examples of request / response models the generator produces? thank you!

@reta Yes, they're included in this PR, the remote store ones. Under the java-client/src/generated/java directory

Sorry missed that, they look pretty close to what we have inherited (comparing to just template changes).

[harshavamsi]

@Xtansia this is amazing work! The generated client and req/res look very similar to the existing APIs.

From a quick glance, I don't see a Response.mustache template. How is the response handler being generated? I see it's being generated from the object shape and the individual pieces are being constructed.

[dblock]

love it, how can we move it closer to something we can merge?

++ stick a header on all generated files that say "this file is generated, don't edit" or something like that

[Xtansia]

@dblock & @reta, I have added the warning header & refactored the transform logic to what I feel is a lot clearer. I've also managed to pull the few more complicated mustache "lambdas"/getters out to primarily templates, with some more minor code logic just to handle recursiveness.
There's a handful of features that'll need to be implemented eventually, for example when we want to regenerate some of the operations that merge multiple endpoints into one operation/request shape (ie. index a doc with or without ID).
Otherwise it's primarily waiting for having an actual published spec from https://github.com/opensearch-project/opensearch-api-specification and probably some kind of configuration for filtering down the desired operations to generate.

[reta]

@Xtansia this is pretty cool, just from your perspective as implementor: how difficult it would be to keep OpenAPI and our own generator in sync (if it makes at all)? Specifically, I have in mind bugfixes and missing OpenAPI 3.1 features (the 3.1 support is still experimental and not fully implemented).

As for a spec, I've found this outdated but pretty comprehensive snippet [1], I converted it to 3.x using Swagger Editor, we could try to use it meantime (for a sake to filling any implementation gaps), thanks

[1] https://gist.github.com/rodripf/cd7a410b5ca7a91a1a4331d4ac7b1bcb

[dblock]

@Xtansia interest in finishing this or #336 ?

[Xtansia]

@dblock @reta @VachaShah This is ready to be looked at when you all have time.

Thanks @Xtansia , I will take a look shortly, just wanted to confirm you think that having this project as part of opensearch-java is a way to go? The reason I am asking is that I expect there would be a lot of changes there and the release cadence of generator and client might be very different (at least initially), having a separate repo would mitigate that quite a lot.

That's a fair point and I think in the long run it would make sense to have this in a separate repo. However at the moment it's not yet at a stable point where this makes sense to actively publish for others. The first primary goal is in getting part or all of the Java client being generated so this initial period of iteration and development is going to be inextricably tied to attempts to generate more operations and covering any missed edge cases etc. So I think it's reasonable for it to live here until it has stabilised somewhat as long as we aren't releasing it from here.

[VachaShah]

This is great!! The only comment I have (after a high level review) looking from previous comments is: if at all there is a chance for this generator to be moved out of this repo, could we make the generator classes restricted to use for consumers? Mostly no user would end up using them but if we move out public classes out of the repo, we might end up with breaking changes and a major version increment.

[dblock]

Merge?

[reta]

Merge?

No objections, we are not going to backport it to 2.x, right?

[dblock]

Merge?

No objections, we are not going to backport it to 2.x, right?

Why not? Assuming we don't introduce breaking changes we should be able to swap manual authoring for a generator.

Leaving this to @Xtansia.

[reta]

Why not? Assuming we don't introduce breaking changes we should be able to swap manual authoring for a generator.

If we do, we need clearly document that this is new experimental module with all the limitations it has now - we do make releases from 2.x and the new module will be published

[dblock]

Why not? Assuming we don't introduce breaking changes we should be able to swap manual authoring for a generator.

If we do, we need clearly document that this is new experimental module with all the limitations it has now - we do make releases from 2.x and the new module will be published

I don't think I'm following, the generator is just a tool, the actual code of the client being released will not change.

[reta]

I don't think I'm following, the generator is just a tool, the actual code of the client being released will not change.

Correct, but will we release the tool as part of client release?

[dblock]

I don't think I'm following, the generator is just a tool, the actual code of the client being released will not change.

Correct, but will we release the tool as part of client release?

I don't think so. Extracting the tool into something releasing on itself could make sense, but otherwise no.

[reta]

I don't think so. Extracting the tool into something releasing on itself could make sense, but otherwise no.

No objections, sounds good to me

[Xtansia]

I don't think I'm following, the generator is just a tool, the actual code of the client being released will not change.

Correct, but will we release the tool as part of client release?

@reta I've removed the publish config from the build.gradle of the generator so it won't get published

[dblock]

@Xtansia merge at will

[Xtansia]

@dblock I'm happy for it to be merged if everyone else is, I can't do it myself as I'm not a maintainer

[dblock]

@dblock I'm happy for it to be merged if everyone else is, I can't do it myself as I'm not a maintainer

Oh yeah? ;) Looking forward to that spec to be fully generated.

[Xtansia]

This is great!! The only comment I have (after a high level review) looking from previous comments is: if at all there is a chance for this generator to be moved out of this repo, could we make the generator classes restricted to use for consumers? Mostly no user would end up using them but if we move out public classes out of the repo, we might end up with breaking changes and a major version increment.

@VachaShah I would think this should not be an issue as long as we aren't publishing the generator to maven etc. For someone to rely on the classes of the generator they'd need to be building from source. AFAIK Java doesn't have a good mechanism to restrict access to the classes in a useful way as they need to be public to be accessed by the classes in other parts of the generator.