publish minimal distribution to maven

[rursprung]

Is your feature request related to a problem? Please describe

currently, automated builds (e.g. of in-house docker images) don't have a good way to automatically pull in the .tar.gz file of the minimal distribution: the artifact is just offered via HTTPS, the link has to be known.

Describe the solution you'd like

the minimal distribution artifact (e.g. https://artifacts.opensearch.org/releases/core/opensearch/1.3.1/opensearch-min-1.3.1-linux-x64.tar.gz) should be published to maven.

that way, automated build systems can be used to pull in the artifact w/o having to know an exact URL or requiring direct internet access (it can then go via an in-house maven server which serves as a proxy and also caches the artifact for later re-use).

since maven artifacts can be accessed using HTTP(S) this also means that this could - theoretically - even be used as the primary download location in the future (i.e. replacing the current download paths for (new) artifacts).

Describe alternatives you've considered

any form of using HTTP(S) fails for several reasons:

• the link isn't known / would have to be stored in the build (not an absolute show-stopper, but definitely not something nice if you can also have the maven approach where you just know the artifact name & version)
• the build server might not have internet access
• there's no built-in caching support, i.e. automated builds would always hit the public download rather than re-using a locally cached version (be that cached on the build server or some intermediate proxy)

there would be workarounds, e.g. having a dedicated proxy server for this, but that'd be a huge overkill and would require this to be in place everywhere rather than just having the artifacts deployed to maven once.

Additional context

• coming from Publish plugin distribution ZIPs to Maven, get rid of checked in snapshot builds #716 (comment)
• Publish plugin distribution ZIPs to Maven, get rid of checked in snapshot builds #716 (comment)
• Publish snapshots and release artifacts to maven #20 (comment)
• Publish snapshots and release artifacts to maven #20 (comment)
• this is essentially the same as publish full distribution to maven #2069 but for the minimal distribution

[prudhvigodithi]

[Triage] We will update more about this issue once this meta #1916 to publish zips to maven repo is taken care.

[rursprung]

FYI: it seems that someone (atlassian?) at some point already uploaded some tar fields of some releases (lots of "some" in this sentence 😆) to the atlassian maven repo: https://mvnrepository.com/artifact/org.opensearch.distribution

once this is done properly centrally then those other usages can go away.

[prudhvigodithi]

Adding @elfisher @setiah, Since min's are not production grade distributions, can you add your thoughts for publishing min distribution to maven repo?
Thank you

[dblock]

I don't see why not, except that it would be duplicate authoritative content. But since we publish plugin zips, we might as well publish a min distribution the same way.

[setiah]

I don't see any problem with this either. We already publish min artifacts on artifact.opensearch.org, so publishing it to maven does not seem like additional risk to me. It would rather help simplify building things for the community like plugin development

[bbarani]

@setiah @dblock I don’t see an issue either but we were specifically requested not to publish min bundles on other distribution channels (including Docker) as it lacks security features. We won’t be able to control / advertise the usage once it’s in Maven though.

[rursprung]

see my comment on #2069 on the general usefulness of having this on maven.

@bbarani: for the min artefacts: i see a big difference between a docker image (which you can just grab & run and don't think twice about) and an artefact on maven (generally only used by enterprise-y setups / people who hopefully know what they're doing). if you're grabbing something like this off of maven you have a pretty good idea of what you're doing and will be adding the necessary security plugin (OpenSearch Security, Search Guard FLX or whatever else might be on the market in the future) or explicitly don't add one if such a use-case would exist.

[bbarani]

@rursprung thanks for your inputs. @elfisher @CEHENKLE @dblock @setiah Please provide your inputs as well to proceed further.

[dblock]

I am fine giving up pushing back on also publishing everything to maven - people (@rursprung really) are asking for it, so YOLO.

[CEHENKLE]

It makes my palms sweat, because I don't want someone accidentally using the min to store data they care about, but I see the utility of it. @davidlago let's hurry up and get security into core :)

Is there anything else we can do to make it really, really clear "only use this for testing/dev"?

[rursprung]

Is there anything else we can do to make it really, really clear "only use this for testing/dev"?

you could in general add a README file to the distribution ZIPs which give a quick-start for them. for min it could say "this has no security built in, we strongly recommend you use the full distribution unless you know exactly what you're doing", and for both of them it could give some links to the quick-start guide in the documentation?

[prudhvigodithi]

Just a thought how about in the startup logs? A warning message that throws a CAUTION message (just for min artifact), but this should have a code added to the min distribution.

[rursprung]

Just a thought how about in the startup logs? A warning message that throws a CAUTION message (just for min artifact), but this should have a code added to the min distribution.

how do you know if you have a security plugin or not? unless you have some "is security plugin enabled" API which the security plugin has to implement to tell OpenSearch that it's there? it might not be the OpenSearch Security plugin which is running and providing it, so you also can't just filter the plugin list to check it.

[prudhvigodithi]

Some mechanism if its not a full bundle vs min, if not full bundle then add a message.

[rursprung]

Some mechanism if its not a full bundle vs min, if not full bundle then add a message.

but we're not using a full bundle but still add OpenSearch Security. i don't want a warning in that case (probably nobody would notice it... but if somebody does, good luck trying to explain to them why they get a warning on startup about missing security even though security is in place)

[dblock]

@CEHENKLE want to write up an issue in OpenSearch that says "help users understand they should not use min without security"?

I propose not to over-engineer this and spend all our time doing what's best for users, ie. having security enabled by default in core so that those who disable it do it deliberately.

[rursprung]

note: once #99 is fully implemented the no-jdk version of the distribution should be published to maven as part of this step here.