feat: allow overwrite opensearch home

[shyim]

Description

Right now it is not possible to set a fixed OPENSEARCH_HOME directory.

This is required for Nix packaging as the OpenSearch files lives inside the nix store (/nix/store/random-string-opensearch/bin/opensearch). Therefore we needed to patch this to allow us to set a custom directory.

If you need maybe more background to understand we setup a own empty Opensearch home and symlink required files: https://github.com/NixOS/nixpkgs/blob/fc46a9d1e2b30321a2497e413eb1562c9cfda5b6/nixos/modules/services/search/opensearch.nix#L185-L217

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/13322/
• CommitID: 3c796e3
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[shyim]

Missing SHA for lucene-core-9.6.0-snapshot-a3ae27f.jar. Run "gradle updateSHAs" to create them

I guess this is not related to this PR

[reta]

Missing SHA for lucene-core-9.6.0-snapshot-a3ae27f.jar. Run "gradle updateSHAs" to create them

I guess this is not related to this PR

Nope, #6959

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/13363/
• CommitID: cb95c96

[dblock]

Needs a CHANGELOG. Would also search documentation for where we mention this, and/or edit/add this, and see if we're breaking any contracts. I wish we could actually test this, too ...

I think we can backport to 2.x, @reta lmk if you feel like this is a breaking change. IMO while the behavior of setting OPENSEARCH_HOME before was "has no effect", we never spelled it out anywhere.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/13409/
• CommitID: 8165c9f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[reta]

I think we can backport to 2.x, @reta lmk if you feel like this is a breaking change. IMO while the behavior of setting OPENSEARCH_HOME before was "has no effect", we never spelled it out anywhere.

@dblock I am a bit concerned about this change (@shyim I think I do understand the issue you are trying to solve): the opensearch-env has a strict assumptions that all the bits (shell scripts, libs, bundled jdk) are located in the exact same folder as this file itself. But with this change the opensearch-env assumptions become irrelevant: you could run the script from one location but it could pick the bits from anywhere literally (whatever OPENSEARCH_HOME says).

It opens a whole class of issues (possibly even security ones?), @peterzhuamazon do you folks have an opinion on that?

[peterzhuamazon]

Thanks @shyim for the contribution and @dblock @reta for the review!

[peterzhuamazon]

Hi @opensearch-project/opensearch-core could you give an approval on this?
Thanks.

[peterzhuamazon]

@reta I think this is more of a minor verison change than patch version.
Thoughts? Currently only adding backport to 2.x and 1.x.

Thanks.

[opensearch-trigger-bot]

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-1.x 1.x
# Navigate to the new working tree
pushd ../.worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-6956-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0202de5849dcdcef61e1a2ad3b6a270574fbf2ed
# Push it to GitHub
git push --set-upstream origin backport/backport-6956-to-1.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-6956-to-1.x.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-6956-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0202de5849dcdcef61e1a2ad3b6a270574fbf2ed
# Push it to GitHub
git push --set-upstream origin backport/backport-6956-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-6956-to-2.x.

[dblock]

Looks like this will need manual backports if you want it in 2.x and 1.x :(

[andrross]

This should be in the [Unreleased 2.x] section since it is to be backported and released. This is also likely the reason the cherry pick failed on the automated backport.

[peterzhuamazon]

Hi @shyim can you take a look at above comments?
Thanks!

[shyim]

Do I have to make now the cherry picks manually for 1.x/2.x?

[vamsimanohar]

sql windows integration tests are failing with below error.

Exec output and error:
| Output for cmd:The following usage of the path operator in batch-parameter
| substitution is invalid: %~nxI
|
|
| For valid formats type CALL /? or FOR /?
| The syntax of the command is incorrect.

Is this error related to these changes. I haven't made any changes to batch file in sql code base. I still couldn't figure out the root cause. @shyim Could you please help me out here?

Integ tests kind of spin up an opensearch cluster, so it seems to me these are related. I will confirm if this is caused by my changes, by creating a new PR in sql with almost no changes.

This PR has only read me change: https://github.com/opensearch-project/sql/actions/runs/4660531784/jobs/8248701010#step:4:2063 but still fails, so probably the issue should be in upstream. @peterzhuamazon @dblock

The other related issues : opensearch-project/security#2657

[stephen-crawford]

To echo @vamsi-amazon, Security and Job Scheduler are also failing. I looked into this on the Security end and it does not appear to be related. I am going to check some stuff here, but this may require an expedited patch or revert.

The updated code could cause the error reported if the %I variable does not contain a valid path. We need to make sure %I contains a valid path before attempting to use the ~nx modifier. If %OPENSEARCH_HOME% does not contain a valid path, then the ~nx modifier will not work.

Something like this needs to be added:

for %%I in (opensearch.bat) do (
    set "OPENSEARCH_HOME=%%~dpI"
)

to fix the issue.

I think that #7079 should fix the issue but I am not sure how to test its impact on downstream repos without merging. @dblock what would you recommend?

[peterzhuamazon]

Hi @shyim @vamsi-amazon @DarshitChanpura @scrawfor99 ,

Have a solution in this PR:

• Resolve windows opensearch-env.bat having variable delayed expansion issues on OPENSEARCH_HOME assignment introduced in #6956 #7080