[FEATURE] Make the job scheduler lock index a system index

[bowenlan-amzn]

The job scheduler lock index (.opendistro-job-scheduler-lock) is used fully internally and shouldn't be changed by user.
If user inadvertently change this index, it could cause jobs to not running anymore.
Making it a system index can protect it from being changed by user.

[cwperks]

@joshpalis This may be a good first issue and should be straightforward. There is always a chance when adding a system index that the plugin needs to be re-factored to stash the thread context before trying to interact with the system index so testing needs to be done to ensure the plugin works as expected.

For background, a system index has special protections so that not even the admin can delete it. It is there to prevent a cluster admin from inadvertently corrupting the cluster and have opensearch and its plugins not behave as expected.

Assuming that no re-factoring is needed, then this index needs to be added in the following 2 files of the security plugin:

• https://github.com/opensearch-project/security/blob/main/tools/install_demo_configuration.sh#L386
• https://github.com/opensearch-project/security/blob/main/tools/install_demo_configuration.bat#L318

This setting in opensearch.yml is what gives indices system index protection provided by the security plugin.

[cwperks]

I don't see any instances of stashContext in the code so that would need to be added before the system index can be added in the security plugin. Can I take on this task?

Here are 2 places where the index is interacted with:

• LockService.createLockIndex - https://github.com/opensearch-project/job-scheduler/blob/main/spi/src/main/java/org/opensearch/jobscheduler/spi/utils/LockService.java#L79-L95
• LockService.acquireLockWithId - https://github.com/opensearch-project/job-scheduler/blob/main/spi/src/main/java/org/opensearch/jobscheduler/spi/utils/LockService.java#L144-L176

[joshpalis]

Hi @cwperks sure, Ill assign this to you