Merge branch 'main' into ignore-hidden-indexes

CHANGELOG.md

@@ -3,9 +3,7 @@
 Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ## [Unreleased]
-
 ### Added
-
 - Add support for s390x architecture ([#4001](https://github.com/opensearch-project/OpenSearch/pull/4001))
 - Github workflow for changelog verification ([#4085](https://github.com/opensearch-project/OpenSearch/pull/4085))
 - Point in time rest layer changes for create and delete PIT API ([#4064](https://github.com/opensearch-project/OpenSearch/pull/4064))
@@ -29,15 +27,25 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Added release notes for 1.3.6 ([#4681](https://github.com/opensearch-project/OpenSearch/pull/4681))
 - Added precommit support for MacOS ([#4682](https://github.com/opensearch-project/OpenSearch/pull/4682))
 - Recommission API changes for service layer ([#4320](https://github.com/opensearch-project/OpenSearch/pull/4320))
+- Update GeoGrid base class access modifier to support extensibility ([#4572](https://github.com/opensearch-project/OpenSearch/pull/4572))
+- Add a new node role 'search' which is dedicated to provide search capability ([#4689](https://github.com/opensearch-project/OpenSearch/pull/4689))
+- Introduce experimental searchable snapshot API ([#4680](https://github.com/opensearch-project/OpenSearch/pull/4680))
+- Recommissioning of zone. REST layer support. ([#4624](https://github.com/opensearch-project/OpenSearch/pull/4604))
+- Added in-flight cancellation of SearchShardTask based on resource consumption ([#4565](https://github.com/opensearch-project/OpenSearch/pull/4565))
+- Apply reproducible builds configuration for OpenSearch plugins through gradle plugin ([#4746](https://github.com/opensearch-project/OpenSearch/pull/4746))
+- Add groupId value propagation tests for ZIP publication task ([#4772](https://github.com/opensearch-project/OpenSearch/pull/4772))
+
 ### Dependencies
 - Bumps `log4j-core` from 2.18.0 to 2.19.0
 - Bumps `reactor-netty-http` from 1.0.18 to 1.0.23
 - Bumps `jettison` from 1.5.0 to 1.5.1
 - Bumps `azure-storage-common` from 12.18.0 to 12.18.1
 - Bumps `forbiddenapis` from 3.3 to 3.4
-
-
-### Dependencies
+- Bumps `gson` from 2.9.0 to 2.9.1
+- Bumps `protobuf-java` from 3.21.2 to 3.21.7k
+- Bumps `azure-core` from 1.31.0 to 1.33.0
+- Bumps `avro` from 1.11.0 to 1.11.1
+- Bumps `woodstox-core` from 6.3.0 to 6.3.1
 - Bumps `xmlbeans` from 5.1.0 to 5.1.1 ([#4354](https://github.com/opensearch-project/OpenSearch/pull/4354))
 - Bumps `azure-core-http-netty` from 1.12.0 to 1.12.4 ([#4160](https://github.com/opensearch-project/OpenSearch/pull/4160))
 - Bumps `azure-core` from 1.27.0 to 1.31.0 ([#4160](https://github.com/opensearch-project/OpenSearch/pull/4160))
@@ -50,9 +58,12 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Bumps `jempbox` from 1.8.16 to 1.8.17 ([#4550](https://github.com/opensearch-project/OpenSearch/pull/4550))
 - Bumps `hadoop-hdfs` from 3.3.3 to 3.3.4 ([#4644](https://github.com/opensearch-project/OpenSearch/pull/4644))
 - Bumps `jna` from 5.11.0 to 5.12.1 ([#4656](https://github.com/opensearch-project/OpenSearch/pull/4656))
+- Update Jackson Databind to 2.13.4.2 (addressing CVE-2022-42003) ([#4779](https://github.com/opensearch-project/OpenSearch/pull/4779))
+- Bumps `tika` from 2.4.0 to 2.5.0 ([#4791](https://github.com/opensearch-project/OpenSearch/pull/4791))
+- Exclude jettison version brought in with hadoop-minicluster. ([#4787](https://github.com/opensearch-project/OpenSearch/pull/4787))
+- Bump protobuf-java to 3.21.7 in repository-gcs and repository-hdfs ([#]())
 
 ### Changed
-
 - Dependency updates (httpcore, mockito, slf4j, httpasyncclient, commons-codec) ([#4308](https://github.com/opensearch-project/OpenSearch/pull/4308))
 - Use RemoteSegmentStoreDirectory instead of RemoteDirectory ([#4240](https://github.com/opensearch-project/OpenSearch/pull/4240))
 - Plugin ZIP publication groupId value is configurable ([#4156](https://github.com/opensearch-project/OpenSearch/pull/4156))
@@ -65,23 +76,32 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - GET api for weighted shard routing([#4275](https://github.com/opensearch-project/OpenSearch/pull/4275/))
 - Unmute test RelocationIT.testRelocationWhileIndexingRandom ([#4580](https://github.com/opensearch-project/OpenSearch/pull/4580))
 - Add DecommissionService and helper to execute awareness attribute decommissioning ([#4084](https://github.com/opensearch-project/OpenSearch/pull/4084))
+- Delete api for weighted shard routing([#4400](https://github.com/opensearch-project/OpenSearch/pull/4400/))
 - Further simplification of the ZIP publication implementation ([#4360](https://github.com/opensearch-project/OpenSearch/pull/4360))
 - Relax visibility of the HTTP_CHANNEL_KEY and HTTP_SERVER_CHANNEL_KEY to make it possible for the plugins to access associated Netty4HttpChannel / Netty4HttpServerChannel instance ([#4638](https://github.com/opensearch-project/OpenSearch/pull/4638))
 - Load the deprecated master role in a dedicated method instead of in setAdditionalRoles() ([#4582](https://github.com/opensearch-project/OpenSearch/pull/4582))
 - Include Windows OS in Bootstrap initializeNatives() check for definitelyRunningAsRoot() ([#4656](https://github.com/opensearch-project/OpenSearch/pull/4656))
 - Add APIs (GET/PUT) to decommission awareness attribute ([#4261](https://github.com/opensearch-project/OpenSearch/pull/4261))
 - Improve Gradle pre-commit checks to pre-empt Jenkins build ([#4660](https://github.com/opensearch-project/OpenSearch/pull/4660))
 - Update to Apache Lucene 9.4.0 ([#4661](https://github.com/opensearch-project/OpenSearch/pull/4661))
+- Controlling discovery for decommissioned nodes ([#4590](https://github.com/opensearch-project/OpenSearch/pull/4590))
 - Backport Apache Lucene version change for 2.4.0 ([#4677](https://github.com/opensearch-project/OpenSearch/pull/4677))
-
+- Use ReplicationFailedException instead of OpensearchException in ReplicationTarget ([#4725](https://github.com/opensearch-project/OpenSearch/pull/4725))
+- Fix weighted routing metadata deserialization error on process restart ([#4691](https://github.com/opensearch-project/OpenSearch/pull/4691))
+- Refactor Base Action class javadocs to OpenSearch.API ([#4732](https://github.com/opensearch-project/OpenSearch/pull/4732))
+- Migrate client transports to Apache HttpClient / Core 5.x ([#4459](https://github.com/opensearch-project/OpenSearch/pull/4459))
+- Refactored BalancedAllocator.Balancer to LocalShardsBalancer ([#4761](https://github.com/opensearch-project/OpenSearch/pull/4761))
 ### Deprecated
-
 ### Removed
 - Remove deprecated code to add node name into log pattern of log4j property file ([#4568](https://github.com/opensearch-project/OpenSearch/pull/4568))
 - Unused object and import within TransportClusterAllocationExplainAction ([#4639](https://github.com/opensearch-project/OpenSearch/pull/4639))
+- Remove LegacyESVersion.V_7_0_* and V_7_1_* Constants ([#2768](https://https://github.com/opensearch-project/OpenSearch/pull/2768))
+- Remove LegacyESVersion.V_7_2_ and V_7_3_ Constants ([#4702](https://github.com/opensearch-project/OpenSearch/pull/4702))
+- Always auto release the flood stage block ([#4703](https://github.com/opensearch-project/OpenSearch/pull/4703))
+- Remove LegacyESVersion.V_7_4_ and V_7_5_ Constants ([#4704](https://github.com/opensearch-project/OpenSearch/pull/4704))
+- Remove Legacy Version support from Snapshot/Restore Service ([#4728](https://github.com/opensearch-project/OpenSearch/pull/4728))
 
 ### Fixed
-
 - `opensearch-service.bat start` and `opensearch-service.bat manager` failing to run ([#4289](https://github.com/opensearch-project/OpenSearch/pull/4289))
 - PR reference to checkout code for changelog verifier ([#4296](https://github.com/opensearch-project/OpenSearch/pull/4296))
 - `opensearch.bat` and `opensearch-service.bat install` failing to run, missing logs directory ([#4305](https://github.com/opensearch-project/OpenSearch/pull/4305))
@@ -121,36 +141,37 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Fix flaky DecommissionControllerTests.testTimesOut ([4683](https://github.com/opensearch-project/OpenSearch/pull/4683))
 - Fix new race condition in DecommissionControllerTests ([4688](https://github.com/opensearch-project/OpenSearch/pull/4688))
 - Fix SearchStats (de)serialization (caused by https://github.com/opensearch-project/OpenSearch/pull/4616) ([#4697](https://github.com/opensearch-project/OpenSearch/pull/4697))
+- Fixing Gradle warnings associated with publishPluginZipPublicationToXxx tasks ([#4696](https://github.com/opensearch-project/OpenSearch/pull/4696))
+- [BUG]: Remove redundant field from GetDecommissionStateResponse ([#4751](https://github.com/opensearch-project/OpenSearch/pull/4751))
+- Fixed randomly failing test ([4774](https://github.com/opensearch-project/OpenSearch/pull/4774))
+- Update version check after backport ([4786](https://github.com/opensearch-project/OpenSearch/pull/4786))
+- Fix decommission status update to non leader nodes ([4800](https://github.com/opensearch-project/OpenSearch/pull/4800))
+- Fix recovery path for searchable snapshots ([4813](https://github.com/opensearch-project/OpenSearch/pull/4813))
+- Fix bug in AwarenessAttributeDecommissionIT([4822](https://github.com/opensearch-project/OpenSearch/pull/4822))
 
 ### Security
-
 - CVE-2022-25857 org.yaml:snakeyaml DOS vulnerability ([#4341](https://github.com/opensearch-project/OpenSearch/pull/4341))
 
 ## [2.x]
-
 ### Added
-
 - Github workflow for changelog verification ([#4085](https://github.com/opensearch-project/OpenSearch/pull/4085))
 - Label configuration for dependabot PRs ([#4348](https://github.com/opensearch-project/OpenSearch/pull/4348))
 - Added RestLayer Changes for PIT stats ([#4217](https://github.com/opensearch-project/OpenSearch/pull/4217))
 - Added GeoBounds aggregation on GeoShape field type.([#4266](https://github.com/opensearch-project/OpenSearch/pull/4266))
   - Addition of Doc values on the GeoShape Field
   - Addition of GeoShape ValueSource level code interfaces for accessing the DocValues.
   - Addition of Missing Value feature in the GeoShape Aggregations.
+- Install and configure Log4j JUL Adapter for Lucene 9.4 ([#4754](https://github.com/opensearch-project/OpenSearch/pull/4754))
 - Added feature to ignore indexes starting with dot during shard limit validation.([#4695](https://github.com/opensearch-project/OpenSearch/pull/4695))
-
 ### Changed
-
 ### Deprecated
-
 ### Removed
-
 ### Fixed
-
 - PR reference to checkout code for changelog verifier ([#4296](https://github.com/opensearch-project/OpenSearch/pull/4296))
 - Commit workflow for dependabot changelog helper ([#4331](https://github.com/opensearch-project/OpenSearch/pull/4331))
+- Better plural stemmer than minimal_english ([#4738](https://github.com/opensearch-project/OpenSearch/pull/4738))
+- Disable merge on refresh in DiskThresholdDeciderIT ([#4828](https://github.com/opensearch-project/OpenSearch/pull/4828))
 
 ### Security
-
 [Unreleased]: https://github.com/opensearch-project/OpenSearch/compare/2.2.0...HEAD
 [2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.2.0...2.x

buildSrc/src/main/groovy/org/opensearch/gradle/plugin/PluginBuildPlugin.groovy

@@ -29,13 +29,13 @@
 package org.opensearch.gradle.plugin
 
 import com.github.jengelman.gradle.plugins.shadow.ShadowPlugin
+import org.gradle.api.tasks.bundling.AbstractArchiveTask
 import org.opensearch.gradle.BuildPlugin
 import org.opensearch.gradle.NoticeTask
 import org.opensearch.gradle.Version
 import org.opensearch.gradle.VersionProperties
 import org.opensearch.gradle.dependencies.CompileOnlyResolvePlugin
 import org.opensearch.gradle.info.BuildParams
-import org.opensearch.gradle.plugin.PluginPropertiesExtension
 import org.opensearch.gradle.test.RestTestBasePlugin
 import org.opensearch.gradle.testclusters.RunTask
 import org.opensearch.gradle.util.Util
@@ -134,6 +134,12 @@ class PluginBuildPlugin implements Plugin<Project> {
         }
         project.configurations.getByName('default')
                 .extendsFrom(project.configurations.getByName('runtimeClasspath'))
+        project.tasks.withType(AbstractArchiveTask.class).configureEach { task ->
+            // ignore file timestamps
+            // be consistent in archive file order
+            task.preserveFileTimestamps = false
+            task.reproducibleFileOrder = true
+        }
         // allow running ES with this plugin in the foreground of a build
         project.tasks.register('run', RunTask) {
             dependsOn(project.tasks.bundlePlugin)

buildSrc/src/main/java/org/opensearch/gradle/PublishPlugin.java

@@ -92,7 +92,7 @@ public String call() throws Exception {
                 return String.format(
                     "%s/distributions/%s-%s.pom",
                     project.getBuildDir(),
-                    getArchivesBaseName(project),
+                    pomTask.getName().toLowerCase().contains("zip") ? project.getName() : getArchivesBaseName(project),
                     project.getVersion()
                 );
             }
@@ -130,7 +130,6 @@ public String call() throws Exception {
             publication.getPom().withXml(PublishPlugin::addScmInfo);
 
             if (!publication.getName().toLowerCase().contains("zip")) {
-
                 // have to defer this until archivesBaseName is set
                 project.afterEvaluate(p -> publication.setArtifactId(getArchivesBaseName(project)));
 
@@ -139,6 +138,8 @@ public String call() throws Exception {
                     publication.artifact(project.getTasks().getByName("sourcesJar"));
                     publication.artifact(project.getTasks().getByName("javadocJar"));
                 }
+            } else {
+                project.afterEvaluate(p -> publication.setArtifactId(project.getName()));
             }
 
             generatePomTask.configure(

buildSrc/src/main/java/org/opensearch/gradle/pluginzip/Publish.java

@@ -13,12 +13,14 @@
 import org.gradle.api.publish.maven.MavenPublication;
 
 import java.nio.file.Path;
+import java.util.Set;
+import java.util.stream.Collectors;
+
 import org.gradle.api.Task;
 import org.gradle.api.publish.maven.plugins.MavenPublishPlugin;
 
 public class Publish implements Plugin<Project> {
 
-    // public final static String PLUGIN_ZIP_PUBLISH_POM_TASK = "generatePomFileForPluginZipPublication";
     public final static String PUBLICATION_NAME = "pluginZip";
     public final static String STAGING_REPO = "zipStaging";
     public final static String LOCAL_STAGING_REPO_PATH = "/build/local-staging-repo";
@@ -67,10 +69,15 @@ public void apply(Project project) {
                 if (validatePluginZipPom != null) {
                     validatePluginZipPom.dependsOn("generatePomFileForNebulaPublication");
                 }
-                Task publishPluginZipPublicationToZipStagingRepository = project.getTasks()
-                    .findByName("publishPluginZipPublicationToZipStagingRepository");
-                if (publishPluginZipPublicationToZipStagingRepository != null) {
-                    publishPluginZipPublicationToZipStagingRepository.dependsOn("generatePomFileForNebulaPublication");
+
+                // There are number of tasks prefixed by 'publishPluginZipPublication', f.e.:
+                // publishPluginZipPublicationToZipStagingRepository, publishPluginZipPublicationToMavenLocal
+                final Set<Task> publishPluginZipPublicationToTasks = project.getTasks()
+                    .stream()
+                    .filter(t -> t.getName().startsWith("publishPluginZipPublicationTo"))
+                    .collect(Collectors.toSet());
+                if (!publishPluginZipPublicationToTasks.isEmpty()) {
+                    publishPluginZipPublicationToTasks.forEach(t -> t.dependsOn("generatePomFileForNebulaPublication"));
                 }
             } else {
                 project.getLogger()

buildSrc/src/main/resources/forbidden/http-signatures.txt

@@ -15,31 +15,14 @@
 # language governing permissions and limitations under the License.
 
 @defaultMessage Explicitly specify the ContentType of HTTP entities when creating
-org.apache.http.entity.StringEntity#<init>(java.lang.String)
-org.apache.http.entity.StringEntity#<init>(java.lang.String,java.lang.String)
-org.apache.http.entity.StringEntity#<init>(java.lang.String,java.nio.charset.Charset)
-org.apache.http.entity.ByteArrayEntity#<init>(byte[])
-org.apache.http.entity.ByteArrayEntity#<init>(byte[],int,int)
-org.apache.http.entity.FileEntity#<init>(java.io.File)
-org.apache.http.entity.InputStreamEntity#<init>(java.io.InputStream)
-org.apache.http.entity.InputStreamEntity#<init>(java.io.InputStream,long)
-org.apache.http.nio.entity.NByteArrayEntity#<init>(byte[])
-org.apache.http.nio.entity.NByteArrayEntity#<init>(byte[],int,int)
-org.apache.http.nio.entity.NFileEntity#<init>(java.io.File)
-org.apache.http.nio.entity.NStringEntity#<init>(java.lang.String)
-org.apache.http.nio.entity.NStringEntity#<init>(java.lang.String,java.lang.String)
+org.apache.hc.core5.http.io.entity.StringEntity#<init>(java.lang.String)
+org.apache.hc.core5.http.io.entity.StringEntity#<init>(java.lang.String,java.nio.charset.Charset)
 
 @defaultMessage Use non-deprecated constructors
-org.apache.http.nio.entity.NFileEntity#<init>(java.io.File,java.lang.String)
-org.apache.http.nio.entity.NFileEntity#<init>(java.io.File,java.lang.String,boolean)
-org.apache.http.entity.FileEntity#<init>(java.io.File,java.lang.String)
-org.apache.http.entity.StringEntity#<init>(java.lang.String,java.lang.String,java.lang.String)
+org.apache.hc.core5.http.io.entity.FileEntity#<init>(java.io.File,org.apache.hc.core5.http.ContentType)
 
 @defaultMessage BasicEntity is easy to mess up and forget to set content type
-org.apache.http.entity.BasicHttpEntity#<init>()
-
-@defaultMessage EntityTemplate is easy to mess up and forget to set content type
-org.apache.http.entity.EntityTemplate#<init>(org.apache.http.entity.ContentProducer)
+org.apache.hc.core5.http.io.entity.BasicHttpEntity#<init>(java.io.InputStream,org.apache.hc.core5.http.ContentType)
 
 @defaultMessage SerializableEntity uses java serialization and makes it easy to forget to set content type
-org.apache.http.entity.SerializableEntity#<init>(java.io.Serializable)
+org.apache.hc.core5.http.io.entity.SerializableEntity#<init>(java.io.Serializable,org.apache.hc.core5.http.ContentType)

buildSrc/src/test/java/org/opensearch/gradle/plugin/PluginBuildPluginTests.java

@@ -31,6 +31,7 @@
 
 package org.opensearch.gradle.plugin;
 
+import org.gradle.api.tasks.bundling.AbstractArchiveTask;
 import org.opensearch.gradle.BwcVersions;
 import org.opensearch.gradle.test.GradleUnitTestCase;
 import org.gradle.api.Project;
@@ -64,6 +65,10 @@ public void testApply() {
         assertNotNull("plugin extensions has the right type", project.getExtensions().findByType(PluginPropertiesExtension.class));
 
         assertNull("plugin should not create the integTest task", project.getTasks().findByName("integTest"));
+        project.getTasks().withType(AbstractArchiveTask.class).forEach(t -> {
+            assertFalse(String.format("task '%s' should not preserve timestamps", t.getName()), t.isPreserveFileTimestamps());
+            assertTrue(String.format("task '%s' should have reproducible file order", t.getName()), t.isReproducibleFileOrder());
+        });
     }
 
     @Ignore("https://github.com/elastic/elasticsearch/issues/47123")