[Feature] Cross cluster field mappings query

[seankao-az]

Is your feature request related to a problem? Please describe.
Yes. We’re trying to support cross cluster search in the OpenSearch sql plugin. As part of our sql and ppl query execution flow, some metadata is required for query analysis and plan generation. We need field mappings for each index at remote clusters.

Related issue: opensearch-project/sql#789

Here are some example responses of REST api for mappings request. Though we will not be using the REST api directly for our use case, this shows the problem more clearly:

• GET /remote-cluster:index/_mappings

{
    "error": {
        "root_cause": [
            {
                "type": "index_not_found_exception",
                "reason": "no such index [ccs-cluster:index]",
                "index": "ccs-cluster:index",
                "resource.id": "ccs-cluster:index",
                "resource.type": "index_or_alias",
                "index_uuid": "_na_"
            }
        ],
        "type": "index_not_found_exception",
        "reason": "no such index [ccs-cluster:index]",
        "index": "ccs-cluster:index",
        "resource.id": "ccs-cluster:index",
        "resource.type": "index_or_alias",
        "index_uuid": "_na_"
    },
    "status": 404
}

• GET /*:index/_mappings

{
}

A side note on the second request: it returns nothing, which we consider a bug.

Describe the solution you'd like
We need some transport layer actions to support querying field mappings on cross clusters. We have no clear solution on how to achieve that. The requirement is that we can, for example, provide indices name with cross cluster syntax (*:index, ccs-cluster:index) for get mappings requests. For example, in this function.

Describe alternatives you've considered
We could get such info by making individual connections to each of the remote clusters, but that defeats the purpose of using cross cluster search in the first place. Following this path, we’ll also have to reinvent the wheel of cluster name resolving and request re-routing.

Additional context
Some context on how we’re querying field mappings currently:

• https://github.com/opensearch-project/sql/blob/18661b4ed0a6f613ae91b052774751a0ea4f17c7/opensearch/src/main/java/org/opensearch/sql/opensearch/client/OpenSearchNodeClient.java#L90
• https://github.com/opensearch-project/sql/blob/18661b4ed0a6f613ae91b052774751a0ea4f17c7/opensearch/src/main/java/org/opensearch/sql/opensearch/client/OpenSearchRestClient.java#L71

[minalsha]

@mch2 , @andrross. This requires some more deep dive. Can this be done without field mappers not registered on a cluster? This we think would be possible with Extensions, however this would be long term. cc @dagneyb

[vamsimanohar]

Want to revive this issue, as we are planning to develop cross cluster tracing solutions.
cc: @anirudha @msfroh

[msfroh]

One thought is that you have a NodeClient from the plugin, which has the following method:

    @Override
    public Client getRemoteClusterClient(String clusterAlias) {
        return remoteClusterService.getRemoteClusterClient(threadPool(), clusterAlias);
    }

Of course, you'd still need to piece out the cluster alias from the index name. It would be nice if TransportGetMappingsAction could transparently search remote clusters the way that TransportSearchAction does.

Actually, while digging a big, I learned about the field capabilities (or _field_caps) API, which may give you exactly the mapping info that you're looking for. As a bonus, it looks like it already does have cross-cluster fetching:

OpenSearch/server/src/main/java/org/opensearch/action/fieldcaps/TransportFieldCapabilitiesAction.java

Lines 145 to 147 in aca2e9d

	// this is the cross cluster part of this API - we force the other cluster to not merge the results but instead
	// send us back all individual index results.
	for (Map.Entry<String, OriginalIndices> remoteIndices : remoteClusterIndices.entrySet()) {