can not write type [class org.elasticsearch.Version]

[LeeDr]

Elasticsearch version (bin/elasticsearch --version): 7.16.0 BC3

Plugins installed: []

JVM version (java -version):

OS version (uname -a if on a Unix-like system): Cloud GCP

Description of the problem including expected versus actual behavior: a call to GET /_migration/deprecations gives an error response;

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "can not write type [class org.elasticsearch.Version]"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : ".ml-anomalies-shared",
        "node" : "czQpoMmrReqHYj1MO35irQ",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "can not write type [class org.elasticsearch.Version]"
        }
      }
    ],
    "caused_by" : {
      "type" : "",
      "reason" : ": can not write type [class org.elasticsearch.Version]",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "can not write type [class org.elasticsearch.Version]"
      }
    }
  },
  "status" : 400
}

Steps to reproduce:

Much of this was done by the Cloud infrastructure so I can only say the steps I performed

1. Create a new 7.16.0 Cloud deployment on GCP Belgium region and select to restore snapshots from my existing 7.15.1 Cloud deployment (it's a bit large so in my case it took around 2 hours)
2. Check in Kibana Upgrade Assistant for Elasticsearch critical deprecations and it shows 16 critical deprecations about _tier_preference (1 of the same for each of 16 indices). I updated 1 ilm index and the count of critical deprecations went down from 16 to 15. I didn't do anything with the .ml-anomalies-shared index.
3. The next day, the Upgrade Assistant has an error showing the Elasticsearch deprecations. It was suggested I try this in the Kibana Dev Tools console GET /_migration/deprecations. See the error.

Provide logs (if relevant):

[instance-0000000000] failed to serialize outbound message [Request{indices:data/read/search[phase/query]}{3087052}{false}{false}{false}]
java.lang.IllegalArgumentException: can not write type [class org.elasticsearch.Version]
	at org.elasticsearch.common.io.stream.StreamOutput.writeGenericValue(StreamOutput.java:847) ~[elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.search.sort.FieldSortBuilder.writeTo(FieldSortBuilder.java:163) ~[elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1117) ~[elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.search.builder.SearchSourceBuilder.writeTo(SearchSourceBuilder.java:313) ~[elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:978) ~[elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.search.internal.ShardSearchRequest.innerWriteTo(ShardSearchRequest.java:333) ~[elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.search.internal.ShardSearchRequest.writeTo(ShardSearchRequest.java:319) ~[elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:75) ~[elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:180) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.transport.OutboundHandler.sendRequest(OutboundHandler.java:109) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.transport.TcpTransport$NodeChannels.sendRequest(TcpTransport.java:288) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:975) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:206) [x-pack-security-7.16.0.jar:7.16.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$300(SecurityServerTransportInterceptor.java:53) [x-pack-security-7.16.0.jar:7.16.0]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:167) [x-pack-security-7.16.0.jar:7.16.0]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:874) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:937) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:925) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:250) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:104) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$3(AbstractSearchAsyncAction.java:328) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.performPhaseOnShard(AbstractSearchAsyncAction.java:364) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:517) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:64) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:343) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:651) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.transport.TransportService$4.handleException(TransportService.java:853) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1481) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.transport.TransportService$5.doRun(TransportService.java:1016) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:771) [elasticsearch-7.16.0.jar:7.16.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.0.jar:7.16.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]

[dnhatn]

I took a look at the serialization of FieldSortBuilder. I think the issue is related to the below usage. We can't use an Elasticsearch version as a missing value in a sort spec. @benwtrent Can you take a look at this issue please? Thank you!

elasticsearch/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/job/persistence/JobResultsProvider.java

Lines 1273 to 1275 in bcfbf00

	if (sortField.equals(ModelSnapshot.MIN_VERSION.getPreferredName())) {
	sb.missing(Version.fromString("6.3.0")).unmappedType("keyword");
	}

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[benwtrent]

@dnhatn that is almost 100% the cause. Confirming with a test, will have a fix ASAP

[benwtrent]

@LeeDr, Remind me to get you 🍻 next time we get together :)