[BUG] "Request failed to get to the server (status code: 200)" by specific message

[aelimport]

Describe the bug

I was debuggin an error which appeared after a service wrote a log message to opensearch what resulted in Request failed to get to the server (status code: 200) in the search (by query inside the DevTools) and JSON.parse: expected ',' or '}' after property value in object at line 1 column 9444 of the JSON data on the discover-dashboard.
I have then tried to find out which part of the message makes the failure appear.
It seems, that a value of any field which is equal ,14130001000004100, results in the failure I can not identify.
In the "Discover" it results in a failure if this document is in the range of the results (like in the screenshot). The application logs don't show anything what could explain this.

Related component

Indexing

To Reproduce

POST message in the Dev Tools

POST <index>/_doc
{"request": ",14130001000004100,"}

Response

{
  "_index": "<index>",
  "_id": "CHfYN44BjF2d8J26W8hQ",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 15680046,
  "_primary_term": 1
}

Try to find the message in Dev-Tools

GET <index>/_search
{
  "query": {
    "match": {
      "_id": "CHfYN44BjF2d8J26W8hQ"
    }
  }
}

Response
Request failed to get to the server (status code: 200)

And an error in UI should appear

Expected behavior

Remove any sign of the value. In the following message '0' before the last ',' is removed

POST message in the Dev Tools

POST <index>/_doc
{"request": ",1413000100000410,"}

Response

{
  "_index": "<index>",
  "_id": "dnfgN44BjF2d8J26ssi0",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 15680107,
  "_primary_term": 1
}

Try to find the message in Dev-Tools

GET <index>/_search
{
  "query": {
    "match": {
      "_id": "dnfgN44BjF2d8J26ssi0"
    }
  }
}

Response

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "<index>",
        "_id": "dnfgN44BjF2d8J26ssi0",
        "_score": 1,
        "_source": {
          "request": ",1413000100000410,"
        }
      }
    ]
  }
}

Additional Details

Addition informations
Opensearch version: v.2.12.0
Browser: Firefox 123.0.1

• instance installed in the docker environment
• accessed through reverse proxy with self-signed SSL-Certificate
• message is collected by Filebeat and send to logstash
• original format is json so no transformations inside logstash pipeline except the addition of tags

[andrross]

[Triage - attendees 1 2 3]
@opensearch-project/triage Can you please transfer this to the https://github.com/opensearch-project/OpenSearch-Dashboards repository?

[ElectricWeasel]

We are experiencing the same issue.

Regards
Rafał

[AMoo-Miki]

#6245 and #6253 should take care of this for ever. These are included in the upcoming 2.13 release. Please reopen if you continue you to face the problem wen using the new release.

[rlueckl]

I've just tried 2.13.0 and it seems that the bug is still there (at least partially). Please check again! @AMoo-Miki

My minimal_example.json doesn't result in an error anymore, but with a sufficiently long message OpenSearch Dashboards still goes into a(n endless?) loop.

In my case it's a Cassandra log (same as before), specifically from PaxosCleanupLocalCoordinator.java:185, if you happen to have a Cassandra installed. The log line ("message" field) looks like this:

"message": "INFO [OptionalTasks:1] 2024-04-11 13:44:32,607 PaxosCleanupLocalCoordinator.java:185 - Completed 0 uncommitted paxos instances for dbname.tblname on ranges [(-9183943694846826806,-9178949053182314612], (-9178949053182314612,-9174491011562720949], (-9174491011562720949,-9169014240013922742], (-9169014240013922742,-9154280125678606431], (-9154280125678606431,-9154213578127245161], (-9154213578127245161,-9140967271019555306], (-9140967271019555306,-9133960543506102611], (-9133960543506102611,-9131172695337741062], (-9131172695337741062,-9110143864827552341], (-9110143864827552341,-9089115034317363619], (-9089115034317363619,-9081718208802390781], (-9081718208802390781,-9038399725264351592], (-9038399725264351592,-9037586368092877291], (-9037586368092877291,-9036773010921402990], (-9036773010921402990,-9034940736886749152], (-9034940736886749152,-9031693738204765577], (-9013153787639449114,-8992467188075380278], (-8992467188075380278,-8987196154876405162], ... [HUGE BUNCH OF NUMBERS] ... (8910982110278044393,8960900989155763951], (8960900989155763951,8972606730535503180], (8972606730535503180,8977452888941169379], (8977452888941169379,9008253241338722153], (9008253241338722153,9008475595953247050], (9008475595953247050,9008697950567771948], (9008697950567771948,9049032138977479669], (9049032138977479669,9066825071560905034], (9066825071560905034,9069511475811258334], (9069511475811258334,9074987722729151089], (9074987722729151089,9094374407204864338], (9094374407204864338,9113761091680577587], (9113761091680577587,9119842770282429270], (9119842770282429270,9125924448884280953]] for session 00000000-0000-0000-0000-000000000000",