Discover shows whole json object

[dosera]

I just noticed - after upgrading heartbeat from 5.x to 6.x - hat the discover view displays a whole json object as a string instead of the simple fields.

Following scenario:

I have an index pattern with

• monitor - string [this is the legacy field for 5.6, when monitor was a string]
• monitor.duration.us - number
• monitor.host - string
• monitor.ip - string
• ....

When looking at the discover view, it looks like this:

I would expect - since monitor is no string here - one entry per field, e.g.

"monitor.name": "name"
"monitor.id": "id"
...

[Bargs]

Strange. What does your mapping for the monitor field look like? Also what does the monitor field look like on the JSON tab?

[dosera]

Mapping for the monitor field:
GET _template/metrics-heartbeat

"monitor": {
    "properties": {
        "duration": {
            "properties": {
                "us": {
                    "type": "long"
                }
            }
        },
        "host": {
            "ignore_above": 1024,
            "type": "keyword"
        },
        "id": {
            "ignore_above": 1024,
            "type": "keyword"
        },
        "ip": {
            "ignore_above": 1024,
            "type": "keyword"
        },
        "name": {
            "ignore_above": 1024,
            "type": "keyword"
        },
        "scheme": {
            "ignore_above": 1024,
            "type": "keyword"
        },
        "status": {
            "ignore_above": 1024,
            "type": "keyword"
        },
        "type": {
            "ignore_above": 1024,
            "type": "keyword"
        }
    }
}

Used to be

"monitor": {
    "type": "keyword"
}

The monitor in the JSON tab:

"monitor": {
    "name": "name",
    "id": "id",
      "duration": {
        "us": 1616
      },
      "type": "tcp",
      "host": "dev",
      "scheme": "tcp"
}

Regards

[Bargs]

What does Kibana show as the "type" of the monitor field on the index pattern management page? You should be able to see the sub-fields listed here as well (e.g. monitor.name). If monitor was once mapped as a keyword I wonder if Kibana still thinks it's a string. If that's the case, you can just refresh the field list by clicking the refresh button on that page.

[dosera]

monitor has the type string; the subfields are for example monitor.duration.us=number, monitor.host=string etc.

If that's the case, you can just refresh the field list by clicking the refresh button on that page.

That I did of course, but it doesnt change a thing. I guess it may be due to the legacy data.

[Bargs]

Yes, if you have old indices with different mappings for monitor, this will cause troubles in Kibana.

[Bargs]

Does Kibana say anything about the field having a mapping conflict on the management screen? I'd be surprised if it didn't.

[dosera]

Does Kibana say anything about the field having a mapping conflict on the management screen? I'd be surprised if it didn't.

It doesnt.

[Bargs]

Ok, so after much delay I had some time to try to reproduce this. It was pretty easy following the info @dosera provided above. I created two indices, one where monitor is mapped as a keyword and another where monitor is an object. I created an index pattern matching both and when I look at it in Discover I'm seeing the same behavior @dosera described. I haven't looked at the code, but apparently having monitor listed as type string in the index pattern is throwing things off. It's a bit of an odd configuration since it would only show up when you have indices with heterogeneous mappings. That said, I bet we could handle it more gracefully.

Here are the Console requests I used, in case anyone wants to quickly reproduce this and take a swing at fixing it:

PUT object-string
{
  "mappings": {
    "doc": {
      "properties": {
        "monitor": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT object-object
{
  "mappings": {
    "doc": {
      "properties": {
        "monitor": {
          "properties": {
            "duration": {
              "properties": {
                "us": {
                  "type": "long"
                }
              }
            },
            "host": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "ip": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "scheme": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "status": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "type": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}


POST object-object/doc
{
  "monitor": {
    "name": "name",
    "id": "id",
    "duration": {
      "us": 1616
    },
    "type": "tcp",
    "host": "dev",
    "scheme": "tcp"
  }
}

POST object-string/doc
{
  "monitor": """{"name":"name","id":"id","duration":{"us":1616},"type":"tcp","host":"dev","scheme":"tcp"}"""
}

[tsg]

This ES ticket is related and I think required to solve this one: elastic/elasticsearch#33237

[darkmoon03]

I have the same issue, without having the heterogeneous objects. I can delete the index completely and it gets recreated with the proper mappings, but shows in Discover as json blobs only.

[darkmoon03]

Mapping:
` "meta": {
"dynamic": "true",
"properties": {
"delay": {
"type": "integer"
},
"filter": {
"type": "keyword"
},
"filterhost": {
"type": "keyword"
},
"filtertime": {
"type": "long"
},

etc...
`

In Discover:
{ "type": "syslog", "target": { "topic": "logqueue-pprd", "index": "log_syslog-2019.05.07" }, "queuehost": [ "logqueue-prod-03", "logqueue-pprd-01" ], "queuetime": 1557256848, "filter": [ "20-syslog-base", "78-category" ], "input": [ "beats", "prodsample" ], "size": 107, "filtertime": 1557256850, "delay": 8, "filterhost": "logfilter-pprd-02.cas-1.cls.vt.edu" }

Before deleting and recreating indices (to fix the API/FORBIDDEN issue), these all showed as individual fields. Same versions of everything.

[darkmoon03]

Kibana is adding an extraneous entry for the object itself when it creates the index pattern. Removing this manually fixes that object, but it comes back if you refresh the pattern.

Clearly a bug in Kibana's index pattern creation. Can we get a fix, please?

[darkmoon03]

Please, does anyone know how to make Kibana display objects properly, again?

[darkmoon03]

https://www.elastic.co/guide/en/kibana/6.7/document-data.html#adding-columns

You can see an example here of what should be happening.

Here's the documentation for the elastic mapping:
https://www.elastic.co/guide/en/elasticsearch/reference/6.7/object.html

[Bargs]

@darkmoon03 could you provide exact steps to reproduce? When I look at a basic object field in Discover I'm still seeing the individual subfields correctly.

[timroes]

Fixed via #83891