[BUG] Findings not getting generated for the nested JSON data

[jinalp-crest]

What is the bug?
Findings are not generated for the documents containing nested fields.
While detectors are created for nested Sigma rules, when we ingest documents with the nested field objects, no findings get generated. But when we ingest data with a flattened structure it works properly.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to the 'Security Analytics' plugin's detector section under the OpenSearch dashboard.
2. Click on 'Create detector'.
   i. Configure detector details, select one of the log types as Cloudtrail, specify the index, and select all Sigma rules provided.
   ii. Provide the mappings necessary for the configured rules.
   Note: If we had ingested data as per the ECS schema in the configured index, then fields would get mapped automatically.
   iii. Specify alert trigger conditions with any rules. (By default it will select all configured rules for that log category)
   iv. Click on 'Save changes' to generate a detector.
3. Ingest data to the specified index with both flattened and nested JSON structure.
4. See the findings and alerts for the flattened data, but no findings get generated for the nested log structure.

What is the expected behavior?
If the Sigma rule has nested field specifications such as "userIdentity.type" matching some value, then findings should be generated for both the flattened and nested kind of ingested data.
Sample flattened doc:

{
    "userIdentity.type": "AssumedRole",
    "userIdentity.sessionContext.sessionIssuer.type": "Role"
}

Sample nested doc:

{
      "eventVersion": "1.08",
      "userIdentity": {
        "type": "AssumedRole",
        "principalId": "aaa:hained-role",
        "arn": "arn:aws:sts::xxx:assumed-role/chained-role-ciem/chained-role",
        "accountId": "xxx",
        "accessKeyId": "aaa",
        "sessionContext": {
          "sessionIssuer": {
            "type": "Role",
            "principalId": "aaa",
            "arn": "arn:aws:iam::xxx:role/chained-role-ciem",
            "accountId": "xxx",
            "userName": "chained-role-ciem"
          },
          "webIdFederationData": {},
          "attributes": {
            "creationDate": "2022-09-30T10:19:19Z",
            "mfaAuthenticated": "false"
          }
        }
      }
      ...
}

What is your host/environment?

• OS: Windows
• Version: OpenSearch 2.6x
• Plugins: Security Analytics Plugin (SAP)

Do you have any screenshots?

• Though, findings are only generated for the flattened data following ECS schema as per the field mappings provided.
• When, I checked the SAP detector query and removed additional index context from the query, and tried to run on the same index, both documents got matched with nested as well as flattened structures.

Sample detector query

{
  "query": {
    "query_string": {
      "query": """(aws-cloudtrail-user_identity-type_spike_index_rahul_lEUpVYcB-q1Aw194M_Bs: "AssumedRole") AND (aws-cloudtrail-user_identity-session_context-session_issuer-type_spike_index_rahul_lEUpVYcB-q1Aw194M_Bs: "Role")"""
    }
  },
  "monitor_id": "lEUpVYcB-q1Aw194M_Bs",
  "index": "spike_index_rahul"
}

Querying the index with detector query

API Request:

API Response:

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 0.17402273,
    "hits": [
      {
        "_index": "spike_index_rahul",
        "_id": "100",
        "_score": 0.17402273,
        "_source": {
          "aws.cloudtrail.request_parameters.username": "rahul test flat",
          "event.provider": "iam.amazonaws.com",
          "event.action": "UpdateLoginProfile",
          "aws.cloudtrail.user_identity.type": "AssumedRole",
          "aws.cloudtrail.user_identity.session_context.session_issuer.type": "Role",
          "event": {
            "ingested": "2023-04-06T06:24:11.897632850Z"
          },
          "error": {
            "message": "field [message] doesn't exist"
          }
        }
      },
      {
        "_index": "spike_index_rahul",
        "_id": "101",
        "_score": 0.17402273,
        "_source": {
          "aws": {
            "cloudtrail": {
              "user_identity": {
                "session_context": {
                  "session_issuer": {
                    "type": "Role"
                  }
                },
                "type": "AssumedRole"
              },
              "request_parameters": {
                "username": "jp test nested"
              }
            }
          },
          "event": {
            "ingested": "2023-04-06T06:25:41.480019934Z",
            "provider": "iam.amazonaws.com",
            "action": "UpdateLoginProfile"
          },
          "error": {
            "message": "field [message] doesn't exist"
          }
        }
      }
    ]
  }
}

[dblock]

Sorry for the late reply here. Looks like a bug, or at least like a non-feature - @jinalp-crest maybe try turning this into a failing unit test?

[praveensameneni]

Looking into the issue

[eirsep]

This is an issue in document level monitors
we have fixed it and it has been released in 2.7

[eirsep]

opensearch-project/alerting#844