Rules failed after upgrade: [security_exception] Reason: missing authentication credentials for REST request [/_security/user/_has_privileges]

[wwang500]

Step to reproduce:

1. Deploy a 8.2.3 cluster,
2. Create a few simple rules, cases and connectors,

      "connectors": [
        {
          "name": "server_log_connector",
          "type": "SERVER_LOG"
        }
      ],
      "cases": [
        {
          "title": "stack_case",
          "description": "This is a case in stack",
          "owner": "cases",
          "tags": ["stack", "generic-tag"]
        },
        {
          "title": "observability_case",
          "description": "This is a case in observability",
          "owner": "observability",
          "tags": ["observability", "generic-tag"]
        },
        {
          "title": "security_case",
          "description": "This is a case in security",
          "owner": "securitySolution",
          "tags": ["security", "generic-tag"]
        }
      ],
      "rules": [
        {
          "name": "cluster_health_rule",
          "notification_type": "ON_ACTIVE_ALERT",
          "consumer": "alerts",
          "schedule": {"interval": "1m"},
          "rule_type": "CLUSTER_HEALTH",
          "tags": ["cluster", "health"],
          "actions": [
            {
              "connector_name": "server_log_connector",
              "group": "default",
              "params": {"level": "info", "message": "{{context.internalFullMessage}}"}
            }
          ]
        },
        {
          "name": "disabled_cluster_health_rule",
          "notification_type": "ON_ACTIVE_ALERT",
          "consumer": "alerts",
          "schedule": {"interval": "1m"},
          "rule_type": "CLUSTER_HEALTH",
          "tags": ["cluster", "health"],
          "actions": [
            {
              "connector_name": "server_log_connector",
              "group": "default",
              "params": {"level": "info", "message": "{{context.internalFullMessage}}"}
            }
          ],
          "enabled": false
        },
        {
          "name": "security_rule",
          "notification_type": "ON_ACTIVE_ALERT",
          "consumer": "siem",
          "schedule": {"interval": "1m"},
          "rule_type": "SIEM_THRESHOLD",
          "tags": [
            "__internal_rule_id:some_internal_rule_id",
            "__internal_immutable:false"
          ],
          "actions": [],
          "params": {
            "author": [],
            "description": "A security threshold rule",
            "falsePositives": [],
            "from": "now-3660s",
            "license": "",
            "outputIndex": ".alerts-security.alerts-default",
            "riskScore": 21,
            "riskScoreMapping": [],
            "severity": "low",
            "severityMapping": [],
            "threat": [],
            "to": "now",
            "references": [],
            "type": "threshold",
            "language": "kuery",
            "index": ["kibana_sample_data_logs"],
            "query": "*",
            "filters": [],
            "threshold": {
              "field": ["bytes"],
              "value": 1,
              "cardinality": []
            },
            "ruleId": "some_internal_rule_id",
            "immutable": false,
            "maxSignals": 100,
            "version": 1,
            "exceptionsList": [],
            "meta":{"from":" 1h", "kibana_siem_app_url": "some_url"}
          },
          "enabled": false
        },
        {
          "name": "observability_rule",
          "notification_type": "ON_ACTIVE_ALERT",
          "consumer": "alerts",
          "schedule": {"interval": "1m"},
          "rule_type": "LOGS_THRESHOLD",
          "tags": ["metrics", "threshold"],
          "actions": [],
          "params":{
            "timeSize": 1,
            "timeUnit": "h",
            "count": {"value": 1, "comparator": "more than"},
            "criteria": [{"field": "bytes", "comparator": "more than", "value": 1}],
            "groupBy": ["geo.dest"],
            "outputIndex": ".alerts-observability.logs.alerts-default"
          },
          "alertCase": "observability_case"
        }
      ],

3. Upgrade to 8.3.0

Observed:

After upgrade, two of the rules: observability_rule and cluster_health_rule reporting errors:

logs.alert.document.count:270b74f0-f3c2-11ec-9a6e-414bc063a8ba: execution failed - security_exception: [security_exception] Reason: missing authentication credentials for REST request [/_security/user/_has_privileges], caused by: ""

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

I'm assuming this may be a DUP of #118520

[mikecote]

Closing as fixed by #135663.