[Security Solution] [Sourcerer] Sourcerer matched indices not picking up .alerts index on fresh kibana + ES

[dhurley14]

When the .alerts index does not exist, sourcerer does not find it as part of the matched indices on the security default data view which results in some features not working as expected (opening 'inspect in timeline' from the alerts table doesn't load the data, some issues around the execution log, etc..). The fix is to simply perform a browser refresh on the page and the matched indices on the data view are reloaded however it would be great if the matched indices were updated when a new index matching the index patterns defined on the data view is created.

[dhurley14]

The only way to go about fixing this is to refresh the data view every time we fetch it in sourcerer (https://github.com/elastic/kibana/pull/151954/files#diff-508f840edbd45c95151db939b1894ca52ec682fb0863eccdf420eed427f53663R15). This is a fairly heavy operation. And by replacing sourcerer with the ad-hoc data view component we will no longer need to rely on the matched indices property for fetching data. My focus for 8.8 will be on expediting the removal and replacement of sourcerer.

[dhurley14]

Fixed here: #154076

[MadameSheema]

@karanbirsingh-qasource @sukhwindersingh-qasource please validate the fix on current BC. Thanks!

[ghost]

@MadameSheema @dhurley14 can we get steps to regress this issue we have checked this comment #150818 (comment) in which it is mentioned we dont have to do own browser refresh but if we can exact step to check this issue is fixed or not

[dhurley14]

Sure thing! I just want to preface this by saying this is still an issue for the non-default space. So maybe this isn't "fixed" necessarily. But either way, here are the steps:

1. Clean kibana
2. Provide data
3. Create a single rule that will alert on the given data
4. navigate to the alerts page
5. Alerts should appear without having to refresh the page.

[MadameSheema]

@karanbirsingh-qasource do you have now all the needed information to validate the fix? Thanks!

[ghost]

Here are observation to the above steps:

1. Clean kibana

Created the fresh instance

2. Provide data

Installed Audit beat Data Shipper to Kibana instance

3. Create a single rule that will alert on the given data

Create a Custom Rule for process.name: "cmd.exe"
After Clicking on Create and Enable button
Run the cmd on machine

4. navigate to the alerts page
5. Alerts should appear without having to refresh the page.

Rules.-.Kibana.-.Google.Chrome.2023-05-09.16-19-06.mp4

[MadameSheema]

Thanks @karanbirsingh-qasource!! Pending to be validated in different spaces as soon as the fix is complete.

[yctercero]

Hey @MadameSheema ! Could we get this re-validated to see if we can close out?

[MadameSheema]

Sure!! @karanbirsingh-qasource can you please take care of this? thx!

[ghost]

Hi @MadameSheema

This is issue if fixed ✔️ . Alert entries show up without refreshing the page

Default Space

237075060-f87fa8ef-bfa2-47ef-9f36-4356ba77a3fb.mp4

Non Default Space

Create.new.rule.-.Kibana.Mozilla.Firefox.2024-06-05.15-34-54.mp4