[Dataset quality] Introduce Dataset health page

[yngrdyn]

📓 Summary

In order to make ongoing maintenance of log collection easy we want to introduce the concept of dataset health, where users can easily get an overview on the datasets they have with information such as integration, size, last activity, among others.

A hint on the datasources have been laid out in this comment.

With tooltips:

• 🎨 Figma: https://www.figma.com/file/aaJNMsOklYhjwtu5TP7UDK/%5BLogs%2B%5D-Organize-and-Admin?node-id=2106%3A381912&mode=dev

✔️ Acceptance criteria

• The datasets management page shows a table of the datasets owned by the installed integrations as well as all uncategorized datasets.
• The table of datasets has the columns
  • "Name", which is...
    derived from the human-readable name if the dataset is owned by an integration
    derived from the plain dataset name otherwise
    ℹ️ is derived from the plain dataset name
  • "Integration", which is...
    derived from the icon and name of the integration and empty If no integration is found (uncategorized)
  • "Size", which is...
    derived the data stream stats API if a backing data stream exists and 0 otherwise formatted for easy human consumption
  • "Last activity", which is...
    derived from the data stream stats API
• Table columns are sortable and the initial sorting parameter is Name.
• The table of the datasets has pagination
• The dataset management page has an "Add data" link to the on-boarding page in the top right.
• Dataset management page is hidden by default, no reference is added to the side nav or Explorer.

💡 Implementation hints

• In the meantime the dataset management should live in observability. Consider creating an architecture such as log-explorer vs observability-logs-explorer where the dataset management page could be extracted in the future from observability smoothly.
• To create the new plugin we can follow kibana documentation.

Tasks

Preview Give feedback

1. [Dataset quality] Adding datastreams stats end point #170838
   Team:Fleet Team:obs-ux-logs backport:skip release_note:skip v8.12.0 +5
   
2. [Dataset quality] Creating initial plugin #171009
   Team:obs-ux-logs backport:skip release_note:skip v8.12.0 +4
   
3. [Dataset quality] Including integration information within stats #171663
   Team:obs-ux-logs backport:skip release_note:skip v8.12.0 +4
   
4. [Dataset quality] Use synthtrace for api integration tests #171621
   Team:obs-ux-logs backport:skip release_note:skip v8.12.0 +4
   
5. [Dataset quality] Implementing Dataset Table + consuming plugin from observability log explorer #171777
   Team:obs-ux-logs backport:skip release_note:skip v8.12.0 +4
   

[yngrdyn]

@weltenwort, @tonyghiani, @Kerry350, @mohamedhamed-ahmed in favour of abstracting, or at least creating a black box component at the beginning, I was thinking on maybe creating a plugin just for the dataset health (sort of what you have between log_explorer and observability_log_explorer).

What do you think about then creating a page, sort of a wrapper, in observability_log_explorer where you can instantiate the dataset health component? would that even be possible? The idea is for this page to live under observability as far as I understand from the breadcrumbs in the mockup

[tonyghiani]

In favour of abstracting, or at least creating a black box component at the beginning, I was thinking on maybe creating a plugin just for the dataset health (sort of what you have between log_explorer and observability_log_explorer).

I agree it has its pros to abstract into a new plugin.
Particularly, the main benefit I see is having a reusable page that can be embedded in different apps, as we thought for the Log Explorer to be a reusable one that different apps can consume (apm, fleet, etc.)

What do you think about then creating a page, sort of a wrapper, in observability_log_explorer where you can instantiate the dataset health component? would that even be possible? The idea is for this page to live under observability as far as I understand from the breadcrumbs in the mockup

In my understanding, we want to treat the dataset health page as we do for the log explorer, creating a new client route that embeds the previously created component.

flowchart TD

apm(Apm)
fleet(Fleet)
log_explorer(Log Explorer)
dataset_health(Dataset Health)
observability_log_explorer(Observability Log Explorer)

observability_log_explorer --> log_explorer
observability_log_explorer --> dataset_health
apm --> log_explorer
fleet --> log_explorer
fleet --> dataset_health

Loading