Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Discover] Add DiscoverSavedSearchContainer #153528

Merged
merged 69 commits into from
May 8, 2023

Conversation

kertal
Copy link
Member

@kertal kertal commented Mar 23, 2023

Summary

This PR adds the new DiscoverSavedSearchContainer. It fundamentally changes the way Discover treats the handling of the actual saved search.

  • Before this PR the saved search was updated only when data was fetched or when it was persisted or shared. With this PR, it's updated but not persisted with every relevant depending state change (user changing query, user showing/hiding the histogram, ...).
  • In the new DiscoverSavedSearchContainer we store the initial and the updated version. This allows us diff between both versions and can be used to provide an indication in the UI that there have been changes to the saved search, removing a common pain point in Discover (One it's implemented)
  • Generally DiscoverState has been initialized 2 times before this PR, now it's a singleton high up in the component tree, so when Discover Main is displayed it's just initialized 1x.
  • We accessed the savedSearch seachSource a lot in the UI before this PR, no this this much more limited

Context

This is the final PR refactoring Discover's state management, that was scattered around the code base (saved search, appState, dataState via observables), it's cleaning, and centralizing stuff. Its focus is to get a clearer structure, better separation from UI, removing of hook dependencies making it hard to debug the code. So refactoring without regressions, which makes it easier to further migrate to a future state management solution if needed. One of the biggest advantage is that we can now write unit test what kind of URL creates which kind of state and triggers which behavior.

Generally, the state container consists of several other state containers

export interface DiscoverStateContainer {
/**
* App state, the _a part of the URL
*/
appState: DiscoverAppStateContainer;
/**
* Data fetching related state
**/
dataState: DiscoverDataStateContainer;
/**
* Internal state that's used at several places in the UI
*/
internalState: DiscoverInternalStateContainer;
/**
* kbnUrlStateStorage - it keeps the state in sync with the URL
*/
kbnUrlStateStorage: IKbnUrlStateStorage;
/**
* State of saved search, the saved object of Discover
*/
savedSearchState: DiscoverSavedSearchContainer;
/**
* Service for handling search sessions
*/
searchSessionManager: DiscoverSearchSessionManager;
/**
* Complex functions to update multiple containers from UI
*/
actions: {

Those can be directly accessed by the UI. For complex state manipulation which need more sub states, actions can be used from the UI

actions: {
/**
* Triggers fetching of new data from Elasticsearch
* If initial is true, then depending on the given configuration no fetch is triggered
* @param initial
*/
fetchData: (initial?: boolean) => void;
/**
* Initialize state with filters and query, start state syncing
*/
initializeAndSync: () => () => void;
/**
* Load current list of data views, add them to internal state
*/
loadDataViewList: () => Promise<void>;
/**
* Load a saved search by id or create a new one that's not persisted yet
* @param savedSearchId
* @param dataView
*/
loadSavedSearch: (
savedSearchId?: string | undefined,
dataView?: DataView | undefined,
dataViewSpec?: DataViewSpec
) => Promise<SavedSearch | undefined>;
/**
* Create and select a ad-hoc data view by a given index pattern
* @param pattern
*/
onCreateDefaultAdHocDataView: (pattern: string) => Promise<void>;
/**
* Triggered when a new data view is created
* @param dataView
*/
onDataViewCreated: (dataView: DataView) => Promise<void>;
/**
* Triggered when a new data view is edited
* @param dataView
*/
onDataViewEdited: (dataView: DataView) => Promise<void>;
/**
* Triggered when a saved search is opened in the savedObject finder
* @param savedSearchId
*/
onOpenSavedSearch: (savedSearchId: string) => void;
/**
* Triggered when the unified search bar query is updated
* @param payload
* @param isUpdate
*/
onUpdateQuery: (
payload: { dateRange: TimeRange; query?: Query | AggregateQuery },
isUpdate?: boolean
) => void;
/**
* Triggered when the user selects a different data view in the data view picker
* @param id
*/
onChangeDataView: (id: string) => Promise<void>;
/**
* Triggered when an ad-hoc data view is persisted to allow sharing links and CSV
* @param dataView
*/
persistAdHocDataView: (dataView: DataView) => Promise<DataView>;
/**
* Set the currently selected data view
*/
setDataView: (dataView: DataView) => void;
/**
* Undo changes made to the saved search
*/
undoChanges: () => void;
/**
* When saving a saved search with an ad hoc data view, a new id needs to be generated for the data view
* This is to prevent duplicate ids messing with our system
*/
updateAdHocDataViewId: () => void;
};

With this state container we can write unit tests like e.g. for the loadSavedSearch action

test('loadSavedSearch data view handling', async () => {
const { state } = await getState('/', savedSearchMock);
await state.actions.loadSavedSearch(savedSearchMock.id);
expect(state.savedSearchState.getState().searchSource.getField('index')?.id).toBe(
'the-data-view-id'
);
state.savedSearchState.load = jest.fn().mockReturnValue(savedSearchMockWithTimeField);
await state.actions.loadSavedSearch('the-saved-search-id-with-timefield');
expect(state.savedSearchState.getState().searchSource.getField('index')?.id).toBe(
'index-pattern-with-timefield-id'
);
});

Reviewing

First dear reviewer, apologies, this is a big PR ... but also, if I would have opened the initial draft PR fro review ... it would have been much worse. So, the best way to start reviewing is the first commit, which is introducing the new state container:

071fa37

And since the creator of this PR has been working on that for a while, he might not see the fires for tree.
There's certainly potential for follow up improvements, but it was time to bring the scope over the finishing line. Ideally we could use this state container also in the saved search embeddable, but this is Zukunfsmusik (🚀 🎼 )

Testing

The new version needs to be tested for regressions. Functional tests caught lots of stuff during development, not only Discover tests, also ML & alerting test were helpful. Nothing should have changed but since it's also a fundamental change, more testing is better than less. What should be tested

  • using the New menu bar entry to create new saved searches (ideally after loading persisted ones)
  • persisting a new saved search, editing it, saving it, switching to New, loading the saved search
  • creating alert rules
  • creating CSVs
  • Sharing URLs

Resolves #142131
Fixes #155193

Checklist

- A container to centralize functionality around the usage of saved searches in Discover
@kertal kertal added Feature:Discover Discover Application Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. labels Mar 23, 2023
@kertal kertal self-assigned this Mar 23, 2023
@kertal kertal added the ci:cloud-deploy Create or update a Cloud deployment label Mar 23, 2023
@kertal kertal added the release_note:skip Skip the PR/issue when compiling release notes label Mar 23, 2023
* Can also be used to track changes to the saved search
* It centralizes functionality that was spread across the Discover main codebase
*/
export interface DiscoverSavedSearchContainer {
Copy link
Member Author

@kertal kertal Mar 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dear reviewer, my I introduce the new (sub) state container to you ... most of the code in this PR is about using this ... so this is the ❤️ of the PR

@kertal kertal marked this pull request as ready for review March 23, 2023 12:42
@kertal kertal requested a review from a team as a code owner March 23, 2023 12:42
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

Comment on lines +54 to +59
const stateContainer = useSingleton<DiscoverStateContainer>(() =>
getDiscoverStateContainer({
history,
services,
})
);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from adding a new state container, this is the biggest change, before this, DiscoverStateContainer was initialized 2x on every change to different saved search

*/
useEffect(() => {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've heard rumors that useEffect isn't popular anymore (right @mattkime ), happy to say, that I removed some, and there might be potential for more ... oh yes, there a big hook gone use_discover_state, 👋 goodbye and thx for all the fish.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, this effect is a bad example because it was just moved to L90 😄

@@ -17,10 +18,12 @@ export const FieldStatisticsTab: React.FC<Omit<FieldStatisticsTableProps, 'query
const querySubscriberResult = useQuerySubscriber({
data: services.data,
});
const savedSearch = useSavedSearch();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not using the state container here ... it's not available in the saved search embeddable

Copy link
Contributor

@jughosta jughosta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow! So many changes! Leaving some questions to clarify 💫

DiscoverLayoutProps,
'navigateTo' | 'savedSearch' | 'searchSource'
> & {
export type DiscoverTopNavProps = Pick<DiscoverLayoutProps, 'navigateTo'> & {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Top nav disappeared for me after adding a query and a filter:

Screenshot 2023-03-27 at 14 35 18

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is odd, I couldn't reproduce ... were there any error message in console? can you reproduce? many thx!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, also can't reproduce, sorry!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's the hardest one, I suspect it broke for the same reason alerting broke, causing a silent exception here

const initialFetchStatus = shouldSearchOnPageLoad
? FetchStatus.LOADING
: FetchStatus.UNINITIALIZED;
const getInitialFetchStatus = () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When SEARCH_ON_PAGE_LOAD_SETTING is off, it still loads data on page refresh and after pressing New

Mar-27-2023 14-41-09

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Working on a fix, moving the listener for the / path, that creates a new saved search or resets an existing one to discover_main_route.tsx

5ba1e2b

The way it was implemented before caused a duplicate fetch command before ... will follow up, first CI needs to reply

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue is still present

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, for me it seems fixed, something to follow up after 🥚 & 🇳🇱

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davismcphee I can't reproduce it, but this needs definitely another pair of 👀

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, well, found and fixed it (needed a query to reproduce) 98efa69

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx!

Copy link
Contributor

@jughosta jughosta May 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this issue with ignored discover:searchOnPageLoad is happening again

Copy link
Member Author

@kertal kertal May 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We chatted "offline" and it was because the refresh interval was "On" ... So it works like before. However me might consider, switching the refresh interval to "Off" when triggering New ... When I start a new search I think I first want to apply data, query, filter, before I want to have the data refreshed? ... not part of this PR of course

@kertal kertal marked this pull request as draft March 28, 2023 08:32
@kertal
Copy link
Member Author

kertal commented Mar 28, 2023

Wow! So many changes! Leaving some questions to clarify 💫

Yes, sorry for that, also seems I rushed to the review state. Apologies for that. Flagging it to draft again, until fixes are applied, 🙇 thx a lot for this first review!

@kertal kertal requested review from jughosta and davismcphee April 28, 2023 12:50
@kertal
Copy link
Member Author

kertal commented May 2, 2023

@elasticmachine merge upstream

@@ -141,21 +145,45 @@ export function resolveDataView(
});
return ownDataView;
}
if (!Boolean(isTextBasedQuery)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this condition redundant?

Copy link
Member Author

@kertal kertal May 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it wasn't but I guess it was a change that got lost since it was added during the long period of time of this PR ... This message doesn't make sense when loading a new saved search with SQL in the state URL. I reverted this change to consider text based queries , also added a test for it: 53ec89d

const initialFetchStatus = shouldSearchOnPageLoad
? FetchStatus.LOADING
: FetchStatus.UNINITIALIZED;
const getInitialFetchStatus = () => {
Copy link
Contributor

@jughosta jughosta May 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this issue with ignored discover:searchOnPageLoad is happening again

@kertal
Copy link
Member Author

kertal commented May 4, 2023

@elasticmachine merge upstream

Copy link
Contributor

@jughosta jughosta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did another test round: looking good! 👍
Thanks for improving the state management!

@kertal
Copy link
Member Author

kertal commented May 4, 2023

@jughosta thx a lot for taking care of this 🐰 🕳️ in terms of a review, for catching so much stuff 👍
@davismcphee no priority, but a final look at this would be great, because the size of this justifies 2 LGTM , many thx 🙇

@davismcphee
Copy link
Contributor

@jughosta thx a lot for taking care of this 🐰 🕳️ in terms of a review, for catching so much stuff 👍 @davismcphee no priority, but a final look at this would be great, because the size of this justifies 2 LGTM , many thx 🙇

@kertal I was planning on doing a final review/round of testing on this today but didn't get the chance -- will definitely take a look at it tomorrow though!

Copy link
Contributor

@davismcphee davismcphee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just finished a final round of reviewing and testing, and I'm happy to say it LGTM 🥳 That was a long time coming, but this was much needed work and I'm excited to finally see it merged to main. It's also good that it's early in the minor so we have time to catch and fix anything we might have missed. Great work, and I'd say we're good to push the button whenever you're ready!

@kertal
Copy link
Member Author

kertal commented May 7, 2023

@elasticmachine merge upstream

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
discover 460 462 +2

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
discover 77 75 -2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
discover 426.2KB 423.8KB -2.4KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
savedSearch 5.5KB 5.6KB +52.0B
Unknown metric groups

API count

id before after diff
savedSearch 55 58 +3

ESLint disabled line counts

id before after diff
discover 39 40 +1
enterpriseSearch 19 21 +2
securitySolution 398 401 +3
total +6

References to deprecated APIs

id before after diff
discover 27 29 +2

Total ESLint disabled count

id before after diff
discover 39 40 +1
enterpriseSearch 20 22 +2
securitySolution 478 481 +3
total +6

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kertal

@kertal kertal merged commit 8f41ab0 into elastic:main May 8, 2023
@kertal
Copy link
Member Author

kertal commented May 8, 2023

Thx so much @jughosta @davismcphee for taking care of the reviews of this 👹 🙇

kertal added a commit that referenced this pull request May 10, 2023
## Summary

Fixes an issue introduced #153528 by removing a change 
introduced in #156652 that enables sharing for text-based languages
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting Feature:Discover Discover Application release_note:skip Skip the PR/issue when compiling release notes Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. v8.9.0
Projects
None yet
6 participants