feat(dataProcessInstance): Support data process instance entity page and lineage #12499

asikowitz · 2025-01-30T01:54:56Z

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

… filtering nodes

jjoyce0510 · 2025-01-30T02:02:59Z

...hub-web-react/src/app/entityV2/shared/components/styled/search/EmbeddedListSearchSection.tsx

@@ -19,6 +19,30 @@ import {
 import { decodeComma } from '../../../utils';

 const FILTER = 'filter';
+const SEARCH_ENTITY_TYPES = [
+    EntityType.Dataset,


Why not use the backend to do this? Via default search entity types?

Don't want them to appear in main search, only container page / impact analysis for now

jjoyce0510 · 2025-01-30T02:03:21Z

...hub-web-react/src/app/entityV2/shared/components/styled/search/EmbeddedListSearchSection.tsx

@@ -146,6 +170,7 @@ export const EmbeddedListSearchSection = ({

    return (
        <EmbeddedListSearch
+            entityTypes={SEARCH_ENTITY_TYPES}


Again, we've been managing this on the backend for some time. Why move to the UI layer?

codecov · 2025-01-30T02:07:59Z

Codecov Report

Attention: Patch coverage is 29.72350% with 305 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...-react/src/app/lineageV2/pruneAllDuplicateEdges.ts	31.85%	92 Missing ⚠️
...t/src/app/lineageV2/useComputeGraph/filterNodes.ts	9.09%	70 Missing ⚠️
.../dataProcessInstance/DataProcessInstanceEntity.tsx	43.18%	25 Missing ⚠️
...pp/lineageV2/LineageFilterNode/computeOrFilters.ts	0.00%	18 Missing ⚠️
datahub-web-react/src/app/lineageV2/common.ts	38.09%	13 Missing ⚠️
...rc/app/lineageV2/controls/LineageSearchFilters.tsx	0.00%	13 Missing ⚠️
.../dataProcessInstance/DataProcessInstanceEntity.tsx	33.33%	12 Missing ⚠️
...b-react/src/app/lineageV2/useColumnHighlighting.ts	0.00%	12 Missing ⚠️
...containers/profile/sidebar/SidebarEntityHeader.tsx	10.00%	9 Missing ⚠️
...p/entityV2/dataProcessInstance/preview/Preview.tsx	20.00%	8 Missing ⚠️
... and 10 more

❌ Your patch status has failed because the patch coverage (29.72%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Files with missing lines	Coverage Δ
.../containers/profile/header/DefaultEntityHeader.tsx	`87.84% <100.00%> (ø)`
...b-react/src/app/ingest/source/builder/constants.ts	`100.00% <100.00%> (ø)`
datahub-web-react/src/app/lineageV2/NodeBuilder.ts	`28.39% <ø> (ø)`
...onents/styled/search/EmbeddedListSearchSection.tsx	`40.29% <96.15%> (ø)`
...app/lineageV2/useComputeGraph/getDisplayedNodes.ts	`19.02% <0.00%> (ø)`
...tahub-web-react/src/app/previewV2/EntityHeader.tsx	`89.68% <50.00%> (ø)`
...eb-react/src/app/lineageV2/useBulkEntityLineage.ts	`14.28% <50.00%> (ø)`
.../app/lineageV2/useComputeGraph/useComputeGraph.tsx	`18.81% <0.00%> (ø)`
...-react/src/app/lineageV2/useSearchAcrossLineage.ts	`16.80% <25.00%> (ø)`
...ub-web-react/src/app/lineageV2/LineageExplorer.tsx	`20.71% <0.00%> (ø)`
... and 13 more

... and 1147 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c2a4358...ffd5ca4. Read the comment docs.

jjoyce0510 · 2025-01-30T02:12:20Z

datahub-web-react/src/app/entityV2/shared/containers/profile/header/DefaultEntityHeader.tsx

@@ -148,6 +149,14 @@ export const DefaultEntityHeader = ({
    const displayedEntityType = getDisplayedEntityType(entityData, entityRegistry, entityType);
    const { platform, platforms } = getEntityPlatforms(entityType, entityData);

+    const containerPath =


nice refactor

jjoyce0510 · 2025-01-30T02:13:23Z

datahub-web-react/src/app/lineageV2/controls/LineageSearchFilters.tsx

@@ -75,6 +77,17 @@ export default function LineageSearchFilters() {
                    />
                </Tooltip>
            </ToggleWrapper>
+            <ToggleWrapper>


Thanks for putting this behind a control. This is really nice!

jjoyce0510 · 2025-01-30T02:14:02Z

datahub-web-react/src/app/previewV2/EntityHeader.tsx

                        </CardEntityTitle>
                    </Tooltip>
                ) : (
                    <EntityTitle title={name} onClick={onClick} $titleSizePx={titleSizePx} data-testid="entity-title">
-                        <SearchTextHighlighter field="name" text={name || ''} />
+                        <SearchTextHighlighter field="name" text={name || urn} />


nice change

jjoyce0510 · 2025-01-30T02:14:34Z

datahub-web-react/src/app/lineageV2/useSearchAcrossLineage.ts

@@ -80,6 +78,7 @@ export default function useSearchAcrossLineage(
                    platforms: [DBT_URN],
                },
                { entityType: EntityType.DataJob },
+                { entityType: EntityType.DataProcessInstance },


Always? I guess so

jjoyce0510 · 2025-01-30T02:15:38Z

datahub-web-react/src/app/lineageV2/useComputeGraph/filterNodes.ts

+    if (hideDataProcessInstances) {
+        // Note: Will only pick one query node if there is lineage t1 -> q1 -> dpi1 -> q2 -> t2
+        // Currently data process instances can't have lineage to queries so this is fine
+        newNodes = new Map(


I thought we were ignoring dpi as hops.. Do they still come back? So we have to filter them out and connect through them?

I built this like I built the hide transformations toggle, which doesn't alter the query and instead filters entirely on the frontend. If we alter the query, like the show hidden edges (show ghost entities) toggle, then we have to refresh the graph when you toggle, because your graph state might be the result of many search across lineage calls and currently I don't store which ones have been made so we could re-issue them with the new params. I think it's a better experience to get the instant feedback on toggling + not having to re-expand your graph, but there is a concern that a graph could have many data process instances and thus cause slowdown. I think for now, this is ok.

jjoyce0510

This looks great. Also relatively low risk...

The main question i had was around moving search entity types to the UI side instead of keeping it at the server.. Is there a reason I am missing? (I'm sure there is)

shirshanka

LGTM!

Remove lineage_features

This reverts commit 0e7add9.

asikowitz added 3 commits January 29, 2025 17:54

cherry pick

4a1682a

fix(ui/lineageV2): Properly set adjacencyList edges for queries after…

14f17bf

… filtering nodes

revert

f4d2823

asikowitz requested a review from shirshanka January 30, 2025 01:54

github-actions bot added ingestion PR or Issue related to the ingestion of metadata product PR or Issue related to the DataHub UI/UX devops PR or Issue related to DataHub backend & deployment smoke_test Contains changes related to smoke tests labels Jan 30, 2025

jjoyce0510 reviewed Jan 30, 2025

View reviewed changes

datahub-cyborg bot added the pending-submitter-response Issue/request has been reviewed but requires a response from the submitter label Jan 30, 2025

jjoyce0510 reviewed Jan 30, 2025

View reviewed changes

vercel bot deployed to Preview January 30, 2025 02:13 View deployment

jjoyce0510 reviewed Jan 30, 2025

View reviewed changes

jjoyce0510 approved these changes Jan 30, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-merge and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jan 30, 2025

shirshanka approved these changes Jan 30, 2025

View reviewed changes

asikowitz added 2 commits January 29, 2025 18:37

Update DataProcessInstanceType.java

69ebafa

Remove lineage_features

Update test_data_process_instance.py

6975c92

vercel bot deployed to Preview January 30, 2025 03:21 View deployment

fix flake

0e7add9

vercel bot deployed to Preview January 30, 2025 05:40 View deployment

asikowitz added 2 commits January 29, 2025 21:45

Revert "fix flake"

666d775

This reverts commit 0e7add9.

redo

ffd5ca4

vercel bot deployed to Preview January 30, 2025 06:03 View deployment

asikowitz merged commit aaaa655 into datahub-project:master Jan 30, 2025
193 of 197 checks passed

asikowitz deleted the as--ai-changes-on-top branch January 30, 2025 08:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(dataProcessInstance): Support data process instance entity page and lineage #12499

feat(dataProcessInstance): Support data process instance entity page and lineage #12499

asikowitz commented Jan 30, 2025

jjoyce0510 Jan 30, 2025

asikowitz Jan 30, 2025

jjoyce0510 Jan 30, 2025

codecov bot commented Jan 30, 2025 •

edited

Loading

jjoyce0510 Jan 30, 2025

jjoyce0510 Jan 30, 2025

jjoyce0510 Jan 30, 2025

jjoyce0510 Jan 30, 2025

jjoyce0510 Jan 30, 2025

asikowitz Jan 30, 2025

jjoyce0510 left a comment

shirshanka left a comment

feat(dataProcessInstance): Support data process instance entity page and lineage #12499

feat(dataProcessInstance): Support data process instance entity page and lineage #12499

Conversation

asikowitz commented Jan 30, 2025

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 30, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jjoyce0510 left a comment

Choose a reason for hiding this comment

shirshanka left a comment

Choose a reason for hiding this comment

codecov bot commented Jan 30, 2025 •

edited

Loading