Unable to fetch lineage when running databricks notebook from azure data factory

[GouravJ123]

Hello,

while running the databricks notebook directly, i am able to get the lineage in azure purview but when i run the same notebook using azure data factory and select databricks notebook as activity then its throwing error in purviewout azure function. The error is : AdbClient-isTokenExpired: Token cache hit and "AdbClient-GetSingleAdbJobAsync: error, message: Response status code does not indicate success: 404 (Not Found)." Can you please help me in resolving this issue

[gianlucadardia]

I am facing the same issue. Any thoughts?

[wjohnson]

@GouravJ123 and @gianlucadardia by any chance are you using multiple databricks workspaces? There may be an issue if you have a Function that handles multiple databricks workspaces that could be causing this error.

[jogacolhue]

Notebooks called by Data Factory are considered Databricks Jobs, so you can try to follow the last part of the stand-alone guide