feat(ingest/lookml): CLL support for sql set in sql_table_name attribute of lookml view

[sid-acryl]

Summary by CodeRabbit

• New Features

  • Introduced a new SQL view for analyzing rental income as a source of employee income.
  • Added new functionalities for improved handling of SQL patterns and derived views in Looker integration.

• Enhancements

  • Enhanced the LookerViewContext class for better SQL query classification.
  • Streamlined SQL query processing through modular functions for cleaner code.
  • Improved dataset metadata with additional aspects for better usability and data management.

• Bug Fixes

  • Addressed completeness issues in metadata processing related to Looker source integration.

• Documentation

  • Expanded documentation to include updated patterns for SQL table definitions within Looker views.

[coderabbitai]

Walkthrough

This update refines the Looker integration by optimizing metadata handling and enhancing SQL view capabilities. Key modifications include the removal of specific metadata work unit generation, improved SQL string processing, and the addition of new LookML models and views related to employee income sources. These changes enrich the dataset's metadata structure, improving usability and clarity for data analysis.

Changes

Files	Change Summary
.../looker/looker_source.py, .../looker/looker_template_language.py	Removed specific metadata work unit generation and added _drop_derived_view_pattern for SQL cleaning.
.../looker/lookml_concept_context.py	Enhanced LookerViewContext with new SQL handling patterns and added is_direct_sql_query_case.
.../lookml/vv-lineage-and-liquid-templates/data.model.lkml, .../rent_as_employee_income_source.view.lkml	Added new data model and explore for rental income sources in LookML.
.../vv_lineage_liquid_template_golden.json	Added multiple aspects to a dataset, enhancing its metadata and usability.
.../looker/golden_test_ingest.json	Removed redundant dataPlatformInstance entries, streamlining the JSON structure.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant Looker
    participant Metadata
    User->>Looker: Request data analysis
    Looker->>Metadata: Retrieve dataset metadata
    Metadata-->>Looker: Return enriched dataset info
    Looker-->>User: Present analysis results

Loading

🐰 In the garden of code where changes bloom bright,
A rabbit hops joyfully, a wondrous sight!
With views and models, we dance and we play,
Transforming the data in a magical way.
New paths for income, the metadata sings,
A world full of insights, oh, what joy it brings! 🌼✨

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 1

Outside diff range, codebase verification and nitpick comments (1)

metadata-ingestion/src/datahub/ingestion/source/looker/lookml_concept_context.py (1)

287-288: Remove unnecessary blank line.

The blank line at the beginning of the method is unnecessary and can be removed for better readability.

- 
  sql_table_name: Optional[str] = self._get_sql_table_name_field()

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 66ecfae and 8d79701.

Files selected for processing (7)

• metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (1 hunks)
• metadata-ingestion/src/datahub/ingestion/source/looker/looker_template_language.py (3 hunks)
• metadata-ingestion/src/datahub/ingestion/source/looker/lookml_concept_context.py (10 hunks)
• metadata-ingestion/src/datahub/ingestion/source/looker/view_upstream.py (6 hunks)
• metadata-ingestion/tests/integration/lookml/vv-lineage-and-liquid-templates/data.model.lkml (2 hunks)
• metadata-ingestion/tests/integration/lookml/vv-lineage-and-liquid-templates/rent_as_employee_income_source.view.lkml (1 hunks)
• metadata-ingestion/tests/integration/lookml/vv_lineage_liquid_template_golden.json (1 hunks)

Additional context used

Ruff

metadata-ingestion/src/datahub/ingestion/source/looker/lookml_concept_context.py

6-6: sqlglot imported but unused

Remove unused import: sqlglot

(F401)

---

465-468: Return the condition directly

Inline condition

(SIM103)

Additional comments not posted (24)

metadata-ingestion/tests/integration/lookml/vv-lineage-and-liquid-templates/rent_as_employee_income_source.view.lkml (4)

13-16: Verify the dimension definition for id.

Ensure that the dimension definition for id is correct and follows best practices. Consider the following points:

• Verify that ${TABLE}.id correctly references the id field in the SQL table.
• Ensure that the id field is of type number in the SQL table.

Verification successful

Let's gather more information to verify the dimension definition for id.

---

The dimension definition for id is correct.

The id field is consistently defined as an INTEGER in the SQL files, which aligns with the number type specified in the LookML dimension.

• metadata-ingestion/tests/integration/trino/setup/setup.sql
• metadata-ingestion/tests/integration/kafka-connect/setup/mysql-setup.sql
• metadata-ingestion/tests/integration/great-expectations/setup/setup.sql

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the dimension definition for `id`.

# Test: Check if the id field is correctly referenced and is of type number. Expect: Field exists and is of type number.
rg --type sql -A 5 $'id.*number'

Length of output: 33

---

Script:

#!/bin/bash
# Description: Verify the dimension definition for `id`.

# Search for the `id` field in the SQL files and display its type.
rg --type sql -A 5 'id'

Length of output: 30003

---

23-26: Verify the dimension definition for source.

Ensure that the dimension definition for source is correct and follows best practices. Consider the following points:

• Verify that ${TABLE}.source correctly references the source field in the SQL table.
• Ensure that the source field is of type string in the SQL table.

---

2-10: Verify the SQL query.

Ensure that the SQL query used in sql_table_name is correct and follows best practices. Consider the following points:

• Ensure that the table ${employee_income_source.SQL_TABLE_NAME} exists and is correctly referenced.
• Verify that the source field is indexed to optimize the WHERE clause.
• Consider removing the LIMIT 10 clause if it is not necessary for the view definition.

---

18-21: Verify the dimension definition for name.

Ensure that the dimension definition for name is correct and follows best practices. Consider the following points:

• Verify that ${TABLE}.name correctly references the name field in the SQL table.
• Ensure that the name field is of type string in the SQL table.

Verification successful

Let's refine our search to ensure we capture the definition of the name field correctly. We'll search for the name field in both SQL and LookML files to cover all bases.

---

To ensure the LookML dimension correctly references the name field, we will first locate LookML files using fd and then search for the dimension: name definition within those files.

---

To ensure the SQL reference ${TABLE}.name is correctly used in the LookML dimension definitions, we will search for this exact reference within the LookML files.

---

Verified the dimension definition for name.

The dimension name in the LookML file rent_as_employee_income_source.view.lkml correctly references the SQL field ${TABLE}.name, which is defined as a string in the SQL files.

• LookML file: metadata-ingestion/tests/integration/lookml/vv-lineage-and-liquid-templates/rent_as_employee_income_source.view.lkml
• SQL reference: ${TABLE}.name

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the dimension definition for `name`.

# Test: Check if the name field is correctly referenced and is of type string. Expect: Field exists and is of type string.
rg --type sql -A 5 $'name.*string'

Length of output: 35

---

Script:

#!/bin/bash
# Description: Verify the dimension definition for `name`.

# Search for the `name` field in SQL files and LookML files.
rg --type sql -A 5 'name'
rg --type lookml -A 5 'dimension: name'

Length of output: 19815

---

Script:

#!/bin/bash
# Description: Verify the dimension definition for `name`.

# Find LookML files and search for the `dimension: name` definition.
fd --type f --extension lkml | xargs rg 'dimension: name'

Length of output: 961

---

Script:

#!/bin/bash
# Description: Verify the SQL reference for `name` dimension.

# Search for the exact SQL reference `${TABLE}.name` within LookML files.
fd --type f --extension lkml | xargs rg '\$\{TABLE\}\.name'

Length of output: 610

metadata-ingestion/tests/integration/lookml/vv-lineage-and-liquid-templates/data.model.lkml (2)

29-30: Verify the explore block for rent_as_employee_income_source.

Ensure that the explore block for rent_as_employee_income_source is correct and follows best practices. Consider the following points:

• Verify that the explore block is correctly defined.
• Ensure that the explore block is necessary and used in the model.

Verification successful

---

The explore block for rent_as_employee_income_source is correctly defined and necessary.

The explore block is referenced and used in the codebase, indicating its necessity and correctness.

• metadata-ingestion/tests/integration/lookml/vv-lineage-and-liquid-templates/data.model.lkml
• metadata-ingestion/tests/integration/lookml/vv-lineage-and-liquid-templates/rent_as_employee_income_source.view.lkml

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the explore block for `rent_as_employee_income_source`.

# Test: Check if the explore block is correctly defined and used. Expect: Block is correctly defined and used.
rg --type lkml -A 5 $'explore: rent_as_employee_income_source'

Length of output: 92

---

Script:

#!/bin/bash
# Description: Find all .lkml files and search for the explore block for `rent_as_employee_income_source`.

# Step 1: Find all .lkml files
lkml_files=$(fd --extension lkml)

# Step 2: Search for the explore block within the found .lkml files
for file in $lkml_files; do
  echo "Searching in $file"
  rg --type-add 'lkml:*.lkml' -A 5 'explore: rent_as_employee_income_source' "$file"
done

Length of output: 26891

---

Script:

#!/bin/bash
# Description: Search for references to `rent_as_employee_income_source` in the codebase.

# Step 1: Search for references to `rent_as_employee_income_source` in .lkml files
rg --type-add 'lkml:*.lkml' 'rent_as_employee_income_source'

Length of output: 3325

---

9-9: Verify the inclusion of rent_as_employee_income_source.view.lkml.

Ensure that the inclusion statement for rent_as_employee_income_source.view.lkml is correct and follows best practices. Consider the following points:

• Verify that the included file path is correct.
• Ensure that the included file is necessary and used in the model.

Verification successful

---

Inclusion of rent_as_employee_income_source.view.lkml verified.

The inclusion statement for rent_as_employee_income_source.view.lkml is correct. The file path is accurate, and the elements defined within the file are used in the model, confirming its necessity and appropriate usage.

• rent_as_employee_income_source.view.lkml is correctly included and referenced in the model.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the inclusion of `rent_as_employee_income_source.view.lkml`.

# Test: Check if the included file path is correct. Expect: File exists.
fd 'rent_as_employee_income_source.view.lkml'

Length of output: 162

---

Script:

#!/bin/bash
# Description: Verify the usage of `rent_as_employee_income_source.view.lkml` in the model.

# Test: Check if `rent_as_employee_income_source.view.lkml` is referenced in the model files.
rg 'rent_as_employee_income_source.view.lkml' metadata-ingestion/tests/integration/lookml/vv-lineage-and-liquid-templates

Length of output: 265

---

Script:

#!/bin/bash
# Description: Verify the usage of elements defined in `rent_as_employee_income_source.view.lkml` in the model.

# Test: Extract the names of elements (like views or fields) defined in `rent_as_employee_income_source.view.lkml` and search for their references in the model files.

# Extract element names from `rent_as_employee_income_source.view.lkml`
element_names=$(rg 'view:|dimension:|measure:' metadata-ingestion/tests/integration/lookml/vv-lineage-and-liquid-templates/rent_as_employee_income_source.view.lkml | awk '{print $2}')

# Check if these element names are referenced in the model files
for name in $element_names; do
  rg $name metadata-ingestion/tests/integration/lookml/vv-lineage-and-liquid-templates
done

Length of output: 10679

metadata-ingestion/src/datahub/ingestion/source/looker/looker_template_language.py (3)

98-100: Verify the function definition _drop_derived_view_pattern.

Ensure that the function definition for _drop_derived_view_pattern is correct and follows best practices. Consider the following points:

• Verify that the regular expression pattern DERIVED_VIEW_PATTERN is correctly defined and used.
• Ensure that the function correctly removes the derived view pattern from the input string.

Verification successful

---

The function _drop_derived_view_pattern is correctly defined and follows best practices.

• The regular expression pattern DERIVED_VIEW_PATTERN is correctly defined as r"\$\{([^}]*)\}" in metadata-ingestion/src/datahub/ingestion/source/looker/lookml_config.py.
• The function correctly uses this pattern to remove ${ and } from the input string, while retaining the content inside the curly braces.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the function definition for `_drop_derived_view_pattern`.

# Test: Check if the regular expression pattern `DERIVED_VIEW_PATTERN` is correctly defined and used. Expect: Pattern is correctly defined and used.
rg --type python -A 5 $'DERIVED_VIEW_PATTERN'

Length of output: 75

---

Script:

#!/bin/bash
# Description: Verify the function definition for `_drop_derived_view_pattern`.

# Test: Check if the regular expression pattern `DERIVED_VIEW_PATTERN` is correctly defined and used. Expect: Pattern is correctly defined and used.
rg -A 5 'DERIVED_VIEW_PATTERN'

Length of output: 2715

---

117-117: Verify the return statement in _complete_incomplete_sql.

Ensure that the return statement in _complete_incomplete_sql is correct and follows best practices. Consider the following points:

• Verify that _drop_derived_view_pattern is correctly applied to the SQL query.
• Ensure that the SQL query is correctly transformed and returned.

Verification successful

---

The return statement in _complete_incomplete_sql is correct.

The function _complete_incomplete_sql correctly applies _drop_derived_view_pattern to the sql_query and returns the transformed query. The logic follows the intended purpose of ensuring the SQL query is complete before applying the pattern.

• The function ensures the SQL query has SELECT and FROM clauses.
• The transformed query is then passed through _drop_derived_view_pattern and returned.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the return statement in `_complete_incomplete_sql`.

# Test: Check if `_drop_derived_view_pattern` is correctly applied to the SQL query. Expect: Pattern is correctly applied.
rg --type python -A 5 $'_complete_incomplete_sql'

Length of output: 79

---

Script:

#!/bin/bash
# Description: Verify the return statement in `_complete_incomplete_sql`.

# Test: Check if `_drop_derived_view_pattern` is correctly applied to the SQL query. Expect: Pattern is correctly applied.
rg --type py -A 5 '_complete_incomplete_sql'

Length of output: 1207

---

Script:

#!/bin/bash
# Description: Retrieve the complete implementation of the `_complete_incomplete_sql` function.

# Test: Extract the full function body of `_complete_incomplete_sql` to verify the return statement.
ast-grep --lang python --pattern $'def _complete_incomplete_sql($_, $_) -> $_:\n    $$$'

Length of output: 1965

---

129-135: Verify the modifications in resolve_liquid_variable_in_view_dict.

Ensure that the modifications in resolve_liquid_variable_in_view_dict are correct and follow best practices. Consider the following points:

• Verify that view["sql_table_name"] is correctly transformed and assigned to view["datahub_transformed_sql_table_name"].
• Ensure that _drop_derived_view_pattern is correctly applied to view["datahub_transformed_sql_table_name"].

metadata-ingestion/src/datahub/ingestion/source/looker/lookml_concept_context.py (4)

Line range hint 62-196:
LGTM! The updated docstring is comprehensive and well-detailed.

The addition of Pattern 7 is clearly explained and enhances the documentation.

---

315-321: LGTM! The changes enhance the functionality.

The inclusion of a check for direct SQL queries is a useful enhancement.

---

408-412: LGTM! The changes enhance the functionality.

The inclusion of a check for direct SQL queries is a useful enhancement.

---

423-425: LGTM! The changes enhance the functionality.

The inclusion of a check for direct SQL queries is a useful enhancement.

metadata-ingestion/src/datahub/ingestion/source/looker/view_upstream.py (4)

240-240: LGTM! The changes enhance the structure of the class.

Making SqlBasedDerivedViewUpstream an abstract base class and adding an abstract method get_sql_query enhances its structure and extensibility.

Also applies to: 393-395

---

398-406: LGTM! The implementation aligns with the new pattern.

The get_sql_query method is correctly implemented for the DirectQueryUpstreamSource class.

---

407-414: LGTM! The implementation aligns with the new pattern.

The get_sql_query method is correctly implemented for the DerivedQueryUpstreamSource class.

---

Line range hint 636-671:
LGTM! The changes enhance the control flow.

The updates to instantiate the new subclasses based on the view context are correctly implemented.

metadata-ingestion/tests/integration/lookml/vv_lineage_liquid_template_golden.json (7)

1583-1600: LGTM!

The addition of the subTypes aspect specifying the dataset type as "View" is correct and enhances the metadata.

---

1601-1618: LGTM!

The addition of the viewProperties aspect with detailed LookML view logic is correct and enhances the dataset's metadata.

---

1619-1634: LGTM!

The addition of the container aspect linking the dataset to a specific container is correct and enhances the hierarchical metadata structure.

---

1635-1806: LGTM!

The addition of the proposedSnapshot aspect with detailed metadata, including BrowsePaths, Status, and UpstreamLineage, is correct and enhances the dataset's traceability and understanding of data flow.

---

1807-1830: LGTM!

The addition of the browsePathsV2 aspect outlining the navigational paths for the dataset is correct and enhances the dataset's usability.

---

Line range hint 1831-1842: LGTM!

The addition of the tagKey aspect specifying the tag name as "Dimension" is correct and enhances the metadata.

---

Line range hint 1843-1854: LGTM!

The addition of the tagKey aspect specifying the tag name as "Measure" is correct and enhances the metadata.

[coderabbitai]

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 8d79701 and 0d0866c.

Files selected for processing (4)

• metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py (2 hunks)
• metadata-ingestion/src/datahub/ingestion/source/looker/looker_template_language.py (4 hunks)
• metadata-ingestion/src/datahub/ingestion/source/looker/lookml_concept_context.py (9 hunks)
• metadata-ingestion/tests/integration/looker/golden_test_ingest.json (2 hunks)

Files skipped from review due to trivial changes (1)

• metadata-ingestion/tests/integration/looker/golden_test_ingest.json

Files skipped from review as they are similar to previous changes (3)

• metadata-ingestion/src/datahub/ingestion/source/looker/looker_source.py
• metadata-ingestion/src/datahub/ingestion/source/looker/looker_template_language.py
• metadata-ingestion/src/datahub/ingestion/source/looker/lookml_concept_context.py