feat(source-google-sheets): migrate low code #50843
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…xtractor that matches schema properties with values
…in components resolver in order to get an indexed object of properties where index is the order. - Add partition router to retriver in components resolver so we can slice.
- Remove old SourceClass to temp file to be deleted and - Remove unused transformations - Make extractor ha transform data from component mapping
…anges to discard empty cells in first row
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
/format-fix
|
lazebnyi
approved these changes
Jan 30, 2025
|
|
maxi297
reviewed
Jan 30, 2025
airbyte-integrations/connectors/source-google-sheets/source_google_sheets/config_migrations.py
Outdated
Show resolved
Hide resolved
airbyte-integrations/connectors/source-google-sheets/integration_tests/expected_records.txt
Show resolved
Hide resolved
… github.com:airbytehq/airbyte into aldogonzalez8/source-google-sheets/migrate-low-code
maxi297
approved these changes
Jan 30, 2025
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
To migrate
source-google-sheetsconnector relying ongoogle-api-python-client/google-auth-httplib2to communicate with API to manifest using ourHttpRequesterand latest CDK features.Fixed: https://github.com/airbytehq/airbyte-internal-issues/issues/10962
How
Removed helper and client file dependencies to use new CDK features like
ComponentResolver,DynamicSchemaandDynamicStreams.Review guide
Components Resolver
The component resolver has its own
SimpleRetrieverwith apartition_routerthat depends on a static stream,get_spreadsheet_info_and_sheets. This stream is responsible for requesting spreadsheet information and retrieving all the sheets.Request:
{{ config["spreadsheet_id"] }}?includeGridData=false&alt=jsonNow, using the sheet IDs, we utilize the component resolver retriever/requester to obtain information, mainly the first row of data and the row count.
Request:
{{ config["spreadsheet_id"] }}?includeGridData=true&ranges={{ stream_partition.sheet_id }}!1:1&alt=jsonWith the response, we can start passing information using the
component_mapping.Schema Loader
The schema loader has its own
SimpleRetriever, which makes the following request:Request:
{{ config["spreadsheet_id"] }}?includeGridData=true&ranges={{ stream_partition.sheet_id }}!1:1&alt=jsonAdditionally, the schema loader has a custom record extractor,
DpathSchemaExtractor, which is responsible for:Parsing sheet headers from the provided raw schema. This method assumes that the data is contiguous, i.e., every cell contains a value, and the first cell that does not contain a value denotes the end of the headers.
Performing name conversions.
Removing duplicated fields.
Dynamic Streams -
DpathSchemaMatchingExtractorThis extractor performs similar functions to the one in the
SchemaLoader, but it also handles the matching between the schema (first row of the sheet) and the data (rows fetched), ensuring that record data is correctly assigned to the fields. A sample of the process can be found in the class description.RangePartitionRouter
Sheet data is fetched in ranges, such as
2-202, 203-303, 404-504and so on, until the entire sheet is scanned. The router then creates slices for each range.User Impact
No
Can this PR be safely reverted and rolled back?