Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(source-amazon-seller-partner): Convert to low-code and uptake concurrency with the upgrade to CDK v6 #49293

Open
wants to merge 68 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
70a2587
Amazon SP: bump version
artem1205 Dec 12, 2024
51e5445
Amazon SP: ref imports
artem1205 Dec 12, 2024
199818c
Amazon SP: migrate auth
artem1205 Dec 13, 2024
2f3ccdb
Amazon SP: migrate config migrations
artem1205 Dec 13, 2024
2695af2
Amazon SP: migrate requests to http client
artem1205 Dec 13, 2024
ab8a0b6
Amazon SP: fix mocker tests
artem1205 Dec 13, 2024
5a120f8
Amazon SP: add noauth authenticator for test
artem1205 Dec 13, 2024
499cde4
Amazon SP: add noauth authenticator for test
artem1205 Dec 13, 2024
cd7f25c
Amazon SP: add noauth authenticator for test
artem1205 Dec 13, 2024
393c9e1
Amazon SP: fix unit tests
artem1205 Dec 13, 2024
a7e2894
Amazon SP: fmt
artem1205 Dec 16, 2024
7a865e0
Amazon SP: add manifest
artem1205 Dec 16, 2024
f78a5d9
Amazon SP: add orderitems
artem1205 Dec 16, 2024
c94fcc1
Amazon SP: add list_financial_events + backoff strategy
artem1205 Dec 17, 2024
80d4702
Amazon SP: add decoder
artem1205 Dec 18, 2024
39f8fd9
Amazon SP: add decoder
artem1205 Dec 18, 2024
e291d96
Amazon SP: add report stream
artem1205 Jan 2, 2025
7139b56
Amazon SP: add typetransformer
artem1205 Jan 2, 2025
695c6ea
Amazon SP: add init
artem1205 Jan 2, 2025
1725226
Amazon SP: ref
artem1205 Jan 3, 2025
14c94ca
Amazon SP: ref
artem1205 Jan 3, 2025
f753327
Amazon SP: add decoders & type transformers
artem1205 Jan 6, 2025
d812d39
Amazon SP: add incremental reporting streams
artem1205 Jan 6, 2025
58aa06e
Amazon SP: add GET_V2_SETTLEMENT_REPORT_DATA_FLAT_FILE
artem1205 Jan 7, 2025
5d29dd9
Amazon SP: add vendor_orders & vendor_direct_fulfillment_shipping
artem1205 Jan 8, 2025
240008a
Amazon SP: ref all streams except AnalyticsStream
artem1205 Jan 8, 2025
1e646a2
Merge remote-tracking branch 'origin/master' into artem1205/source-am…
artem1205 Jan 8, 2025
fecf6fa
Amazon SP: ref
artem1205 Jan 8, 2025
e01326c
Amazon SP: ref
artem1205 Jan 8, 2025
775f969
Amazon SP: ref test_transform
artem1205 Jan 8, 2025
2028d28
Amazon SP: fix unit tests
artem1205 Jan 8, 2025
8af5875
Amazon SP: fix transformer
artem1205 Jan 8, 2025
4d82f58
Amazon SP: add missing typetransformer
artem1205 Jan 8, 2025
3f2b7bb
Amazon SP: fmt init
artem1205 Jan 8, 2025
3af5854
Amazon SP: ref tests
artem1205 Jan 8, 2025
217d118
Amazon SP: ref
artem1205 Jan 8, 2025
9d162c1
Merge remote-tracking branch 'origin/master' into artem1205/source-am…
artem1205 Jan 9, 2025
4fc626d
Amazon SP: ref
artem1205 Jan 9, 2025
c4757df
CDK: update poetry
artem1205 Jan 9, 2025
87bb771
Amazon SP: ref
artem1205 Jan 9, 2025
7771020
Amazon SP: ref headers
artem1205 Jan 10, 2025
67104c7
Amazon SP: ref url path
artem1205 Jan 10, 2025
1f2d3d4
Amazon SP: fix vendor orders
artem1205 Jan 10, 2025
83c1c74
Amazon SP: fix vendor orders
artem1205 Jan 10, 2025
8fc30ac
Amazon SP: fix vendor orders
artem1205 Jan 10, 2025
2da3502
Amazon SP: ref
artem1205 Jan 14, 2025
e70e03a
Amazon SP: update poetry.lock
artem1205 Jan 14, 2025
03de7e5
fixing end time, formatting, renaming, adding formatting for timezone…
brianjlai Jan 17, 2025
d2ecc20
Adding cursor value to GET_MERCHANT_LISTINGS_ALL_DATA records
maxi297 Jan 22, 2025
c74bbea
A lotttttt of fixes to the reports streams to fix cursor values, add …
brianjlai Jan 23, 2025
13eaa8f
Fix GET_BRAND_ANALYTICS_MARKET_BASKET_REPORT
maxi297 Jan 23, 2025
97465ae
fix relative paths in test
maxi297 Jan 23, 2025
fe50c49
don't sleep on tests
maxi297 Jan 23, 2025
aea7aeb
Removing analytic streams
maxi297 Jan 23, 2025
b8fdfbd
fix test_given_report_when_read_then_state_message_produced_and_state…
maxi297 Jan 23, 2025
bee1458
fixing a million tests, implementing streams that were missing like G…
brianjlai Jan 24, 2025
e2302a9
remove some unneeded tests rebase to latest CDK w/ fixes, enforce max…
brianjlai Jan 24, 2025
2da6092
Merge branch 'master' into artem1205/source-amazon-SP-update-cdk
brianjlai Jan 24, 2025
7c1f810
increase code coverage, ignore unused files, fix date ranges for fina…
brianjlai Jan 25, 2025
5fd2327
transform into date instead of datetime
brianjlai Jan 25, 2025
60b75c8
accept incoming state in form of %Y-%m-%d for certain streams and fix…
brianjlai Jan 25, 2025
7666fad
fix state format emitting for a few streams
brianjlai Jan 25, 2025
673f71e
fixing date format for incoming state values
brianjlai Jan 26, 2025
b5808f8
remove GET_V2_SETTLEMENT_REPORT_DATA_FLAT_FILE from expected records …
brianjlai Jan 26, 2025
70a836a
update expected records for ListFinancialEventGroups
brianjlai Jan 26, 2025
3bb9685
update expected records and make test window smaller to speed up acce…
brianjlai Jan 26, 2025
bba4049
skip empty streams
brianjlai Jan 26, 2025
a23d239
removed expected
brianjlai Jan 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Amazon SP: ref test_transform
Signed-off-by: Artem Inzhyyants <artem.inzhyyants@gmail.com>
  • Loading branch information
artem1205 committed Jan 8, 2025
commit 775f96943537f578a353118e6ddd026991171b7c
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,47 @@


import pytest
from source_amazon_seller_partner.streams import (
FlatFileSettlementV2Reports,
LedgerDetailedViewReports,
MerchantListingsFypReport,
MerchantListingsReports,
SellerFeedbackReports,
from source_amazon_seller_partner.components import (
FlatFileSettlementV2ReportsTypeTransformer,
LedgerDetailedViewReportsTypeTransformer,
MerchantListingsFypReportTypeTransformer,
MerchantReportsTypeTransformer,
SellerFeedbackReportsTypeTransformer,
)

from airbyte_cdk.test.state_builder import StateBuilder

def reports_stream(marketplace_id):
stream = SellerFeedbackReports(
stream_name="SELLER_FEEDBACK_REPORTS",
url_base="https://test.url",
replication_start_date="2010-01-25T00:00:00Z",
replication_end_date="2017-02-25T00:00:00Z",
marketplace_id=marketplace_id,
authenticator=None,
period_in_days=0,
report_options=None,
)
return stream

@pytest.fixture(name="config")
def config_fixture():
config = {
"replication_start_date": "2021-07-01T00:00:00Z",
"refresh_token": "<refresh_token>",
"lwa_app_id": "<lwa_app_id>",
"lwa_client_secret": "<lwa_client_secret>",
"aws_access_key": "<aws_access_key>",
"aws_secret_key": "<aws_secret_key>",
"role_arn": "<role_arn>",
"aws_environment": "PRODUCTION",
"region": "US",
}

return config


@pytest.fixture()
def stream_by_name(config):
# use local import in favour of global because we need to make imports after setting the env variables
from source_amazon_seller_partner.source import SourceAmazonSellerPartner

def mocker(stream_name, source_config=config):
source = SourceAmazonSellerPartner(None, source_config, StateBuilder().build())
streams = source.streams(source_config)
for stream in streams:
if stream.name == stream_name:
return stream

return mocker


INPUT_DATES = {
Expand All @@ -39,7 +59,7 @@ def reports_stream(marketplace_id):

def parametrize_seller_feedback():
result = []
for marketplace_id, date_format in SellerFeedbackReports.MARKETPLACE_DATE_FORMAT_MAP.items():
for marketplace_id, date_format in SellerFeedbackReportsTypeTransformer.MARKETPLACE_DATE_FORMAT_MAP.items():
for index, input_date in enumerate(INPUT_DATES.get(date_format)):
expected_date = EXPECTED_DATES[index]
result.append(
Expand All @@ -54,10 +74,9 @@ def parametrize_seller_feedback():


@pytest.mark.parametrize("marketplace_id,input_data,expected_data", parametrize_seller_feedback())
def test_transform_seller_feedback(marketplace_id, input_data, expected_data):
stream = reports_stream(marketplace_id)
transformer = stream.transformer
schema = stream.get_json_schema()
def test_transform_seller_feedback(marketplace_id, input_data, expected_data, stream_by_name):
transformer = SellerFeedbackReportsTypeTransformer(config={"marketplace_id": marketplace_id})
schema = stream_by_name("GET_SELLER_FEEDBACK_DATA").get_json_schema()
transformer.transform(input_data, schema)

assert input_data == expected_data
Expand All @@ -76,10 +95,9 @@ def test_transform_seller_feedback(marketplace_id, input_data, expected_data):
),
),
)
def test_transform_merchant_reports(report_init_kwargs, input_data, expected_data):
stream = MerchantListingsReports(**report_init_kwargs)
transformer = stream.transformer
schema = stream.get_json_schema()
def test_transform_merchant_reports(input_data, expected_data, stream_by_name):
transformer = MerchantReportsTypeTransformer()
schema = stream_by_name("GET_MERCHANT_LISTINGS_ALL_DATA").get_json_schema()
transformer.transform(input_data, schema)
assert input_data == expected_data

Expand All @@ -97,10 +115,9 @@ def test_transform_merchant_reports(report_init_kwargs, input_data, expected_dat
),
),
)
def test_transform_merchant_fyp_reports(report_init_kwargs, input_data, expected_data):
stream = MerchantListingsFypReport(**report_init_kwargs)
transformer = stream.transformer
schema = stream.get_json_schema()
def test_transform_merchant_fyp_reports(input_data, expected_data, stream_by_name):
transformer = MerchantListingsFypReportTypeTransformer()
schema = stream_by_name("GET_MERCHANTS_LISTINGS_FYP_REPORT").get_json_schema()
transformer.transform(input_data, schema)
assert input_data == expected_data

Expand All @@ -115,10 +132,9 @@ def test_transform_merchant_fyp_reports(report_init_kwargs, input_data, expected
({"Date": "", "dataEndTime": "2022-07-31"}, {"Date": "", "dataEndTime": "2022-07-31"}),
),
)
def test_transform_ledger_reports(report_init_kwargs, input_data, expected_data):
stream = LedgerDetailedViewReports(**report_init_kwargs)
transformer = stream.transformer
schema = stream.get_json_schema()
def test_transform_ledger_reports(input_data, expected_data, stream_by_name):
transformer = LedgerDetailedViewReportsTypeTransformer()
schema = stream_by_name("GET_LEDGER_DETAIL_VIEW_DATA").get_json_schema()
transformer.transform(input_data, schema)
assert input_data == expected_data

Expand All @@ -133,9 +149,8 @@ def test_transform_ledger_reports(report_init_kwargs, input_data, expected_data)
({"posted-date": "", "dataEndTime": "2022-07-31"}, {"posted-date": None, "dataEndTime": "2022-07-31"}),
),
)
def test_transform_settlement_reports(report_init_kwargs, input_data, expected_data):
stream = FlatFileSettlementV2Reports(**report_init_kwargs)
transformer = stream.transformer
schema = stream.get_json_schema()
def test_transform_settlement_reports(report_init_kwargs, input_data, expected_data, stream_by_name):
transformer = FlatFileSettlementV2ReportsTypeTransformer()
schema = stream_by_name("GET_V2_SETTLEMENT_REPORT_DATA_FLAT_FILE").get_json_schema()
transformer.transform(input_data, schema)
assert input_data == expected_data