Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TDL-23979] Fix the existing schema and discovery #28

Merged
merged 54 commits into from
Sep 28, 2023
Merged

[TDL-23979] Fix the existing schema and discovery #28

merged 54 commits into from
Sep 28, 2023

Conversation

shantanu73
Copy link
Contributor

Description of change

  1. Fixed Discovery by adding "valid-replication-keys" parameter in catalog
  2. Schema changes as per API documentation

Manual QA steps

  • Ran discovery and validated schema in catalog.json

Risks

  • Incorrect schema may cause integration tests to fail.

Rollback steps

  • revert this branch

@shantanu73 shantanu73 changed the title Fixed discovery. [TDL-23979] Fix the existing schema and discovery Sep 6, 2023
@RushiT0122 RushiT0122 self-requested a review September 6, 2023 10:28
RushiT0122
RushiT0122 reviewed Sep 6, 2023
View reviewed changes
tap_outreach/discover.py Outdated Show resolved Hide resolved
@shantanu73 @sgandhi1311
Added creatorId & updaterId in teams schema.
d0ba75a
@shantanu73 @sgandhi1311
CHanges:
288c61a 
1. Fixed accounts, calls & teams schema.
2. Added phoneNumberId in STREAM_CONFIGS.
@shantanu73 @sgandhi1311
Fixed content categories schema.
c131222
@shantanu73 @sgandhi1311
Fixed events schema.
594ccc3
@shantanu73 @sgandhi1311
Fixed mailboxes schema.
c4829e0
@shantanu73 @sgandhi1311
Fixed mailings schema.
0cfffda
@shantanu73 @sgandhi1311
Fixed opportunities schema.
e51a815
@shantanu73 @sgandhi1311
Fixed prospects schema.
9783c78
@shantanu73 @sgandhi1311
Fixed sequences schema.
518e7ba
@shantanu73 @sgandhi1311
Fixed sequence_templates schema.
009919d
@shantanu73 @sgandhi1311
Fixed sequence_steps schema.
e4c33d3
@shantanu73 @sgandhi1311
Fixed sequence_states schema.
56244a0
@shantanu73 @sgandhi1311
Fixed tasks schema.
7175e84
@shantanu73 @sgandhi1311
Fixed review comments.
5f27f5e
@shantanu73 @sgandhi1311
Changes:
e761390 
1) Fixed users schema.
2) Changed teams & sequenceTemplates streams to incremental.
@shantanu73 @sgandhi1311
Fixed sequence_steps schema and added fixture for tasks & events stream.
28d943c
@shantanu73 @sgandhi1311
Removed redundant tasks stream from skip list.
38863a1
@shantanu73 @sgandhi1311
Changes:
d97868a 
1) Making sequence_templates to full table as updatedAt is not filterable & sortable.
@shantanu73 @sgandhi1311
Fixed bookmarking.
61b7a53
@sgandhi1311 sgandhi1311 force-pushed the TDL-23979-Fix-schema-and-discovery branch from 513f264 to 61b7a53 Compare September 11, 2023 13:20
shantanu73 and others added 6 commits September 11, 2023 13:23
@shantanu73
Merge branch 'TDL-23979-Fix-schema-and-discovery' of github.com:singe…
f456f7a 
…r-io/tap-outreach into TDL-23979-Fix-schema-and-discovery
@shantanu73
Fixed bookmarking logic.
c053d88
@sgandhi1311
sort the streams while storing in catalog file
14d8976
@shantanu73
Fixed sync_endpoint test by modifying bookmark value.
93ee200
@shantanu73
Changes:
3be0832 
1) Added logger for currently syncing.
2) Fixed formatting for all unit tests.
@shantanu73
[TDL-20774] Integration tests (#27)
a2ba6e9 
* Added replication keys in catalog during discovery.
* Added base test suite & discovery test.
* Added new lines at the end of test files.
* Changed teams & sequence_templates from full to incremental.
* Added automatic and all fields tests.
* Fixed MISSING_FIELDS & streams_to_exclude in all fields test.

* Fixed MISSING_FIELDS in automatic fields test.

* Changes:
1) Added new pagination test.
2) Changed docstring for all fields test.

* Added start date test.

* Changed sequence_templates stream to full table in base.py

* Changes:
1) Added new bookmark test.
2) Fixed a field in automatic fields test.

* Added new Sync Canary Test.

* Fixed formatting for all integration tests.

* Changes:
1) Fixed boomark & start date test.
2) Added new interrupted sync test.

* Fixed names for Bookmark & Start date test.

* Fixed bookmark & start date tests.

* fix bookmark, interrupted sync and start date tests

* add integration tests in circle ci config

* update tap-tester image version in cicle ci config

* Fix integration tests

* Fix all fields test and minor fixes

* fix bookmark test review

---------

Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com>
@RushiT0122 RushiT0122 self-requested a review September 28, 2023 06:29
@shantanu73 shantanu73 merged commit fff263c into master Sep 28, 2023
@dsprayberry dsprayberry deleted the TDL-23979-Fix-schema-and-discovery branch November 27, 2023 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RushiT0122 RushiT0122 RushiT0122 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@shantanu73 @RushiT0122 @sgandhi1311