[TDL-23979] Fix the existing schema and discovery (#28)

* Fixed discovery.

* Schema changes for accounts stream.

* Fixed accounts schema.

* Fixed STREAM_CONFIG.

* Fixed teams stream.

* Added creatorId & updaterId in teams schema.

* CHanges:
1. Fixed accounts, calls & teams schema.
2. Added phoneNumberId in STREAM_CONFIGS.

* Fixed content categories schema.

* Fixed events schema.

* Fixed mailboxes schema.

* Fixed mailings schema.

* Fixed opportunities schema.

* Fixed prospects schema.

* Fixed sequences schema.

* Fixed sequence_templates schema.

* Fixed sequence_steps schema.

* Fixed sequence_states schema.

* Fixed tasks schema.

* Fixed review comments.

* Changes:
1) Fixed users schema.
2) Changed teams & sequenceTemplates streams to incremental.

* Fixed sequence_steps schema and added fixture for tasks & events stream.

* Removed redundant tasks stream from skip list.

* Changes:
1) Making sequence_templates to full table as updatedAt is not filterable & sortable.

* Fixed bookmarking.

* Fixed discovery.

* Schema changes for accounts stream.

* Fixed accounts schema.

* Fixed STREAM_CONFIG.

* Fixed teams stream.

* Added creatorId & updaterId in teams schema.

* CHanges:
1. Fixed accounts, calls & teams schema.
2. Added phoneNumberId in STREAM_CONFIGS.

* Fixed content categories schema.

* Fixed events schema.

* Fixed mailboxes schema.

* Fixed mailings schema.

* Fixed opportunities schema.

* Fixed prospects schema.

* Fixed sequences schema.

* Fixed sequence_templates schema.

* Fixed sequence_steps schema.

* Fixed sequence_states schema.

* Fixed tasks schema.

* Fixed review comments.

* Changes:
1) Fixed users schema.
2) Changed teams & sequenceTemplates streams to incremental.

* Fixed sequence_steps schema and added fixture for tasks & events stream.

* Removed redundant tasks stream from skip list.

* Changes:
1) Making sequence_templates to full table as updatedAt is not filterable & sortable.

* Fixed bookmarking.

* Fixed bookmarking logic.

* sort the streams while storing in catalog file

* Fixed sync_endpoint test by modifying bookmark value.

* Changes:
1) Added logger for currently syncing.
2) Fixed formatting for all unit tests.

* [TDL-20774] Integration tests (#27)

* Added replication keys in catalog during discovery.
* Added base test suite & discovery test.
* Added new lines at the end of test files.
* Changed teams & sequence_templates from full to incremental.
* Added automatic and all fields tests.
* Fixed MISSING_FIELDS & streams_to_exclude in all fields test.

* Fixed MISSING_FIELDS in automatic fields test.

* Changes:
1) Added new pagination test.
2) Changed docstring for all fields test.

* Added start date test.

* Changed sequence_templates stream to full table in base.py

* Changes:
1) Added new bookmark test.
2) Fixed a field in automatic fields test.

* Added new Sync Canary Test.

* Fixed formatting for all integration tests.

* Changes:
1) Fixed boomark & start date test.
2) Added new interrupted sync test.

* Fixed names for Bookmark & Start date test.

* Fixed bookmark & start date tests.

* fix bookmark, interrupted sync and start date tests

* add integration tests in circle ci config

* update tap-tester image version in cicle ci config

* Fix integration tests

* Fix all fields test and minor fixes

* fix bookmark test review

---------

Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com>

---------

Co-authored-by: Sourabh Gandhi <sgandhi@talend.com>
Co-authored-by: RushiT0122 <rtodkar@stitchdata-talend.com>

.circleci/config.yml

@@ -5,7 +5,7 @@ orbs:
 jobs:
   build:
     docker:
-      - image: 218546966473.dkr.ecr.us-east-1.amazonaws.com/circle-ci:tap-tester-v4
+      - image: 218546966473.dkr.ecr.us-east-1.amazonaws.com/circle-ci:stitch-tap-tester
     steps:
       - checkout
       - run:
@@ -33,6 +33,13 @@ jobs:
             nosetests --with-coverage --cover-erase --cover-package=tap_outreach --cover-html-dir=htmlcov tests/unittests
             coverage html
           when: always
+      - run:
+          name: 'Integration Tests'
+          command: |
+            aws s3 cp s3://com-stitchdata-dev-deployment-assets/environments/tap-tester/tap_tester_sandbox dev_env.sh
+            source dev_env.sh
+            source /usr/local/share/virtualenvs/tap-tester/bin/activate
+            run-test --tap=tap-outreach tests
       - store_test_results:
           path: test_output/report.xml
       - store_artifacts:

tap_outreach/discover.py

@@ -15,8 +15,8 @@ def get_schemas():
     schemas_metadata = {}
     schemas_path = get_abs_path('schemas')
 
-    file_names = [f for f in os.listdir(schemas_path)
-                  if os.path.isfile(os.path.join(schemas_path, f))]
+    file_names = sorted([f for f in os.listdir(schemas_path)
+                  if os.path.isfile(os.path.join(schemas_path, f))])
 
     for file_name in file_names:
         stream_name = file_name[:-5]
@@ -28,9 +28,11 @@ def get_schemas():
             singer.resolve_schema_references(schema, refs)
 
         replication = STREAM_CONFIGS[stream_name]['replication']
+        replication_key = STREAM_CONFIGS[stream_name].get('filter_field', None)
         meta = metadata.get_standard_metadata(
             schema=schema,
             key_properties=['id'],
+            valid_replication_keys=[replication_key] if replication_key else [],
             replication_method='FULL_TABLE' if replication == 'full' else replication.upper()
         )