[EPIC] Ensure Source and Destination errors are distinguished and useful (Defogging phase 0)

[evantahler]

From Connector Quality & Observability - Defogging Connector Metrics

The goal of Phase 0 is to ensure that are able to store and syndicate relevant failure reasons for every failed sync. This includes gaining specific messages from the connector when something goes wrong. This Epic introduces AirbyteTraceMessage to allow connectors to share better information with the platform.

Analyitcs Team

• https://github.com/airbytehq/airbyte-warehouse/issues/92
• https://github.com/airbytehq/airbyte-warehouse/issues/83
• Determine which OSS versions to include in reports

OSS Platform Team

• Push Failure Reason info to Segment #10190
• Add AirbyteTraceMessage to Airbyte protocol #12458
• Add AirbyteTraceMessage to the airbyte protocol #12519
• Display sync error messages from connectors (use AirbyteTraceMessage to populate failure reason) #12161
• Migrate OSS to the new scheduler #10021 (until this happens no failure reason will persist in the OSS db)

UI Team

• Log View: Allow linking to a specific line in a log #12130

Connectors/sources Team

• [EPIC] All CDK Connectors emit AirbyteTraceMessage when something goes wrong #12322
• New source-dockerhub connector from docker hub registry API #12773

Connectors/databases Team

Future Work

• [EPIC] Improve database sources & destinations error messages #11322
• Report StandardSyncSummary outcomes at the end of the logs #12864
• Update XXX Connector with detailed log messages when crashing

[sherifnada]

fyi @evantahler this is a relevant ticket being done by the platform team: #11279

[evantahler]

The final goal of this story is to end up with logic in our datawarehouse similar to this sheet

[evantahler]

Not part of this epic, but related:
This epic which will attribute problems when the source emits RECORDS that don't match the SPEC properly to the Source, not Normalization - #11279

[evantahler]

I'm calling #12322 done, as both CDKs now emit AirbyteLogMessage generically when something goes wrong, and new versions of GA connectors have been released. #11322 remains open, but that's enhancement future work.

[evantahler]

Optional story - Add a sync summary to the end of the logs - #12864

[sherifnada]

Calling this one done as the bulk of the work is completed