Display sync error messages from connectors (use AirbyteTraceMessage to populate failure reason)

[lmossman]

What

See the tech spec for full context on this issue: https://docs.google.com/document/d/1nDVU-6_lDHkNVHPFHwfozaEoGl9hmV42RaAbv__6Bqo/edit

We would like to offer the ability for connectors to send the platform "display" messages that will be shown to the user, as described in the above tech spec. We will be going with Proposal 2.

This ticket only focuses on changing the protocol and setting the "failure reason" for a sync attempt to one of these messages if they exist for that attempt.

How

This ticket is composed of two changes, as described in the tech spec linked above:

• Add the AirbyteDisplayMessage object to the airbyte protocol, with a matching AirbyteMessage type (e.g. DISPLAY)
• Make the Airbyte UI use the first ERROR AirbyteDisplayMessage as the failure reason of the attempt (if none, default to current behavior)
  • This will involve modifying the DefaultReplicationWorker / DefaultAirbyteStreamFactory to look for DISPLAY AirbyteMessages, and if it finds any with level: ERROR, set add those to the failure reasons of the attempt.
  • Note: as described in the implementation notes of the tech spec, the UI currently selects the first failureReason of the attempt to show in the UI. Therefore, the change here will need to ensure that the first ERROR AirbyteDisplayMessage comes first in the attempt's failureReasons list, so that the UI selects that one to use (alternatively, the UI code could be changed to use a different selection strategy)

[evantahler]

Some notes since this story was initially written:

• A few extra fields have been added to the AirbyteDisplayMessage spec to better enable reporting and error correlation: internal_message, stack_trace, failure_type, and log_timestamp.
• As noted above, this additional information should be to fill out the sync attempt's failure reasons - and now we have data for most of those fields.

[evantahler]

#12322 is the epic we can use to track the work to add AirbyteDisplayMessage to the CDK and connectors. This replaces the current behavior of logging and throwring when something goes wrong.

[evantahler]

Noting a spec update - it's AirbyteTraceMessage now!

[sherifnada]

@alovew did this PR #12619 close this issue?

[alovew]

@sherifnada yep!