[EPIC] All CDK Connectors emit AirbyteTraceMessage when something goes wrong

[evantahler]

Using AirbyteTraceMessage will set failure_reason metadata automagically. This replaces the current failure behavior of logging and throwing when something goes wrong. This story is to update the CDK as naively as possible such that any crash produces a log message in the right format before existing. e.g. this would be wrapping the calls to read() and write() in a try/catch.

Relevant Spec: https://docs.google.com/document/d/1ctrj3Yh_GjtQ93aND-WH3ocqGxsmxyC3jfiarrF6NY0/edit#

Class AirbyteLoggedException () {
  constructor(e:Error, display_message:string, failureType=ENUM<failure_types>=”system_error”) {
    log(new AirbyteLogMessage(e, level)) ;   
    log(new AirbyteTraceMessage(type=”error”, display_message, internalMessage=e, stack_trace=e.stack, failureType)) ;
    process.exit(1)
  }
}

// usage

try {
  await read(config, configured_catalog);
} catch (error) {
  throw AirbyteLoggedException(error, ‘could not connect to API, check your password’, “user_error”);
}

Which produces:

 AirbyteTraceMessage:
    type: error_status
    emitted_at: 123456
    error_status: 
        message: Cannot connect to postgres.  Check your credentials
        internal_message: JDBC Error - ssl cert invalid
        stack_trace: JDBC Error - ssl cert invalid (line 101)\nfrom connect() on line 101\nfrom read() on line 202
        failure_type: config_error

CDKs

• Update the Java CDK to emit AirbyteTraceMessage #12471
• Update the Python CDK to emit AirbyteTraceMessage #12472
• [EPIC] Improve database sources & destinations error messages #11322
• SAT Testing confirms that a crash produces at least one AirbyteTraceMessage #12473

---

SAT Ideas:

• All connectors have an invalid config file - so, if you run a sync with the invalid config file, an AirbyteTraceMessage should be emitted

[evantahler]

This story is naïve by design in this first pass so what we can have filled-out failureReasons to work with. Future stories will work within each connector to produce better and more meaningful failure reasons.

[sherifnada]

note for implementer: we need to first publish the protocol / java/ python generated files to unblock this ticket. It's a small part of #12161

[Phlair]

Context link: Here's the PR adding AirbyteTraceMessage to the protocol