Destination S3-V2: avro schema matches legacy CDK

[johnny-schmidt]

What

Client reported that _airbyte_extracted_at was missing logical type timestamp-millis. During investigation I realized that _airbyte_raw_id was also missing logical type uuid.

Fixed both issues and added tests confirming that avro schemas match legacy schemas.

Why

I'm hacking this in surgically for now. However, the correct fix is to introduce TimestampMillis and UUID as discrete extended types (assuming this gets approved).

Also

For reference, a legacy schema pulled from the logs of the escalating client's previous runs:

{
         "name":"_airbyte_raw_id",
         "type":{
            "type":"string",
            "logicalType":"uuid"
         }
      },
      {
         "name":"_airbyte_extracted_at",
         "type":{
            "type":"long",
            "logicalType":"timestamp-millis"
         }
      },
      {
         "name":"_airbyte_generation_id",
         "type":"long"
      },
      {
         "name":"_airbyte_meta",
         "type":{
            "type":"record",
            "name":"_airbyte_meta",
            "namespace":"",
            "fields":[
               {
                  "name":"changes",
                  "type":{
                     "type":"array",
                     "items":{
                        "type":"record",
                        "name":"change",
                        "fields":[
                           {
                              "name":"field",
                              "type":"string"
                           },
                           {
                              "name":"change",
                              "type":"string"
                           },
                           {
                              "name":"reason",
                              "type":"string"
                           }
                        ]
                     }
                  }
               },
               {
                  "name":"sync_id",
                  "type":"long"
               }
            ]
         }
      },

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 15, 2025 0:07am