Source based metadata management workflow approach

[kaypro4]

Hello, I've been working through designing and beginning to implement an approach to exporting changes made in a "development" org out to source control (git) and then using automated pipelines to import those changes into "staging" and "production" orgs.

My goal is to get as few preconditions in an org as possible so that there is as little need for manual configuration changes related to TML managed items.

The workflow I envision is developers work in the development org to make changes to connections, tables, worksheets, answers and liveboards, and when they are ready to publish changes into staging, they create a feature branch in git locally and run a command such as this.

cs_tools tools scriptability export --org 144009899 --tags "source-controlled" --export-associated .

This has a concept of using a tag to identify what to pull, but you could also specify guids, limit the types, etc if desired. You would want to do that in cases where you had multiple parallel changes happening in development and only a subset are related to your changes and are ready to go to the next environment.

I was then envisioning a commit, push and PR review process happening where the TML can be reviewed and merged into a "staging" branch when it looks good. Possibly we also add in a VALIDATE_ONLY merge check to validate that the metadata is deployable. Either way, the merge would fire off an automated import to the staging org via a command like this:

cs_tools tools scriptability import --org 1146874994 --source 144009899 --dest 1146874994 --show-mapping --import-policy ALL_OR_NONE --tags "source-deployed" .

I have been moving this along and testing things out. I did discover that I believe the connection may first need to exist in the target org, so it can't be entirely "fresh".

But the next challenge I'm running into is this error when testing the import command above locally.

Error decoding connection 2a018fc8-747d-4926-b0d9-2420fe63793e: __init__() got an unexpected keyword argument 'code' supplied data does not produce a valid TML (Connection) document

Mitigation is Verify TS connection and GUID.

I think I may be missing a concept related to GUIDs, and specifically guid mapping, and I've been reviewing the docs and source and trying to understand but can't seem to get a handle on it. I was hoping that buy specifying the source and dest in the import, that the mapping would not need to live in source, but I'm questioning if that is true. Ideally, the deployment process to staging and production for example would not need to write anything back to source...but it could if necessary with a little bit of work.

Anyway, I'd appreciate any feedback about overall approach and any support around this place where I seem to be stuck.

Thanks!
Matt

[billdback-ts]

@kaypro4 there are two topics here.

For the flow, this should be fine. Connections should work with scriptability (I use it myself), but you have to set the secure information that isn't exported. If you don't want to do this in the TML (for good reasons), then you would create it first. Note that you might have to update the mapping file manually when you create new connections.

For the second issue, that seems like a TML issue. I suspect there were changes to the TML that are not yet reflected in the parser.