Sharing dbt models across multi-repos

[brandon-segal]

Contact Details

bsegal@spotify.com

I have read the dbt Developer Blog contribution guidelines.

• I have read the dbt Developer Blog contribution guidelines.

Which of these best describes you?

• I am a dbt Community member or partner contributing to the Developer Blog
• I work for dbt Labs and am creating this issue for a community or marketing approved piece.

What is the topic of your post?

Spotify distributes Insights teams across many different parts of the organization, and as a result, insights teams work closely with their product teams to generate datasets to describe their features. As analytics engineering has grown as a practice and more groups adopt dbt, it has become common for squads to share work to create authoritative data sources and informative data products to describe a business area or product suite. Teams have typically shared data by making it discoverable through our data catalog in Backstage. Still, insights teams using dbt can make sharing models across projects using packages even more accessible. This article will share how our group of squads at Spotify expose their external models using packages that allow quick use of those models in child projects while communicating data contracts for transparent versioning of those models using commitizen.

Link to an initial outline.

https://docs.google.com/document/d/1mblB1pq8poBcpnB5ommzSIMMaT_YlAhuTG0MqYQAcNQ/edit?usp=sharing

[KiraFuruichi]

Hey @brandon-segal, sorry for the delayed response. My name is Kira and I’m a technical writer here at dbt Labs and one of the folks who helps manage the Developer Blog. First off, thank you for opening up this issue (and the other one)! We’re always excited to learn how the Community is tackling real issues using dbt.

The timing of this issue could not be more fortuitous! The dbt Core team is currently actively thinking about what the future of multi project deployments should look like in dbt. Would you and your team be open to chat with @jtcohen6, the Core PM, and @dbeatty10, the Core Developer Experience Advocate, on their thinking and your team’s thinking around multi projects? If so, I’m happy to schedule up some time for all of us.

Since multi-project deployments is something the team is actively thinking and working on, it makes sense for this blog post idea to be posted around/after dbt comes out with recommended implementation patterns for these types of problems.
In the meantime, we’d love to work with you on getting the Type 2 Dimensions with snapshots article issue you opened up and live on the Developer Blog! I’ll comment on that issue for next steps there 😄

If you are open for a chat with the team (Jeremy and Doug) around yours and their thinking with multi-project deployments, please let me know and I can schedule something up!

[brandon-segal]

Hey Kira, finally got a second to come back to this! Let's schedule some time for early December to go over our thoughts around multi-repo projects. Would love to see what your team thinks about this kind of workflow and what your team is proposing in your design!

[KiraFuruichi]

Awesome, sounds great! Please let me know what your availability is for the first two weeks in December and I'll send out invites from our end :)

[runleonarun]

Hey @brandon-segal I am going to close this out, but please re-open if you want to draft a blog post on this topic.