diff --git a/podcast/2.7-2.md b/podcast/2.7-2.md new file mode 100644 index 00000000000..93ce6b01d70 --- /dev/null +++ b/podcast/2.7-2.md @@ -0,0 +1,435 @@ +--- +title: 'Decomposing a massive Rails monolith with Kirsten Westeinde, software development manager, Shopify' +publishDate: 2021-08-17T10:01-07:00 +tags: [podcast] +slug: refactoring-shopify-codebase-kirsten-westeinde +published: true +--- + + + + + + + + + +Kirsten Westeinde, Beyang Liu + + + + + +What's it like to deconstruct one of the largest Rails codebases (3 million lines of code, 500,000+ lifetime commits, 40,000 files) on the planet? And why didn't Shopify follow the standard path to microservices, but instead choose to modularize their monolith? + +In this episode, [Kirsten Westeinde](https://www.kirstenwesteinde.com/), software development manager at Shopify, describes how her team led the charge in refactoring and re-architecting Shopify's massive codebase, sharing the winding path they took to make this massive change and the way they tackled both the technical and human sides of this challenge. + + + + + +Ruby on Rails: https://rubyonrails.org/ + +React: https://reactjs.org/ + +React Native: https://reactnative.dev/ + +Go: https://golang.org/ + +Python: https://www.python.org/ + +Scala: https://www.scala-lang.org/ + +Kirsten Westeinde: https://www.kirstenwesteinde.com/ + +(Blog post) Deconstructing the Monolith: Designing Software that Maximizes Developer Productivity by Kirsten Westeinde: https://shopify.engineering/deconstructing-monolith-designing-software-maximizes-developer-productivity + +(2019 Shopify Unite conference talk) Deconstructing the Monolith: https://www.youtube.com/watch?v=ISYKx8sa53g + +(Blog post) Under Deconstruction: The State of Shopify’s Monolith by Philip +Müller: https://shopify.engineering/shopify-monolith + +Wedge: Deprecated internal Shopify tool that was built to score engineering teams on how well they are doing on their componentization journey. + +Packwerk: https://github.com/Shopify/packwerk + +Shipit presents Packwerk: https://www.youtube.com/watch?v=olEA157z7kU + +(Blog post) Enforcing Modularity in Rails App with Packwerk by Maple Ong: https://shopify.engineering/enforcing-modularity-rails-apps-packwerk + +Zeitwerk: https://github.com/fxn/zeitwerk + +Martin Fowler’s design stamina hypothesis: https://martinfowler.com/bliki/DesignStaminaHypothesis.html + +Shopify Fulfillment Network (SFN): https://www.shopify.com/fulfillment + +Kafka: https://kafka.apache.org/ + + + + + +**Beyang Liu:** +All right. I'm here with Kirsten Westeinde. Kirsten is a software development manager at Shopify where I understand, Kirsten, you've been involved in a years-long effort to decompose Shopify's massive Rails monolith, or perhaps like multiple monoliths now, into smaller, modular components. Is that right? + +**Kirsten Westeinde:** +That's right. + +**Beyang Liu:** +Cool. Thanks so much for taking the time. Sounds like a massive effort and I'll bet that Shopify engineering has been very busy over the past year. So I really appreciate you taking the time to come chat with us and share things that you've learned along the way. + +**Kirsten Westeinde:** +Absolutely. Thanks for having me. + +**Beyang Liu:** +So I think the focal point of our conversation is really going to be this massive decomposition effort. It's going to be very fascinating to hear about because I think a lot of engineering teams try to undertake projects like this, especially as the company grows and some succeed, but a lot end in failure. And so I think you have a lot to share in terms of how you can pull this off successfully. But before we dive into all of that, we have kind of a tradition on the podcast of first asking people how they got into programming. Right now you're this software development manager who's in charge of this very complex project. You have a lot of responsibility, but at some point you had your first contact with code. So, for you, what was that? And how did that start your journey into this world of bits and ones and zeros that we live in? + +**Kirsten Westeinde:** +Absolutely. Yeah. If you had asked me when I was a kid, I definitely would not have thought that this would have been where I ended up. I actually didn't even really know what programming was before I went into university, but I did end up studying engineering just because of my love for math and science. And the school that I went to ended up doing a general first year where we basically took classes from all of the different engineering tracks to kind of learn a bit about them and decide where we wanted to go. And so in my first year I took a programming class, we learned C++ and it turns out that I really enjoyed it and I was really good at it. And the more that I looked into it, it seemed like a really interesting field to be in. So I had originally thought I would be a civil engineer, but I'm very glad that I ended up going the direction I did. + +**Beyang Liu:** +C++—that's kind of like diving off the deep end as your first programming language. + +**Kirsten Westeinde:** +Yeah. It's always funny to hear what people learn in school. It's rarely what we end up using in the field I find. + +**Beyang Liu:** +Okay, cool. So from your start in university, what was kind of like the rough journey from there to joining Shopify? + +**Kirsten Westeinde:** +I actually joined Shopify as an intern while I was still in school. So I did two internships with other companies, but I joined Shopify in 2013 out of the Ottawa office as an intern. And then I ended up doing three different internships with Shopify before joining full-time after I graduated. And through those internships, I really used the time to try out different things and try to understand where my interests lay. So I did some web development using Rails, I did some mobile development, and I worked on some internal tools and external products. So I've really had my hands in a lot of different places at Shopify over the years. + +**Beyang Liu:** +Awesome. So you've been there since 2013, which means you've kind of had a front-row seat to this whole rocket ship trajectory that the company has been on. + +**Kirsten Westeinde:** +Yeah, it's been crazy. When I joined, I think we had 150 people working for the company and we're a private company. We had only a handful of merchants. We had one small office and fast forward to today, we're an international company. We have... It's hard to keep track of it, but I think like 7,000+ employees. At this time, we're a public company, so much has changed. I like to tell people I haven't worked at the same company for eight years, I've actually worked at like eight different companies because it really has been changing that quickly. + +**Beyang Liu:** +Yeah, that's crazy. And you know, I bet most of our listeners don't need any introduction to Shopify, but you know, for the one or two people who've been living under a rock for the past seven years. Chances are... If you bought anything on the internet, chances are you use Shopify, right? Because you power a lot of the kind of small businesses and independent operators that sell things online right? + +**Kirsten Westeinde:** +Yeah, that's correct. We actually don't force the Shopify brand on our merchant sites. So a lot of people buy things from Shopify stores without actually knowing that they have. But basically if you bought something from Amazon and it was enjoyable or—sorry—from anything other than Amazon and it was an enjoyable experience, that was probably a Shopify store. + +**Beyang Liu:** +Yeah. And actually for me personally, increasingly I find myself buying more and more things off Amazon because there's like... A lot of times the things that you can get in these kinds of independent stores are just like higher quality or it just feels closer to the seller. And so thank you for enabling all that. So diving into this large-scale refactoring and re-architecture project, I thought maybe we could start off by giving people an overview of what Shopify looks like as an engineering organization and as a codebase. So in broad strokes, what are the major teams and what are the major parts of the code, and what does it all kind of do as an engineering system? + +**Kirsten Westeinde:** +Definitely. So we talk about the core Shopify product as the core codebase. And so that's basically what powers merchants to manage their orders, to manage shipping, etc. So all of the core pieces of the product live in one Rails codebase. And it's actually one of the largest, probably the largest Rails codebase on the planet because we got started with Rails like right when Rails became a thing, and it's been under continuous development since at least 2006. And the last time I checked, we were pushing like 3 million lines of code, more than 500,000 lifetime commits and about 40,000 files. So it's massive. And in terms of an engineering org, I think we're at about 1,500 or so engineers and we're actually aiming to hire another 2021 this year, so it'll be continuing on this massive pace of growth. And if I had to guess, I'd guess probably about half of the engineering team works daily in the core codebase and the other half works on applications outside of core. + +**Beyang Liu:** +That's crazy, that's so much code. Does it... Have you gotten to the point where Git operations start to slow down at all? Like if you were to clone the entire codebase to your local machine... Like do people even do that? Or how long does that take? + +**Kirsten Westeinde:** +Yeah, people do it, it works. The test suite definitely can get slow, but development itself is not a problem. We actually... We always joke, like we host Kylie Jenner's shop on Shopify and it's like one of the largest stores. And we joke that like we are the Kylie to so many other systems that we use. So we're hosted on Google Cloud and like we are by far the largest product hosted on Google Cloud and so we're often stretching the boundaries of their systems. And similarly, like with Rails itself, we are often the ones that find those bugs that you only find when you're operating at this scale. And so what we do is we really make choices about what technologies we want to use and then really double down on them. So we actually have a full team at Shopify who are Rails core contributors, and we try to actually just evolve the tools that we use to be better and to be better able to support us and others. + +**Beyang Liu:** +What languages are in use? So with Rails, I assume Ruby is a big language, are there any others that are big parts of the stack? + +**Kirsten Westeinde:** +Yeah. Shopify has always been very opinionated about technology choices. So essentially any problem that can be solved with Ruby on Rails should be, there are some exceptions. If applications have really, really high performance constraints or need parallelization, sometimes we use Go, the front ends are in React. And for mobile development, we've just standardized around React Native going forward as well. And then in data land, we have some Python and Scala and in infrastructure we have Go and Lua as well. + +**Beyang Liu:** +So you wrote this blog post about how you were in charge of this project to kind of decompose Shopify’s main Ruby on Rails monolith into smaller, more modular components, not microservices, but different modularized components, we'll get into that later. Tell us at a high level, what was the mandate of that project and how did you come to be involved in that. + +**Kirsten Westeinde:** +Yeah. So in, I think it was around early 2017, it was becoming very clear that our core codebase was reaching a tipping point. Where it was becoming more and more challenging to build functionality into it because of all of the interdependencies. So projects were taking longer, adding things that used to be simple were causing test failures and big headaches. And also it was really... We're finding it really challenging to onboard new people into the codebase because there was just so much to learn. And so we knew we had all of these problems, but we didn't want to get into solutioning without really understanding what the problems were. So we actually sent out a large survey to all the engineers working in the core codebase to hear their feedback about what the pain points were before deciding what to do. + +**Kirsten Westeinde:** +And it was that survey that actually led us to start this project. We originally called it a very clever name of, "Break core up into multiple pieces." But that name eventually evolved into componentization where essentially we wanted to make this one large codebase feel like many smaller codebases. And so we wanted to separate the code by business domain basically because that is the code that more often needs to change together. And so what that would mean is if we were hiring someone onto say the shipping team, they should be able to just understand the shipping component without having to understand all of the other elements of the core codebase. So at a high level, that was the goal. + +**Beyang Liu:** +Got it. So if I were a buzzword-driven engineering manager and you came to me and said, "Let's decompose this monolith into different independent components." I would say, "Well, that's obvious, you know, we live in 2021 and the obvious solution to decomposing a monolith is to break it down into these things called microservices. But I understand that that's not the approach that you ended up taking. So can you talk about why you didn't follow the naive path of like, let's write a bunch of microservices. What did you end up with instead? What is this modular monolith that you talk about? + +**Kirsten Westeinde:** +Yeah. I mean, I think for one thing we had been developing on Shopify core for so long that pausing everything and building out net new microservices just wasn't feasible for us. Whereas the direction we ended up going was making incremental changes in the existing codebase while teams continued to develop and add features to it. So that was a big reason, but the other thing is that unless we wanted to start from scratch, to be able to start extracting some of these pieces out into microservices, we would need to have them well encapsulated and have boundaries in place, which is what we ended up doing. But we ended up deciding that we wanted to stop there because monoliths have a lot of pros, a lot of benefits. + +**Kirsten Westeinde:** +Having only one codebase means you only have to do all of your gem bumps and your security fixes in one place. It also means you only have to manage one deployment infrastructure and one test pipeline. There's also a lot less surface area in terms of places that you could be attacked, and I think that makes the system a lot more resilient. And we found it helpful to keep all of our data in one place, in one database and allow the different components to have access to the data that they needed instead of trying to synchronize it across many different systems. + +**Beyang Liu:** +Yeah. + +**Kirsten Westeinde:** +And lastly, with microservices, the communication between the different components has to happen over the network, which introduces a whole bunch of complexity around API version management and backwards compatibility, as well as just dealing with the network in general. It can be slow. There's lots of ways it can fail. So those were all the things that we liked about the monolith and didn't want to just give those all up. And we kind of felt like moving to microservices was more of a topology change whereas what we wanted was an architecture change. There was nothing necessarily wrong with our monolith, it was just that there were no well-defined boundaries between different things. + +**Beyang Liu:** +So you had this principled, thoughtful approach of "We have developer pain, let's address that directly, but we're not going to involve it with changing the topology of production, introducing all these unnecessary network boundaries because that introduces its own complexity and its own baggage" in a way. + +**Kirsten Westeinde:** +Yeah, exactly. Our monolith had gotten us so far. We, I think, had learned so much about what is good about it, that it gave us pause to really ask the question and really think through whether microservices were right for us. And the answer at the time was no. + +**Kirsten Westeinde:** +That being said, we are believers of service-oriented architecture, so we do have other applications outside of the core codebase. However, everything related to the core functionalities of commerce, we keep in one place. But there have been some examples where we actually have extracted components out of the core codebase because they weren't related to the core functionality of commerce and we pulled them out into their own services, but that was exponentially easier to do post componentization. + +**Beyang Liu:** +So what did this modularization of the monolith actually entail? I assume you started with... My knowledge of Ruby On Rails is fairly limited, but I understand the standard directly out is: you have a model directory, you have a view directory, you have a controller directory. Was that the structure of the codebase at the start of the project? + +**Kirsten Westeinde:** +Yeah, that's correct. So we started before this, we basically just had a Rails app out of the box, so you're exactly right. They were organized by our software concepts, but the first thing that we wanted to do was restructure our codebase to be organized by real-world concepts like orders, shipping, inventory, billing. And what we ended up doing was we actually audited every single Ruby class in the code base, which was like 6,000 at the time, in a spreadsheet and manually labeled which component we thought that they belonged in based on the initial list of components we had. And then we built up one massive PR with automated scripts from the spreadsheet to basically just create those top-level folders. And then within the orders folder it would still be models, views, controllers, but we would just move all of the specific models, views and controls into that folder. So we did that all in one big PR, which was kind of terrifying, but ended up being fine. We have good test coverage. So it is all good. + +**Beyang Liu:** +What was the size of that diff? It must have been like... + +**Kirsten Westeinde:** +It was every file, every file was touched. + +**Beyang Liu:** +Wow. + +**Kirsten Westeinde:** +Yeah. So that was the first step. + +**Beyang Liu:** +Why wasn't that sufficient? The straw man is like, "Okay, if we put everything in its own directories and now each team gets to work in their own directory and that's done. They're like separate codebases." Why was that not just the end of the project? + +**Kirsten Westeinde:** +Well, all that had achieved was moving files around, right? Like in Ruby On Rails, everything is automatically globally accessible. So just because all the code for orders was within an orders folder, it could still use whatever it wanted from any of the other folders. So we still had all the same problems with fragility of the test suite and really if I was working in the orders component, I still had to understand about the taxes component and the shipping component because of how tightly coupled they were and how there was no clearly defined API for them to communicate between the folders. Nothing had structurally changed about how the code was interacting. + +**Kirsten Westeinde:** +So it was a great first step in the sense that I now knew if I'm looking for something orders related, I can find it. And that the top-level component folders had explicit owners, which was actually a huge win. But beyond that, we still hadn't solved a lot of the existing problems. + +**Beyang Liu:** +You mentioned something in your blog post about how one of the initial pain points was as the codebase grew, the amount of code that a given developer would have to understand to get a thing done was like something like O of N, it grew linearly with the size of the codebase. And that obviously is untenable as you grow, because it just means your productivity approaches asymptotically down to zero. So how'd you go from O of N to sub linear? What were the things that you had to make in terms of codebase structure to enable that? + +**Kirsten Westeinde:** +Yeah, definitely. So once we moved all the code into their separate folders, the next ask was for each component to define a public API. And ideally that public API should be easy enough to understand that I can understand what it's doing without having to go under the hood and look into the component. And so that would allow me to get to know one component really well. We also asked each component to explicitly say which other components they depended on. So say I was joining the orders team, I could really deeply get to know the orders component. And then as my next step, I could look at what the dependency is listed out for the orders component is, and maybe just start getting to know their public API so I know what methods are available to me from those components without needing to understand all the nuances of how they get the things done exposing the public API. + +**Beyang Liu:** +Yeah. What about the human side of this kind of change? In the organizations that we've worked with that have attempted something like this, there's the technical side, which involves making the proper changes to the code, in this case, introducing a bunch of interfaces and making those interface boundaries well-defined, but then there's also this human, semi-political aspect to it where you're going to each of these engineering teams and each of them has their own goals and objectives that they're working toward, business goals, and you're coming to them and essentially asking them to add work onto their plate to satisfy this change. + +**Beyang Liu:** +Can you talk about that element of it? Did you have to do a lot of horse trading or, "Do me a favor and I'll do you a favor?" What does that look like from your perspective? + +**Kirsten Westeinde:** +Yeah, it's interesting. I've actually talked about this project at a few different conferences and the two questions I always get are, "How did you get buy-in from the business to invest in this? And how did you get buy-in from the individual teams to invest in this?" I guess there's no really easy answer or perfect way to do it and it's going to be so nuanced depending on the organization that you're within. But one of Shopify's core cultural values that were kind of taught from the beginning, is build for the long term. And it's really driven home that we're trying to build this business for a hundred years, not for 10 years. And so with that kind of mindset, it is easier to get buy-in for these short-term pain for long-term gain type projects. + +**Kirsten Westeinde:** +I think as well, the developers that had been working in the core codebase had all been feeling this pain. And so they were eager for opportunities to make it better and easier to develop within. So it wasn't always easy. And to be honest with you, different components are at completely different points in their componentization journey based on how well their team has been able to prioritize this type of work. One thing that we did was we built this tool called Wedge, a lot of complexity under the hood, but it basically gave components scores as to how well they were doing on their componentization journey. And we made it like a little fun competition between teams. + +**Beyang Liu:** +You gamified it. + +**Kirsten Westeinde:** +Exactly. Yeah. As a technical initiative that the whole engineering org was working towards. + +**Beyang Liu:** +That's really interesting. So you kind of built a tool that condensed all these complex refactorings into something that you can measure, like a numerical score and that made it easier to track and also verify. How did you do that? How do you convert adding an interface or defining a good interface or eliminating a cyclical dependency or something into a number that is meaningful that the engineering teams don't shake their heads at and that also is useful for tracking it at a top level. + +**Kirsten Westeinde:** +Yeah. So, Wedge is a really interesting tool. What we did was we used it to hook into Ruby trace points during our test suite run to get a full call graph of all the calls that were being made as our code was being executed. And then we sorted the callers and callees by component and basically pulled out all of the calls that were happening across component boundaries and then sent them to Wedge. And we'd also send some other data like code analysis and active record associations and inheritance and such. And then Wedge basically just went through everything that was sent to it and determined which of these things were okay, and which ones were violating. So if a cross-component call was being made, if there was no dependency explicitly declared between those two components, it would be violating. And if it was anything other than the public interface it would be violating. + +**Kirsten Westeinde:** +We actually found though that it really is hard to get a number that is right. Especially with as much complexity as there is in call graph logging. So in the end we actually ended up canning Wedge because we didn't find it to be a useful feedback cycle. And also mostly because we had to run the full test suite to get this feedback and that takes way too long for it to be helpful. So we ended up going in a different direction. But in the early days it was really helpful to, like you said, gamify it. + +**Beyang Liu:** +Yeah. That makes a lot of sense. As you learn more about what was useful and productive, what kind of stepped into the place that Wedge occupied? Was there another tool that you used or was there some different set of criteria that you used to gauge overall progress and success? + +**Kirsten Westeinde:** +Yeah. We ended up building this tool called Packwerk, which actually as of September of last year is open source. And it's a really cool tool that basically analyzes static constant references. So we found that there's less ambiguity in static references and because these references are always explicitly introduced by developers, it's more actionable to highlight them, and it's much faster. So we're able to run a full analysis on our largest codebase in a few minutes, which means that we can actually put it in as part of our pull request workflow, which is a way more helpful feedback loop for developers. + +**Beyang Liu:** +That's really interesting. So the Wedge was this dynamic tool. It kind of built up this call graph or reference graph from runtime by observing what actually got called and then Packwerk was building up that graph, but statically based on imports and references in source code. Is that right? + +**Kirsten Westeinde:** +Yeah, that's correct. Packwerk uses the same assumptions as Zeitwerk, which is the Rails code loader. And the way that Rails is able to make constants globally accessible is it basically infers the file location based on the constant's name. And we did the exact same thing with Packwerk. + +**Beyang Liu:** +So the programming languages part of my brain is saying, "But wait a minute, Ruby is a dynamically typed language, not all types are known at compile time." But were you able to solve that problem in Packwerk? Like inferring types or was it some element of best effort? Like, this is good enough. + +**Kirsten Westeinde:** +Yeah. A hundred percent the latter. We understand that it's an imperfect solution and there's a lot of crazy stuff that can happen in Rails that will not be detected through this, but it's good enough essentially. And it does catch a lot of things. So we decided that the fact that it can happen so much faster and provide that faster feedback loop is a trade-off we're willing to accept for the fact that yes, maybe some things will slide under the radar. + +**Beyang Liu:** +I guess it's important to recognize that there's humans in the loop in this process. So in some sense, it'd be a much harder problem to completely automate it because then you'd have to get everything precise. But as long as there's humans in the loop, they can step in and say, "Well, we recognize this tool's a little bit fuzzy, as long as the signal to noise ratio is decently high, we can work with this." + +**Kirsten Westeinde:** +Absolutely, and the magic of this tool is not only within the tool itself, but it's within the conversations that it sparks, because if on my PR I get told, this is a cross-component violation, and maybe I don't know what that is, or maybe I'm a more junior programmer, and I don't have an idea of how to do inversion of control, for example, there's been a lot more conversations around software architecture across Shopify because of this tool. + +**Kirsten Westeinde:** +So, what that means is sometimes maybe the tool won't catch cross-component boundaries, but because it's now becoming a part of our engineering culture, maybe someone will in a PR review. Building that muscle of good software design really is the overarching goal of this whole thing. So, yes, the tool's not perfect, but it's been really valuable for us. + +**Beyang Liu:** +You just mentioned inversion of control, and I think that along with some other things you called out in your blog post were good rules of thumb or good general principles that you may have discovered along the way of doing this. Can you talk about some of those principles or maybe empirical patterns, in terms of if someone were to attempt this on a Rails codebase in general, what are the maybe tactical things? Tactical things, tricks that you can apply to make the codebase more decomposed and modular. + +**Kirsten Westeinde:** +Yeah. I think the first thing I would say is that if someone's going to be trying this on a Rails codebase, they'll probably be in a similar situation that we are, or that we were, which is basically, if you look at the dependency graph that has just naturally happened, it's crazy. Every component was basically depending on every other component. There's a ton of cyclic dependency. Rails lends itself naturally to have high dependency. + +**Kirsten Westeinde:** +So, I'll say that first. Even just being able to visualize that dependency graph allows you to reason at a much higher level about, does it make sense for this thing to depend on this thing? Sometimes it does, and other times it doesn't, but when it doesn't, that's when you start to use some of these tactical approaches for decoupling those things. + +**Kirsten Westeinde:** +I mentioned inversion of control, which basically is saying ... So, I'll give an example. I work on a product called the Shopify Fulfillment Network, where, basically, some merchants, we host their products on their behalf. When orders come in, we'll choose the most optimal warehouse to fulfill it out of to get it to the buyer as quickly as possible. + +**Kirsten Westeinde:** +SFN, that's the acronym for the Shopify Fulfillment Network. It needs to know when an order comes in on a merchant store. One way to do that could be for the orders component to make a call to a method in SFN and say, "New order has been created," and then SFN responds with, like, "Great. We got it." + +**Kirsten Westeinde:** +That's a hard dependency from the orders component to SFN, whereas if we want to get rid of that dependency, we can actually just use a method like a publish subscribe mechanism. There's a bunch of different ones to do that, but at Shopify, we use Kafka events a lot. Basically, the orders component can just say, like, "An order happened," and then SFN can choose to subscribe to that and do its processing without needing to necessarily reply, and that has now broken that dependency between the two. + +**Beyang Liu:** +It's like, at run time, you want this one thing to provide behavior to this other thing that is going to run, but at compile time, you don't actually want an import going from that thing to this other thing, because that adds a hard dependency, and it means that someone working on the first thing will have to go through the second thing and understand how it behaves. There's not this interface boundary that says, "Okay, don't worry about the stuff behind this curtain." + +**Kirsten Westeinde:** +Yeah. That's exactly right, and had there been a hard dependency, like, say, from the orders component to SFN, then the orders component would have to care every time SFN changed. Now, having done this inversion of control, it does not at all. It just has to fire its event. + +**Beyang Liu:** +Another thing that you talked about in the post was this kind of loose coupling, but high cohesion pattern that got adopted. Do you mind explaining what that means? + +**Kirsten Westeinde:** +Absolutely, yeah. This was one of the core principles of this project that we were striving towards. Coupling is basically the level of dependency between modules, or classes, or in this case, components. So, you want that to be low, because you want your dependency graph to be as light as possible, but you want the cohesion to be high. The cohesion is basically the measure of how much elements within a certain encapsulation belong together. So, whether you're looking at a module level, a class level, or the package level. + +**Kirsten Westeinde:** +One thing to note is that it's really hard to get the component interface right when the classes within it have interfaces that are not well-defined, and the modules within it. It really starts with good software design from the bottom up. That makes it easier, a lot easier to get the component interfaces right. + +**Kirsten Westeinde:** +Then, on the point of cohesion, there’s two different types of cohesion that we think about. The first is functional cohesion, so that basically means code that performs the same task list together. You can think of service objects as being functionally cohesive, whereas data or informational cohesion is code that operates on the same object living together. + +**Kirsten Westeinde:** +Again, Rails really lends itself to data or informational cohesion, because of the way that active record models are built up to interact with one database model. We often add a lot of methods relating to that object on its model, even though they might be parts of completing completely different tasks. I'm sure different languages lend themselves to different ones, but we found ourselves way over-skewed in the direction of data and informational cohesion. + +**Beyang Liu:** +Where is this project today? You mentioned it started in 2017, and it's been a multi-years-long effort. Are things still being componentized in the main Rails codebase, or at this point, has it been componentized, and it's more or less in a done state, you just have to maintain the status quo? + +**Kirsten Westeinde:** +Yeah, I wish I could tell you we were in a done state, but the reality is that we're not. All of the code has been moved into components, and all of the components do have publicly defined entry points, but the reality was that we were starting from a place where a lot of these calls were already violating the rules. So, one of the nice features that Packwerk built was that you could essentially declare bankruptcy, and start from a certain point, and only start tracking violations beyond that. We have a list of deprecated references, and the rule is basically that you can't add any new violations. + +**Beyang Liu:** +Got it. + +**Kirsten Westeinde:** +We know that there are some existing ones, but no new ones can be added. Over time, that deprecated references list basically gives us a tech debt to-do list to work off of. Like I said, different components are in different shape, but we haven't yet gotten to the point where all of those violations are gone. Only once all of those violations are gone will we be able to actually enforce those boundaries. + +**Beyang Liu:** +Fascinating. You have this list of tech debt to-do items. In a given iteration cycle, how do you decide how many of those to tackle versus how many, I guess, more product-oriented things to tackle? Because that's something that our teams at Sourcegraph, I assume, like probably any software team struggles with. What is the right balance of tech debt versus new feature development to take on? + +**Kirsten Westeinde:** +Yeah. That's a hard question. + +**Beyang Liu:** +Can you put a price on a tech debt item? It's almost an impossible exercise. + +**Kirsten Westeinde:** +Yeah, no, it's really a hard question. Honestly, the answer's going to be different team to team at Shopify. One of the patterns that I've seen in my years at Shopify is that we built a lot of features. We move really quickly leading up to Black Friday and Cyber Monday, because that's when our merchants make the vast majority of their sales for the year. So, it's really important that we've given them the tools that they need to be successful during that time. + +**Kirsten Westeinde:** +Come the holiday season, we actually tend to cool down a bit on feature development, just because we don't want to be breaking things when our merchants are having their most important sales of the year. So, often, that feature cooldown can be a good time to tackle some of these larger units of tech debt. That said, I'm not saying that we just leave tech debt to do at one point in the year. We definitely try to find some pause between projects, or even just parallel tracks of work, where some people are working on more technical debt, and some people are working on product features. It's definitely a hard balance, but we have been slowly chipping away at it. + +**Beyang Liu:** +I understand that the Shopify Fulfillment Network, which you mentioned earlier, that's a separate codebase. It also is or was a Rails monolith. You're currently tackling a similar project in that codebase, but adjusting some things based on what you've learned from this first major... + +**Kirsten Westeinde:** +Decomposition. + +**Beyang Liu:** +... decomposition componentization effort. + +**Kirsten Westeinde:** +Exactly. + +**Beyang Liu:** +Tell me about round two. What's different this time in the sequel? + +**Kirsten Westeinde:** +Yeah. We actually took a pretty different philosophical approach. What we did in Core was we moved all of the code into the components, and then slowly, over time, started trying to break some of those dependencies, whereas what we've done in the Shopify Fulfillment Network was we've introduced separate components. We've actually used Rails engines, which are like the one modularity feature that comes out of the box with Rails, and that can allow us to enforce those boundaries a little bit more strictly. Then, over time, piece by piece, moved bits of code into it that are always respecting the boundaries. + +**Kirsten Westeinde:** +So, we flipped it on its head that way, whereas we still do have the main app controllers, models, etc., but the hard rule is we can't add any more code into the app folder. All new code goes into component folders. Over time, we're pulling code from the app folder out into its, basically, correct component, so that anything that's in a component is respecting the component boundaries, whereas that is not the case in Core. + +**Beyang Liu:** +Interesting, so, with Core, the approach was move all the files around into their appropriate directories, and then, over time, reduce the linkedness or monolithic-ness of each component, whereas with Fulfillment, you basically said, keep the existing monolith as is, but each new feature has to be added to something that's external in a Rails engine. Then, we'll have some connection between the two, so that the functionality makes it into the application, but we always preserve this, I guess, invariance because all of the new components are modularized. Is that about right? + +**Kirsten Westeinde:** +Yeah, that's right. One of the main benefits of that is that with so many developers working in a codebase, people tend to want to follow the patterns that they see existing. In Core, there is a mix of good patterns and bad patterns, because some stuff has been adjusted to break that interdependency, and some stuff hasn't. So, it's hard to know which pattern to follow, which way is the right way, when there are both examples, whereas in SFN, anything that has been componentized is following the patterns that we're wanting to strive for, so there's more good examples to be able to learn from, and demonstrate to other developers, and follow. + +**Beyang Liu:** +I get all the learnings here. This is really fantastic insight. One thing I'd love to hear about is like, what is the process on the ground that you arrive at these insights? Like, for this new project, we don't want to move a bunch of stuff in the existing codebase around, but we want to start with something new, or moving from the dynamic dependency call graph to the static version, what are those conversations like? Is it you and a bunch of other engineers or engineering managers in a room, whiteboarding, or someone coming up with a bright idea, and slowly convincing other people? Are there any, I guess, war stories that come to mind when you reach these insights? + +**Kirsten Westeinde:** +Yeah. It's always a conclusion that we come to over time. In the example of Wedge, it was working well for a while, but over time, the problems with it became louder, and louder, and louder until we couldn't ignore them. So with that one, it was really, as our test suite was getting larger and slower, it was really just that it wasn't a helpful feedback loop to have to wait for that entire test suite to run to be able to get that insight. And so we knew there was a problem with that. But then the question is, what's the solution? + +**Kirsten Westeinde:** +So we have a few different avenues for brainstorming these ideas. One is actually a team whose mandate is architecture patterns, and they're the ones that built Packwerk and provided a lot of the patterns to follow. And so sometimes it's just projects that get resourced, and we do explore and prototype phases. And so being armed with the learnings from Wedge, it led us down the road to Packwerk. + +**Kirsten Westeinde:** +But the other thing is we have a software architecture guild at Shopify, which is basically anyone from the company who is interested in software architecture. We have a Slack channel and a bi-weekly meetup. And that's where a lot of these conversations come to a head. Because I think there were a lot of learnings from Core, but then for example, the developers who were starting to build out SFN may not have had access to that frontline information. + +**Kirsten Westeinde:** +And so we did a lot of pairing actually as a way to share this knowledge and have a lot of discussions with the architecture guild. Because Shopify is very, I guess each team has a lot of autonomy. So the SFN team had the autonomy to decide what the right solution for it was, but definitely wanted to leverage the learnings that were present from Core. So that's the type of discussion that we would have at an architecture guild. Or we also do technical design reviews for any big tech projects we're kicking off. So that would be another. We have templates for those, and we have filled those templates with some prompting and questions to try to make people think about modularization as part of the upfront design. + +**Beyang Liu:** +Interesting. Tell me more about this guild. So what is a guild exactly? Is it just a collection of individuals who have a common technical interest area? Or how is it organized? Is there a process associated with it? How does it fit into the overall org structure of the engineering team? + +**Kirsten Westeinde:** +Yeah. Actually it doesn't fit into the org structure of the engineering team. It's composed of people from across everywhere in the engineering organization. It's completely opt-in, and it grew organically from people who were curious to be having conversations like this. I think one big learning actually that my coworker Phillip called out in his blog post was that he wished that we had started this sooner in the componentization process. So more minds could be involved and had buy-in basically for them to apply the approach that we were aligning upon in their different areas of the company. + +**Kirsten Westeinde:** +But it really is just people that want to nerd out about software architecture, and people come with presentations every other week. Sometimes we'll just share the architecture of one of our systems. Sometimes we'll have more pointed discussions about certain software design strategies. It's pretty organic, but people like it. + +**Beyang Liu:** +That's awesome. So the guild formed after the start of the original project. + +**Kirsten Westeinde:** +Yeah. That's right. + +**Beyang Liu:** +At what stage do you think it makes sense to form a guild? I think that we run into the same issue, and I think a lot of other engineering organizations run into this issue, where you have your org chart and you try your best to produce an org chart that reflects the needs of the product and how knowledge and information needs to flow. But no chart is ever going to be perfect, and there's all these areas of expertise that don't neatly fit into that. And I guess the question in my mind has always been, at what point do you start to need these guild structures? Is it at 100 engineers? Is it at 1,000 engineers? Or what, in your mind, is the right size? + +**Kirsten Westeinde:** +Yeah. We've actually experimented with a few different things. The guild is more of a meetup. We less drive change through the guild, I would say, other than grassroots movements. But we also have what we call a technical leadership team, which is on a rotational basis. Technical leaders from across the company will be on that team. And anytime we're doing a tech design review, you'll get paired up with someone from the technical leadership team. You can bring them your gnarliest problems, and it's made up of people from all different parts of the org with all different perspectives on the problem. + +**Kirsten Westeinde:** +So it's a way to try to, I guess, make sure that we're leveraging the learnings that are available at the company level. And then there's also what we were chatting about before, I forget what it's... the tools and patterns team. And their actual mandate and what they work on day in and day out is building out some of these tools and patterns. + +**Kirsten Westeinde:** +And I think I would guess that team spun up probably in 2018 just based on this project having started in 2017. And yeah, we were probably 500 to 600 engineers around then. It's going to be different for every organization, and I think it depends on how much your organization already has it baked into their DNA to be thinking about these things, to be thinking about good software design, technical initiatives, etc., or not. Sometimes it might make sense to start earlier if that's not as organically present. + +**Kirsten Westeinde:** +There's trade-offs to each approach. One of the challenges that the tools and patterns team has is that they're just building tools and patterns, but they aren't necessarily using those tools and patterns to build a product. So they need to make sure that they are in touch with the people that are using those tools and patterns and actually solving the right problems. And so we do that through pairing and internal documentation presentations, etc. But we really try to make sure that the problems that need to be solved are being solved, and we're not just adding friction to developers' lives because we think it's nicer. + +**Beyang Liu:** +Yeah. That makes a ton of sense. Looking back on both projects that you've been involved in with decomposition, to what extent are the patterns that you discovered, do you think, specific to those particular codebases? And to what extent do you feel like there's some general principles here that would be widely applicable? That's one question. A more proximal question would be the way that you're approaching modularization on the fulfillment network codebase. Do you wish that you had done that with the main codebase initially, knowing what you know now? Or do you think that each codebase found its proper approach? + +**Kirsten Westeinde:** +Yeah. It's a challenging question because they both have their trade-offs. So I think that you just need to know in your situation which trade-offs you're willing to accept. For the Shopify Core case, what we were really struggling with and really wanting to optimize for was that we had a lot of code that didn't have ownership. And so having moved the code into folders that have explicit owners is a big win for that. Whereas in SFN, we still have a lot of code living in the app component that doesn't have as clear owners, but for us that's okay because it's a smaller group of people working on it. And so it's easier to share the load across our team, which is smaller. Whereas for core, it was really important that we get it right in terms of code ownership. + +**Kirsten Westeinde:** +So I think you really have to ask yourself, what is the intermediate state that you're most comfortable with? Because these things take a long time. And so I think Phillip has an amazing quote in his blog post. He says, "My experience tells me that a temporary incomplete state will at least last longer than you expect. So choose an approach based on which intermediate state is most useful for your situation." And it's 100% true in this case. Both of those codebases are in intermediate states that are correct for them, and I think you just need to think about what that means for your codebase. + +**Beyang Liu:** +Yeah. That's very fair. Somewhere out there there's a person listening who wants to propose a big refactoring re-architecture project, but you often run into resistance from Product, from other stakeholders in the business because the value of such a project is not always clear. What advice would you have for people in that position, just in the very early stages of thinking through starting such a project and getting organizational buy-in? + +**Kirsten Westeinde:** +Yeah. One thing that I think about often as it relates to this is, I don't know if you're familiar with Martin Fowler's design stamina hypothesis, but it basically says that in the beginning— + +**Beyang Liu:** +I read that in your post. + +**Kirsten Westeinde:** +Yeah, I love it. I talk about it all the time. It basically says that in the beginning, no design is actually the best design, and you'll be able to move more quickly without design. But at some point you're going to reach a point where not having a design is going to slow you down more and more and more, and adding incremental functionality just gets harder and harder. + +**Kirsten Westeinde:** +So the first question that I would ask is, are we past that point? He calls it the design payoff line. Are we really at a point where we need this design? Because honestly, it's worse to have a bad design than to have no design, in my opinion. So you want to make sure that you have enough information and have built enough features to understand what design your codebase needs. So that's the first thing. + +**Kirsten Westeinde:** +But if you think that you are in that position, I always find that the business and the product managers are very motivated by data. So the more data that you can capture, the better. So I don't know that your organization uses story points, but say you could point to an issue of story point three that used to take half a day. Now it takes a day-and-a-half. Anything that you can point to to show that you are getting slower is a really good starting point. + +**Kirsten Westeinde:** +Often they will have been feeling similar things. It probably won't come out of left field to them. They might feel like projects are taking longer than they should and are harder than they should. So the more you socialize the idea of, "X would be easier if we did this" and really just keep being the squeaky wheel, that can be really helpful. + +**Kirsten Westeinde:** +The other thing too is we would absolutely never have gotten buy-in to stop feature development and do this. So if you can do it in an incremental way that you can chip away at over time, it'll be much more likely that you can get buy-in. And if you're in the lucky situation where your company has enough engineers to be working on parallel tracks of work, then if some people can still be building features while some people do this, it's going to be a lot easier of a pill to swallow for the business. + +**Beyang Liu:** +Yeah. That makes a lot of sense. Kirsten, thanks so much for taking the time today. If there's people listening to this and they're interested in learning more about this effort or more about projects and learnings that you've undertaken, where should they go on the internet to find out more and learn more? + +**Kirsten Westeinde:** +Well, Shopify has an engineering blog where we've published a few different blog posts on this topic. So I would definitely start with checking those out. We also have a ShipIt Presents new YouTube series about some of the engineering efforts at Shopify, and there's an episode that speaks about this as well. So check those out. And there's lots of other interesting non-componentization-related stuff on the shelf for the engineering blog. So I would say give that a look. + +**Beyang Liu:** +All right. My guest today has been Kirsten Westeinde. Kirsten, thanks for being on the show. + +**Kirsten Westeinde:** +Thanks for having me. It was fun. + + diff --git a/podcast/3.1.md b/podcast/3.1.md index 93ce6b01d70..af082578032 100644 --- a/podcast/3.1.md +++ b/podcast/3.1.md @@ -1,435 +1,574 @@ --- -title: 'Decomposing a massive Rails monolith with Kirsten Westeinde, software development manager, Shopify' -publishDate: 2021-08-17T10:01-07:00 +title: 'Max Howell, creator of Homebrew and founder of tea' +publishDate: 2022-05-18T07:00-07:00 tags: [podcast] -slug: refactoring-shopify-codebase-kirsten-westeinde +slug: max-howell published: true --- - - - - - - -Kirsten Westeinde, Beyang Liu +Max Howell, Beyang Liu -What's it like to deconstruct one of the largest Rails codebases (3 million lines of code, 500,000+ lifetime commits, 40,000 files) on the planet? And why didn't Shopify follow the standard path to microservices, but instead choose to modularize their monolith? +Beyang talks with Max Howell, creator of Homebrew, about his new package manager, tea, which aims to solve the problem of open-source funding. + +Max shares his beginnings in programming and what led him to work on early music players in Linux, Last.fm, and eventually get into Mac development. Max discusses the frustrations he experienced in cross-platform development that were the impetus for the creation of Homebrew and explains how Homebrew became the de facto package manager for macOS. + +Max talks about his latest project, tea, a successor to Homebrew that aims to solve the open-source funding problem with a decentralized protocol that uses NFTs and an understanding of the package dependency graph to distribute funding to open-source maintainers. -In this episode, [Kirsten Westeinde](https://www.kirstenwesteinde.com/), software development manager at Shopify, describes how her team led the charge in refactoring and re-architecting Shopify's massive codebase, sharing the winding path they took to make this massive change and the way they tackled both the technical and human sides of this challenge. +
+ +
+ +
+ Subscribe: + YouTube + Apple + Spotify + Google + RSS +
-Ruby on Rails: https://rubyonrails.org/ +Beyang and Max discuss Max’s early years in programming experimenting with the BBC Micro and BBC BASIC (1:55-9:25) -React: https://reactjs.org/ +Max talks about his experience helping to create Amarok, an early open-source music player on Linux (9:35-13:42) -React Native: https://reactnative.dev/ +Max explains how he transitioned from open source to programming for pay by getting hired at last.fm (13:47-16:52) -Go: https://golang.org/ +Max discusses how his work on Last.fm apps led him into Mac development (16:54-20:13) -Python: https://www.python.org/ +Beyang and Max discuss user experience and Max’s work in the macOS ecosystem (20:15-24:55) -Scala: https://www.scala-lang.org/ +Max discusses the frustrations he experienced in cross-platform development that led him to start Homebrew (27:15-33:59) -Kirsten Westeinde: https://www.kirstenwesteinde.com/ +Max explains how Homebrew became the de facto package manager for macOS (33:30-41:00) -(Blog post) Deconstructing the Monolith: Designing Software that Maximizes Developer Productivity by Kirsten Westeinde: https://shopify.engineering/deconstructing-monolith-designing-software-maximizes-developer-productivity +Max introduces his new project, a successor to Homebrew, called Tea (41:04-50:34) -(2019 Shopify Unite conference talk) Deconstructing the Monolith: https://www.youtube.com/watch?v=ISYKx8sa53g +Beyang and Max discuss NFTs and how selling an NFT prompted Max to consider the problem of paying developers in open source (50:35- 54:00) -(Blog post) Under Deconstruction: The State of Shopify’s Monolith by Philip -Müller: https://shopify.engineering/shopify-monolith +Max discusses how Tea aims to solve the problem of compensation in open source (54:04-56:33) -Wedge: Deprecated internal Shopify tool that was built to score engineering teams on how well they are doing on their componentization journey. +Max describes the “Nebraska problem” and walks through how Tea can fund projects like OpenSSL and its dependencies. (56:34-57:43) -Packwerk: https://github.com/Shopify/packwerk +Beyang and Max discuss how NFTs and DAOs fit into the Tea protocol (57:45-1:08:20) -Shipit presents Packwerk: https://www.youtube.com/watch?v=olEA157z7kU + -(Blog post) Enforcing Modularity in Rails App with Packwerk by Maple Ong: https://shopify.engineering/enforcing-modularity-rails-apps-packwerk + -Zeitwerk: https://github.com/fxn/zeitwerk +**Beyang:** All right, welcome back to another iteration of the Sourcegraph Podcast. Today I’m here with Max Howell, creator of Homebrew, the popular package manager for macOS, a prominent contributor to the Mac and Swift open-source communities, and most recently, the founder of Tea, a new decentralized package manager that aims to solve the hairy problem of open-source funding. Max, welcome to the show. -Martin Fowler’s design stamina hypothesis: https://martinfowler.com/bliki/DesignStaminaHypothesis.html +**Max:** Hey, great to be here. -Shopify Fulfillment Network (SFN): https://www.shopify.com/fulfillment +**Beyang:** Awesome. It’s so good to have you. We have so much ground to cover, but the way we always kick things off here on Sourcegraph Podcast is by asking people how they got into programming. So, going back to, you know, your beginnings as a programmer, what was the thing that got you into this world? -Kafka: https://kafka.apache.org/ +**Max:** Yeah. When I was six, my dad brought home this computer that we had in the UK, called the BBC Micro, and it was, in fact, indeed made, in some manner at least, by the BBC, or they sponsored it. Certainly, they stuck ‘BBC’ all over it. And one of the things about it that was interesting, and I think is kind of a shame that this doesn’t still happen is that it booted straight into a prompt that was not, like, Bash, not a shell, it was a programming language. It was BBC BASIC, which was a very typical BASIC and probably didn’t deserve to be branded BBC. I’m sure this was like, you know, someone else’s BASIC, I’m not really sure. - +But because it booted straight into the prompt, my dad started playing around with it, and then he introduced me to what you could do. And obviously, the first few programs we wrote were the ones that you do see from time to time when you go to computer museums, the PRINT HELLO, on line 10, and then on line 20, GOTO 10. Because BASICs used to number all their lines, and then you referred to them that way, which was a very easy way to get the hang of what program flow was like. And it is actually kind of a shame that there isn’t any major BASICs around nowadays, I think, for people to learn from. Like, it seems a lot harder to jump straight into JavaScript or Python than it was for me to get into BASIC. - +So yeah, from a very young age, I was into it, but just as something that I was doing with everything else that a child does. So, I made video games. And [laugh] remember, I got in and out of it, and I remember when I was 11, 12-ish, at school, I’d go to the library at lunch and just work on this game that I was making. And me and some friends were, like, competing to figure out how to make any kind of game scroll. We started off making this, like, Asteroids kind of game, it was like a maze. You had a ship and you moved through this maze. + +And we couldn’t figure out how to make the screen scroll so we have this, like, little competition among us. I was the first to figure out how to make a game, like how to make it so that keypresses moved something on the screen. And this was before the internet, of course, or the internet was extremely nascent then. Like, we did not have it. So, we had to scour the manuals and figure it out. + +And then my friend, I came in one lunch, and he was like, “I figured out how to make it scroll.” He was just so thrilled at himself. Yeah, it opened up a whole new sort of era in our BASIC video game-making. So yeah, for a long time, like, that was it for me. I never considered it as a career because nobody in my life programmed for a job. + +**Beyang:** What did your parents do? + +**Max:** My mom was an interior designer and my dad was just like someone who did, like, construction, essentially, like, internal inside houses work. But my dad was very into math and electronics and, you know, programming to a certain extent. Like, he used to play chess with me until I got better than him and then he just suddenly stopped playing with me. [laugh]. + +**Beyang:** [laugh]. + +**Max:** So yeah, my mom always, like, gave me this impression that the only things worth doing were, like, being a lawyer or a doctor. It wasn’t, you know, something she did deliberately, but it never was something I considered. And I remember, I went through a career fair when I was 16, 17, and I went to the programmers who were there and talked to them. And I came away feeling that they were just so geeky that I didn’t want to do it. [laugh]. + +**Beyang:** [laugh]. You were turned off by the nerdiness. + +**Max:** Yeah. Yeah. + +**Beyang:** It’s just not your personality type. + +**Max:** Well, things I was quite nerdy as a teenager, but I didn’t want to be. + +**Beyang:** [laugh]. It’s not really an aspirational thing. It’s just a—[laugh]. + +**Max:** I wanted to pick a career where I was moving away from that to some extent. Well, I don’t know. I picked chemistry. So, I went to university and did a Masters in Chemistry. And well, I did one year in industry and hated it. + +I… I really enjoyed the first month where I got to know the machines and was measuring surfactant surface tension and, like, the organic chemistry would make me new surfactants to measure, and I’d measure them. And when I got the hang of the machine, it was really fun mastering that. And then after a month, I’d mastered it, and I realized that if I did this, as a career, I would be using this machine for the next five to ten years of my life and that’s it. There would be nothing else. I could try and write some papers and get myself, like, out there, et cetera, but I realized it was a very slow-moving area. + +And in a way, it’s fortunate that I ended up doing this boring physical chemistry rather than something a little more interesting because I might have stuck with it. But I became extremely depressed and started looking for other things to do. And I remembered that I really enjoyed programming. And at this point, it was possible to install Linux and then look at the source code. And I was, it seemed, like, natural for me to get into something where other people were collaborating, contributing because I wanted to understand how to be part of something like that. + +It was rewarding. And so, as a result, I got into programming from open-source, like that was my route. I never did computer science. I never really found it that interesting is the thing. Obviously, I’m famous for snubbing it at this point. + +**Beyang:** The kind of theory behind it, not your cup of tea? You’re more of kind of like— + +**Max:** Yeah. + +**Beyang:** —in practice, hands-on. + +**Max:** I like building things. I like making something that someone else is using, and, like, get something out of. And my [unintelligible 00:07:00] I got a design background as well. I almost went to university and did product design or industrial design. I got a scholarship… in UK we call the sixth form, which is… there isn’t really an equivalent in the States, but it’s like, after 16, 17, and 18, you can do two more years. + +So, I almost did that, but as result, I have a design basis; I have a product background. So, the algorithms, I find them fun occasionally. I certainly it’s not as though I haven’t had to do some algorithms of a Data Science and Computer Science, kind of, nature here and there. Like, when I need to I do. But it was never the, like, part which interested me. I like building things. + +**Beyang:** Nice. So, when you got into open-source, was computing just sort of this hobby that you had kept on through your college years, and your— + +**Max:** Mm-hm. + +**Beyang:** Years as a professional chemist? Were you just always, like, installing the like, how did you get introduced to Linux in the first place? + +**Max:** [laugh]. Yeah well, I got fed up with Windows. [laugh]. + +**Beyang:** [laugh]. Well, a lot of people got fed up with Windows. + +**Max:** And the more I studied, like, what was going on in computing—like, I found Slashdot, like, one year, I remember finding Slashdot. Like, during university, I became obsessed with this mp3 player called Sonique, which was like the main competitor to Winamp, but not as popular. It was very pretty. And I think that’s probably what got me into it. But not as good, frankly, with hindsight. + +But there was a bunch of people that I used to hang with on those forums, and they introduced me to Linux as, like, this free version of Windows. And at the time, Microsoft was evil, right? I don’t know if people remember this, but they were a tr— + +**Beyang:** [laugh] the old Microsoft. + +**Max:** [laugh]. Yeah, right. Nowadays, I have a reasonable amount respect for them, but at the time, like, IE6 was destroying the internet and they were doing everything they could to make sure that Windows was the only operating system that existed, and that included trying to take out Linux. So like, I got into this sort of revolutionary spirit, I think, that was part of the Linux movement. And I’d watch Stallman on his movies and, like, this—for people who don’t know, the guy is kind of—you know, I don’t want to— + +**Beyang:** RMS. + +**Max:** —offend him. Yeah. He’s uh… he’s certainly passionate, and inspiring in a way. And he got me into it. Was like, “Yeah, I want to be part of freeing software.” And I read more about Unix and went back and read about, like, the people who created it and it really inspired me. + +So, I found this Music Player because, like I say, like, music was something that I felt that the software wasn’t very good. It was called Amarok, and it only just been released version point-six, I believe. And the guy released it, there was a mailing list for this sort of stuff—it was for Linux—and he was like, “I’d like some other people to join me and make this.” So, I found the IRC channel—I miss IRC, incidentally. [laugh]. Things aren’t quite the same without it. Like, obviously it still exists, but other technologies much better now, so I don’t dwell on stuff that has nostalgia for me. If it’s not as good. Slack is better. + +And I said, “I’d like to get involved.” And he’s like, “Great. Just download the source code; start doing stuff.” So, I started hacking on it. And it came to be there was three of us, and we got known as the Three Ms—because I was mxcl, and the founder—we don’t call them founders in open-source, like, [unintelligible 00:10:59]— + +**Beyang:** Creator? + +**Max:** Released initially—yeah, creator—markey and mueslix. And still great friends with mueslix. Markey kind of fell off the map about ten years ago, unfortunately. And well, I don’t know how, but none of us had jobs and [laugh] so we just worked on it the whole time. Worked on it continuously. + +And we made it into something that was really impressive. People loved it. At the time, I think it was probably the best music player that there was. Like, you know, Winamp had been purchased by AOL and so, like, the innovation had moved to other areas. And, you know, I’d stay up all night because we have people contributing to it from all time zones, and I never wanted to miss anything because I was worried they would ruin it. [laugh]. + +But it was very open-source so nobody really had complete control over what was going on, which I learned from that experience is bad. Like, I took that into account and saw how Linux had Linus, and he controlled the contributions to a certain extent. He was very allowing and very open to whatever you wanted to build, but if he thought it was wrong, he’d say no. And Amarok had a problem where there wasn’t anyone who was willing to say no. Should have been markey. It was his initially, but he believed in the open-source spirit. So, I did learn that from that, but it can slow things down and derail the project. And I took that into account with Homebrew quite sincerely. + +**Beyang:** Yeah, yeah. So, I want to get into Homebrew and kind of how you transitioned over to the Mac. But before we get into that, like, so what was your life situation at that time? You said, “None of you had jobs,” so like, what were you doing for money at that point? And what was that like? + +**Max:** Yeah, so I was living with my parents. After the year in industry and quitting, and not being able to… handle the idea of going back to that, I moved back in with my parents, and, you know, just told them, I was going to try and figure out what I was doing myself. And still, I wasn’t thinking that I could make a career out of programming. But that’s all that I did. And so, I turned it into a night owl so that I wouldn’t have to see my parents and interact with them or explain to them what the hell I was doing with my life. + +**Beyang:** Were they understanding or were they kind of judgmental about the whole thing? + +**Max:** Well, first, they were understanding. And then, after about six months, and I’d been working on Amarok for, like, probably 18 months at this point, they sat me down just after I got up and just before they were about to go to bed and were like, “This cannot go on. You need to move out and get a job.” [laugh]. + +**Beyang:** [laugh]. Oh, now. And did you or did you just kind of continue as it was? + +**Max:** Well, I moved out. [laugh]. + +**Beyang:** [laugh]. Got it. + +**Max:** I went to live with my girlfriend at the time. She was living with her parents. Her parents were far less fussed about this sort of thing. But— + +**Beyang:** I can—yeah. + +**Max:** I don’t know why they were less fussed. Do you think they’d want me to make something with myself, but well, no, they liked me more than my parents apparently. [laugh]. + +**Beyang:** Interesting. [laugh]. + +**Max:** That’s not true. That’s not true. My parents have been extremely supportive. Mostly. This one exception. + +**Beyang:** I think they were just concer—it’s just like the parental concern, you know? They see you making a decision that they perceive to be, you know, a bad life decision and they kind of want to course correct. + +**Max:** Yeah. Like, they didn’t really understand what I was doing. They knew it wasn’t earning any money and they didn’t think it could. So, I lived with the girlfriend for six months and her dad gave me a job. So, you know. + +And I was making Excel spreadsheets [laugh] and I made them programmable, and so I was getting into that stuff. It was after about six months of that, the Last.fm, this startup in London, wrote to me and asked if I wanted to come and work for them. And it was because of Amarok. Amarok was very popular there. They were a music social network, like, early Web2.0, trying to figure out what music as a social network could be. + +And in many ways, really the only ones who ever figured it out, even though they’re not a big anymore, unfortunately. So, that’s how I got into doing it for pay, through open-source, so I don’t think it’s very typical. + +**Beyang:** Yeah, that’s interesting. And when did you get into macOS development? You know, what was that transition like? As I understand it, you’re mostly Mac these days. So, what made you kind of go from the Linux world into the very proprietary, very non-open world of Mac development? + +**Max:** Yeah. So, it was about the Last.fm period, and I was getting fed up with Linux because didn’t matter what you did on Linux, you had to read a tutorial or a blog post about it. Because nothing ever worked. Nothing just worked. + +And it’s not as though I could go back to Windows, having become a Unix diehard at this point. And it was about then, a bit before that when they—Apple announced that the next Mac would be Intel. And the fact that it was Intel and thus, all the software that I was using on Linux could easily be compiled for it—this was before it was very common for everything to be able to compile for ARM as well, although people made a lot of effort to do that. And I—and it was macOS Tiger, and it was clear that Apple were putting a lot of effort into making Mac OS X really nice, and, like, as a product person, you know, I appreciate the fact that they put all this time and effort into it, while Linux was a mess. And part of the reason I was involved in open-source on Linux is I was trying to fix it. + +But I kept seeing how these big organizations without anyone in charge were having trouble making something as large as a desktop environment. That’s what they call them on Linux. So, I bought one of the new Mac Minis with an Intel to see what I’d think of it. And I loved it. Every part of macOS that I looked at was clearly very well designed. + +Even the Unix integrations, how the file system was laid out, I just loved how everything was so consistent on the file system. There was a directory called library, and everything was capitalized, like, everything was CamelCase capitalized. Like, on Linux, everything was like whatever that particular dev liked. So, underscores, hyphens, CamelCase, spaces if they hated everybody. And Apple have chosen to CamelCase everything and very rarely put spaces in. Sometimes, sometimes there’s spaces in there, but really the spaces were more consistent, really. + +And I was like, okay, well, so they saying you always must quote in shell scripts. Like, I forgave them for that. I think with hindsight now, that was a bad call, but at the time, I was like, willing to, like, see how there was a positive to some of these choices. And so yeah, I really liked every part of it, but the part that was missing was a decent package manager. I came from Linux so I was used to having, like, power. + +I came from Arch Linux, actually, which I still think has one of the best package managers. So, that seeded the idea, certainly, that I could perhaps make the macOS Package Manager that was how I needed it to be. + +**Beyang:** Do you think—you know, you mentioned you had a design background. Do you think that aspect of the Mac world appealed to you? Like, do you feel like you were kind of unique in the Linux world in terms of being motivated by that, kind of like, end-user product experience? + +**Max:** Well, certainly at the time in Linux world. Things like Ubuntu was starting to turn up and they cared more about the user experience for their users. But yeah, it was frustrating, and I used to put a lot of effort into thinking about the usability and user experience, even though I had no formal training. I made this app, it was really the first open-source project that I made was an app called Filelight, which was this—it’s like Disk Daisy on Mac, it shows a concentric pie chart, essentially, so you can see where disk space was. And of course, back in, like, 2003 when we had 30 gigabyte hard drives, it was very important to know who was using your disk space because you needed it. [laugh] + +But I remember one time getting this comment from someone accusing the app of having no appreciation for user experience and it just annoyed me so much because that’s all I was thinking about. [laugh]. I was like, this is not like your other apps on Linux. Can you not see that? So, it occurred to me at that point that everyone was always complaining about the user experience on Linux, but you can’t even listen to these people because they don’t know good user experience. And that was frustrating. + +And I found on macOS, that the developers on macOS really cared about user experience and also really knew what it was, and the users also knew what it was. So, my work would be appreciated, rather than being told that I didn’t have a good eye for user experience by someone who didn’t know any better, apparently, what the term was. So, it’s just like, also—I hate to say it—like, the quality of developers, the quality of users, the quality of the ecosystem is excellent. And I think it’s a bit sad, actually. I don’t think Apple is that quality anymore. I don’t. + +**Beyang:** Interesting. So, when you got into macOS development, did you jump straight into, kind of, replicating that package manager experience? Or— + +**Max:** No. + +**Beyang:** Did you work on other things and then, you know, this became—those became motivation for developer experience? + +**Max:** Yeah, I certainly didn’t jump straight into it. It was years later, while I was at Last.fm, I became the lead developer for the client team there, and Last.fm had six client apps, and so—when we meant client, it was app. Last.fm was a website predominantly, but we had an app portion. + +And over the years, the number of apps increased from just Mac, Linux, and Windows. And you know, it’s another reason I liked Last.fm, the people there were Linux people, but—so we had apps for every platform so it meant that I would be making apps for every platform. We used this cross-platform toolkit called Qt—Q-T—which still exists, but I don’t know if it’s very—like, you know, the Bitcoin client was originally Qt—no, it wasn’t originally Qt, but it became Qt. + +**Beyang:** [laugh]. + +**Max:** And so, [my friend 00:22:22]— + +**Beyang:** Yeah I remember using—yeah, sorry. Go ahead. + +**Max:** So, when iPhone came out, there was an avid community of people who reverse-engineered it in order to make apps. And I think part of the reason that the App Store came out so quickly after the iPhone was originally announced was because people were clearly showing they really wanted to make their own apps for this device. They reverse-engineered the whole thing, open-sourced that, and put it online, and you could make apps sort of. You had to jailbreak the phone, which was a lot easier back then obviously. [laugh]. But Apple were a lot less bothered about whether or not you did it. + +And we hired the guy who made the Last.fm app for iPhone. So, we were really early to iPhone, and so I got into iPhone development super-early relative to a lot of people which was instrumental in where my career went. And then we made Android and we also had a Blackberry app because BlackBerry did have an App Store on their, like, newer phones for a bit. It was terrible. I hated their SDK. C++ and it was gross. But, you know, iPhone dev was the first— + +**Beyang:** They probably had a fair bit of market share at that point. They might have still been dominant. No? + +**Max:** They were doing well because people didn’t want to give up their old BlackBerrys and then they made a huge mistake by removing the keyboard and trying to copy of your iPhone, and it bombed. + +**Beyang:** They should have—yeah, yeah. + +**Max:** Know why your users like your product. [laugh]. It’s very important. I think touchscreens have gone way too many places. I think it’s ludicrous that Elon Musk thinks that Tesla should have such a big touchscreen in a car when you don’t want to have to be looking at these things. + +**Beyang:** Tactile feedback. + +**Max:** Yeah. I like Elon Musk. I read his autobiography at Christmas, but I think sometimes you make bad calls. He’s not as much of a product person as he thinks he is. He needs to listen to his CPO more. + +On the other hand, maybe I’m completely wrong, right? Like, everyone thinks touchscreens are so great, so he put them in the Tesla, and then they think the Tesla’s great. Sometimes I’m not the best person to ask. That is certainly something I’ve learned over the years. Like, I’ll make it so the product is all that matters, and then it won’t sell because I totally misread the market. + +**Beyang:** Well, you know what it is? I think sometimes they’re just like, waves that sweep the collective consciousness, or whatever, where everyone just agrees, like, that’s the way it should be done. Like, my kind of like, controversial take is, like, actually, you know, computers would be easier to use if they’re more like keyboard-based inputs. You know, and that’s a very, like, developer-y, you know, mindset. + +**Max:** Well, you’re right, it would be easier to use, just harder to learn. + +**Beyang:** Yeah. Yeah. + +**Max:** And that’s, of course, a big factor in whether or not a product succeeds. + +**Beyang:** Yeah, definitely. Okay, so you were working on these different mobile clients, but you liked iOS the best for Last.fm? + +**Max:** Yeah, I really enjoyed Objective-C. So, my experience with Objective-C had been more limited up until then because we used this cross-platform toolkit. And even though I bought a Mac, I haven’t jumped straight into making apps on it at that point. But we used Mac for all of the development because it was the only platform where we could have a VM for Linux and a VM for Windows, and easily—you know, because the tools for making those virtual machines were great on Mac. Like, that was the thing, everything was always greatest on Mac, even if, like, it might be greater on Windows for one thing, overall it was easier to just develop on Mac. So, we all did that in the client team. + +Like most of the company even. Like, you know, this was right at the beginning of the era, where Mac became the choice for developers. So, we did everything on Mac, and like, you know, so like Android SDK, you didn’t need to VM it, it worked well on Mac. You could pick Mac or Windows and that was a universal thing. Even the BlackBerry SDK, like, you could pick Windows or Mac. + +So, at that point, we were trying to build the six different platforms and the amount of crap we had to go through to make it work on six platforms was prohibitive. Like, I estimated we were wasting 40% of our time just messing around with build systems and trying to try to make it consistently work. We had a bunch of open-source that was underpinning the whole, like, all of the work we did was open-source, all these libraries trying to make it work. And it annoyed me, you know? This led me back to thinking about doing a package manager. Well, the truth is, I’d go to the pub and complain about it every evening [laugh] until my coworkers got fed up with me complaining about it, and one of them said, “Max, maybe you should just fix this problem.” And uh, it’s— + +**Beyang:** And you’re like, “Challenge accepted.” + +**Max:** Yeah. It occurred to me that I could and that I didn’t have to just complain about it. So, that weekend, I started what would later become Homebrew. Just thinking through what, as a developer, I needed in order to do this cross-platform dev. And so, you know, this is strange how that Homebrew is not a cross-platform development tool now and indicative of how what a product is changes during its development. But that was how it started. + +**Beyang:** Interesting. So, what were some of the—you know, when you’re first getting the project off the ground, you know, what were the ideas that you took from your experience with the Linux package management world, and what were the changes or new ideas you brought in? + +**Max:** Yeah. So, most package managers are really inflexible, like, famously so. Like, at the time, apt was probably the biggest one and forums online were always filled with people trying to get something installed and failing because it had no flexibility. So, you have installed foo, and this thing you’re trying to install wants foo slightly newer and I refuse to install it. [laugh]. I will not do it for you. + +And now that annoyed me, I felt that developers knew what they were doing, so they needed, like, some freedom to, you know, do that, get things working, that you can trust them. So, from the go, I made like that was why the formula system was built from the go, and why I built it on top of Git. I wanted you to go in there and hack on these package recipes yourself. I wanted it to be extremely obvious how you did it and for you to have the power to do whatever you want and to make it work for you. Like, the idea originally was you cloned a version of Homebrew into your source directory and then made it install the packages you needed in order to get going. + +And then you could have, like, several copies of the copies of it scattered around, which meant compiling everything from source which didn’t seem a problem because unlike the main competitor to Homebrew, which is called MacPorts. I realized that the Mac itself came with a huge amount of Unix software that Apple will maintain for us, and they were obligated to make it binary compatible and safe and consistent so that all the apps that you installed would continue working, you know, for ten years. So, I took advantage of that. We didn’t reinstall all those things. And it meant using Homebrew was vastly quicker than using MacPorts, which certainly helped with early adoption. + +And the symlinking system, I think there was another package manager that used it, but I didn’t know about it at the time, but I had heard about the symlinking system from another tool that GNU made, but it wasn’t the [unintelligible 00:30:34] typical, it’s a typical GNU thing, right, like, almost good. [laugh]. + +**Beyang:** [laugh]. + +**Max:** Because I wanted to have multiple versions of things available so you could switch back and forth as you needed for different projects, or if something was broken. And I wanted the file system layout to be really nice, just like the rest of Mac. I wanted it so you could go into the cellar and there would be all the software you’d installed, and it’s all nicely versioned and you can see what each package contains. I like the idea that the file system was the database, and so I very carefully designed everything so I didn’t need another database. Because also at that point, I’d realized that every time you add a new system—call it a database or whatever—a load of stuff breaks, and you’ve added complexity that’s just going to slow down development to the rest of time. + +And so, if you can avoid it, like, it wasn’t going to, like, insist we avoid it. Like, if we needed it, I’d be like, “Okay, now we have a database.” But if I could think of some clever way to avoid it that worked well enough and made it so people could use regular tools to inspect what packages they’d installed, I really liked that. I liked that you could just use the terminal to see what you have installed, you can use ls, you can use find, and you didn’t have to learn a cryptic command like all the other package managers, in order to just use it. Like, I really believe that the package manager is just this fundamental tool that nobody wants. + +And I wanted to build something that just got out of your way so you could get on with what mattered. You wanted to build an app, you wanted to use an app, you wanted to do something else. And the package manager shouldn’t be sitting there going, “Hey, I’m important. You have to learn me. You have to understand how I work. You have to tread on eggshells while you’re using me. And if you don’t, you’re going to be sorry.” [laugh]. I didn’t want that. + +Another reason that it was such a successful product, I think, is because people saw just by reading the readme that I was trying to make the thing they wanted. And while everyone else who made a package manager, it became their lives. And, like, when you build something, it becomes important to you, and you want everyone to know how important this thing you’ve built is. I think, like, the human ego side of things was holding back similar tools. + +**Beyang:** Mm-hm. So, how did Homebrew, I guess, go from this thing was originally intended for, you know, multi-platform package management, geared towards your mobile development issues to, like, the thing that became the de facto package manager for macOS? Was that, like, a sudden transition? Was there, like, a particular inflection point where it kind of changed from one to the other? Or was it more just, like, you know, one day you woke up and you realized that this was now mainly for macOS, and it was basically ubiquitous. + +**Max:** Yeah, it was definitely gradual. And it was following what users wanted was the truth. So, there was huge demand for making it work on Mac and making it work as a replacement for MacPorts so it could install all the tools that people needed and not just the things you might need as a developer to build apps. So yeah, it was following the community and allowing the community to influence it. I was—from the way it was designed, I always wanted other people to get involved, like, heavily involved because there are tens of thousands of packages, and I’ll be damned if I personally had to write all the package recipes. + +So, I built it from the start so that it was super easy to get involved. Like, it was just a git clone. So, you could go in there, edit it, it was working out of the clone, there was no installation of the package manager itself, and you could just push back, like, straightaway, commit and push back. Now, this is also partly so that you could have maintained your own fork so that you could customize it to what you wanted, but it was also because I wanted people to get involved straight away. And so, I created two commands ‘brew create,’ which took a URL and then would download that URL, assuming it was a source tarball, and then try and create the formula, the template, the recipe—we called them formula; well, I called them formula. The namings system is something—another thing it’s famous for. + +And it would probably work. You could probably just push that back, open a pull request—after pull requests existed; like, Homebrew was before pull requests, initially. They turned up a few months later, thank God. It was very hard to work with open-source before that. Better than SourceForge, but. + +And ‘brew edit,’ which meant that if something wasn’t working, you could—I wanted people to fix things themselves, and it’s still a big problem with open-source is that a lot of projects don’t tell their users, their developer—they don’t, they assume the developers who are using it, A, haven’t got the talent to fix things in the way that they present themselves, but they don’t tell the users how to get involved either or what they need to do. I still do not know how to contribute to a Ruby Gem. I built Homebrew with Ruby. I couldn’t find any documentation for it, and I gave up. + +How do you do it? Do you clone it somewhere? Then how do you make Ruby Gems use your version rather than vended version? I don’t know, and I never learned and I don’t really use Ruby anymore, so I’ll never know. But I think everything that’s open-source should have at the top how to get involved, how to contribute, and it should be two or three steps at most. + +With Homebrew it was one step, so everyone got involved. And if you look up, like, the video that someone’s put online about showing how Homebrew’s contribution exploded, it’s one of those videos where people’s heads fly around and they activate dots and the dots represent files. So, the Homebrew one is for, like, the first few months while I was getting it up to speed, before I even really open-sourced, just me darting around. And then a few other people turned up, and then within no time at all, there’s thousands of people participating. + +**Beyang:** That’s awesome. + +**Max:** And it’s—I don’t think there can be another open-source project that got as many unique contributors as quickly. And it is the project that has the most unique contributors of any open-source project, and thus, almost certainly any software project that exists. Because I designed it to do that, from the get-go. + +**Beyang:** Because the formulas for Homebrew which, you know, if you’re contributing to another package manager, you might not think of as, like, contributing per se, you’re just you know, writing a formula, but with Homebrew it’s all just in the same Git repository, so the boundaries are— + +**Max:** Or was. + +**Beyang:** —kind of fuzzy—or was, sorry. But it was easy to go from one to the other. And I think this gets back to your earlier point about the advantages of just using a system that’s already there, you know, file system on your computer, or you know, Git for hosting things that matter to devs. It wasn’t some other system; it was just a Git repository so it’s easy to, like, introspect and, like, you know, pull open the covers. There were no covers; you could just see how it worked. + +I can think of, like, a couple other open-source projects, like, you know, TypeScript, the way they did, their type definitions with definitely typed that repository in the early days. The Go package management in the early days, where it was just a Git URL. I think there’s something to be said for just, like, using the tool that people are already familiar with instead of trying to reinvent something that people then have to learn; it becomes, like, this whole, like, thing they have to grok before they can start contributing to your tool. + +**Max:** It’s absolutely true. There’s so much open-source released every week. Nobody wants to learn anything new. [laugh] so if you’re going to— + +**Beyang:** [laugh]. We’re also busy. Yeah. + +**Max:** You have to be extremely compelling to make them jump through hoops to learn how it works. Like, you know, we’re all lazy in that respect. But also, we don’t trust that anything is really going to be that good. You can say it’s going to be great all you want but we don’t trust it. So yeah, but people aren’t willing to sit down and read an instruction manual. + +I think that’s an issue with a huge amount of software at the moment. Like, Docker is a good example of something that’s so compelling everybody uses it, but the interface that we use to interact with it is awful. I have to Google every time how to do everything. I can never remember it; it’s not intuitive in the slightest. So yeah, you’re I’d have to be so compelling—like Docker—that people willing to tolerate a terrible interface, or you have to have an incredibly good interface. And preferably both. [laugh]. Right? + +**Beyang:** [laugh]. Yeah, that makes sense. Okay, so you’re the author of the most successful package manager on macOS. Some might argue the most successful package manager in existence, given the ubiquity it has for Mac users and Mac devs. You know, a less ambitious person might be content with that, and just, you know, ride off into the sunset being satisfied that they created this awesome package manager that everyone uses. + +But no. You are working on a new project which in some ways is a bit of a successor to Homebrew, kind of a Homebrew 2, called Tea. So, talk a little bit about what Tea is and what the motivations were for creating this new system that you’re working on now. + +**Max:** Good name, right? + +**Beyang:** Yeah. It—[laugh] I love the naming scheme throughout. Yeah. + +**Max:** Yeah, I’ve always been fond of names. And it goes back to intuitiveness, again. Humans need labels. Computers don’t. Like, I talk about this with my girlfriend on a regular basis about what a potential AI is going to be like, and one of the things I talked about is how it doesn’t need to name things. It will just use IDs, it will make a UUID for everything. + +**Beyang:** [laugh]. + +**Max:** Humans need names in order to understand what something is, and so that limits our creative ability for sure. When everything needs a name, it means you to have less things that need names or you overload the human. So, I’m a big believer in picking names that evoke the right metaphor so that someone reading about your tool, your app, your library has a preconceived notion of what it is. Obviously, this is no good for children, but children learn things extremely quickly so I’m not worried about them. It’s adults who are the problem. + +So, I’ve always been a big person for names. And Homebrew has great naming scheme; we can talk about that some other time, I guess. But yeah, Tea is, essentially Brew 2 for Web3 is what we’re saying. And the genesis of this came because I haven’t worked on Homebrew for quite a few years at this point. I burned out on the project. I felt that it was good enough, it did everything that it should do, it was extremely useful to people, and there was a limited amount of value I could continue adding to it. -**Beyang Liu:** -All right. I'm here with Kirsten Westeinde. Kirsten is a software development manager at Shopify where I understand, Kirsten, you've been involved in a years-long effort to decompose Shopify's massive Rails monolith, or perhaps like multiple monoliths now, into smaller, modular components. Is that right? +But the main thing was that in order to work on it, I had to have two full-time jobs. Like, I got to confess to a lot of my past employers at this point that there was a lot of times when I was working on Homebrew and I was not working on the thing you paid me to do. [laugh]. Sorry about that, but I hope you were happy with what I produced otherwise, anyway. And so, I got into this cycle. -**Kirsten Westeinde:** -That's right. +I quit my job to work on Homebrew initially, after it became successful because it was so compelling to work on this huge project that was getting people really excited, and it was by far, like, the most interesting thing I could be working on at that point. But then I ran out of money. I remember going to the bank and asking them how going overdrawn works because I’d never done it before, and I didn’t know, like, would I be paying interest? And how much money could I go overdrawn by? -**Beyang Liu:** -Cool. Thanks so much for taking the time. Sounds like a massive effort and I'll bet that Shopify engineering has been very busy over the past year. So I really appreciate you taking the time to come chat with us and share things that you've learned along the way. +**Beyang:** Wow. -**Kirsten Westeinde:** -Absolutely. Thanks for having me. +**Max:** And were they going to report me to the government as some kind of delinquent? [laugh]. I was 28. So, I kept going through the cycle of getting jobs—I had to pick jobs that I liked, I can’t work on things I don’t like. That’s how I got into open-source, right? Like, chemistry didn’t turn out to be what I wanted, and it made me incredibly depressed. -**Beyang Liu:** -So I think the focal point of our conversation is really going to be this massive decomposition effort. It's going to be very fascinating to hear about because I think a lot of engineering teams try to undertake projects like this, especially as the company grows and some succeed, but a lot end in failure. And so I think you have a lot to share in terms of how you can pull this off successfully. But before we dive into all of that, we have kind of a tradition on the podcast of first asking people how they got into programming. Right now you're this software development manager who's in charge of this very complex project. You have a lot of responsibility, but at some point you had your first contact with code. So, for you, what was that? And how did that start your journey into this world of bits and ones and zeros that we live in? +So, I had to find jobs that I wanted, which is difficult. And then I wanted to go back and work on Homebrew or my other open-source because that was what I felt was my calling, really. That was where I was making the most impact. So, I cycled through jobs, like, doing something for a bit, quitting, working on open-source, and then going back. And, you know, a few years ago, I was like, “This is getting ridiculous. I need to figure out how to try and make money with open-source.” -**Kirsten Westeinde:** -Absolutely. Yeah. If you had asked me when I was a kid, I definitely would not have thought that this would have been where I ended up. I actually didn't even really know what programming was before I went into university, but I did end up studying engineering just because of my love for math and science. And the school that I went to ended up doing a general first year where we basically took classes from all of the different engineering tracks to kind of learn a bit about them and decide where we wanted to go. And so in my first year I took a programming class, we learned C++ and it turns out that I really enjoyed it and I was really good at it. And the more that I looked into it, it seemed like a really interesting field to be in. So I had originally thought I would be a civil engineer, but I'm very glad that I ended up going the direction I did. +And there’s a few people who do and they’re quite successful, and I tried to copy their system but didn’t manage to get enough people to sponsor my Patreon and found that even marketing myself for Patreon was an enormous amount of time that I didn’t want to be spending on that. -**Beyang Liu:** -C++—that's kind of like diving off the deep end as your first programming language. +**Beyang:** Yeah. -**Kirsten Westeinde:** -Yeah. It's always funny to hear what people learn in school. It's rarely what we end up using in the field I find. +**Max:** So, went back to get another job. -**Beyang Liu:** -Okay, cool. So from your start in university, what was kind of like the rough journey from there to joining Shopify? +**Beyang:** [laugh]. You don’t strike me much as a company man. -**Kirsten Westeinde:** -I actually joined Shopify as an intern while I was still in school. So I did two internships with other companies, but I joined Shopify in 2013 out of the Ottawa office as an intern. And then I ended up doing three different internships with Shopify before joining full-time after I graduated. And through those internships, I really used the time to try out different things and try to understand where my interests lay. So I did some web development using Rails, I did some mobile development, and I worked on some internal tools and external products. So I've really had my hands in a lot of different places at Shopify over the years. +**Max:** No. No. No. Apple knew that full well. I feel bad for Apple, really. I worked there for a year and I just couldn’t handle it. Too corporate, at least part I was at. Tried to make something useful and important—hopefully did anyway. -**Beyang Liu:** -Awesome. So you've been there since 2013, which means you've kind of had a front-row seat to this whole rocket ship trajectory that the company has been on. +Yeah, so—well indeed. So, startups mostly. But it’s an old problem. Everyone knows it, that people who make an open-source may not find a way to get paid for that. Like, a lot of people who work on Homebrew now work at GitHub and great, I’m glad that there are companies that exist who understand the importance of open-source and willing to basically fund it. -**Kirsten Westeinde:** -Yeah, it's been crazy. When I joined, I think we had 150 people working for the company and we're a private company. We had only a handful of merchants. We had one small office and fast forward to today, we're an international company. We have... It's hard to keep track of it, but I think like 7,000+ employees. At this time, we're a public company, so much has changed. I like to tell people I haven't worked at the same company for eight years, I've actually worked at like eight different companies because it really has been changing that quickly. +I don’t think that’s good for open-source to have big companies like Microsoft controlling how open-source works. -**Beyang Liu:** -Yeah, that's crazy. And you know, I bet most of our listeners don't need any introduction to Shopify, but you know, for the one or two people who've been living under a rock for the past seven years. Chances are... If you bought anything on the internet, chances are you use Shopify, right? Because you power a lot of the kind of small businesses and independent operators that sell things online right? +**Beyang:** Sure. Yeah, yeah. -**Kirsten Westeinde:** -Yeah, that's correct. We actually don't force the Shopify brand on our merchant sites. So a lot of people buy things from Shopify stores without actually knowing that they have. But basically if you bought something from Amazon and it was enjoyable or—sorry—from anything other than Amazon and it was an enjoyable experience, that was probably a Shopify store. +**Max:** Like, let’s not forget what Microsoft were, like, 20 years ago. Even ten years ago. They could easily go back to that and then you have huge portions of what builds the internet. And let’s face it, there isn’t an app that exists isn’t composed of open-source in some capacity, even if it’s just the operating system itself. Like, 80, 90% of all applications are an open-source bed, and then some proprietary source code on top, which is, like, the cherry, which makes the thing into the app and makes it into Instagram. -**Beyang Liu:** -Yeah. And actually for me personally, increasingly I find myself buying more and more things off Amazon because there's like... A lot of times the things that you can get in these kinds of independent stores are just like higher quality or it just feels closer to the seller. And so thank you for enabling all that. So diving into this large-scale refactoring and re-architecture project, I thought maybe we could start off by giving people an overview of what Shopify looks like as an engineering organization and as a codebase. So in broad strokes, what are the major teams and what are the major parts of the code, and what does it all kind of do as an engineering system? +But a huge amount of open-source software and the people who have built this stuff often would love to work on it. They’re passionate enough they built it in the first place. But there’s no mechanism for how that works. And last year, while I was—well here’s the truth. My girlfriend and me were trying for a baby and succeeded in record time. [laugh]. The gynecologist told us it could take up to a year and not to expect anything, so like, two weeks later, when she was showing me the pregnancy tests, I was like, “Oh, great. This is what we wanted, but I thought we had a bit more time to figure out how—what I was going to do next.” -**Kirsten Westeinde:** -Definitely. So we talk about the core Shopify product as the core codebase. And so that's basically what powers merchants to manage their orders, to manage shipping, etc. So all of the core pieces of the product live in one Rails codebase. And it's actually one of the largest, probably the largest Rails codebase on the planet because we got started with Rails like right when Rails became a thing, and it's been under continuous development since at least 2006. And the last time I checked, we were pushing like 3 million lines of code, more than 500,000 lifetime commits and about 40,000 files. So it's massive. And in terms of an engineering org, I think we're at about 1,500 or so engineers and we're actually aiming to hire another 2021 this year, so it'll be continuing on this massive pace of growth. And if I had to guess, I'd guess probably about half of the engineering team works daily in the core codebase and the other half works on applications outside of core. +**Beyang:** Cash flow and yeah. [laugh]. -**Beyang Liu:** -That's crazy, that's so much code. Does it... Have you gotten to the point where Git operations start to slow down at all? Like if you were to clone the entire codebase to your local machine... Like do people even do that? Or how long does that take? +**Max:** Yeah, exactly. [laugh]. So, I kind of panicked and went through all the ideas I’ve ever written down. I’m a bit of an obsessive notetaker, and even though I hadn’t worked on Homebrew for, you know, five, six years at this point, I’d been recording what would be, like, a Brew 2, what I could do. And people have often asked me if I make another Brew 2, and I always said no, but I’d made the notes anyway. -**Kirsten Westeinde:** -Yeah, people do it, it works. The test suite definitely can get slow, but development itself is not a problem. We actually... We always joke, like we host Kylie Jenner's shop on Shopify and it's like one of the largest stores. And we joke that like we are the Kylie to so many other systems that we use. So we're hosted on Google Cloud and like we are by far the largest product hosted on Google Cloud and so we're often stretching the boundaries of their systems. And similarly, like with Rails itself, we are often the ones that find those bugs that you only find when you're operating at this scale. And so what we do is we really make choices about what technologies we want to use and then really double down on them. So we actually have a full team at Shopify who are Rails core contributors, and we try to actually just evolve the tools that we use to be better and to be better able to support us and others. +And I said no because I didn’t see anything—like, all my ideas were good, but they weren’t good enough that it made sense for me to do something new. Homebrew is good. It’s still a great tool. And I have a friend—and he’s my co-founder—Timothy Lewis, and I’ve known him for ten years or so. -**Beyang Liu:** -What languages are in use? So with Rails, I assume Ruby is a big language, are there any others that are big parts of the stack? +And he’s been trying to get me into Web3 and crypto. Crypto of course before Web3 because that’s what it was. Like, the fundamental—you know, blockchain is the technology underneath it, and for a long time, its only purpose was crypto, cryptocurrencies. And so, I started diving down there as well because it’s like, okay. You know, whatever I think about cryptocurrencies, there’s definitely money there. Like, last year was the year I think a lot of developers saw that. -**Kirsten Westeinde:** -Yeah. Shopify has always been very opinionated about technology choices. So essentially any problem that can be solved with Ruby on Rails should be, there are some exceptions. If applications have really, really high performance constraints or need parallelization, sometimes we use Go, the front ends are in React. And for mobile development, we've just standardized around React Native going forward as well. And then in data land, we have some Python and Scala and in infrastructure we have Go and Lua as well. +**Beyang:** Yeah. But you’re coming from a point of skepticism it sounds like? -**Beyang Liu:** -So you wrote this blog post about how you were in charge of this project to kind of decompose Shopify’s main Ruby on Rails monolith into smaller, more modular components, not microservices, but different modularized components, we'll get into that later. Tell us at a high level, what was the mandate of that project and how did you come to be involved in that. +**Max:** Yes. -**Kirsten Westeinde:** -Yeah. So in, I think it was around early 2017, it was becoming very clear that our core codebase was reaching a tipping point. Where it was becoming more and more challenging to build functionality into it because of all of the interdependencies. So projects were taking longer, adding things that used to be simple were causing test failures and big headaches. And also it was really... We're finding it really challenging to onboard new people into the codebase because there was just so much to learn. And so we knew we had all of these problems, but we didn't want to get into solutioning without really understanding what the problems were. So we actually sent out a large survey to all the engineers working in the core codebase to hear their feedback about what the pain points were before deciding what to do. +**Beyang:** Like you. -**Kirsten Westeinde:** -And it was that survey that actually led us to start this project. We originally called it a very clever name of, "Break core up into multiple pieces." But that name eventually evolved into componentization where essentially we wanted to make this one large codebase feel like many smaller codebases. And so we wanted to separate the code by business domain basically because that is the code that more often needs to change together. And so what that would mean is if we were hiring someone onto say the shipping team, they should be able to just understand the shipping component without having to understand all of the other elements of the core codebase. So at a high level, that was the goal. +**Max:** Well— -**Beyang Liu:** -Got it. So if I were a buzzword-driven engineering manager and you came to me and said, "Let's decompose this monolith into different independent components." I would say, "Well, that's obvious, you know, we live in 2021 and the obvious solution to decomposing a monolith is to break it down into these things called microservices. But I understand that that's not the approach that you ended up taking. So can you talk about why you didn't follow the naive path of like, let's write a bunch of microservices. What did you end up with instead? What is this modular monolith that you talk about? +**Beyang:** He was more—he was earlier in the space—and you were kind of— -**Kirsten Westeinde:** -Yeah. I mean, I think for one thing we had been developing on Shopify core for so long that pausing everything and building out net new microservices just wasn't feasible for us. Whereas the direction we ended up going was making incremental changes in the existing codebase while teams continued to develop and add features to it. So that was a big reason, but the other thing is that unless we wanted to start from scratch, to be able to start extracting some of these pieces out into microservices, we would need to have them well encapsulated and have boundaries in place, which is what we ended up doing. But we ended up deciding that we wanted to stop there because monoliths have a lot of pros, a lot of benefits. +**Max:** Yeah. -**Kirsten Westeinde:** -Having only one codebase means you only have to do all of your gem bumps and your security fixes in one place. It also means you only have to manage one deployment infrastructure and one test pipeline. There's also a lot less surface area in terms of places that you could be attacked, and I think that makes the system a lot more resilient. And we found it helpful to keep all of our data in one place, in one database and allow the different components to have access to the data that they needed instead of trying to synchronize it across many different systems. +**Beyang:** “Eh, I don’t”— -**Beyang Liu:** -Yeah. +**Max:** He was there very early. Like, good on him. Like, you know, you got to say good on all the people who saw it a long time ago, at this point because they’ve done very well out of it. So, good on them. But yeah, I wasn’t so much skeptical—well, I was, but not, like, totally skeptical. -**Kirsten Westeinde:** -And lastly, with microservices, the communication between the different components has to happen over the network, which introduces a whole bunch of complexity around API version management and backwards compatibility, as well as just dealing with the network in general. It can be slow. There's lots of ways it can fail. So those were all the things that we liked about the monolith and didn't want to just give those all up. And we kind of felt like moving to microservices was more of a topology change whereas what we wanted was an architecture change. There was nothing necessarily wrong with our monolith, it was just that there were no well-defined boundaries between different things. +I’d read the Bitcoin whitepaper years ago and I was like, “This is a good piece of technology.” Like, clearly, I liked the whitepaper format. Like, very good way for getting people who don’t know a lot about the technicalities of something. Like you didn’t have to understand crypto—cryptography, I mean—in order to understand the Bitcoin whitepaper. But I thought it was just money, and money has never really interested me as—I like having it because then I don’t have to worry about it. [laugh]. -**Beyang Liu:** -So you had this principled, thoughtful approach of "We have developer pain, let's address that directly, but we're not going to involve it with changing the topology of production, introducing all these unnecessary network boundaries because that introduces its own complexity and its own baggage" in a way. +You know, I like the things I make. That’s my motivation. So, I didn’t realize that it had become what we’re now calling Web3, of course. And that there’s all these ways that blockchain as a base technology can be used that are exciting. And so, I dove down that rabbit hole, bought a few NFTs just to see what that was about because that was, you know, obviously it was really becoming a craze then and, like, now I honestly I think NFTs are a little bit h— -**Kirsten Westeinde:** -Yeah, exactly. Our monolith had gotten us so far. We, I think, had learned so much about what is good about it, that it gave us pause to really ask the question and really think through whether microservices were right for us. And the answer at the time was no. +**Beyang:** [laugh]. -**Kirsten Westeinde:** -That being said, we are believers of service-oriented architecture, so we do have other applications outside of the core codebase. However, everything related to the core functionalities of commerce, we keep in one place. But there have been some examples where we actually have extracted components out of the core codebase because they weren't related to the core functionality of commerce and we pulled them out into their own services, but that was exponentially easier to do post componentization. +**Max:** —they’ve ha—they’ve peaked. But you know, don’t tell some other people that, I guess. But I bought them to see what it was about. And when I sold one and there was this 10% royalty that was enforced by the digital contract, I was like, “Ah, this is interesting.” There was no way to not—well, I now know there is a way to not pay the 10% but that’s because [laugh] you can— -**Beyang Liu:** -So what did this modularization of the monolith actually entail? I assume you started with... My knowledge of Ruby On Rails is fairly limited, but I understand the standard directly out is: you have a model directory, you have a view directory, you have a controller directory. Was that the structure of the codebase at the start of the project? +**Beyang:** Oh, that’s interesting. -**Kirsten Westeinde:** -Yeah, that's correct. So we started before this, we basically just had a Rails app out of the box, so you're exactly right. They were organized by our software concepts, but the first thing that we wanted to do was restructure our codebase to be organized by real-world concepts like orders, shipping, inventory, billing. And what we ended up doing was we actually audited every single Ruby class in the code base, which was like 6,000 at the time, in a spreadsheet and manually labeled which component we thought that they belonged in based on the initial list of components we had. And then we built up one massive PR with automated scripts from the spreadsheet to basically just create those top-level folders. And then within the orders folder it would still be models, views, controllers, but we would just move all of the specific models, views and controls into that folder. So we did that all in one big PR, which was kind of terrifying, but ended up being fine. We have good test coverage. So it is all good. +**Max:** Yeah. -**Beyang Liu:** -What was the size of that diff? It must have been like... +**Beyang:** There’s a loophole. -**Kirsten Westeinde:** -It was every file, every file was touched. +**Max:** Yeah. Well, it’s because the NFT is on Ethereum, shall we say? So, it’s just part of the blockchain and you can just transfer that. The digital contract enforces the royalties, OpenSeas. But they’re making a new proposal which will make it so it’s part of the blockchain itself, then you won’t be able to get around it. -**Beyang Liu:** -Wow. +**Beyang:** Ah, I see. They had their own centralized, kind of, system of record for NFT ownership. Now, they’re decentralizing it. -**Kirsten Westeinde:** -Yeah. So that was the first step. +**Max:** Well, essentially, you had to sell the—you still have to sell your NFT through OpenSea for the 10% royalty to be paid to the original creator because you have to use their digital contract. And some people, you know, save 10%, effectively, by going around it, and it’s just, you know, [unintelligible 00:50:42]. Shouldn’t do that. -**Beyang Liu:** -Why wasn't that sufficient? The straw man is like, "Okay, if we put everything in its own directories and now each team gets to work in their own directory and that's done. They're like separate codebases." Why was that not just the end of the project? +**Beyang:** Yeah. Yeah. And 10% on some of these NFT purchases, that’s quite a substantial sum of money. -**Kirsten Westeinde:** -Well, all that had achieved was moving files around, right? Like in Ruby On Rails, everything is automatically globally accessible. So just because all the code for orders was within an orders folder, it could still use whatever it wanted from any of the other folders. So we still had all the same problems with fragility of the test suite and really if I was working in the orders component, I still had to understand about the taxes component and the shipping component because of how tightly coupled they were and how there was no clearly defined API for them to communicate between the folders. Nothing had structurally changed about how the code was interacting. +**Max:** Yeah. Let’s not talk about it. I saw [unintelligible 00:50:52]. -**Kirsten Westeinde:** -So it was a great first step in the sense that I now knew if I'm looking for something orders related, I can find it. And that the top-level component folders had explicit owners, which was actually a huge win. But beyond that, we still hadn't solved a lot of the existing problems. +**Beyang:** [laugh]. -**Beyang Liu:** -You mentioned something in your blog post about how one of the initial pain points was as the codebase grew, the amount of code that a given developer would have to understand to get a thing done was like something like O of N, it grew linearly with the size of the codebase. And that obviously is untenable as you grow, because it just means your productivity approaches asymptotically down to zero. So how'd you go from O of N to sub linear? What were the things that you had to make in terms of codebase structure to enable that? +**Max:** I saw some of those come and go, and I was like, “This is stupid, isn’t it?” Like everyone else, I thought it was stupid. I now understand that NFTs as a thing is more useful, and like, well, we’re going to use them for Tea, right? We’re not using JPEGed NFTs. But I’ll get to that. -**Kirsten Westeinde:** -Yeah, definitely. So once we moved all the code into their separate folders, the next ask was for each component to define a public API. And ideally that public API should be easy enough to understand that I can understand what it's doing without having to go under the hood and look into the component. And so that would allow me to get to know one component really well. We also asked each component to explicitly say which other components they depended on. So say I was joining the orders team, I could really deeply get to know the orders component. And then as my next step, I could look at what the dependency is listed out for the orders component is, and maybe just start getting to know their public API so I know what methods are available to me from those components without needing to understand all the nuances of how they get the things done exposing the public API. +**Beyang:** Yeah. Yeah. -**Beyang Liu:** -Yeah. What about the human side of this kind of change? In the organizations that we've worked with that have attempted something like this, there's the technical side, which involves making the proper changes to the code, in this case, introducing a bunch of interfaces and making those interface boundaries well-defined, but then there's also this human, semi-political aspect to it where you're going to each of these engineering teams and each of them has their own goals and objectives that they're working toward, business goals, and you're coming to them and essentially asking them to add work onto their plate to satisfy this change. +**Max:** [laugh]. But it occurred to me that this mandatory royalty was very interesting. And like, sponsorship and bounties are two systems people try to use to fund open-source, but it only rewards, really, the popular projects, stuff that people know about. And you know, your average node application is 600 to 1000 node dependencies and then that goes down underneath that all the way down to libc, the most fundamental ones. And you could say the kernel, even, is the most fundamental part that’s doing the literal hardware operations. -**Beyang Liu:** -Can you talk about that element of it? Did you have to do a lot of horse trading or, "Do me a favor and I'll do you a favor?" What does that look like from your perspective? +And so, you got, you know, thousands of open-source dependencies and it’s just infeasible for anyone to sponsor their entire stack. We give these companies a hard time because they got the money to sponsor their stack, but how they literally going to do that? Like, I have this popular open-source library called PromiseKit—and I’ve used this as an example throughout talking about Tea—it’s used by 100,000 apps so if every one of those apps just give me one dollar a year— -**Kirsten Westeinde:** -Yeah, it's interesting. I've actually talked about this project at a few different conferences and the two questions I always get are, "How did you get buy-in from the business to invest in this? And how did you get buy-in from the individual teams to invest in this?" I guess there's no really easy answer or perfect way to do it and it's going to be so nuanced depending on the organization that you're within. But one of Shopify's core cultural values that were kind of taught from the beginning, is build for the long term. And it's really driven home that we're trying to build this business for a hundred years, not for 10 years. And so with that kind of mindset, it is easier to get buy-in for these short-term pain for long-term gain type projects. +**Beyang:** [laugh]. -**Kirsten Westeinde:** -I think as well, the developers that had been working in the core codebase had all been feeling this pain. And so they were eager for opportunities to make it better and easier to develop within. So it wasn't always easy. And to be honest with you, different components are at completely different points in their componentization journey based on how well their team has been able to prioritize this type of work. One thing that we did was we built this tool called Wedge, a lot of complexity under the hood, but it basically gave components scores as to how well they were doing on their componentization journey. And we made it like a little fun competition between teams. +**Max:** All right, that’s, you know, that’s a salary. -**Beyang Liu:** -You gamified it. +**Beyang:** Solve your money problems. Yeah. Yeah, yeah. -**Kirsten Westeinde:** -Exactly. Yeah. As a technical initiative that the whole engineering org was working towards. +**Max:** Well, it’s not a great salary. I think we have to give open-source developers good salaries, otherwise they’re not going to do open-source full-time. So— -**Beyang Liu:** -That's really interesting. So you kind of built a tool that condensed all these complex refactorings into something that you can measure, like a numerical score and that made it easier to track and also verify. How did you do that? How do you convert adding an interface or defining a good interface or eliminating a cyclical dependency or something into a number that is meaningful that the engineering teams don't shake their heads at and that also is useful for tracking it at a top level. +**Beyang:** Totally. Yeah. -**Kirsten Westeinde:** -Yeah. So, Wedge is a really interesting tool. What we did was we used it to hook into Ruby trace points during our test suite run to get a full call graph of all the calls that were being made as our code was being executed. And then we sorted the callers and callees by component and basically pulled out all of the calls that were happening across component boundaries and then sent them to Wedge. And we'd also send some other data like code analysis and active record associations and inheritance and such. And then Wedge basically just went through everything that was sent to it and determined which of these things were okay, and which ones were violating. So if a cross-component call was being made, if there was no dependency explicitly declared between those two components, it would be violating. And if it was anything other than the public interface it would be violating. +**Max:** —that’s part of what we’re trying to do. So yeah, like—but the thing is, it was buried in the stack, it was hard for people to find, I got a good sponsor once and they said that they went through all their dependencies and tried to find ones to look for, and, like, it was a three to four-week process for them to figure out how to sponsor me. Now, there is GitHub sponsorship now, but still, like, difficult to figure out your whole stack. It’s impossible. So, sponsorship doesn’t work, in my opinion. -**Kirsten Westeinde:** -We actually found though that it really is hard to get a number that is right. Especially with as much complexity as there is in call graph logging. So in the end we actually ended up canning Wedge because we didn't find it to be a useful feedback cycle. And also mostly because we had to run the full test suite to get this feedback and that takes way too long for it to be helpful. So we ended up going in a different direction. But in the early days it was really helpful to, like you said, gamify it. +Bounties are interesting and they will continue to exist for sure. Because you should be able to provide an incentive for someone to build the future you want on [unintelligible 00:52:58] open-source. But they still mostly only reward the top of the stack. Like, no one’s going to do a bounty on, like, some random logging library that sitting underneath all there, probably. -**Beyang Liu:** -Yeah. That makes a lot of sense. As you learn more about what was useful and productive, what kind of stepped into the place that Wedge occupied? Was there another tool that you used or was there some different set of criteria that you used to gauge overall progress and success? +**Beyang:** Sure. -**Kirsten Westeinde:** -Yeah. We ended up building this tool called Packwerk, which actually as of September of last year is open source. And it's a really cool tool that basically analyzes static constant references. So we found that there's less ambiguity in static references and because these references are always explicitly introduced by developers, it's more actionable to highlight them, and it's much faster. So we're able to run a full analysis on our largest codebase in a few minutes, which means that we can actually put it in as part of our pull request workflow, which is a way more helpful feedback loop for developers. +**Max:** So, it occurred to me that a tool like Homebrew knows the entire open-source ecosystem, it understands the graph, it understand—it sits underneath all of those tools. And because we understand the graph, we could thus, as a result, spread some, you know, token around. So, if something at the top of the stack—let’s say React—gets some token, we can feed a percentage to its dependencies, and then a percentage of that to all their dependencies as well, and thus, like, fund the entire graph. And I was like, “Oh, wow. This is an important understanding, an important realization.” -**Beyang Liu:** -That's really interesting. So the Wedge was this dynamic tool. It kind of built up this call graph or reference graph from runtime by observing what actually got called and then Packwerk was building up that graph, but statically based on imports and references in source code. Is that right? +I went and woke my girlfriend up because it was like two in the morning. I told her that I had just figured out how we could fund open-source software, the entire open-source ecosystem. She does not remember this. [laugh]— -**Kirsten Westeinde:** -Yeah, that's correct. Packwerk uses the same assumptions as Zeitwerk, which is the Rails code loader. And the way that Rails is able to make constants globally accessible is it basically infers the file location based on the constant's name. And we did the exact same thing with Packwerk. +**Beyang:** [laugh]. -**Beyang Liu:** -So the programming languages part of my brain is saying, "But wait a minute, Ruby is a dynamically typed language, not all types are known at compile time." But were you able to solve that problem in Packwerk? Like inferring types or was it some element of best effort? Like, this is good enough. +**Max:** —sadly. -**Kirsten Westeinde:** -Yeah. A hundred percent the latter. We understand that it's an imperfect solution and there's a lot of crazy stuff that can happen in Rails that will not be detected through this, but it's good enough essentially. And it does catch a lot of things. So we decided that the fact that it can happen so much faster and provide that faster feedback loop is a trade-off we're willing to accept for the fact that yes, maybe some things will slide under the radar. +**Beyang:** Maybe it was a dream, you know? It was late at night, and it was just you’re in a dream state. [laugh]. -**Beyang Liu:** -I guess it's important to recognize that there's humans in the loop in this process. So in some sense, it'd be a much harder problem to completely automate it because then you'd have to get everything precise. But as long as there's humans in the loop, they can step in and say, "Well, we recognize this tool's a little bit fuzzy, as long as the signal to noise ratio is decently high, we can work with this." +**Max:** [laugh]. Like—well, it’s a story all the same, now. [laugh].Even if it was. I guess it’s possible. It was late. Like, I’ve been staying up late just going through everything, trying to figure out what I could make into something that would be a startup or way to fund me working on open-source. And yeah, it was just a combination [crosstalk 00:54:34]— -**Kirsten Westeinde:** -Absolutely, and the magic of this tool is not only within the tool itself, but it's within the conversations that it sparks, because if on my PR I get told, this is a cross-component violation, and maybe I don't know what that is, or maybe I'm a more junior programmer, and I don't have an idea of how to do inversion of control, for example, there's been a lot more conversations around software architecture across Shopify because of this tool. +**Beyang:** So, just to clarify, the eureka moment was thinking about open-source funding and connecting some of the ideas in the Web3 space into, like, a token that kind of understands the structure of the dependency tree? Is that kind of the idea? -**Kirsten Westeinde:** -So, what that means is sometimes maybe the tool won't catch cross-component boundaries, but because it's now becoming a part of our engineering culture, maybe someone will in a PR review. Building that muscle of good software design really is the overarching goal of this whole thing. So, yes, the tool's not perfect, but it's been really valuable for us. +**Max:** Yes. -**Beyang Liu:** -You just mentioned inversion of control, and I think that along with some other things you called out in your blog post were good rules of thumb or good general principles that you may have discovered along the way of doing this. Can you talk about some of those principles or maybe empirical patterns, in terms of if someone were to attempt this on a Rails codebase in general, what are the maybe tactical things? Tactical things, tricks that you can apply to make the codebase more decomposed and modular. +**Beyang:** And then you, like, fund the token and that automatically spreads the funding to all parts of the tree, not just the most prominent— -**Kirsten Westeinde:** -Yeah. I think the first thing I would say is that if someone's going to be trying this on a Rails codebase, they'll probably be in a similar situation that we are, or that we were, which is basically, if you look at the dependency graph that has just naturally happened, it's crazy. Every component was basically depending on every other component. There's a ton of cyclic dependency. Rails lends itself naturally to have high dependency. +**Max:** Yes. -**Kirsten Westeinde:** -So, I'll say that first. Even just being able to visualize that dependency graph allows you to reason at a much higher level about, does it make sense for this thing to depend on this thing? Sometimes it does, and other times it doesn't, but when it doesn't, that's when you start to use some of these tactical approaches for decoupling those things. +**Beyang:** —or the most well-known name? -**Kirsten Westeinde:** -I mentioned inversion of control, which basically is saying ... So, I'll give an example. I work on a product called the Shopify Fulfillment Network, where, basically, some merchants, we host their products on their behalf. When orders come in, we'll choose the most optimal warehouse to fulfill it out of to get it to the buyer as quickly as possible. +**Max:** Exactly. -**Kirsten Westeinde:** -SFN, that's the acronym for the Shopify Fulfillment Network. It needs to know when an order comes in on a merchant store. One way to do that could be for the orders component to make a call to a method in SFN and say, "New order has been created," and then SFN responds with, like, "Great. We got it." +**Beyang:** Got it. -**Kirsten Westeinde:** -That's a hard dependency from the orders component to SFN, whereas if we want to get rid of that dependency, we can actually just use a method like a publish subscribe mechanism. There's a bunch of different ones to do that, but at Shopify, we use Kafka events a lot. Basically, the orders component can just say, like, "An order happened," and then SFN can choose to subscribe to that and do its processing without needing to necessarily reply, and that has now broken that dependency between the two. +**Max:** Like, yeah, the top is where people will probably still target their money, but we can filter it to all the other dependencies. And because of the way the law of exponentials works, if you’re a good dependency, you’re going to get a ton of those smaller percentages anyway so the funding will be relatively equal. And, like, a good part of what we’re talking about in our whitepaper is figuring out the math so that we are funding things in a fair and reasonable way, and, like, not over-funding things as well. Like, we don’t want—although I’m—there’s all sorts of interesting things that come out of what we’re trying to do. -**Beyang Liu:** -It's like, at run time, you want this one thing to provide behavior to this other thing that is going to run, but at compile time, you don't actually want an import going from that thing to this other thing, because that adds a hard dependency, and it means that someone working on the first thing will have to go through the second thing and understand how it behaves. There's not this interface boundary that says, "Okay, don't worry about the stuff behind this curtain." +One of the things is, so a token gets into the system, and then there’s a dependency. Let’s say, OpenSSL; it’s my favorite example. OpenSSL, it’s used by so much of the internet, very important security layer. And it’s also terrible. Every year, there’s a new exploit found that, like, causes huge problems for the entire internet. -**Kirsten Westeinde:** -Yeah. That's exactly right, and had there been a hard dependency, like, say, from the orders component to SFN, then the orders component would have to care every time SFN changed. Now, having done this inversion of control, it does not at all. It just has to fire its event. +With our system, we would actually be incentivizing people to replace OpenSSL. That’s never happened before. Open-source, typically things get inserted into this tower of open-source—we use the XKCD comic called “Dependency” a lot are met—you know, just to demonstrate to people what we’re talking about. And it has a tower of bricks like a toddler might make, and right near the bottom, there’s this tiny little pillar that seems to be holding up, like, 80% of the internet. And has an arrow pointing to it, and it says, “This dependency is maintained by some guy from Nebraska since 2004, and thanklessly.” -**Beyang Liu:** -Another thing that you talked about in the post was this kind of loose coupling, but high cohesion pattern that got adopted. Do you mind explaining what that means? +**Beyang:** Yeah, and has never realized a dime from the work that has gone into that. -**Kirsten Westeinde:** -Absolutely, yeah. This was one of the core principles of this project that we were striving towards. Coupling is basically the level of dependency between modules, or classes, or in this case, components. So, you want that to be low, because you want your dependency graph to be as light as possible, but you want the cohesion to be high. The cohesion is basically the measure of how much elements within a certain encapsulation belong together. So, whether you're looking at a module level, a class level, or the package level. +**Max:** Exactly. So, OpenSSL is famous, but there are examples of the Nebraska problem—that’s what we call it—and one was Log4j which had that security exploit in December. And a huge amount of the internet, a surprising amount, became suddenly zero-day exploitable. Like Minecraft, you could type something into the chat and it would root the server. Or you could, you know, you got access to do that if you wanted it to. -**Kirsten Westeinde:** -One thing to note is that it's really hard to get the component interface right when the classes within it have interfaces that are not well-defined, and the modules within it. It really starts with good software design from the bottom up. That makes it easier, a lot easier to get the component interfaces right. +And nobody had ever heard of this thing. It was one of those deep dependencies they didn’t even know they were using, something was built on top of it, which added a layer which made it nicer in some manner or whatever, and then like that was built into something else. And then people were angry. And there was a lot—you can look on Twitter; it was abuse. People were abusive towards these dev. -**Kirsten Westeinde:** -Then, on the point of cohesion, there’s two different types of cohesion that we think about. The first is functional cohesion, so that basically means code that performs the same task list together. You can think of service objects as being functionally cohesive, whereas data or informational cohesion is code that operates on the same object living together. +And they were like, “Look, we’re going to fix the bug. We’re sorry about the bug. We do care about this, but nobody’s ever given us a cent for making this thing so maybe we should fix that.” [laugh]. -**Kirsten Westeinde:** -Again, Rails really lends itself to data or informational cohesion, because of the way that active record models are built up to interact with one database model. We often add a lot of methods relating to that object on its model, even though they might be parts of completing completely different tasks. I'm sure different languages lend themselves to different ones, but we found ourselves way over-skewed in the direction of data and informational cohesion. +**Beyang:** Yeah. -**Beyang Liu:** -Where is this project today? You mentioned it started in 2017, and it's been a multi-years-long effort. Are things still being componentized in the main Rails codebase, or at this point, has it been componentized, and it's more or less in a done state, you just have to maintain the status quo? +**Max:** So, with Tea, that’s our objective. We’re making this trustless decentralized protocol which understands the open-source ecosystem. Effectively, the package manager’s registry, with Homebrew was a Git repository; with Tea, it’s the blockchain. It’s a blockchain. We’re putting all the packages on-chain as NFTs. It’s a good use of NFTs. -**Kirsten Westeinde:** -Yeah, I wish I could tell you we were in a done state, but the reality is that we're not. All of the code has been moved into components, and all of the components do have publicly defined entry points, but the reality was that we were starting from a place where a lot of these calls were already violating the rules. So, one of the nice features that Packwerk built was that you could essentially declare bankruptcy, and start from a certain point, and only start tracking violations beyond that. We have a list of deprecated references, and the rule is basically that you can't add any new violations. +We’re not exactly calling them NFTs at this point because they’ve got so much of a bad stigma, but an NFT is just a token, right? It’s just a non-fungible token. It doesn’t have to be a JPEG. Ours are representing package releases, a very good use for NFTs. I think over the years, people will start to see that NFT is just a utility technology, and it doesn’t mean that you’re getting a Bored Ape or something else, some sort of collectible jpg. -**Beyang Liu:** -Got it. +**Beyang:** Yeah. -**Kirsten Westeinde:** -We know that there are some existing ones, but no new ones can be added. Over time, that deprecated references list basically gives us a tech debt to-do list to work off of. Like I said, different components are in different shape, but we haven't yet gotten to the point where all of those violations are gone. Only once all of those violations are gone will we be able to actually enforce those boundaries. +**Max:** And so, the NFTs are going to be on-chain will guide how a token is distributed through that network, however we decide to insert it. Now, our whitepaper is coming out soon. The way that we’re going to have insertion happen is an inflationary staking mechanism. And I’m super excited about it because it means that we’re not fundamentally changing how open-source works. -**Beyang Liu:** -Fascinating. You have this list of tech debt to-do items. In a given iteration cycle, how do you decide how many of those to tackle versus how many, I guess, more product-oriented things to tackle? Because that's something that our teams at Sourcegraph, I assume, like probably any software team struggles with. What is the right balance of tech debt versus new feature development to take on? +People have asked me this a lot, and it’s good question: “Are you going to charge for open-source now?” The answer is no, not at all. You can’t charge for open-source. No one would accept that. [laugh]. You can’t take something that has worked a certain way for more than 20 years and fundamentally change it. -**Kirsten Westeinde:** -Yeah. That's a hard question. +So, Tea is free to use. Our protocol is going to be run by DAO, not by the company. It’s not funded by the—it’s not run by the funded company. Obviously, initially, a few of us at the company will have large voting rights because we’re going to make sure that it goes in the right direction, but over time, our voting rights will fall to be in line with the rest of the open-source community. So, people who make open-source will get voting rights by publishing into our chain. -**Beyang Liu:** -Can you put a price on a tech debt item? It's almost an impossible exercise. +And so, the chain itself is going to be managed by the open-source community, managed by open-source ecosystem. Tea, the package manager and the other tools that we’re going to build on top of it are just going to be clients onto that protocol. We’re hoping to make the best clients, but very much like Twitter used to be, where everyone and their son made their own Twitter app because the API was public, and it was very interesting, and like, people make better things all the time, we’re going to have the same ecosystem, except we don’t own the API like Twitter did. So, they just started closing it at one point and made it private. -**Kirsten Westeinde:** -Yeah, no, it's really a hard question. Honestly, the answer's going to be different team to team at Shopify. One of the patterns that I've seen in my years at Shopify is that we built a lot of features. We move really quickly leading up to Black Friday and Cyber Monday, because that's when our merchants make the vast majority of their sales for the year. So, it's really important that we've given them the tools that they need to be successful during that time. +**Beyang:** Okay. So, you’re going to have a decentralized protocol for managing package dependencies and then your client is going to be the equivalent of the Brew command-line tool, or like an apt-get where this will talk to that distributed chain and fetch the dependencies from that chain— -**Kirsten Westeinde:** -Come the holiday season, we actually tend to cool down a bit on feature development, just because we don't want to be breaking things when our merchants are having their most important sales of the year. So, often, that feature cooldown can be a good time to tackle some of these larger units of tech debt. That said, I'm not saying that we just leave tech debt to do at one point in the year. We definitely try to find some pause between projects, or even just parallel tracks of work, where some people are working on more technical debt, and some people are working on product features. It's definitely a hard balance, but we have been slowly chipping away at it. +**Max:** Yes. -**Beyang Liu:** -I understand that the Shopify Fulfillment Network, which you mentioned earlier, that's a separate codebase. It also is or was a Rails monolith. You're currently tackling a similar project in that codebase, but adjusting some things based on what you've learned from this first major... +**Beyang:** For purposes of installing an actual—okay, got it. And then you’ll also have a, sort of like, one-to-one mapping between an NFT-type token and each release of each package on this chain? -**Kirsten Westeinde:** -Decomposition. +**Max:** Uh-huh. -**Beyang Liu:** -... decomposition componentization effort. +**Beyang:** I guess one of my questions is, for these tokens, what gives them value? Is it just like in the kind of current NFT sense of, like, you don’t technically—well, I guess, some some NFTs have rights attached to them, right? Is that kind of idea? Are there any— -**Kirsten Westeinde:** -Exactly. +**Max:** Yeah. -**Beyang Liu:** -Tell me about round two. What's different this time in the sequel? +**Beyang:** Like, rights or privileges or— -**Kirsten Westeinde:** -Yeah. We actually took a pretty different philosophical approach. What we did in Core was we moved all of the code into the components, and then slowly, over time, started trying to break some of those dependencies, whereas what we've done in the Shopify Fulfillment Network was we've introduced separate components. We've actually used Rails engines, which are like the one modularity feature that comes out of the box with Rails, and that can allow us to enforce those boundaries a little bit more strictly. Then, over time, piece by piece, moved bits of code into it that are always respecting the boundaries. +**Max:** Oh yeah. -**Kirsten Westeinde:** -So, we flipped it on its head that way, whereas we still do have the main app controllers, models, etc., but the hard rule is we can't add any more code into the app folder. All new code goes into component folders. Over time, we're pulling code from the app folder out into its, basically, correct component, so that anything that's in a component is respecting the component boundaries, whereas that is not the case in Core. +**Beyang:** Like, a sense of ownership attached to owning the NFT associated with a particular open-source project in this world? -**Beyang Liu:** -Interesting, so, with Core, the approach was move all the files around into their appropriate directories, and then, over time, reduce the linkedness or monolithic-ness of each component, whereas with Fulfillment, you basically said, keep the existing monolith as is, but each new feature has to be added to something that's external in a Rails engine. Then, we'll have some connection between the two, so that the functionality makes it into the application, but we always preserve this, I guess, invariance because all of the new components are modularized. Is that about right? +**Max:** Yes. So, for it to be an NFT, it has to be transferable, and we actually expect people to do interesting things with that. Like, my chief product officer has suggested that people—someone will probably make a digital contract which allows you to lease a release of open-source. So, you can say, “Okay, I’m building this open-source package and I need, like, 100 grand now in order to build it.” It’s like, you know, writing a book and getting upfront stipend— -**Kirsten Westeinde:** -Yeah, that's right. One of the main benefits of that is that with so many developers working in a codebase, people tend to want to follow the patterns that they see existing. In Core, there is a mix of good patterns and bad patterns, because some stuff has been adjusted to break that interdependency, and some stuff hasn't. So, it's hard to know which pattern to follow, which way is the right way, when there are both examples, whereas in SFN, anything that has been componentized is following the patterns that we're wanting to strive for, so there's more good examples to be able to learn from, and demonstrate to other developers, and follow. +**Beyang:** An advance. -**Beyang Liu:** -I get all the learnings here. This is really fantastic insight. One thing I'd love to hear about is like, what is the process on the ground that you arrive at these insights? Like, for this new project, we don't want to move a bunch of stuff in the existing codebase around, but we want to start with something new, or moving from the dynamic dependency call graph to the static version, what are those conversations like? Is it you and a bunch of other engineers or engineering managers in a room, whiteboarding, or someone coming up with a bright idea, and slowly convincing other people? Are there any, I guess, war stories that come to mind when you reach these insights? +**Max:** Yeah and advance. Exactly. And so, they’ll say, “Okay, for 12 months, you own the NFT or you’re leasing it from me.” And this is the beauty of digital contracts, right? You can—it’s a way to build law into the money system. -**Kirsten Westeinde:** -Yeah. It's always a conclusion that we come to over time. In the example of Wedge, it was working well for a while, but over time, the problems with it became louder, and louder, and louder until we couldn't ignore them. So with that one, it was really, as our test suite was getting larger and slower, it was really just that it wasn't a helpful feedback loop to have to wait for that entire test suite to run to be able to get that insight. And so we knew there was a problem with that. But then the question is, what's the solution? +And so, “Someone else is going to do this. We’re not going to do this. We’re just suggesting someone else build this system.” And so, they’ll be able to get 100 grand of token upfront and then the person who’s leasing the NFT gets all the profits off of the use. -**Kirsten Westeinde:** -So we have a few different avenues for brainstorming these ideas. One is actually a team whose mandate is architecture patterns, and they're the ones that built Packwerk and provided a lot of the patterns to follow. And so sometimes it's just projects that get resourced, and we do explore and prototype phases. And so being armed with the learnings from Wedge, it led us down the road to Packwerk. +But the NFTs themselves are not the primary way that money is going to be made. We’re going to use the NFTs as, like, a map. So, they’re like signposts that show the protocol how to filter and funnel and direct token that enters it. The primary way that token—I expect—to enter it—and this is in the whitepaper, by the time this podcast is released, we should have released the whitepaper so I can go into it. -**Kirsten Westeinde:** -But the other thing is we have a software architecture guild at Shopify, which is basically anyone from the company who is interested in software architecture. We have a Slack channel and a bi-weekly meetup. And that's where a lot of these conversations come to a head. Because I think there were a lot of learnings from Core, but then for example, the developers who were starting to build out SFN may not have had access to that frontline information. +**Beyang:** Cool. We’ll link to it in the [show notes 01:03:12]. -**Kirsten Westeinde:** -And so we did a lot of pairing actually as a way to share this knowledge and have a lot of discussions with the architecture guild. Because Shopify is very, I guess each team has a lot of autonomy. So the SFN team had the autonomy to decide what the right solution for it was, but definitely wanted to leverage the learnings that were present from Core. So that's the type of discussion that we would have at an architecture guild. Or we also do technical design reviews for any big tech projects we're kicking off. So that would be another. We have templates for those, and we have filled those templates with some prompting and questions to try to make people think about modularization as part of the upfront design. +**Max:** Yeah. But we’re using a proof-of-stake chain, and the proof-of-stake system creates new token periodically at epochs. So, it’s an inflationary system. This new token is what we’re going to funnel into the open-source ecosystem. So, it’s an inflationary system. So, every epoch, we’ll give some token to the people that stake in to reward them for staking their token and securing the network. Because what you stake it, it means that you cannot spend it, you cannot use that token. You’re putting it into a digital contract that gives you 24 hours, at least, to withdraw it. It’s like a bank saying you don’t have your money for 24 hours. -**Beyang Liu:** -Interesting. Tell me more about this guild. So what is a guild exactly? Is it just a collection of individuals who have a common technical interest area? Or how is it organized? Is there a process associated with it? How does it fit into the overall org structure of the engineering team? +**Beyang:** Sure. -**Kirsten Westeinde:** -Yeah. Actually it doesn't fit into the org structure of the engineering team. It's composed of people from across everywhere in the engineering organization. It's completely opt-in, and it grew organically from people who were curious to be having conversations like this. I think one big learning actually that my coworker Phillip called out in his blog post was that he wished that we had started this sooner in the componentization process. So more minds could be involved and had buy-in basically for them to apply the approach that we were aligning upon in their different areas of the company. +**Max:** And this means that the price fluctuates less so it’s good for the system. So, we reward them for doing that by giving them a bit of a token, but we’re also going to give, like, a percentage—probably half or 60%, like, we haven’t got the exact figure yet—to the graph, the open-source ecosystem graph. And the people staking would choose the packages that they’re staking against, and we’re going to incentivize staking ones that are less staked because we don’t want everyone to just stake their favorites; we want people to actively find new exciting open-source projects. So, when you find it, you’ll get a better reward initially. -**Kirsten Westeinde:** -But it really is just people that want to nerd out about software architecture, and people come with presentations every other week. Sometimes we'll just share the architecture of one of our systems. Sometimes we'll have more pointed discussions about certain software design strategies. It's pretty organic, but people like it. +**Beyang:** Interesting. And what is a reward here? Like, is it— -**Beyang Liu:** -That's awesome. So the guild formed after the start of the original project. +**Max:** The token. -**Kirsten Westeinde:** -Yeah. That's right. +**Beyang:** Is it mainly just being—the token? And the token allows you to say that I am sponsoring this project? -**Beyang Liu:** -At what stage do you think it makes sense to form a guild? I think that we run into the same issue, and I think a lot of other engineering organizations run into this issue, where you have your org chart and you try your best to produce an org chart that reflects the needs of the product and how knowledge and information needs to flow. But no chart is ever going to be perfect, and there's all these areas of expertise that don't neatly fit into that. And I guess the question in my mind has always been, at what point do you start to need these guild structures? Is it at 100 engineers? Is it at 1,000 engineers? Or what, in your mind, is the right size? +**Max:** Yes. -**Kirsten Westeinde:** -Yeah. We've actually experimented with a few different things. The guild is more of a meetup. We less drive change through the guild, I would say, other than grassroots movements. But we also have what we call a technical leadership team, which is on a rotational basis. Technical leaders from across the company will be on that team. And anytime we're doing a tech design review, you'll get paired up with someone from the technical leadership team. You can bring them your gnarliest problems, and it's made up of people from all different parts of the org with all different perspectives on the problem. +**Beyang:** Like, I am contributing to—okay, got it. -**Kirsten Westeinde:** -So it's a way to try to, I guess, make sure that we're leveraging the learnings that are available at the company level. And then there's also what we were chatting about before, I forget what it's... the tools and patterns team. And their actual mandate and what they work on day in and day out is building out some of these tools and patterns. +**Max:** The token is governance so you will be able to help govern how this entire system works. And we’re going to give governance priority to open-source packages so if you have some actual NFTs in the graph that are being staked, we’re calling it ‘steeping’ incidentally, for this particular part. -**Kirsten Westeinde:** -And I think I would guess that team spun up probably in 2018 just based on this project having started in 2017. And yeah, we were probably 500 to 600 engineers around then. It's going to be different for every organization, and I think it depends on how much your organization already has it baked into their DNA to be thinking about these things, to be thinking about good software design, technical initiatives, etc., or not. Sometimes it might make sense to start earlier if that's not as organically present. +**Beyang:** [laugh]. I love the terminology. -**Kirsten Westeinde:** -There's trade-offs to each approach. One of the challenges that the tools and patterns team has is that they're just building tools and patterns, but they aren't necessarily using those tools and patterns to build a product. So they need to make sure that they are in touch with the people that are using those tools and patterns and actually solving the right problems. And so we do that through pairing and internal documentation presentations, etc. But we really try to make sure that the problems that need to be solved are being solved, and we're not just adding friction to developers' lives because we think it's nicer. +**Max:** It’s nice and the metaphor works really well. So, we have staking and steeping in our system. The whitepaper will go into a lot more detail here. -**Beyang Liu:** -Yeah. That makes a ton of sense. Looking back on both projects that you've been involved in with decomposition, to what extent are the patterns that you discovered, do you think, specific to those particular codebases? And to what extent do you feel like there's some general principles here that would be widely applicable? That's one question. A more proximal question would be the way that you're approaching modularization on the fulfillment network codebase. Do you wish that you had done that with the main codebase initially, knowing what you know now? Or do you think that each codebase found its proper approach? +**Beyang:** Got it. -**Kirsten Westeinde:** -Yeah. It's a challenging question because they both have their trade-offs. So I think that you just need to know in your situation which trade-offs you're willing to accept. For the Shopify Core case, what we were really struggling with and really wanting to optimize for was that we had a lot of code that didn't have ownership. And so having moved the code into folders that have explicit owners is a big win for that. Whereas in SFN, we still have a lot of code living in the app component that doesn't have as clear owners, but for us that's okay because it's a smaller group of people working on it. And so it's easier to share the load across our team, which is smaller. Whereas for core, it was really important that we get it right in terms of code ownership. +**Max:** But in order to, like, make sure that everything is correct, like, obviously, this is easy to game; we’re going to have a slashing mechanism where we can take away your stake. And we’ll do this if packages have security exploits, for example. So, we’re going to incentivize, like, properly incentivize the open-source ecosystem to make sure that their software doesn’t have security holes. So— -**Kirsten Westeinde:** -So I think you really have to ask yourself, what is the intermediate state that you're most comfortable with? Because these things take a long time. And so I think Phillip has an amazing quote in his blog post. He says, "My experience tells me that a temporary incomplete state will at least last longer than you expect. So choose an approach based on which intermediate state is most useful for your situation." And it's 100% true in this case. Both of those codebases are in intermediate states that are correct for them, and I think you just need to think about what that means for your codebase. +**Beyang:** Interesting. -**Beyang Liu:** -Yeah. That's very fair. Somewhere out there there's a person listening who wants to propose a big refactoring re-architecture project, but you often run into resistance from Product, from other stakeholders in the business because the value of such a project is not always clear. What advice would you have for people in that position, just in the very early stages of thinking through starting such a project and getting organizational buy-in? +**Max:** If a package is found have a security issue, someone can report that to the DAO. So, the DAO is a Decentralized Autonomous Organization. It’s just a governance structure that runs the protocol. If the DAO decides that it’s a legitimate security issue, we can slash that package and the people who is staking it. So, this means that if you—you get more rewards for staking new packages, but if they’re found to be, you know, either fake, for example, like, people—it isn’t a real open-source project, or there’s security issues, it’s not good quality, then you could lose your stake. -**Kirsten Westeinde:** -Yeah. One thing that I think about often as it relates to this is, I don't know if you're familiar with Martin Fowler's design stamina hypothesis, but it basically says that in the beginning— +So, you have to make sure you’re staking the right things. So, we’re effectively making—well for a start the creator economy for open-source, but we’re also making a system where the people are actively involved in making sure open-source continues to be great and is greater and greater going forwards, as well as allowing the people who make this essential infrastructure that builds the internet, powers the internet, makes everything that happens on the internet possible, are able to do that as a job. And they don’t have to go and find someone at Facebook, who will, like, “Okay, well 70% of the time you can work on this because it’s useful, but we also want it to do this,” or, you know, beg for money, effectively, on Patreon, or find bounties. Like, I approve of bounties and I think Gitcoin is going to be a wonderful complement to what we’re doing—Gitcoin is a Web3 bounties program—but it does introduce [agenda 01:07:26]. And you shouldn’t have to chase bounties from people who are then shaping how your product goes. -**Beyang Liu:** -I read that in your post. +**Beyang:** Totally. -**Kirsten Westeinde:** -Yeah, I love it. I talk about it all the time. It basically says that in the beginning, no design is actually the best design, and you'll be able to move more quickly without design. But at some point you're going to reach a point where not having a design is going to slow you down more and more and more, and adding incremental functionality just gets harder and harder. +**Max:** The open-source community should shape how open-source goes, in a transparent and fair way. And I believe what we’re going to build is going to achieve that, and it’s very exciting. -**Kirsten Westeinde:** -So the first question that I would ask is, are we past that point? He calls it the design payoff line. Are we really at a point where we need this design? Because honestly, it's worse to have a bad design than to have no design, in my opinion. So you want to make sure that you have enough information and have built enough features to understand what design your codebase needs. So that's the first thing. +**Beyang:** Awesome. So, you know, we are almost out of time. I wish we had another hour to go into all the intricacies of this plan, but for those listening who want to get involved in this, you know, new package manager, this way of funding open-source, this really interesting take on Web3, from both a, I would say, like, maintainer perspective and an open-source user perspective, what’s the best way to get involved? -**Kirsten Westeinde:** -But if you think that you are in that position, I always find that the business and the product managers are very motivated by data. So the more data that you can capture, the better. So I don't know that your organization uses story points, but say you could point to an issue of story point three that used to take half a day. Now it takes a day-and-a-half. Anything that you can point to to show that you are getting slower is a really good starting point. +**Max:** Yeah, so follow us on Twitter and follow us on GitHub. I haven’t open-sourced the package manager yet. I never open-source it before it has enough functionality because otherwise people get excited and then they lose interest waiting for it to be something. -**Kirsten Westeinde:** -Often they will have been feeling similar things. It probably won't come out of left field to them. They might feel like projects are taking longer than they should and are harder than they should. So the more you socialize the idea of, "X would be easier if we did this" and really just keep being the squeaky wheel, that can be really helpful. +So, I’m open-sourcing it hopefully in May. That’s the plan. The only reason I wouldn’t is it just doesn’t quite get ready—it’s not ready yet, or some other reason, like, there is a company behind the package manager so I have to have considerations there. So, at GitHub, we’re going to have our [test net 01:08:52] up in autumn. So, that will be run by the DAO, but company is building it initially. -**Kirsten Westeinde:** -The other thing too is we would absolutely never have gotten buy-in to stop feature development and do this. So if you can do it in an incremental way that you can chip away at over time, it'll be much more likely that you can get buy-in. And if you're in the lucky situation where your company has enough engineers to be working on parallel tracks of work, then if some people can still be building features while some people do this, it's going to be a lot easier of a pill to swallow for the business. +Whitepaper’s coming out in April so I—you know, soon. So hopefully, we’ll—everyone listening to this will read it and then give us genuine feedback. Like, this is to make a genuinely open, transparent, fair, open-source remuneration and governance system. If it doesn’t achieve those things then we failed and I want you to tell me before we build it, preferably. -**Beyang Liu:** -Yeah. That makes a lot of sense. Kirsten, thanks so much for taking the time today. If there's people listening to this and they're interested in learning more about this effort or more about projects and learnings that you've undertaken, where should they go on the internet to find out more and learn more? +**Beyang:** And then for those listening on audio only, your Twitter handle is @teaxyz\_, right? -**Kirsten Westeinde:** -Well, Shopify has an engineering blog where we've published a few different blog posts on this topic. So I would definitely start with checking those out. We also have a ShipIt Presents new YouTube series about some of the engineering efforts at Shopify, and there's an episode that speaks about this as well. So check those out. And there's lots of other interesting non-componentization-related stuff on the shelf for the engineering blog. So I would say give that a look. +**Max:** Yes. Unfortunately, Twitter being as old as it is now, you can’t get good handles anymore, so [laugh] put the underscore in the end there. But yeah, you can go to tea.xyz and it has all our links. -**Beyang Liu:** -All right. My guest today has been Kirsten Westeinde. Kirsten, thanks for being on the show. +**Beyang:** Cool. Awesome. Max, thanks so much for coming on the show. This was an awesome conversation and I really appreciate you taking the time. -**Kirsten Westeinde:** -Thanks for having me. It was fun. +**Max:** Great. Well, it’s been really good to talk through everything that we’re building. Thanks for having us on. diff --git a/website/src/components/podcast/PodcastSubscribeLinks.tsx b/website/src/components/podcast/PodcastSubscribeLinks.tsx index b11b9d803ec..5ea80f8a0da 100644 --- a/website/src/components/podcast/PodcastSubscribeLinks.tsx +++ b/website/src/components/podcast/PodcastSubscribeLinks.tsx @@ -8,6 +8,13 @@ interface Props { export const PodcastSubscribeLinks: React.FunctionComponent = ({ className = '', linkClassName = '' }) => (
Subscribe:{' '} + + YouTube + = ({ classNam > Google - - YouTube - RSS