Skip to content

Book1

Jump to bottom
Ben Christel edited this page Oct 8, 2022 · 22 revisions

Software Application Engineering

How to Write Code You Know Will Work

What is this book about?

Engineering is the application of science and mathematics to the design of practical things. This book is about engineering software applications—software that people can use to solve particular problems.

The term software engineering is a controversial one in 2022. Programmers, and others involved in the software trade, sometimes muse doubtfully about whether software engineering is a "real" engineering field. I contend that, while much of the software-making that takes place today is not engineering, some of it is, and software could indeed become an engineering discipline in the not-too-distant future. I hope this book will give you a good sense of what such a shift might look like, and get you excited about being involved in it.

Some people worry that if programming becomes engineering, all the fun will be taken out of it. I certainly hope not—and I don't think that's likely, anyway. At "worst," the superficial fun will be replaced by a much deeper joy. The worry seems to come from the idea—unfortunately reinforced by too much of the STEM curriculum in schools today—that math, science, and engineering are dry, soulless disciplines, lovable only by people who want to think like machines. That simply isn't accurate. Science and mathematics are, at their heart, the investigation of reality by engaged and curious minds—investigation that is made much easier by an appreciation of beauty. Nor is reality itself depressing. While twentieth-century philosophy has left us with the idea that reality is fundamentally dismal and inhuman, closer investigation reveals this view to be wrong. There is nothing inherently dismal about reality, and a deep understanding of software has the power to show you that, through quasi-mystical insight. Once you grok that insight, science and mathematics become a window through which you can glimpse the awe-inspiring and inexpressible metapattern that generates all experience. So don't worry!

Software is already an engineering field! You just need to do X!

The software trade abounds with principles and "best practices." Ideas like ObjectOrientedProgramming, FunctionalProgramming, SoftwareTesting, Mocking, and Refactoring are part of the common vocabulary of programmers today. I don't see this book as contradicting or challenging these ideas—but you might.

The fact is, a lot of the information on the Web about software development "best practices" is oversimplified or phrased in a way that's easy to misinterpret.

I see ideas like OOP, not as rules or comprehensive methods, but as techniques that work in certain contexts and address specific problems. If you learned these techniques from a more dogmatic source (or one that was trying to sell you a particular technology) you might find that I'm challenging your beliefs. You might not think that what I'm describing could really work in practice. But the fact is, the techniques I describe in this book really have worked, in large, complex codebases that I've worked on professionally, and they've enabled me to solve problems that I was unable to solve without them.

But don't take my word for it—read the sources I list in the bibliography.

Limitations

I call this book Software Application Engineering because I do not have enough experience with systems programming to feel confident saying anything about it, or giving examples from it. The ideas and techniques in this book may not be applicable to, say, the engineering of operating systems. That said, I would be very surprised if they were not applicable at all to such systems. Most likely they will just require adaptation or reinterpretation—reading between the lines, if you will. Principles still more general than the ones in this book might someday elucidate a comprehensive theory of software engineering, one that includes both applications and systems programming. That will have to wait for another book, though.

The Goal

Code that DoesWhatYouIntend.

This is an easier goal to agree on than code that is "correct" or "high-quality". What "correct" and "high-quality" mean will depend on your context. But we can all agree that if code doesn't do what we intended it to do, it's no good.

Why is this hard?

TheTaskOfProgramming

No one pays for code for code's sake—they pay for the behavior of the running system. Therefore, our job as programmers is to shape our SoftwareSystem's Behavior. In general, the behavior is an infinite set of desirable Interactions. We're tasked with sculpting an infinitely large shape in hyperspace, using finite means—finite code and finite brainpower.

It's easy to lose sight of this while coding, because code is easy to change. The risk of making the wrong change should give us pause—though too often it doesn't. It's easy to forge ahead with false confidence, thinking (or more soberly, hoping) that things will just work out. The reason this seems easy is that obtaining true confidence seems prohibitively hard. It would be nice if our systems had the simplicity and lucidity of textbook examples, but they don't, so we make do.

The Means

What we seek is a workable mental model of our software—what PeterNaur in ProgrammingAsTheoryBuilding calls a theory of the software. Actually, we need several mental models. Each model will give us a wrong or incomplete impression of some details, so having multiple models is necessary to fill the gaps.

A set of techniques that can help solve problems that you might run into while programming. Emphasis on you. Writing code that does what you intend is largely a matter of the experience that you have while programming.

Example problems: Some behavior of your code is hard to test. The test output is hard to understand. You get lost while navigating the codebase and trying to understand what calls what. There are techniques that address all of these problems.

This book is not a set of rules to follow. Not everything in here is appropriate for every situation. Some of it is only appropriate very rarely. The point is not to do what the book says, the point is to use these techniques if and when they improve your experience of programming.

The Shape of the Solution

Though the Wholeness of the system is Ineffable (not amenable to rule-based explication) that doesn't mean it can't be communicated or understood. We can use multiple complementary MentalModels (36Views) to communicate about, and eventually reach a shared understanding of, what we can't explicitly teach.

Mental Models of Programs

Sources of Confidence

  • InformalReasoning - c.f. OutOfTheTarPit
    • GregWilson cited some reasearch showing that reading the code finds more bugs per hour than testing
  • SoftwareTesting - passing tests make us more confident
  • AlgebraicType - proofs of some kinds of internal consistency, ruling out many errors that could happen in a program with dynamic types. If our typechecker outputs no errors, that makes us more confident.
    • easiest to see in a language where we can do an apples-to-apples comparison of typed and untyped forms, e.g. TypeScript vs. JavaScript.
  • CompositionalReasoning with AlgebraicProperties (i.e. "semi-formal" reasoning)

How the Confidence Sources complement each other

  • InformalReasoning
  • SoftwareTesting
    • Flaw: spot-checking is not a proof.
    • Flaw: process-external Effects are hard to test
    • Flaw: duplicate test coverage makes failures hard to interpret and coverage hard to analyze. The opposite, "over-mocking," leads to situations where all the tests pass but the system as a whole doesn't work.
    • Flaw: you're not the Oracle
      • e.g. you need to call an API that returns some complicated data. It's not clearly documented and you misinterpret the meaning of one of the fields when creating a Stub of the API. So your UnitTests pass, but the system as a whole is wrong.
      • Partial fix: ensure you only make this mistake once by transforming incoming data into a form that's self-documenting or otherwise well-documented, and hard to misuse. I.e. ParseDontValidate.
    • Summary: testing is complemented by TDD (writing the simplest code that passes the tests), an architecture that pushes effects to the boundaries or abstracts them behind contracts with convenient algebraic properties, a shallowly layered architecture, and a discipline of understanding the data that's passed across architectural boundaries.
  • AlgebraicType
    • Flaw: certain generic interfaces are very difficult or impossible to express in certain type systems. E.g. generic variadic functions.
      • This is a shortcoming of current, specific type system technologies, not the mathematics of types
      • Even proponents of dynamic typing rely on the idea of types to make sense of their programs—they just don't have automated tools to check their thinking.
      • Possible resolution: something like Clojure's spec? Then you can't write fewer tests, though.
    • Summary: algebraic types are complemented by tests.
  • AlgebraicProperty and CompositionalReasoning
    • Flaw: error propagation threatens the simplicity of algebraic properties when the implementor has process-external Effects.

Programming Tactics

The True Goal

The first goal is the sine qua non of the second.

A sense of oneness with your work. "Oneness" isn't quite the right word, and there are many possible near-synonyms: connectedness, identity, care, kindness. Oneness involves:

  • a feeling of Mastery: you are confident that you can get the code to do what you need it to do. When you get a feature request, you can often develop plans of action that will work.
  • a feeling of Compassion toward your fellow programmers, the users of your software, and the authors of your dependencies. Compassion also involves doing things that help your coworkers achieve oneness with their work (e.g. writing understandable code with APIs that are hard to misuse).
  • a sense of responsibility. If the code's Messy or has a Bug, you take it seriously and fix it. This kind of responsibility can't be forced on you by others. You assume it, naturally and inevitably, when you care about the code and the people it affects. If the other qualities of oneness are present, you'll find it easy to fix any problems you cause, so the responsibility won't be a burden.
  • non-Instrumentality. Instrumentality often appears when you try to get something for free, without putting time or attention into it, or identifying with it. E.g. suppose you use someone else's code that you don't understand very well to try to make a job easier. If what you try doesn't work, it's easy to get frustrated and blame the code or the other person, which causes everyone to suffer. An attitude of non-instrumentality both:
    • recognizes that you may have to put some learning effort in to get the benefit of using the code. Nothing is free.
    • is willing to let go of a dependency on bad code and reimplement the functionality, if that's the pragmatic option.
  • continuous attention and course-correction as you work.
Clone this wiki locally