This project is a proof of concept for demonstrating the potential power and capacity of an open docs-as-code content-management framework, namely one based on the LiquiDoc CMF (LDCMF) standard being developed in unison with this initial use-case project.
“ACMEdoc” is a parody project name, and this entire repo is for purposes of modeling and evaluating some big ideas for keeping documentation source close to the software it references.
LiquiDoc CMF is a way to get technical documentarians working more closely to developers, in terms of virtual space (same or embedded code repositories), types of tools (open source apps and flat files), and workflow (Git and issue-tracking systems). And ACMEdoc is an example of how all this comes together, documenting the power of LDCMF by describing how the ACMEdoc artifacts themselves are sourced and generated.
ACMEdoc is a phony product with real documentation. It’s kind of a wrapper for LiquiDoc CMF, simultaneously demonstrating and describing a way to quite fully implement the CMF’s various features and capabilities.
Unlike a traditional component content management system (CCMS), LiquiDoc CMF will be an open framework for developing documentation architecture. That is, LDCMF is intended to be a set of rules and tools that establish a standard between documentation projects, as well as helping bootstrap those projects. The framework is little more than a convention for how to arrange data and content in source files in ways that can be readily integrated with broader product development and release processes.
This framework will hopefully be flexible enough to accommodate a huge range of docs challenges faced by complex product demands, all without constraining implementers and users into proprietary or overly restrictive alternatives.
A docs framework like LiquiDoc CMF is best suited for product-integrated documentation, but any complex docs project can take advantage of its tooling and conventions. Integrated documentation is not just stored in the same repository as the source code for the product it documents, it actually shares source code and/or data with the product itself. We’ll explore this concept in some depth here with ACMEdoc.
The truth is, most small, straightforward software projects can be well served by existing docs-as-code solutions like ReadTheDocs, MkDocs. Heck, any simple static site generator can handle 90% of the software documentation projects that can currently be addressed in a docs-as-code environment. For the more complex edge cases, with messy repos or highly conditional, interrelated products/editions, a truly open-ended, open-source solution is needed.
The intention for LiquiDoc CMF and this ACMEdoc demo project is to prove out a docs-as-code approach for solving highly complex documentation problems, even if the modestly scaled project at hand (ACMEdoc) is far more complex than it is large. The goal is to make ACMEdoc as complicated to document as possible, then solving it all with open standards and free, open-source tools.
Most of the real-world obstacles faced by our documentation infrastructure have to do with the tendency for source divergence, so ACMEdoc focuses heavily on solving source-divergence challenges. In the next section, we’ll look at the range of divergence points we will simulate and document here.
Additionally, this repository will be used to model several features of the underlying LiquiDoc and Asciidoctor tools to address several additional problems commonly encountered by product teams. These include managing release history and dependency licensing, as well as syncing docs into the product’s continuous integration (CI) routine.
You can imagine any range of complicating factors that cause us to want to split our output. Different audiences (users, admins, developers, etc), different product versions (1.0, 1.1, etc), different product editions (basic, premium, cloud, etc) — you name it, there’s at least a temptation to create and serve more than one document or docset.
And every time we create more than one version of our docs output, we are tempted to duplicate at least some of our source matter. This in turn risks content divergence where it should remain identical.
Each divergence point adds a dimensional challenge to our documentation. Just as source divergence is a core concern of developers, so too does it fluster technical documentarians. Engineers have found various ways to keep their source code canonical and “DRY” (“don’t repeat yourself”). Documentarians need to adopt these methods, partly to keep up, but also to engage our subject-matter experts (SMEs) — the developers themselves — in maintaining the docs.
With ACMEdoc, we will attempt to document LiquiDoc CMF as a framework, of course using that very framework. This reflects the product development concept of “eating our own dog food” — we will use our “product” to document itself, even if the product is intended to handle use cases more complicated than our own. If that’s meta enough for you, let’s dig into the “dimensions” we want this framework to be able to handle and thus will model in this repo.
Enterprise applications have different user roles that usually require different docs. Consumer apps often have tiers that effectively split the audience into “premium” vs “basic” users or other such divisions. Whenever different types of users should be shown different content about the same product, the user-roles dimension is in play.
|
Note
|
How ACMEdoc is multi-role
Our generated docs will be oriented toward two somewhat distinct audiences: people interested in extending and contributing to ACMEdoc as *developers* on one hand, and people simply interested in implementing it as *users*.
ACMEdoc’s docs will be separated into user and developer guides, each with overlapping content but addressed to a distinct audience.
Additionally, a third guide will be generated, this one for *fabulous* users.
Parallel to some products' Premium Users tier, this free Fabulous tier will include additional material, mostly in the form of pretty pictures.
|
Any time the same documentation needs to appear in two or more output formats, a dimension is spawned. This was far more common when every product needed to maintain PDF and HTML versions of their docs. That use case is still fairly common, however, and usually the source for each output format must be distinct in a few ways due to the inherent constraints of each. ACMEdoc addresses these differences without duplicating source.
|
Note
|
Output formats of ACMEdoc’s docs
We will maintain a wiki-like static website that encourages users to submit contributions, as well as PDF and one-page HTML editions of that nearly exact content.
|
It goes without saying that whenever you produce documentation artifacts that cover different subjects but share some common material, you are adding a dimension to your docs output. In practice this idea usually tracks closely with audience roles, but a mature project may be served by special “Advanced” or “Beginner” guides for the same role.
In other cases, publications will differ entirely, possibly even by media, such as a slide deck product walkthrough or perhaps marketing materials that appear on a different part of the company’s website, with delivery needs all their own. These create distinct artifacts that require distinctive handling.
|
Note
|
Multiple ACMEdoc docs
In addition to a static website and PDF user and developer guides, ACMEdoc will produce an introductory slide deck.
We will also post separate material to a different repo, such as a product “landing page”.
|
This one is so obvious, most companies consider it a legitimate point of source divergence. But if your products are heavily interdependent or share common elements, their docs should be at least as tied together. (One could argue in such cases docs are the element that most ties disparate projects together.)
|
Note
|
Product divergence in ACMEdoc
It would be pretty difficult to split a project as simple as ACMEdoc into multiple “products”.
Once we are ready to demonstrate continuous integration (CI), we can pretend there is a “cloud edition” of ACMEdoc and document that as a separate “product”, since CI is not strictly a feature of this framework, per se.
See Continuous Integration for more.
|
This one is pretty straightforward, as well as a major driver of the docs-as-code trend. Engineers have gotten extremely good at tracking their code with the release versions it produces. Keeping all source code in the same or a parallel repository is critical to keeping all product interfaces synchronized, including the docs.
|
Note
|
Version tracking in ACMEdoc
Because ACMEdoc is heavily dependent on LiquiDoc, we will tie ACMEdoc updates to LiquiDoc releases, coordinating which versions of LiquiDoc are required for and supported by each version of ACMEdoc.
This will be reflected in ACMEdoc’s documentation.
|
Are some of your docs managed in DITA and some in Markdown? Do your engineers have a treasure trove of JSON and XML files detailing the project that you have to copy from and reformat in order to convey product info to end users? You’re experiencing source format divergence, and ACMEdoc is your ticket to unifying your docs source.
|
Note
|
Source convergence in ACMEdoc
We will purposely simulate “legacy” source formats for some auxiliary content in order to demonstrate the powers of conversion and manipulation.
|
Beyond our original dilemma of multiple docs for users vs developers, any software team that is not transparent likely has documentation that is prevate to members of its organization. These docs could be as simple as employee policies, and so forth, and they may not tie to the product per se. It may well make sense to integrate policy docs into the same content system as the product.
|
Note
|
Internal docs for ACMEdoc
We will add a “private” section to our site and produce a distinct artifact reflecting internal operations and procedural policies, even though we have no actual operations or procedures.
|
Beyond of the core capacities of the ACMEdoc framework, this repo will model several best practices and automation solutions that can make a documentarian’s life remarkably easier. The framework will also incorporate some kind of a general API, as documented in the “Developer” editions of this site documentation.
Put simply, this task can be a huge burden. Release notes and changelog items need to be coordinated with a team’s task/issue tracker, where they originate. Since every reasonable issue-tracking system has a REST API, we’ll use LiquiDoc’s (upcoming) REST API capabilities to draw content from that system.
|
Note
|
ACMEdoc release history
To demonstrate this, we will sync the ACMEdoc release history up to its own or LiquiDoc’s GitHub Issues database, in order to draw source material for publishing in our ACMEdoc artifacts for this repo.
|
If your product packages third-party dependencies bearing license-notification requirements, compliance is mandatory. It’s not just the legal thing to do, it’s literally the least we can do to repay the hackers who work hard to develop, release, and maintain the software that we get use out of, if not profits from. For big projects, there are bad ways to do dependency management, and there are terrible ways to do it. We’ll advise and instruct the least painful way we know, including tooling and workflow examples.
|
Note
|
ACMEdoc dependency licenses
ACMEdoc’s plain-text NOTICE file is used as the source for a rich-text edition in the ACMEdoc artifacts.
|
Possibly more than any other aspect of a project like this, continuous integration is heavily dependent on the product’s own DevOps/CI tooling/process. Since ACMEdoc is intended to be an element of a broader product develoment process, CI means being conveniently useful to build systems (such as Gradle, Make, or Rake) and deployment systems (such as Travis CI, Jenkins, and CircleCI).
The holy grail of source code, and thus of docs-as-code, is to never include the same chunk of code twice: “Don’t Repeat Yourself” or DRY. This extends to not including unprocessed source in the same repository with anything it produces. We’ll explore the ins and outs of, as well as some caveats to, this possibly unachievable ideal all good engineering teams strive for.
Our insistence on DRYness has one very significant caveat… As of now, GitHub and GitLab do a regretably mediocre job of rendering AsciiDoc code in their online “preview” display modes. (Don’t get me wrong — I am psyched that both major services honor AsciiDoc as well as they do, and I am confident this will improve greatly before long.)
This puts limits on any file that must be displayed as-is in the repository preview on either of these services — and here I mean the README but also a CONTRIBUTORS or NOTICE files.
All these files contain content that should appear in rich-text form when viewed in the repo via browser, but also rendered as final output in product documentation.
For now, because any good README file has to include canonical information about the product, the README itself can be used as a canonical source, and the rest of the docs can draw from it.
This very README.adoc file is littered with AsciiDoc include tags so that AsciiDoc files can draw content from it.
In fact, I am probably abusing the README as we speak (as I’ve been known to do before), in this case mainly to illustrate this very point.
|
Tip
|
Even if your dynamic docs can draw from the README, a README file in any format is unlikely to prove a worthy source for product code.
This means anywhere the README must reflect data_ about the product (pieces of information that are “hard coded” into the product), that data must be duplicated in the README.
Therefore, any content that can be kept out of the README and stored in semi-strucutred formats should be.
|
This repository is yours to clone or fork — hack away! Hopefully it will at least demonstrate some techniques that will help you get unstuck on any single-sourcing/docs-as-code obstacles you may be facing.
Like any good implementation of a framework, this project is just a bunch of flat files. These files establish configuration, data, and content for dummy documentation of a nonexistent product. All you need is the Ruby runtime environment and Bundler, as instructed in the LiquiDoc README.
|
Note
|
This repo is not a framework in and of itself. The framework is being established as this “dummy” project develops to document it. If you have other use cases, please contribute challenges in the Issues tab on GitHub and we can try model solutions in this or a forked repo! |
-
Clone or download this repository to your local machine.
git clone {github_project_git_uri} cd acmedoc-poc -
Obtain the required Ruby dependency gems.
bundle install
-
Run the configured build.
bundle exec liquidoc -c _configs/build-docs.yml
LiquiDoc will compile PDFs (see build/docs/) then builds parallel Jekyll sites under build/site/.
To serve your built Jekyll sites as one:
+
bundle exec jekyll s --destination build/site \ --config _configs/jekyll-global.yml --skip-initial-build
The LiquiDoc CMF has a core structure and uses orderly configuration to manage preprocessing and rendering steps.
acmedoc-poc/
├── _configs/
│ └── build-portals.yml
├── data/
│ ├── asciidoctor.yml
│ ├── company.yml
│ ├── portals.yml
│ ├── products.yml
│ └── terms.yml
├── _templates/
│ ├── asciidoc/
│ │ └── x.adoc
│ └── liquid/
│ ├── product-info.asciidoc
│ └── terms.asciidoc
├── build/*
│ └── [content/]
├── content/
│ ├── includes/
│ ├── portal/
│ ├── topics/
│ └── index.adoc
└── theme/Asciidoctor is pretty awesome, but it has some serious limitations when it comes to working with data in non-native formats. Since most other applications don’t read AsciiDoc’s (fairly sad) data structures, we want AsciiDoc to honor popular open data formats. Therefore, the power of LiquiDoc is mainly in two of its features:
- AsciiDoc preprocessing
-
LiquiDoc massages data into AsciiDoc-formatted files and saves them in the
build/directory, where they can be read when we actually render our AsciiDoc files into documents. This data can be in YAML, JSON, XML, CSV, or even free-form formats that can be parsed with regular expressions. By storing data in flat files using these semi-structured formats (instead of a relational database), data can live in the code repository under source control. By using these small-data formats instead of “hard coding” data into your docs, you keep it available for reformatting (by LiquiDoc or other tools) and make it accessible to other programming languages, most of which can read YAML, JSON, and XML. - YAML attributes ingest
-
AsciiDoc uses variables called attributes, but it’s not very good at consuming these from external files. LiquiDoc fixes this by letting us convert the data in multiple YAML-formatted flat files into attributes when we go to render our final docs.
- Asset migration
-
A big part of coordinating a complicated docs build is making sure source files are where they need to be at every step of the build. ACMEdoc actually moves files around quite a bit, often making two copies of various sources, each where a different rendering component (Asciidoctor, Jekyll, etc) needs it to be for processing.
TipDuplication of source files during the build is standard and acceptable practice. Remember, the build/directory is never committed to the repository, so as long as automation tools are configured to use duplicated files properly, there is no problem with this often necessary approach.
With this preprocessing and data-ingest strategy, we greatly expand the single-source (DRY) potential of our docs-as-code system. We can either write entire sections of AsciiDoc files by massaging data with Liquid-formatted templates, or we can feed AsciiDoc-readable variables right into the rendering engine, pressing them into the content where token placeholders exist.