External content collections repository/folder

[4www]

In the current astro architecture (as far as I could research), the content/data (markdown etc.) is either put:

• in the ./src/pages/* repository; traditional astro style
• in the ./content collections folder (https://docs.astro.build/en/guides/content-collections/); new astro RFC

When using Gatsby, there is a feature I really enjoy, is to having to content folder, be external to the code folder of the project (theme); even having both in two different repositories.

For example:

• content repository https://github.com/joblisttoday/data (only markdown)
• theme repository https://gitlab.com/joblist/profiles-gatsby (only gatsby & javascript code); with this gatsby git-clone plugin configuration https://gitlab.com/joblist/profiles-gatsby/-/blob/main/gatsby-config.js#L39-45

The effect of this setup, is that both at development time and build time, the data repository is cloned into the ./content/* folder of the (gatsby) theme repository/project; which is used to generate all of the static pages of the site.

I wonder, is there already a way to achieve this behavior with astro? Is this a feature that could be envisionned in the astro ecosystem.

One solution is to using git submodule (where the content repo is a submodule of the theme repo). This does not really work, as a submodule is "pinned" to a git reference commit ID, thus having to update to the latest commit, the submodule in the theme repository, each time the content/data repository is updated.

The idea would be that the theme repository, could be triggered for a re-build, either with a cron job, or a CI-job triggered by an action in the data repository (to the theme repository) each time the data changes.

[jasikpark]

One thing I've done is install the content repo as a git npm module and import the markdown from there!

[4www]

Thanks for your answer! It seems to me like a pretty way of having the content/code separation of concerns.

With this technique, do you have a way of having the theme repo, "auto-rebuild" with the lastest commit of the data repo?

Since the data repo is pinned by its commit, we'd have to manually update the (data) submodule in the theme repo (git pull submodule), each time there is a change in the data repository.

When both data and theme are in the same repository, it works naturally well with the CI/CD (rebuild on every change in the repo). When there are two repo, it kind of breaks this pattern of building, with the latest version of both data & theme, when there is a change in either of the repositories.

I'm not sure what would be the best way to achieve this with Astro. With gatsby, since there is a plugin to clone an external repository, the theme repository can be re-built with each change in the data repository, for example by calling the CI/CD API (example here: https://gitlab.com/joblist/data/-/blob/main/.gitlab-ci.yml#L11)

[airtonix]

One thing I've done is install the content repo as a git npm module and import the markdown from there!

Wouldn't you have to do this for every single markdown file?

I guess we'd need an example of how you're doing this.

[Its-Just-Nans]

Could it be possible to add an option in the astro function defineConfig like srcDir, publicDir but for the content folder ?

Something like contentDir ?

@jasikpark @bholmesdev

[levino]

What I am looking for is way to install content from npm. So one would publish a content collection via npm and then use it multiple astro projects. Important part would be that one can use this content collection package additionally to any local content collections and possibly even use the remote content in ones own content collection entries.

[Its-Just-Nans]

Why not using git submodules ?

[levino]

Because it is annoyingly complicated, error prone and just cumbersome. Of course I could all kinds of scripting and tooling myself to achieve this. However I think it is a common use case for people building websites. Imagine you could just do
pnpm add astro-content-collection-thetvdb
and then use

getCollection('@thetvdb/actors')

or something.

[airtonix]

this would solve my use case too

[airtonix]

my monorepo has this structure:

apps/
  web-with-astro/src/
    ????
  web-with-nextjs/src/
    app/
      page.tsx    -> require('@zenobius/content').keystatic.collection.get('me', 'intro')
      me/
        page.tsx  -> require('@zenobius/content').keystatic.collection.get('me', 'resume')
      b/[...slug]/
        page.tsx  -> require('@zenobius/content').keystatic.collection.get('posts', props.slug)
      b/
        page.tsx  -> require('@zenobius/content').keystatic.collection.all('posts')
      p/[...slug]/
        page.tsx  -> require('@zenobius/content').keystatic.collection.get('pages', props.slug)
      p/
        page.tsx  -> require('@zenobius/content').keystatic.collection.all('pages')

apps/
  core/content '@zenobius/content'
    documents
      me/
        intro.mdoc
        resume.mdoc
      posts/
        something.mdoc
        something.mdoc
        something.mdoc
        something.mdoc
        something.mdoc
        something.mdoc
        something.mdoc
        something.mdoc
    keystatic.ts
      // builds KeyStaticConfig({ src: __dirname/../documents })
      // exports require('@zenobius/content').keystatic

How can i make the ???? part use `@zenobius/content/documents'.

things i dont want to do as they are yuck:

• symlinks
• git submodules
• mirror the folder file structure and individually import each file

what I want:

🚀

// @zenobius/content/index.ts

export const documentDir = path.resolve(__dirname, 'documents')

// @zenobius/app-web-with-astro/astro.config.ts

import { defineConfig } from 'astro/config';
import react from '@astrojs/react';
import { vanillaExtractPlugin } from '@vanilla-extract/vite-plugin';
import markdoc from '@astrojs/markdoc';
import { documentDir } from '@zenobius/content';     1️⃣ 🚀 


export default defineConfig({
    // Enable React to support React JSX components.
    integrations: [react(), markdoc()],
    compressHTML: false,
    trailingSlash: 'never',
    contentDir: documentDir,                         2️⃣ 🚀 
    build: {
        format: 'directory'
    },
    vite: {
        ssr: {
            noExternal: ['@astrojs/react']
        },
        plugins: [vanillaExtractPlugin()]
    },

});

or ❤️

// @zenobius/app-web-with-astro/src/content/config.ts

import { defineCollection } from 'astro:content';
import { blogSchema, authorSchema } from '../schemas';
import { documentDir } from '@zenobius/content';     1️⃣ ❤️ 

// 2. Define your collections
const blogCollection = defineCollection({
  type: 'content',
  source: path.resolve(documentDir, 'posts'),        2️⃣ ❤️ 
  schema: blogSchema,
});


// 3. Export multiple collections to register them
export const collections = {
  'blog': blogCollection,
};

I think perhaps ❤️ is more scalable and in line with opinions of the astro leadership.

[kcrwfrd]

We have tens of thousands of markdown content files that would overwhelm source tracking in our Astro repo, so what we're doing is using Git LFS to store a pointer to a compressed content.tar.gz file, and our src/content/* is added to gitignore (with whitelisting for src/content/config.ts).

Then we have some script commands in our package.json to untar content.tar.gz into the appropriate place.

We have a separate repo and database for generating the content markdown files, so tracking individual line changes doesn't matter to us.

[ShelbyJenkins]

My use case for this:

I'm working on a no-code github pages blog tool. I don't want users to have to open the src folder. It's scary for them.

Also, I didn't see this blog post linked in this thread, but it has details for using symlinks for this: https://www.eliostruyf.com/symlink-content-astro-portability/

[Hecatron]

This is a feature I'd like to see myself coming from nuxt / docus where you can do this, just to make things tidy.
Using a windows environment mostly myself, symlinking only really works under linux environments
there are ways to do it under windows but its less than ideal, especially if you want someone else to build docs you've uploaded to github.
git submodules is also yuck

I tried relocating the src dir using the config but that fails to work if its located outside of the main astro directory
I think ideally you'd need some way to pass in a source directory to defineCollection as mentioned above

[adriandpdev]

Its a very good idea

[johnmcguin]

I am looking to do this with monorepo as well. Ideally to have multiple sites and have one package that has core UI components as well as shared content.

[Its-Just-Nans]

I made this package to have list of the files https://github.com/Its-Just-Nans/plugin-astro-content