Improve docs how to test community containers (CC) locally?

[crfroehlich]

The documentation for adding new CCs does not cover how to develop and test containers locally before submitting a PR. I have searched the codebase, the issues and the discussions; and I am not finding documentation on this. If it exists, perhaps it should be linked more clearly?

I am approaching this from the perspective of a consumer of the NextCloud AIO stack in a homelab setting, where my production server is a proxmox cluster running the stack on docker in an Ubuntu VM. My users are my family. I am primarily interested in adding new containers and modifying existing containers, but I don't have a lot of spare time--so I don't want to necessarily have to commit to going through the PR process; but if I do submit a PR, I want to be darn sure everything is as close to spotless as possible before I submit. I also don't want to have to run the codebase locally--I just want to "develop" and test CCs (assuming that is possible to isolate CCs from the rest of the stack for the purpose of development and testing). I imagine that there are more than a few AIO users who share this perspective (homelab or small org admins who run AIO in production and want to extend their instance with new functionality). I do not have a staging environment. I make all of my changes on production. Since everything is running inside a VM, I can snapshot and rollback anytime I want to test/revert changes. I'm comfortable with short periods of downtime that I control.

From this perspective: off the top, there are a few possible use cases that could be documented a little more clearly:

• User wants to manage their own "community containers" locally without submitting changes back to the repo.
• User wants to develop and test container candidate locally before submitting PR.
• User has submitted PR and needs to iterate. Links to code review standards, CI/CD process > code coverage/testing expectations and process for running relevant container tests locally.
• User wants to understand general expectations for community container modifications (e.g. what can or cannot be exposed as a subroute? what changes (if any) can be made to other containers, such as Caddy? what are minimum requirements for a readme? etc)

Once it's clear how to inject new AIO containers from local code, it might be possible to extrapolate how one might support a 3rd party repository of containers (or any arbitrary collection of containers that are not centrally defined in the AIO repo), which would be exceptionally useful for users who are primarily interested in running the NextCloud AIO stack and extending its functionality via CCs.

[szaimen]

The documentation for adding new CCs does not cover how to develop and test containers locally before submitting a PR. I have searched the codebase, the issues and the discussions; and I am not finding documentation on this. If it exists, perhaps it should be linked more clearly?

Hi, thanks a lot for bringing this up! I agree we need more documentation on the topic. Help on this is very appreciated :)
(e.g. a PR that updates the documentation on this topic)

---

• User wants to manage their own "community containers" locally without submitting changes back to the repo.

We don't have plans to document this case for now as you can very easily create additional docker containers without using AIOs CC feature for that. But it shouldn't be much different compared to how to test container candidate locally before submitting PR. See below

• User wants to develop and test container candidate locally before submitting PR.

In general, see #4822 (comment)

• User has submitted PR and needs to iterate. Links to code review standards, CI/CD process > code coverage/testing expectations and process for running relevant container tests locally.

Mostly the following workflows need to pass: https://github.com/nextcloud/all-in-one/blob/main/.github/workflows/json-validator.yml and https://github.com/nextcloud/all-in-one/blob/main/.github/workflows/community-containers.yml

• User wants to understand general expectations for community container modifications (e.g. what can or cannot be exposed as a subroute? what changes (if any) can be made to other containers, such as Caddy? what are minimum requirements for a readme? etc)

In general, the jsons are validated against this json schema: https://github.com/nextcloud/all-in-one/blob/main/php/containers-schema.json. There indirectly all possible values are documented. A very much complete example can be found here: https://github.com/nextcloud/all-in-one/blob/main/php/containers.json

Regarding the community container caddy, new containers that are covered can be easily added by submitting a PR to this repo: https://github.com/szaimen/aio-caddy. See e.g. szaimen/aio-caddy#40

[crfroehlich]

Thanks so much for the detailed response! I am still digesting the links you have provided and will add any additional thoughts I might have as they emerge. So here is a grounded use case that actually got my wheels spinning in the first place. @szaimen, you have posted a number of these wonderful comments with code snippets throughout the discussions in response to various: Add <name> container suggestions, e.g. (sample lists, not exhaustive):

• Netdata: Netdata #392 (reply in thread)
• Scrutiny: Scrutiny (GUI for smartd) #3549 (comment)
• CalCardBackup: Add calcardbackup #3696 (comment)
• LanguageTool: Using LanguageTool for collabora/nextcloud office #4380 (reply in thread)
• Flame: community-container idea: dashboard/startpage #4967 (reply in thread)

I notice that some of these threads are old and have not been implemented but there appears to be interest in having them done, so what is the hold up?

From the perspective of the repo maintainers: Is there anything about the process that is either (a) unclear and/or (b) too difficult and/or (c) too much effort and/or (d) slow? What could be done to enable folks to take these code snippets and turn them into PRs faster?

From the perspective of users (like me), using the Netdata example: "Ok, sweet, I have this idea for a CC and @szaimen has given me this code snippet to try out.... now what do I do with it?" #4822 (Comment) is the secret sauce I need to start connecting the dots.

The documentation for adding new CCs does not cover how to develop and test containers locally before submitting a PR. I have searched the codebase, the issues and the discussions; and I am not finding documentation on this. If it exists, perhaps it should be linked more clearly?

Hi, thanks a lot for bringing this up! I agree we need more documentation on the topic. Help on this is very appreciated :) (e.g. a PR that updates the documentation on this topic)

I can't promise anything, but I will make an honest effort to set aside some time this week to make a PR with some suggested edits that fold all of this information together.

• User wants to manage their own "community containers" locally without submitting changes back to the repo.

We don't have plans to document this case for now as you can very easily create additional docker containers without using AIOs CC feature for that. But it shouldn't be much different compared to how to test container candidate locally before submitting PR. See below

There are a few reasons to be interested in CCs vs running vanilla containers:

• Routing is already baked in
• Backups are baked in
• Building on top of LDAP/Stalwart or other component with interdependencies, i.e. extensions to NextCloud functionality

But it is true that these things can be accomplished without hooking into the CC framework, so perhaps that is just another opportunity to provide better documentation for those who want to add functionality to NextCloud the "old fashioned" way?

Thanks again for your detailed and thoughtful response. It's tremendously helpful.

[szaimen]

Thanks so much for the detailed response! I am still digesting the links you have provided and will add any additional thoughts I might have as they emerge. So here is a grounded use case that actually got my wheels spinning in the first place.

Thank you for taking a look at it! It is very valuable for the project! :)

@szaimen, you have posted a number of these wonderful comments with code snippets throughout the discussions in response to various: Add <name> container suggestions, e.g. (sample lists, not exhaustive):

• Netdata: Netdata #392 (reply in thread)
• Scrutiny: Scrutiny (GUI for smartd) #3549 (comment)
• CalCardBackup: Add calcardbackup #3696 (comment)
• LanguageTool: Using LanguageTool for collabora/nextcloud office #4380 (reply in thread)
• Flame: community-container idea: dashboard/startpage #4967 (reply in thread)

I notice that some of these threads are old and have not been implemented but there appears to be interest in having them done, so what is the hold up?

I'd say mostly that no one volunteered yet for submitting a PR - so no one stepped in as maintainer of the specific component inside AIO. Part of that might indeed be that they don't know how to test it before submitting but just guessing here...

From the perspective of the repo maintainers: Is there anything about the process that is either (a) unclear and/or (b) too difficult and/or (c) too much effort and/or (d) slow? What could be done to enable folks to take these code snippets and turn them into PRs faster?

I currently do not see anything we could improve in the repo here apart from maybe indeed documenting better how to test things before submitting it to the repo. I think it is rather something we might could do outside - e.g. pushing some threads to https://help.nextcloud.com that we are actively looking for maintainers for the above components inside AIO...

From the perspective of users (like me), using the Netdata example: "Ok, sweet, I have this idea for a CC and @szaimen has given me this code snippet to try out.... now what do I do with it?" #4822 (Comment) is the secret sauce I need to start connecting the dots.

Yes indeed :)

The documentation for adding new CCs does not cover how to develop and test containers locally before submitting a PR. I have searched the codebase, the issues and the discussions; and I am not finding documentation on this. If it exists, perhaps it should be linked more clearly?

Hi, thanks a lot for bringing this up! I agree we need more documentation on the topic. Help on this is very appreciated :) (e.g. a PR that updates the documentation on this topic)

I can't promise anything, but I will make an honest effort to set aside some time this week to make a PR with some suggested edits that fold all of this information together.

Cool, thanks a lot :)

• User wants to manage their own "community containers" locally without submitting changes back to the repo.

We don't have plans to document this case for now as you can very easily create additional docker containers without using AIOs CC feature for that. But it shouldn't be much different compared to how to test container candidate locally before submitting PR. See below

There are a few reasons to be interested in CCs vs running vanilla containers:

• Routing is already baked in
• Backups are baked in
• Building on top of LDAP/Stalwart or other component with interdependencies, i.e. extensions to NextCloud functionality

I agree these are indeed good reasons but as I said above, it should also work like this: #4822 (comment)

But it is true that these things can be accomplished without hooking into the CC framework, so perhaps that is just another opportunity to provide better documentation for those who want to add functionality to NextCloud the "old fashioned" way?

Yeah, could be possible...

Thanks again for your detailed and thoughtful response. It's tremendously helpful.

You are welcome! I am glad that you are having a look at this :)

[szaimen]

Hi @crfroehlich do you have some update on the topic? :)

[szaimen]

I've added this to the list in #5251 now