Plans on Docker Node for self-hosting?

[siousu]

My warmest congratulations on your Beta-Launch! It's such a joy to see what you guys are achieving. I'm using Anytype since more than 1 year and it's amazing!!

The ability to self-host was one feature I was longing for since the beginning I'm using it. As I'm not really a hard-core programmer or server user it would be really helpful to use an Anytype Node in a Docker Container. Is this planned or already possible? Then I would like to set it up on my Synology.

Thanks and keep up the great work! 👍

[fuksman]

Hey @siousu,

We plan to invest more time in simplifying self-hosting when we have a better understanding of users' requests, so thank you for the feedback!

Also, I see a possibility that a Docker image for self-hosted Anytype could be the first project contributed by the community. Let's see how the community develops in the coming weeks and maybe someone will volunteer to implement it 🤞

[SamBouwer]

As always, there could be multiple parallel hosting options. The list should probably be consolidated to three options max, but for example:

• Trial/dev: all-in-one docker image
• Starter: App docker image + bring your own database
• Intermediate: docker-compose with db, s3, redis, and one of each any-sync image (1 coordinator, 1 node, 1 filenode)
• Pro: Intermediate, but with custom number of any-sync (file)nodes
• Super-pro: build everything yourself from source

[EternalDeiwos]

An addition to the above:

• Almost-super-pro: Helm Chart

[suntorytimed]

Having redis, S3 and MongoDB within one container or within the app container would defeat the purpose of micro services and sandboxing in containers. This would be the old non-container monolithic approach. Having each service in their own container makes it much easier to update individual parts. Especially for a maintainer of the docker image. I would advocate for a good docker compose example with a thorough documentation instead of having one monolithic container setup. That way redis, mongoDB and a container for an easy S3 solution won't have to be maintained and it only needs to be checked which version of those workers need to be used in the compose file.

Personally I would also like to see not only a helm chart but maybe a Terraform example to setup a S3 (compatible) storage node and a server for hosting the containers on a cloud provider or in KVM. But that's really only interesting for bigger deployments.

I could look into packaging the components on openSUSE, but due to the used Any Source Available License v1.0 it will never end up in a Leap or Tumbleweed distribution, as it's not an open source license per OSI definition. But a development repository could be added in a Tumbleweed based container and the packaged components installed from there.

As I can't ensure that the resulting packages and containers wouldn't be redistributed for a commercial use, I personally refrain from doing any packaging or containerisation right now.

[cswrd]

Regarding user needs: I am looking for a fully self-hosted Confluence Server alternative for home usage (ideally, dockerized out of the box and with a similar feature set). I've heard of Anytype just 1 minute ago and checking if it fits my needs.

[KaKi87]

Hi,
How is this going ?
Thanks

[zxiilw63]

The way to build a self-hosted server would be best if it was simplified a bit, it's a bit hard to know how to get started now that I'm watching the tutorials

[ahmedwael216]

Currently AnyType team will not accept contributions for a waek or so.
So this is a simple list to track the current architectures we are discussing.

If any one succeeded in writing the docker-compose files, please share with us.

• Trial/dev: all-in-one docker image
• Starter: App docker image + bring your own database
• Intermediate: docker-compose with db, s3, redis, and one of each any-sync image (1 coordinator, 1 node, 1 filenode)
• Pro: Intermediate, but with custom number of any-sync (file)nodes
• Super-pro: build everything yourself from source (I don't think this is related to docker, but leaving it to be consistent with the original list).

If you have any ideas/comments don't hesitate to say them 😁.

Thanks to @SamBouwer for the list.

[EternalDeiwos]

As above, we could use a Helm Chart too.

[SamBouwer]

Hi all! I'm working on a first version of the "all-in-one" docker image in this repo:

https://github.com/SamBouwer/any-docker/tree/main

As I have never build and hosted Anytype from source AND I am completely new to building docker images, any-help is appreciated ;). I'm also quite new to github pull requests, so please don't take it personally if I by accident reject your contribution :P

Current state is also written in the repo linked above, but I've managed to take a golang docker image (v1.19) as a base image and I can clone the various any-sync-* repos. I'm currently working on using the any-sync-tool within the Dockerfile to generate the required yaml configuration to be used by the other any-sync-* "modules". I want to list the arguments needed by the any-sync-tool to create a new network and then try to add those as arguments that can be passed to the image.

[Flol0]

The biggest problem I currently have with self-hosting is the need to compile the client myself. I have an iPhone and installing self-compiled apps on it sucks.

[fuksman]

Yes, we receive such requests frequently and plan to implement it in future releases.

[markusstephanides]

Do you have an approximate date for that feature? So we know whether it is worth to wait until it's there or we have to build the clients ourselves anyway for now. If the second option applies, is there a guide somewhere on how do to build the client while pointing it to my selfhosted server?

[fuksman]

@markusstephanides the most precise thing I can say for now is that we aim to deliver this in the next two releases.

Here you can find a self-hosting how-to: https://tech.anytype.io/how-to/self-hosting

[clems4ever]

Here is my contribution on this with a working docker-compose based backup node which has been proven to sync spaces and files on my LAN.

https://github.com/clems4ever/anytype-backup-node

However, it requires to rebuild the clients as mentioned above so if you don't know how to do that, you should probably not even try it.

[fuksman]

@clem4sever, @SamBouwer, sorry for bringing bad news, but we are following a monthly release schedule. This means the upcoming release will include all the work we did in July, and self-hosting is not there. Therefore, we won't have the option to release configurable clients until the following month :(

[SamBouwer]

@fuksman that's in line with my expectations :) am I correct to assume that (simplified) selfhosted will be released end of Sept or the release after?

[fuksman]

@SamBouwer, yes, we aim for the second half of September. It will also depend on the state of other tasks planned for that release.

[dzshch]

@fuksman Is there any update on this?

[fuksman]

@dzshch currently in development, will be delivered in the upcoming release (this month).

[clems4ever]

@fuksman , do you have a history of which version of the services are deployed in production? I made a guess when building the various services and ended up picking compatible versions but that's not so straightforward, it would be easier if we had a way to check which version you are/were running. That would help me keep the versions inline in the Dockerfile and avoid issues for the users because currently I don't really know which versions are compatible or not with each others, moreover there is no changelog in the releases or at least it's not so expressive so it's hard to know the impact of the commits without deeply digging into them.

If there is nothing in place yet, do you think there would be a way to share this information in realtime? Like a repo with submodules containing the version of the 3 services being deployed so that we (community) can have access to the git history of compatible versions deployed and you can keep track of your deployments?

[fuksman]

Hey @clems4ever, sorry it took so long. We have created an endpoint where you can check the current node versions we have in production: https://puppetdoc.anytype.io/api/v1/prod-any-sync-compatible-versions/. Every new deployment will create a new record there with the timestamp as a key.

Also, you might find it useful to check the compatibility of pre-release versions in our staging network: https://puppetdoc.anytype.io/api/v1/stage1-any-sync-compatible-versions/.

[clems4ever]

Awesome, thanks!
Did they exist before my last message or have you just created them?

[fuksman]

Endpoints have just been created based on your request 😃

[clems4ever]

That was quick then, thanks for this reactivity!

[Gandalf-the-Blue]

Hi,
Just wanted to check in on this to see what the status is of this project? Is there an expected ETA?
Also whether clients will be made to be self-hosting compatible?

[Gandalf-the-Blue]

Thanks for this @fuksman, so to self-host Anytype should I just deploy @clems4ever's repository or the ones by anyproto?

Additionally, since I am not that technically proficient regarding building clients from source - are there any plans to have the clients as released include an option to enter the self-hosted url and use that instead of the default anytype servers?
Thanks!

[fuksman]

@Gandalf-the-Blue, our internal tools were published to help people build their own solutions. We don't recommend using them in production. So it's better to use @clems4ever's repo or build your own infrastructure.
Regarding client configuration for self-hosting, yes, it will be possible later.

[Shampra]

You can check here too : https://github.com/SamBouwer/any-docker

[alexfreska]

I am getting started with the software and like @Gandalf-the-Blue I am a little confused that there is no way to self-host / use the software autonomously as advertised, I can certainly deploy https://github.com/clems4ever/anytype-backup-node, but definitely want to use the official client software vs manually build and get on device. Looking forward to the custom backup node feature in the official software.

Really awesome mission and software, thank you for all your work!

[bphenriques]

What is the recommended way for home-users with a humble setup? This is not for business, therefore I do not need HA. Only need a server and Android/iPhone/Desktop apps (which it seems there is!).

I am no stranger to docker and docker-compose but it feels overwhelming... Similar to others, I would prefer to use the official tooling if possible.

I https://github.com/clems4ever/anytype-backup-node the recommended 3rd party setup?

[xiaoland]

Despite that I know Anytype focuses at privacy and security, but I reckon what it's important to make Anytype supports collaboration or team use case. Therefore, Self-hosting should implement some new features on this point that Anytype's cloud can't afford.

[phbrgnomo]

Hey folks. Currently looking for a note taking app to host on my home server to sync all my devices.

Any type looks promising for my use case, so was wondering how is the docker container development status.

Besides that, correct me if I am wrong...

To sync my devices currently, I need to have a mongodb server, and use any-sync as an entry point to this db?

[fuksman]

Hey, there is a https://github.com/anyproto/any-sync-dockercompose, but we don't recommend to use for production yet. Will update this soon.
You can sync data between your devices via Anytype Network, via p2p (if both devices are in the same local network), or via self-hosted network. For self-hosted network you will need a mongodb server, correct.

[fraschm1998]

Just to clarify, Anytype local network sync only works with the pc version and not yet with mobile as per: anyproto/anytype-kotlin#755 and anyproto/anytype-kotlin#836

[fuksman]

@fraschm1998 local network sync works for all platforms. However, there are some bugs on certain Android devices. We are currently investigating them.

@phbrgnomo update regarding my previous comment: any-sync-dockercompose is now suitable for home-scale production use.

[alphasingh]

To anyone who is on the same boat as me, here are a couple of things which I had to do (this might help others if there use case is also similar):

1. I was initially facing issues with Makefile to get started due to my old docker version.
   So I uninstalled my old docker and upgraded my docker Docker version 25.0.3, build 4debf41 along with Docker Compose version v2.24.5 (followed this).
   I was able to resolve Makefile and other docker issues after this upgrade on my Ubuntu 20.04.3 LTS.
2. I opened ingress [1001-1006] ports on my remote server because those ports are mentioned in the client.yml file which is a result of make start setup which you will need to connect and sync client devices with remote node backup.

Right now there is just one minor thing I notice everytime I do docker ps:
the any-sync-dockercompose-mongo-1-1 always shows unhealthy for me. The sync works fine though on my desktop and mobile app for the remote backup node configuration. And I have tried to make restart and individually restart with it with docker but keeps showing unhealthy.

[akippnn]

@fraschm1998 local network sync works for all platforms. However, there are some bugs on certain Android devices. We are currently investigating them.

@phbrgnomo update regarding my previous comment: any-sync-dockercompose is now suitable for home-scale production use.

You guys are doing God's work. Currently recommending this to my partner, team, and peers. Docker Compose and Multiplayer MVP would help push people to move away from something like Notion.

[IngwiePhoenix]

Hello othere! Came across this through r/selfhosted on Reddit and I am very interested.

After having felt let down by Notion and Obsidian for not having a proper self-hosted solution - or one you have to pay for as it was the case with Standart notes, pray tell what will happen to that after Proton bought it now - I have been hoping for a great alternative. It may be a little nitpicky - but I prefer to host things myself to truely "own" my data.

So, I would love some pointers: Where, or what, are the components that make the backend tick exactly? I'd be happy to see what I can automate into a Docker image, or even a k8s deployment. I did spot references to Redis, MongoDB and S3, all of which would easily be containerize-able. However, the licensing situation of and with Redis may become an interesting topic for you guys in the long run, seeking out potential alternatives like KeyDB or Valkey.

Either ways, pointers welcome; I'd be happy to work on that if it means I finally get what I have been hoping to have: a self-hosted, self-owned place to journal and track my personal bits and bops of data. :) I have many use cases for a system like that; be that as a roleplayer to store my characters, relations, stories and resources - or as a developer to store interesting things. But maybe also my school and work during my apprenticeship.

Thanks and kind regards,
Ingwie

[fuksman]

Hey @IngwiePhoenix, there is a docker image maintained by the team: https://github.com/anyproto/any-sync-dockercompose

[adamcnaglich]

I couldn't get the docker compose to deploy with portainer just now. There was an error about a variable not being set or something (I'm a self hosting noob)

[akippnn]

Hey @IngwiePhoenix, there is a docker image maintained by the team: https://github.com/anyproto/any-sync-dockercompose

I think Docker Compose is kind of difficult to deal with when there's different ways to host it. Right now, I'm using Podman (via Fedora Server) for all my containers. For example, the GitLab container (which is currently a Docker Compose but will move to K8s shortly), I have to do it differently by appending the volumes with :Z, but it still works flawlessly as if it's being hosted in Docker. I think it would be nicer if it's simply an image we can grab and then we can use whatever implementation we'd like, even use it in K8s.

Edit: Yes, the current implementation is nice and easy to setup with a simple make command. But it's also the least flexible, right now any-sync-dockercompose is the only thing that I can't use without having Docker installed. It really does scream "home-scale production use" so there's a lot more room to grow when we want to scale up or when we have different ways of managing the containers. I think encouraging people to make their own Compose files would be better in this regard, so instead like a decent documentation on how to setup MongoDB/S3/Redis and the any-sync-network would be nice.

[hellodword]

New to anytype, and I'm migrating my notes and to-dos to it.

So I created a compose file for self-hosting a lightweight instance, no MongoDB, no Minio, no S3, see:

https://github.com/orgs/anyproto/discussions/209

[zmylk]

Hello, I have two questions.

1. I currently put the self-hosted program on my LAN server. If one day I want to put local data on a remote VPS, how should I migrate it?
2. I see that the remote storage cannot be modified during use. Do I need to create a new account for subsequent changes to the self-hosted domain name and IP?