nixos-cluster-config

Cluster of nixos nodes

On the host:

touch /etc/nixos/ncluster.nix; chown philipcristiano /etc/nixos/ncluster.nix

Then

scp ncluster.nix $USER@TARGET_HOST:/etc/nixos/ncluster.nix

Add ./ncluster.nix to the imports

Cluster bootstrapping

Vault

See vault/README.md

Consul Value

Expected consul values

site/domain - Base domain expected for services.

Networking

VLANs

VLANs are used to provide separate interfaces for applications. This is meant to work around limitations in macvlan interfaces in linux where the host cannot reach the macvlan'd interfaces.

In your nixos/configuration

  networking.vlans = {
	vlan110 = { id=110; interface="enp2s0"; };
  };
  networking.interfaces.vlan110.useDHCP = true;

BGP

BGP is used with GoCast to advertise floating IPs

Site configuration

nomad_job.vars

domain Internal domain for services docker_registry Custom registry to use, should be equal to docker-registry.$DOMAIN if you are using this docker registry

Services

(WIP)

bash deploy.sh

Storage

Minio for S3-compatible storage that can be hosted on each node.

NFS (hosted outside this cluster) is used for services that cannot use S3

Database

Postgres deployed for each service.

Reverse-Proxy

Traefik and Let's Encrypt for certs

mktxp / mikrotik monitoring

Consul values:

• credentials/mktxp/influxdb_organization
• credentials/mktxp/influxdb_token
• credentials/mktxp/password
• credentials/mktxp/username

Minio / S3-compatible blog storage

Consul values:

• credentials/minio/root_user
• credentials/minio/root_pass

Baserow

Setup

Consul Values

• credentials/baserow-postgres/USER - Username for the root user
• credentials/baserow-postgres/PASSWORD - Root password
• credentials/baserow-postgres/DB - default DB
• credentials/baserow-redis/password - Username for the root user

Bitcoin (and electrs, bitcoin-rpc-explorer, mempool)

NOT SAFE FOR USAGE AS A WALLET - only using this for an API to bitcoin data

Mempool also requires MariaDB

Consul Values

• credentials/electrs/bitcoind_username - Username from above

• credentials/electrs/bitcoind_password - Password generated by rpcauth

• credentials/bitcoin-rpc-explorer/bitcoind_username - Username from above

• credentials/bitcoin-rpc-explorer/bitcoind_password - Password generated by rpcauth

• credentials/mempool/bitcoind_username - Username from above

• credentials/mempool/bitcoind_password - Password generated by rpcauth

For each service:

• credentials/bitcoind/rpcauth/USERNAME - RPC auth line after rpcauth=USERNAME: Just the salt/password portion!

Folio

Postgres

Consul Values

• credentials/folio-postgres/USER - Username for the root user
• credentials/folio-postgres/PASSWORD - Root password
• credentials/folio-postgres/DB - default DB

Frigate

Setup

Consul Values

• credentials/frigate/mqtt_host - MQTT Host IP
• credentials/frigate/mqtt_username - MQTT Username
• credentials/frigate/mqtt_password - MQTT Password
• credentials/frigate/cameras/* - Key: Camera name, Value: input.path for Frigate

Matrix

Matrix-Hookshot

• credentials/matrix-hookshot/passkey.pem - passkey.pem from openssl genpkey -out passkey.pem -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:4096

Nomad Admin

Set spread scheduling algorithm

curl -s $NOMAD_ADDR/v1/operator/scheduler/configuration |
    jq '.SchedulerConfig | .SchedulerAlgorithm="spread"' |
  curl -X PUT $NOMAD_ADDR/v1/operator/scheduler/configuration -d @-

Allow memory oversubscription

curl -s $NOMAD_ADDR/v1/operator/scheduler/configuration | \
  jq '.SchedulerConfig | .MemoryOversubscriptionEnabled=true' | \
  curl -X PUT $NOMAD_ADDR/v1/operator/scheduler/configuration -d @-