There exists a mutitude of tooling to remotely manage nixos machines, each with its own features and compromises. However you can also just use a simple flake with nixos-rebuild --target-host.

This is the workflow I've been using to manage my systems.

I bootstrap the machines manually by following the nixos install guide, and copy over the generated configuration to config/<hostname>/configuration.nix.

On the root directory create flake.nix:

{
  description = "systems needed";
  inputs = {
     # extra inputs go here
  };

  outputs = { self, nixpkgs }@attrs: {
    # this is where we add new machines
    nixosConfigurations = {
     # host nixmachine
      nixmachine = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        specialArgs = attrs;
        modules = [
          # This points to the actual machine configuration
          (import ./config/nixmachine/configuration.nix)
        ];
      };
    };
  };
}

This can be deployed by

nixos-rebuild switch  --target-host root@<hostname> --flake '.#nixmachine'

A new host can be added by adding a new entry inside nixosConfigurations.

Should I do this?

I think this is an easier workflow for me to manage and debug, However a lot of things in this space has non-linear learning curves and has sparse documentation at best.

So maybe a tool with better documentation might work well for you?

This also relies heavily on flakes, which is “experimental”. In my experience, this translate to everyone uses flakes, but good documenation is hard to come by (I've herd good things about https://nixos-and-flakes.thiscute.world/, but haven't read it)

Comments?

Hit me up @nomycommit

When we ran pkg upgrade, the ruby version got bumped. But mastodon needs a specific ruby version to run, and there goes the server.

Thankfully it was easy to restore.

1. Nuke the pkg and grab the code from git

2. Install the correct ruby version with rbenv

   rbenv install 3.0.6
   

3. setup mastodon again (better check the project readme) with bunch of bundle exec incantations.

4. The side effect of loosing packages was that we also don't have service files anymore. While there were @c@bow.st/110429847337542590">plenty of examples and help, we eventually ran it with supervisor with config, which is far simpler than service files IMO.

   • mastodon-web

[program:mastodon_web]
command=/usr/local/www/mastodon/.rbenv/shims/bundle exec puma -C config/puma.rb ; the program (relative uses PATH, can take args)
directory=/usr/local/www/mastodon/live                ; directory to cwd to before exec (def no cwd)
autostart=true                ; start at supervisord start (default: true)
autorestart=true              ; retstart at unexpected quit (default: true)
startsecs=10                  ; number of secs prog must stay running (def. 1)
startretries=3                ; max # of serial start failures (default 3)
user=mastodon                   ; setuid to this UNIX account to run the program
redirect_stderr=true          ; redirect proc stderr to stdout (default false)
stdout_logfile=/var/log/mastodon/web.log        ; stdout log path, NONE for none; default AUTO
environment=RAILS_ENV=production,PATH=/usr/local/www/mastodon/.rbenv/shims:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin

• sidekiq

[program:mastodon_sidekiq]
command=/usr/local/www/mastodon/.rbenv/shims/bundle exec sidekiq -c 25 ; the program (relative uses PATH, can take args)
directory=/usr/local/www/mastodon/live                ; directory to cwd to before exec (def no cwd)
autostart=true                ; start at supervisord start (default: true)
autorestart=true              ; retstart at unexpected quit (default: true)
startsecs=10                  ; number of secs prog must stay running (def. 1)
startretries=3                ; max # of serial start failures (default 3)
user=mastodon                   ; setuid to this UNIX account to run the program
redirect_stderr=true          ; redirect proc stderr to stdout (default false)
stdout_logfile=/var/log/mastodon/sidekiq.log        ; stdout log path, NONE for none; default AUTO
environment=RAILS_ENV=production,PATH=/usr/local/www/mastodon/.rbenv/shims:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin

• mastodon streaming

[program:mastodon_streaming]
command=/usr/local/bin/node ./streaming ; the program (relative uses PATH, can take args)
directory=/usr/local/www/mastodon/live                ; directory to cwd to before exec (def no cwd)
autostart=true                ; start at supervisord start (default: true)
autorestart=true              ; retstart at unexpected quit (default: true)
startsecs=10                  ; number of secs prog must stay running (def. 1)
startretries=3                ; max # of serial start failures (default 3)
user=mastodon                   ; setuid to this UNIX account to run the program
redirect_stderr=true          ; redirect proc stderr to stdout (default false)
stdout_logfile=/var/log/mastodon/streaming.log        ; stdout log path, NONE for none; default AUTO
environment=NODE_ENV=production,PATH=/usr/local/www/mastodon/.rbenv/shims:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin

Its all very janky and defintily a hotfix but it works for now. Maybe one of these days we will migrate to nomad.

Happy to report it works, and we have notwork.in