Deployment to NixOS failing #444

aveltras · 2019-06-01T22:11:46Z

I'm trying to use ob deploy with the default obelisk skeleton (no code edited here).
The ob deploy push errors with the following message

ob deploy push -v
Starting Obelisk </nix/store/a5l11f3dcgvh13wxn4kfnxq27ydlzrjm-obelisk-command-0.1/bin/.ob-wrapped> args=["deploy","push","-v"] logging-level=Debug
Creating process: 'nix-build' './.obelisk/impl' '-A' 'command' '--no-out-link'
✔ Built  on ./.obelisk/impl [command]
Handing off to /nix/store/a5l11f3dcgvh13wxn4kfnxq27ydlzrjm-obelisk-command-0.1/bin/ob
Starting Obelisk </nix/store/a5l11f3dcgvh13wxn4kfnxq27ydlzrjm-obelisk-command-0.1/bin/.ob-wrapped> args=["--no-handoff","deploy","push","-v"] logging-level=Debug
Creating process: 'nix-build' '/home/romain/Code/obapp-deploy/src' '-A' 'server.system' '--argstr' 'hostName' 'XXXXXXXXX' '--argstr' 'adminEmail' 'myname@myapp.com' '--argstr' 'routeHost' 'XXXXXXXXX' '--argstr' 'version' 'b4330893718db2e11b53e9b158bba50d8d225935' '--arg' 'enableHttps' 'false' '--no-out-link'
trace: WARNING: ignoring dependency on doctest
✔ Built  on /home/romain/Code/obapp-deploy/src [server.system]
Creating process: 'nix-copy-closure' '-v' '--to' '--use-substitutes' 'root@XXXXXXXXX' '--gzip' '/nix/store/f4hbz2k1c1dbj3lyw70rmhsmzi42rajg-nixos-system-XXXXXXXXX-18.09pre-git'
✔ Uploading closures
Creating process: 'rsync' '-e ssh -o UserKnownHostsFile=/home/romain/Code/obapp-deploy/backend_known_hosts -o StrictHostKeyChecking=yes -i /home/romain/Code/obapp-deploy/ssh_key' '-qarvz' '/home/romain/Code/obapp-deploy/config' 'root@XXXXXXXXX:/var/lib/backend'
✔ Uploading config
Creating process: 'ssh' '-o' 'UserKnownHostsFile=/home/romain/Code/obapp-deploy/backend_known_hosts' '-o' 'StrictHostKeyChecking=yes' '-i' '/home/romain/Code/obapp-deploy/ssh_key' 'root@XXXXXXXXX' 'nix-env -p /nix/var/nix/profiles/system --set /nix/store/f4hbz2k1c1dbj3lyw70rmhsmzi42rajg-nixos-system-XXXXXXXXX-18.09pre-git && /nix/var/nix/profiles/system/bin/switch-to-configuration switch'
updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/xvda...
Installing for i386-pc platform.
/nix/store/c6vs2m90d166b8h2b62j3yjyvqp8va8g-grub-2.02/sbin/grub-install: error: cannot find a GRUB drive for /dev/xvda.  Check your device.map.
/nix/store/mwgg6ih5n9rfqc9b1xykbk5g91686fx6-install-grub.pl: installation of GRUB on /dev/xvda failed
Process exited with code 1; 'ssh' '-o' 'UserKnownHostsFile=/home/romain/Code/obapp-deploy/backend_known_hosts' '-o' 'StrictHostKeyChecking=yes' '-i' '/home/romain/Code/obapp-deploy/ssh_key' 'root@XXXXXXXXX' 'nix-env -p /nix/var/nix/profiles/system --set /nix/store/f4hbz2k1c1dbj3lyw70rmhsmzi42rajg-nixos-system-XXXXXXXXX-18.09pre-git && /nix/var/nix/profiles/system/bin/switch-to-configuration switch'
✖ Switching to new configuration

I don't think it's an issue with the ssh credentials since the closures uploading step completes successfully.

Trying to execute the last command directly on the server as root

nix-env -p /nix/var/nix/profiles/system --set /nix/store/f4hbz2k1c1dbj3lyw70rmhsmzi42rajg-nixos-system-XXXXXXXXX-18.09pre-git && /nix/var/nix/profiles/system/bin/switch-to-configuration switch
updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/xvda...
Installing for i386-pc platform.
/nix/store/c6vs2m90d166b8h2b62j3yjyvqp8va8g-grub-2.02/sbin/grub-install: error: cannot find a GRUB drive for /dev/xvda.  Check your device.map.
/nix/store/mwgg6ih5n9rfqc9b1xykbk5g91686fx6-install-grub.pl: installation of GRUB on /dev/xvda failed

The NixOs I'm deploying too isn't ec2. It's an installation generated with https://github.com/elitak/nixos-infect which gave me those pretty barebone config files :

configuration.nix

{ ... }: {
  imports = [
    ./hardware-configuration.nix
  ];

  boot.cleanTmpDir = true;
  networking.hostName = "XXXXXXXX";
  networking.firewall.allowPing = true;
  services.openssh.enable = true;
  users.users.root.openssh.authorizedKeys.keys = [
    "ssh-rsa XXXXXXXXX
  ];
}

hardware-configuration.nix

{ ... }:
{
  imports = [ <nixpkgs/nixos/modules/profiles/qemu-guest.nix> ];
  boot.loader.grub.device = "/dev/sda";
  fileSystems."/" = { device = "/dev/sda1"; fsType = "ext4"; };
}

Also, the NixOs channel used is 19.03.

Are those errors linked in any way to Obelisk scripts or should i turn somewhere else for help ?

The text was updated successfully, but these errors were encountered:

pshemass · 2019-06-03T04:37:21Z

i'm getting the same error on VM builded with https://github.com/NixOS/nixpkgs/blob/master/nixos/maintainers/scripts/azure/create-azure.sh

I couldn't have found any occurrences of /dev/xvda in obelisk code so it's probably someone else code.

Any ideas what that could be?

aveltras · 2019-06-03T10:40:18Z

The /dev/xvda seems to be an expected value for the amazon images according to https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/amazon-image.nix. The failing script is this one https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/loader/grub/install-grub.pl which takes it's device config from an xml file apparently, still haven't found where that config is defined though.

pshemass · 2019-06-04T02:01:48Z

it makes sense if you take a look on https://github.com/obsidiansystems/obelisk/blob/develop/default.nix#L178 which is called by deploy push in https://github.com/obsidiansystems/obelisk/blob/develop/lib/command/src/Obelisk/Command/Deploy.hs#L118

the question is how to work around that?

matthewbauer · 2019-06-05T17:28:59Z

It looks like ob deploy assumes you are deploying to ec2 right now. It shouldn't be too much work to make it work somewhere else though. Just remove amazon-image.nix module when --hostname doesn't end with .compute.amazonaws.com. Everything else should work okay AFAICT.

jrodriguesimg · 2019-08-07T23:15:20Z

I'm trying to deploy to a computer were I installed NixOS and I'm having trouble deploying to it. I guess I should remove amazon-image.nix but I don't know what that means. Can some one help me?

alexfmpe · 2019-08-07T23:21:58Z

I'm guessing

obelisk/default.nix

Line 171 in 7dd157b

(pkgs.path + /nixos/modules/virtualisation/amazon-image.nix)

(likely what @pshemass's link pointed to on an older develop)

vaibhavsagar · 2019-08-07T23:22:02Z

@jrodriguesimg you haven't provided enough information for us to help you. What is the error you are seeing? Can you share your ob deploy configuration?

jrodriguesimg · 2019-08-07T23:38:40Z

@vaibhavsagar it's the same error that started this issue. It shouldn't need to change grub just to deploy a web server.

Thanks. @alexfmpe I removed the amazon-image.nix part. Can you help me install the new ob command and replace the current one?

alexfmpe · 2019-08-07T23:51:40Z

Can you help me install the new ob command and replace the current one?

Added to README

jrodriguesimg · 2019-08-09T00:38:50Z

I tried removing the amazon-image.nix and installed the new ob command, but still got the same error while deploying. I even tried running from the result/bin/ob in the obelisk directory.

Change I made to default.nix:

...
 mkBaseEc2 = { hostName, routeHost, enableHttps, adminEmail, ... }: {...}: {
      imports = [
        pkgs.path  # /nixos/modules/virtualisation/amazon-image.nix)
      ];
      networking = {
        inherit hostName;
        firewall.allowedTCPPorts = if enableHttps then [ 80 443 ] else [ 80 ];
      };
      security.acme.certs = if enableHttps then {
        "${routeHost}".email = adminEmail;
      } else {};
      ec2.hvm = true;
    };
...

The error (the folder ../obelisk/ is where I cloned the obelisk repository):

 ~/progs/obelisk/web-app-test-deploy $../obelisk/result/bin/ob deploy push
✔ Built  on ./.obelisk/impl [command]
✔ Built  on /home/joao/progs/obelisk/web-app-test-deploy/src [server.system]
✔ Uploading closures
✔ Uploading config
updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/xvda...
Installing for i386-pc platform.
Tip: Press Ctrl+e to display full output
/nix/store/c6vs2m90d166b8h2b62j3yjyvqp8va8g-grub-2.02/sbin/grub-install: error: cannot find a GRUB drive for /dev/xvda.  Check your device.map.
/nix/store/mwgg6ih5n9rfqc9b1xykbk5g91686fx6-install-grub.pl: installation of GRUB on /dev/xvda failed
Process exited with code 1; 'ssh' '-o' 'UserKnownHostsFile=/home/joao/progs/obelisk/web-app-test-deploy/backend_known_hosts' '-o' 'StrictHostKeyChecking=yes' '-i' '/home/joao/progs/obelisk/web-app-test-deploy/ssh_key' 'root@192.168.1.200' 'nix-env -p /nix/var/nix/profiles/system --set /nix/store/jwqkckc7fgdzgksd4y3s4hkyb1nl40rx-nixos-system-192.168.1.200-18.09pre-git && /nix/var/nix/profiles/system/bin/switch-to-configuration switch'
✖ Switching to new configuration

ingenieroariel · 2019-12-03T18:41:14Z

I have replicated this issue with the 19.09 .ova file on https://nixos.org/nixos/download.html and rev 81c31de (Nov 28, obelisk develop branch):

The output was:

x@hydrogen /d/o/t/prod> ../../obelisk/result/bin/ob deploy push                                                                                                                                                    
✔ Built  on ./.obelisk/impl [command]
Tip: Press Ctrl+e to display full output
✔ Built  on /d/os/testproject/prod/src [server.system]
✔ Uploading closures
✔ Uploading config
updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/xvda...
Installing for i386-pc platform.
/nix/store/zfhz7fjapm706xb247cqp4ysw9g8360n-grub-2.02/sbin/grub-install: error: cannot find a GRUB drive for /dev/xvda.  Check your device.map.
/nix/store/bldrk53j2yakrvpf25xprz6izz9xzjv8-install-grub.pl: installation of GRUB on /dev/xvda failed
Process exited with code 1; 'ssh' '-o' 'UserKnownHostsFile=/d/os/testproject/prod/backend_known_hosts' '-o' 'StrictHostKeyChecking=yes' '-i' '/d/os/testproject/prod/ssh_key' 'root@192.168.5.185' 'nix-env -p /nix/var/nix/profiles/system --set /nix/store/cb5cjsvsp69jxal47mncd4mlrc0zfx7c-nixos-system-192.168.5.185-18.09pre-git && /nix/var/nix/profiles/system/bin/switch-to-configuration switch'
✖ Switching to new configuration
x@hydrogen /d/o/t/prod>

When deploying to non amazon instances an error related to GRUB was appearing. This commit creates a new option virtualization_type that defaults to `amazon` but can be replaced by `virtualbox`, `azure`, `brightbox`, `digital-ocean`, `docker-image` and `google-compute` image. This behavior is not documented because no options other than amazon are formally supportedi in Obelisk, but makes an implicit assumption explicit. This has been reported on Github and Reddit: https://www.reddit.com/r/reflexfrp/comments/9yzqok/obelisk_deployment_issues/ and #444

3noch · 2020-03-12T21:47:16Z

It's now possible to configure for non-EC2 deployments: https://github.com/obsidiansystems/obelisk/blob/0403aa5e8cac158fa1309c9dc78b146bca6f3206/README.md#custom-non-ec2-deployment

Note that this feature isn't released yet.

hSloan added the bug Something isn't working label Jun 4, 2019

matthewbauer added the High Severity label Nov 26, 2019

ApolloUnicorn added the External label Jan 3, 2020

3noch removed the External label Jan 16, 2020

HectorAE mentioned this issue Feb 20, 2020

Support non-EC2 targets with ob deploy #658

Merged

4 tasks

3noch closed this as completed Mar 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deployment to NixOS failing #444

Deployment to NixOS failing #444

aveltras commented Jun 1, 2019

pshemass commented Jun 3, 2019

aveltras commented Jun 3, 2019

pshemass commented Jun 4, 2019

matthewbauer commented Jun 5, 2019 •

edited

Loading

jrodriguesimg commented Aug 7, 2019

alexfmpe commented Aug 7, 2019

vaibhavsagar commented Aug 7, 2019

jrodriguesimg commented Aug 7, 2019

alexfmpe commented Aug 7, 2019 •

edited

Loading

jrodriguesimg commented Aug 9, 2019 •

edited

Loading

ingenieroariel commented Dec 3, 2019

3noch commented Mar 12, 2020

Deployment to NixOS failing #444

Deployment to NixOS failing #444

Comments

aveltras commented Jun 1, 2019

pshemass commented Jun 3, 2019

aveltras commented Jun 3, 2019

pshemass commented Jun 4, 2019

matthewbauer commented Jun 5, 2019 • edited Loading

jrodriguesimg commented Aug 7, 2019

alexfmpe commented Aug 7, 2019

vaibhavsagar commented Aug 7, 2019

jrodriguesimg commented Aug 7, 2019

alexfmpe commented Aug 7, 2019 • edited Loading

jrodriguesimg commented Aug 9, 2019 • edited Loading

ingenieroariel commented Dec 3, 2019

3noch commented Mar 12, 2020

matthewbauer commented Jun 5, 2019 •

edited

Loading

alexfmpe commented Aug 7, 2019 •

edited

Loading

jrodriguesimg commented Aug 9, 2019 •

edited

Loading