How to attach Packet Storage to the nix servers ?

[abhi18av]

Hi @grahamc

I'd like to know how could I connect the storage to nixos server.

I've been following the guides here

https://help.packet.net/article/63-elastic-block-storage

https://github.com/packethost/packet-block-storage

[root@emilynabhinav:~]# bash /usr/bin/packet-block-storage-attach
grep: /etc/iscsi/initiatorname.iscsi: No such file or directory
/usr/bin/packet-block-storage-attach: line 174: /etc/iscsi/initiatorname.iscsi: No such file or directory
grep: /etc/iscsi/iscsid.conf: No such file or directory
grep: /etc/iscsi/iscsid.conf: No such file or directory
grep: /etc/iscsi/iscsid.conf: No such file or directory
grep: /etc/iscsi/iscsid.conf: No such file or directory
grep: /etc/iscsi/iscsid.conf: No such file or directory
portal: 10.144.136.57 iqn: iqn.2013-05.com.daterainc:tc:01:sn:236f2201702cb96
Error: We couldn't discover targets on 10.144.136.57
Error: We couldn't log in iqn iqn.2013-05.com.daterainc:tc:01:sn:236f2201702cb96
portal: 10.144.140.3 iqn: iqn.2013-05.com.daterainc:tc:01:sn:236f2201702cb96
Error: We couldn't discover targets on 10.144.140.3
Error: We couldn't log in iqn iqn.2013-05.com.daterainc:tc:01:sn:236f2201702cb96
Error: Block device /dev/mapper/volume-2351d492 is NOT available for use

Could you please help me out ?

[bgamari]

I have a NixOS module which appears to work: https://gist.github.com/bgamari/c352cf7b6bada19f53678359308526fe.

Note that systemd.services.multipathd is almost identical to the multipathd.service that ships with nixpkgs.multipath-tools but with the absolute paths fixed. It would probably be cleaner if the multipath-tools derivation were fixed to install a functional service file.

[abhi18av]

Ohh, I see @bgamari!

Thanks for this 👍

[grahamc]

Reopening so it is easier to find later :)

[mmlb]

@bgamari is there more to that module? AFAICS the packet-block-storage-attach script will not be nixos friendly (try to update conf files)

[bgamari]

@mmlb there is not. It's definitely somewhat of a hack but it works well enough that I have been leaving it alone.

However, there is one important caveat: the multipathd.service dependencies defined in the module are excessive and may result in dependency loops, breaking systemd at boot. Specifically, multipathd.service needn't be started before local-fs-pre.target. I'll update the Gist with the current state of the module

[ethindp]

So, after discussing this on Slack with @bgamari and @grahamc, this now is trivial to set up. This is what I do (though you may make differences if you so choose):

1. I download the gist he posted the link to with curl. (Note: for some reason, raw Git gists fail to download, returning blank HTTP bodies, so I'd recommend you paste gist directly into the file I discuss below.)
2. I create a file, /etc/nixos/iscsi.nix, and store the nix expression there.
3. I open /etc/nixos/configuration.nix and add ./iscsi.nix to the import list as the very last item (this ensures that important things are not overridden or lost).
4. I make any other modifications I deem necessary, then close my editor.
5. I run nixos-rebuild switch --upgrade.

Note: if you have iscsid running while your doing this, kill it before running nixos-rebuild switch --upgrade. If you don't, the service will throw complaints because it can't bind itself.

[grahamc]

Current state of the gist is as follows:

{ pkgs, config, lib, ... }:

let 
  packet-block-storage = 
    pkgs.stdenv.mkDerivation {
      name = "packet-block-storage";
      src = pkgs.fetchFromGitHub {
        owner = "packethost";
        repo = "packet-block-storage";
        rev = "4be27cbca7a924b4de7af059d5ac30c2aa5c9e6f";
        sha256 = "0h4lpvz7v6xhl14kkwrjp202lbagj6wp2wqgrqdc6cfb4h0mf9fq";
      };
      nativeBuildInputs = [ pkgs.makeWrapper ];
      installPhase = 
        let
          deps = with pkgs; [
            which jq curl gawk kmod openiscsi multipath-tools
          ];
          wrapIt = prog: ''
            cp ${prog} $out/bin
            chmod ugo+rx $out/bin/${prog}
            substituteInPlace $out/bin/${prog} --replace /lib/udev/scsi_id ${pkgs.udev}/lib/udev/scsi_id
            wrapProgram $out/bin/${prog} --prefix PATH : ${lib.makeBinPath deps}
          '';
        in ''
          mkdir -p $out/bin
          ${wrapIt "packet-block-storage-attach"}
          ${wrapIt "packet-block-storage-detach"}
      '';
    };
  
in {
  environment.systemPackages = [ packet-block-storage pkgs.multipath-tools pkgs.openiscsi ];

  systemd.sockets.multipathd = {
    description = "multipathd control socket";
    before = [ "sockets.target" ];

    listenStreams = ["@/org/kernel/linux/storage/multipathd"];
  };

  systemd.services.multipathd = {
    description = "Device-Mapper Multipath Device Controller";
    before = [
      "iscsi.service" "iscsid.service"
    ];
    after = [ "multipathd.socket" "systemd-udevd.service" ];
    unitConfig = {
      DefaultDependencies = false;
    };
    wants = [ "multipathd.socket" "blk-availability.service" ];
    conflicts = ["shutdown.target"];
    serviceConfig = {
      Type = "notify";
      NotifyAccess = "main";
      LimitCORE = "infinity";
      ExecStartPre = "${pkgs.kmod}/bin/modprobe -a dm-multipath";
      ExecStart = "${pkgs.multipath-tools}/bin/multipathd -d -s";
      ExecReload = "${pkgs.multipath-tools}/bin/multipathd reconfigure";
    };
  };

  systemd.targets.iscsi = {
    description = "iSCSI mounts";
    wants = [ "iscsid.service" "attach-block-storage.service" ];
    wantedBy = [ "remote-fs.target" ];
  };

  systemd.services.iscsid = {
    description = "iSCSI daemon";
    path = [ pkgs.openiscsi ];
    script = "iscsid -f";
  };

  systemd.services.attach-block-storage = {
    description = "Attach Packet.net block storage";
    requires = [ "network-online.target" "iscsid.service" "multipathd.service" ];
    after = [ "network-online.target" "iscsid.service" "multipathd.service" ];
    before = [ ];
    script = ''
      ${packet-block-storage}/bin/packet-block-storage-attach
    '';
  };
}