CSI-related empty directories left behind after client alloc GC

[tgross]

When an allocation that mounts a CSI volume starts, we create directories under $datadir/client/csi/node/$plugin_id to stage and publish the mount point for the volume. This results in a directory tree like the following (where 55113309 consumes the volume and 13bf0a2a and 60e62cfc are plugins).

$ sudo tree /var/nomad/data/client/csi
/var/nomad/data/client/csi
├── controller
│   └── org.democratic-csi.nfs
├── node
│   └── org.democratic-csi.nfs
│       ├── per-alloc
│       │   └── 55113309-8135-fd80-112b-d9f0f2c4cc6f
│       │       └── csi-volume-nfs
│       │           └── rw-file-system-single-node-writer # <- this is a bind-mount mount point
│       │               └── test.txt
│       └── staging
│           └── csi-volume-nfs
│               └── rw-file-system-single-node-writer # <- this is a bind-mount mount point
│                   └── test.txt
└── plugins
    ├── 13bf0a2a-7866-7ded-8436-2c53f1268a41
    │   └── csi.sock
    └── 60e62cfc-0bbd-19ff-8f4d-a97b8e17d5cd
        └── csi.sock

14 directories, 4 files

When the allocation is stopped, we unmount the mount points but the per-alloc directory and staging directory for the volume is left behind in the CSI node plugin's working directory. That's would be ok if we cleaned it up during client GC (we do the same for the alloc dir, after all). But we don't:

$ sudo tree /var/nomad/data/client/csi
/var/nomad/data/client/csi
├── controller
│   └── org.democratic-csi.nfs
├── node
│   └── org.democratic-csi.nfs
│       ├── per-alloc
│       │   └── 55113309-8135-fd80-112b-d9f0f2c4cc6f # <-- this should be gone
│       │       └── csi-volume-nfs
│       └── staging
│           └── csi-volume-nfs # <-- this should be gone
└── plugins
    ├── 13bf0a2a-7866-7ded-8436-2c53f1268a41
    │   └── csi.sock
    └── 60e62cfc-0bbd-19ff-8f4d-a97b8e17d5cd
        └── csi.sock

12 directories, 2 files

The "leak" here is a couple of empty directories, but we should tidy up after ourselves.