Re-establish a way to auto-submit jobs on the server

[adamcantor22]

Is your feature request related to a problem? Please describe.
HPC has let us know that they will not be allowing LSF submissions on web01 anymore. As such, auto-submitting jobs is broken.

Describe the solution you'd like
First we should follow up with HPC, and see if we can get them to create a fix for us. They really should, it's not very nice of them to remove functionality. Maybe there's a way to make specific exceptions, or to get access to another node where the submission would work.

Describe alternatives you've considered
If nothing else, it seems like it should be possible to automate some process to login on the correct node and submit the job there.

[adamcantor22]

Potential solution 2: "watch the watcher", start a new watcher on a node that can submit jobs