Use idiomatic features of Snakemake

[huddlej]

Context

The Snakemake workflow for ingest was written to map as directly from the original ingest shell scripts to Snakemake as possible. As a result, some of the workflow's implementation doesn't follow standard Snakemake idioms.

This issue comes from a code review comment of the original Snakemake workflow PR.

Description

We should replace shell-based idioms with Snakemake's own idioms including:

• replacing lists of shell() function calls inside a run block with a single shell block that calls multiple shell commands
• replacing for loops inside run blocks with a single shell block and wildcards in place of the for loop
• moving conditional logic out of shell script and into standalone scripts or as workflow-level conditionals
• using Snakemake's retry-attempts argument where possible
• replacing rule orders with conditionally defined rules
• ensure temporary files are being removed appropriately and clean-up is occurring as desired. See this PR review comment.