Memory consumption of pusher

[mstump]

We use CircleCI as our CI service which imposes a 4GB memory limit. When pushing large docker images (>1GB) pusher will consume several gigs of memory causing the kernel to kill the process upon memory exhaustion. Pusher should restrict active threads or total memory consumption to fit within the limits of the host, or provide configuration mechanisms to limit resource usage.

INFO: Running command line: bazel-bin/datastax-enterprise-v5.1.7-image-push

asia.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-{GIT_VERSION}-134 was resolved to asia.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-2e4767a-134
asia.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-2e4767a-134 was published with digest: sha256:93ce4c62d1ac3792a14eac10fbcf5a433a68c7602a5ddf9c2f05f69f0a378166
asia.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-{GIT_VERSION} was resolved to asia.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-2e4767a
asia.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-2e4767a was published with digest: sha256:93ce4c62d1ac3792a14eac10fbcf5a433a68c7602a5ddf9c2f05f69f0a378166
asia.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7 was resolved to asia.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7
asia.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7 was published with digest: sha256:93ce4c62d1ac3792a14eac10fbcf5a433a68c7602a5ddf9c2f05f69f0a378166
/home/circleci/.cache/bazel/_bazel_circleci/f16e36219ef33c22efc2ad20f3e3775c/execroot/distroless/bazel-out/k8-fastbuild/bin/datastax-enterprise-v5.1.7-image-push.runfiles/distroless/datastax-enterprise-v5.1.7-image-push.7.push: line 26: 24978 Killed                  ${RUNFILES}/distroless/../containerregistry/pusher --name=containers.prod.noc.vorstella.com/vorstella-noc-dev/datastax-enterprise:v5.1.7-{GIT_VERSION}-134 --stamp-info-file=${RUNFILES}/distroless/stable-status.txt --stamp-info-file=${RUNFILES}/distroless/volatile-status.txt --config=${RUNFILES}/distroless/datastax-enterprise/datastax-enterprise.config --digest=${RUNFILES}/distroless/base/base-layer.tar.gz.sha256 --digest=${RUNFILES}/distroless/base/debug-layer.tar.gz.sha256 --digest=${RUNFILES}/distroless/cc/debug-layer.tar.gz.sha256 --digest=${RUNFILES}/distroless/java-oracle/debug-layer.tar.gz.sha256 --digest=${RUNFILES}/distroless/datastax-enterprise/datastax-enterprise-layer.tar.gz.sha256 --layer=${RUNFILES}/distroless/base/base-layer.tar.gz --layer=${RUNFILES}/distroless/base/debug-layer.tar.gz --layer=${RUNFILES}/distroless/cc/debug-layer.tar.gz --layer=${RUNFILES}/distroless/java-oracle/debug-layer.tar.gz --layer=${RUNFILES}/distroless/datastax-enterprise/datastax-enterprise-layer.tar.gz "$@"
E0427 15:05:08.664109   24983 docker_session_.py:335] Error during upload of: containers.prod.noc.vorstella.com/vorstella-noc-dev/datastax-enterprise:v5.1.7-2e4767a
F0427 15:05:08.667324   24983 fast_pusher_.py:162] Error publishing containers.prod.noc.vorstella.com/vorstella-noc-dev/datastax-enterprise:v5.1.7-2e4767a: 
containers.prod.noc.vorstella.com/vorstella-noc-dev/datastax-enterprise:v5.1.7-{GIT_VERSION} was resolved to containers.prod.noc.vorstella.com/vorstella-noc-dev/datastax-enterprise:v5.1.7-2e4767a
/home/circleci/.cache/bazel/_bazel_circleci/f16e36219ef33c22efc2ad20f3e3775c/execroot/distroless/bazel-out/k8-fastbuild/bin/datastax-enterprise-v5.1.7-image-push.runfiles/distroless/datastax-enterprise-v5.1.7-image-push.8.push: line 26: 24979 Killed                  ${RUNFILES}/distroless/../containerregistry/pusher --name=containers.prod.noc.vorstella.com/vorstella-noc-dev/datastax-enterprise:v5.1.7 --stamp-info-file=${RUNFILES}/distroless/stable-status.txt --stamp-info-file=${RUNFILES}/distroless/volatile-status.txt --config=${RUNFILES}/distroless/datastax-enterprise/datastax-enterprise.config --digest=${RUNFILES}/distroless/base/base-layer.tar.gz.sha256 --digest=${RUNFILES}/distroless/base/debug-layer.tar.gz.sha256 --digest=${RUNFILES}/distroless/cc/debug-layer.tar.gz.sha256 --digest=${RUNFILES}/distroless/java-oracle/debug-layer.tar.gz.sha256 --digest=${RUNFILES}/distroless/datastax-enterprise/datastax-enterprise-layer.tar.gz.sha256 --layer=${RUNFILES}/distroless/base/base-layer.tar.gz --layer=${RUNFILES}/distroless/base/debug-layer.tar.gz --layer=${RUNFILES}/distroless/cc/debug-layer.tar.gz --layer=${RUNFILES}/distroless/java-oracle/debug-layer.tar.gz --layer=${RUNFILES}/distroless/datastax-enterprise/datastax-enterprise-layer.tar.gz "$@"
us.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7 was resolved to us.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7
us.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7 was published with digest: sha256:93ce4c62d1ac3792a14eac10fbcf5a433a68c7602a5ddf9c2f05f69f0a378166
us.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-{GIT_VERSION}-134 was resolved to us.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-2e4767a-134
us.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-2e4767a-134 was published with digest: sha256:93ce4c62d1ac3792a14eac10fbcf5a433a68c7602a5ddf9c2f05f69f0a378166
us.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-{GIT_VERSION} was resolved to us.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-2e4767a
us.gcr.io/vorstella-noc-dev/datastax-enterprise:v5.1.7-2e4767a was published with digest: sha256:93ce4c62d1ac3792a14eac10fbcf5a433a68c7602a5ddf9c2f05f69f0a378166
ERROR: Non-zero return code '1' from command: Process exited with status 1