Skip to content

Commit

Permalink
Update printf code, make benchmark code simpler and more general,
Browse files Browse the repository at this point in the history
explain how to use `.env` file with `make` commands,
use default in default for compose.
  • Loading branch information
veritas9872 committed Dec 27, 2021
1 parent 939a50c commit 576bbf8
Show file tree
Hide file tree
Showing 5 changed files with 32 additions and 23 deletions.
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -302,7 +302,7 @@ ARG PASSWD=ubuntu
RUN groupadd -g ${GID} ${GRP} && \
useradd --shell /bin/zsh --create-home -u ${UID} -g ${GRP} \
-p $(openssl passwd -1 ${PASSWD}) ${USR} && \
printf "%s ALL=(ALL) NOPASSWD:ALL" ${GRP} >> /etc/sudoers && \
printf "${GRP} ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers && \
usermod -aG sudo ${USR}

USER ${USR}
Expand All @@ -317,7 +317,7 @@ ENV PIP_DOWNLOAD_CACHE=$HOME/.cache/pip
# Also expects home directory to be in the default location.
WORKDIR $HOME/.zsh
RUN git clone https://github.com/sindresorhus/pure.git $HOME/.zsh/pure
RUN printf "fpath+=%s/.zsh/pure\nautoload -Uz promptinit; promptinit\nprompt pure" $HOME >> $HOME/.zshrc
RUN printf "fpath+=$HOME/.zsh/pure\nautoload -Uz promptinit; promptinit\nprompt pure" >> $HOME/.zshrc

# `PROJECT_ROOT` is where the project code will reside.
ARG PROJECT_ROOT=/opt/project
Expand Down
17 changes: 14 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
[![GitHub stars](https://img.shields.io/github/stars/veritas9872/PyTorch-Universal-Docker-Template?style=flat-square)](https://github.com/veritas9872/PyTorch-Universal-Docker-Template/stargazers)
[![GitHub issues](https://img.shields.io/github/issues/veritas9872/PyTorch-Universal-Docker-Template?style=flat-square)](https://github.com/veritas9872/PyTorch-Universal-Docker-Template/issues)
[![GitHub forks](https://img.shields.io/github/forks/veritas9872/PyTorch-Universal-Docker-Template?style=flat-square)](https://github.com/veritas9872/PyTorch-Universal-Docker-Template/network)
[![Github All Releases](https://img.shields.io/github/downloads/veritas9872/PyTorch-Universal-Docker-Template/total.svg?style=flat-square)]()
[![GitHub license](https://img.shields.io/github/license/veritas9872/PyTorch-Universal-Docker-Template?style=flat-square)](https://github.com/veritas9872/PyTorch-Universal-Docker-Template/blob/main/LICENSE)
[![Twitter](https://img.shields.io/twitter/url?style=social&url=https%3A%2F%2Fgithub.com%2Fveritas9872%2FPyTorch-Universal-Docker-Template?style=flat-square)](https://twitter.com/intent/tweet?text=Awesome_Project!!!:&url=https%3A%2F%2Fgithub.com%2Fveritas9872%2FPyTorch-Universal-Docker-Template)

Expand Down Expand Up @@ -569,12 +568,24 @@ add `COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1` to any
`docker compose` commands being used.


### Tip

The `.env` file does not work with the `Makefile` by default.
However, typing in the configurations for each run can be tedious.
To use the `.env` file for the `make` commands, use the following technique to
give all the variables in the `.env` file to the `make` command.

`make COMMAND $(tr '\n' ' ' < .env)`

Example: `make all-full $(tr '\n' ' ' < .env)`.


## Compose as Best Practice

Docker Compose is a far superior option to using custom shell scripts for each environment.
Not only does it gather all variables and commands for both build and run into a single file,
but its native integration with Docker means that
it makes complicated Docker setups simple to implement.
but its native integration with Docker means that it makes complicated Docker
build/run setups simple to implement.

I wish to emphasize that using Docker Compose this way is a general-purpose technique
that does not depend on anything about this project.
Expand Down
23 changes: 10 additions & 13 deletions benchmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,24 +17,24 @@
and other antivirus programs for best performance.
The hit to code performance from antivirus programs is nontrivial.
"""
from collections import namedtuple
import subprocess
import warnings
from collections import namedtuple
from typing import Sequence, Union

import torch
from torch import nn, Tensor
from torch import autocast, nn, Tensor
from torchvision.models import (
vgg19,
resnet50,
)
from torchvision.models.detection import (
retinanet_resnet50_fpn,
)
from torchvision.models.segmentation import (
fcn_resnet50,
deeplabv3_resnet50,
)
from torchvision.models.detection import (
retinanet_resnet50_fpn,
)
from torchvision.models.video import r3d_18
# Too useful to do without, even if it is an external library.
from tqdm import tqdm
Expand Down Expand Up @@ -82,19 +82,14 @@ def infer(
inputs = tuple(torch.rand(*s, device=device) for s in input_shapes)
if enable_scripting:
network = torch.jit.trace(network, inputs)
if enable_amp:
with torch.cuda.amp.autocast():
elapsed_time = _infer(
network=network,
inputs=inputs,
num_steps=num_steps
)
else:

with autocast(device_type=device.type, enabled=enable_amp):
elapsed_time = _infer(
network=network,
inputs=inputs,
num_steps=num_steps
)

return elapsed_time


Expand Down Expand Up @@ -132,8 +127,10 @@ def get_device(gpu: int) -> torch.device:
return device


# Configuration container.
Config = namedtuple('Config', ('name', 'network', 'input_shapes'))


def measure(
cfgs: Sequence[Config],
num_steps: int,
Expand Down
7 changes: 4 additions & 3 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ services:
dockerfile: Dockerfile
cache_from: # Useful if cache images have been created with the Makefile commands beforehand.
- pytorch_source:${INSTALL_NAME:-build_install}
- pytorch_source:${TORCH_NAME:-build_torch-v1.10.1}
- pytorch_source:${TORCH_NAME:-build_torch-${PYTORCH_VERSION_TAG:-v1.10.1}}
# All arguments given during the build with must be respecified
# in `args` to prevent a cache miss from occurring.
# Default values of the `Dockerfile` (but not the `Makefile`) may be omitted.
Expand Down Expand Up @@ -90,8 +90,9 @@ services:
# Note that the `INSTALL_NAME_FULL` and `TORCH_NAME_FULL` variables have
# different variable names from the `train` service.
# This may cause a cache miss if set incorrectly.
- pytorch_source:${INSTALL_NAME_FULL:-build_install-ubuntu18.04-cuda10.2-cudnn8-py3.9}
- pytorch_source:${TORCH_NAME_FULL:-build_torch-v.1.10.1-ubuntu18.04-cuda10.2-cudnn8-py3.9}
- # Using defaults inside defaults, which may be hard to read.
- pytorch_source:${INSTALL_NAME_FULL:-build_install-${LINUX_DISTRO:-ubuntu}${DISTRO_VERSION:-18.04}-cuda${CUDA_VERSION:-10.2}-cudnn${CUDNN_VERSION:-8}-py${PYTHON_VERSION:-3.9}}
- pytorch_source:${TORCH_NAME_FULL:-build_torch-${PYTORCH_VERSION_TAG:-v1.10.1}-${LINUX_DISTRO:-ubuntu}${DISTRO_VERSION:-18.04}-cuda${CUDA_VERSION:-10.2}-cudnn${CUDNN_VERSION:-8}-py${PYTHON_VERSION:-3.9}}
args: # Equivalent to `--build-arg`. Set to default values for `*-full`.
LINUX_DISTRO: ${LINUX_DISTRO:-ubuntu}
DISTRO_VERSION: ${DISTRO_VERSION:-18.04}
Expand Down
4 changes: 2 additions & 2 deletions ngc.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ ARG PASSWD=ubuntu
RUN groupadd -g ${GID} ${GRP} && \
useradd --shell /bin/zsh --create-home -u ${UID} -g ${GRP} \
-p $(openssl passwd -1 ${PASSWD}) ${USR} && \
printf "%s ALL=(ALL) NOPASSWD:ALL" ${GRP} >> /etc/sudoers && \
printf "${GRP} ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers && \
usermod -aG sudo ${USR}

USER ${USR}
Expand All @@ -76,7 +76,7 @@ ENV PIP_DOWNLOAD_CACHE=$HOME/.cache/pip

WORKDIR $HOME/.zsh
RUN git clone https://github.com/sindresorhus/pure.git $HOME/.zsh/pure
RUN printf "fpath+=%s/.zsh/pure\nautoload -Uz promptinit; promptinit\nprompt pure" $HOME >> $HOME/.zshrc
RUN printf "fpath+=$HOME/.zsh/pure\nautoload -Uz promptinit; promptinit\nprompt pure" >> $HOME/.zshrc

ARG PROJECT_ROOT=/opt/project
WORKDIR ${PROJECT_ROOT}
Expand Down

0 comments on commit 576bbf8

Please sign in to comment.