Charliecloud

Charliecloud uses Linux user namespaces to run containers with no privileged operations or daemons and minimal configuration changes on HPC-center resources. This simple approach avoids most security risks while maintaining access to the performance and functionality already on offer. The Charliecloud container images can be built from existing Dockerfiles or Docker images with a set of simple commands in a terminal window. For additional details on Charliecloud check out the official Charliecloud documentation.

If you have never worked before with containers and/or Docker, please check out the fabulous and very easy to follow Docker tutorials of our colleagues from DFD-LAX that provide you with all the necessary knowledge in just a few minutes (access only for DLR-EOC). For a more detailed but also easy to follow documentation, see the official Docker docs.

Load terrabyte version of charliecloud

#load charliecloud in DLR Version with sqfs Support
echo "Loading Charliecloud module"
module use /dss/dsstbyfs01/pn56su/pn56su-dss-0020/usr/share/modules/files/
module load charliecloud

Generate a Charliecloud image from (Docker) registry URL

We provide a convenient script to automatically pull a docker image from the registry and convert it into a charliecloud squashfs image. To convert the latest version of the hello-world docker image:

/dss/dsstbyfs01/pn56su/pn56su-dss-0020/usr/local/bin/convertDockerImage2charliecloud.sh hello-world:latest

This will

Pull the latest version from the registry
Convert it to the charliecloud format
Add typical mountpoints at terrabyte (/dss /lrz /gpfs)
Export the charliecloud image to your current working dir as a compressed squashfs image (hello-world+latest.sqfs)

You could also run it manually via:

dockerImage="hello-world:latest"

module use /dss/dsstbyfs01/pn56su/pn56su-dss-0020/usr/share/modules/files/
module load charliecloud

# write intermediate files to local SSD for performance
export CH_IMAGE_STORAGE=/tmp/${USER}/chConvert
# and pull image from Registry
ch-image pull -s $CH_IMAGE_STORAGE $dockerImage

# convert the docker-Name syntax to the one used by charliecloud,
# allows us to keep the information on the registry image and version
chimageName=$(echo $dockerImage | sed 's#/#%#g' | sed 's#:#+#g')

# create empty folders for potential mountpoints
for f in lrz dss gpfs
do
    mkdir ${CH_IMAGE_STORAGE}/img/${chimageName}/$f
done

# recommended but optional: convert the image from the intermediate single file format to a compressed squashfs (RO, but better performance than image directory)
ch-convert /target/folder/containers/test_image /target/folder/containers/test_image.sqfs

# clean up in storageDir
rm -r -f $CH_IMAGE_STORAGE

Generate a Charliecloud image from a Dockerfile

A Charliecloud image can be generated most easily from a Dockerfile, without the necessity to install Charliecloud on your local system:

Log into login node
Create a Dockerfile on the login node or copy an existing Dockerfile from your local machine to the login node.
On the login node execute the following commands:

# load the charliecloud module
module use /dss/dsstbyfs01/pn56su/pn56su-dss-0020/usr/share/modules/files/
module load charliecloud

# create the container image like:
# ch-image build -t <image name> -f <path to Dockerfile> <Path to context directory>
# (The latter being the root of COPY instructions in the Dockerfile.)

# The --force option enables unprivileged build workarounds, which is e.g.
# important for installing and building applications with ap-get install.
ch-image build --force -t test_image -f /path/to/Dockerfile /context/directory/as/root/of/copy/instructions

# the container image will be created in a folder in /var/tmp/<USER_ID>.ch/img.
# Move the image from this location to the location you prefer:
mv /var/tmp/<USER_ID>.ch/img/test_image /target/folder

# in order to be able to bind mount the dss and other folders in a
# later step, create them in the root file tree of the container image
for f in lrz dss gpfs
do
    mkdir /target/folder/containers/test_image/$f
done

# recommended but optional: convert the image from the intermediate single file format to a compressed squashfs (RO, but better performance than image directory)
ch-convert /target/folder/containers/test_image /target/folder/containers/test_image.sqfs

Generate a Charliecloud image from an existing Docker image

If you have a Docker image on your local machine, you can convert it into a Charliecloud image tarball and transfer it to the HPDA terrabyte HPC system:

Make sure that your local system meets the following requirements:
- Linux kernel 4.4 or higher
- C11 compilers and libraries installed (if not, install it with sudo apt install build-essential or sudo yum groupinstall 'Development Tools')
- python3 installed
- Bash 4.1 or higher
- Docker installed (you can install Docker with a convenience script, as explained here: https://docs.docker.com/engine/install/ubuntu/#install-using-the-convenience-script)
Download and install Charliecloud on your local machine (see also the Charliecloud documentation):

# download the tar.gz file of the latest release from https://github.com/hpc/charliecloud/releases
# (check the webpage for the URL of the latest tarball)
wget https://github.com/hpc/charliecloud/releases/download/<VERSION>/charliecloud-<VERSION>.tar.gz

# Untar the tarball
tar -xvf charliecloud-<VERSION>.tar.gz

# Build and install Charliecloud
cd charliecloud-<VERSION>
./configure
make
sudo make install

Convert the local Docker image into a Charliecloud tarball and copy it to the login node:

ch-convert -i docker <docker image name> /path/to/where/to/write/charliecloud_image.tar.gz

scp charliecloud_image.tar.gz <USER_ID>@login.terrabyte.lrz.de:/path/on/the/dss

Log into login node
On the login node untar the Charliecloud tarball and make a folder named "dss"

# load the charliecloud module
module use /dss/dsstbyfs01/pn56su/pn56su-dss-0020/usr/share/modules/files/
module load charliecloud

# Untar the tarball
ch-convert /path/on/the/dss/charliecloud_image.tar.gz /path/on/the/dss/charliecloud_image

# in order to be able to bind mount the dss and other folders in a later step, create them
# in the root file tree of the container image
for f in lrz dss gpfs
do
    mkdir /target/folder/containers/test_image/$f
done

# recommended but optional: convert the image from the intermediate
# single file format to a compressed squashfs (RO, but better performance than image directory)
ch-convert /path/on/the/dss/charliecloud_image /path/on/the/dss/charliecloud_image.sqfs

Generate a Docker image from an existing Charliecloud image

In order to convert a Charliecloud image on terrabyte back to a Docker image, do:

Log into login node
Create a tarball of the Charliecloud image and copy it to your local machine with Docker installed:

# load the charliecloud module
module use /dss/dsstbyfs01/pn56su/pn56su-dss-0020/usr/share/modules/files/
module load charliecloud

# create the tarball:
ch-convert /path/on/the/dss/charliecloud_image /path/on/the/dss/charliecloud_image.tar.gz
# or if you have a .sqfs:
ch-convert /path/on/the/dss/charliecloud_image.sqfs /path/on/the/dss/charliecloud_image.tar.gz

scp /path/on/the/dss/charliecloud_image.tar.gz <USER_ID>@<your.local.machine>:/path/on/the/local/machine

Make sure that Charliecloud is installed on your lcoal machine (see Generate a Charliecloud image from an existing Docker image).
Execute the following commands on your local machine:

ch-convert -i tar -o docker s/path/on/the/local/machine/charliecloud_image.tar.gz docker_image_name

# The image will then appear in your docker image list:
docker image list

Run a Charliecloud container and use it in jobs

The command to run a Charliecloud container is ch-run. Since Charliecloud version 0.33, the $HOME-Directory is not automatically mounted into /home in the container anymore. In order to mount your home directory, the flag --home is needed. The /dss must be bind-mounted manually. The general usage of ch-run is:

# load the charliecloud module
module use /dss/dsstbyfs01/pn56su/pn56su-dss-0020/usr/share/modules/files/
module load charliecloud

###
# Replace /path/to/charliecloud_image by /path/to/charliecloud_image.sqfs if you are using a squashfs file.
###

# run a bash shell in the container (i.e. login to the container), mount $HOME and /dss
ch-run --home -b /dss/.:/dss/ /path/to/charliecloud_image -- bash

# run an executable in the container
ch-run --home -b /dss/.:/dss/ /path/to/charliecloud_image -- /path/to/executable

# run a bash script in the container
ch-run --home -b /dss/.:/dss/ /path/to/charliecloud_image -- bash /path/to/bash/script.sh

# run a python script in the container
ch-run --home -b /dss/.:/dss/ /path/to/charliecloud_image -- python /path/to/python/script.py

A good way to test your container is to start an interactive job and run one of the commands above.

The usage of containers in productive jobs is explained with some examples for serial shared-memory jobs and parallel distributed memory jobs.

GPU-nodes

In order to run a charliecloud image on the GPU nodes, you must inject the necessary GPU-libraries and tools into the container image. For more details and a convenience script, see NVIDIA GPU enabled Charliecloud Containers.

Load terrabyte version of charliecloud​

Generate a Charliecloud image from (Docker) registry URL​

Generate a Charliecloud image from a Dockerfile​

Generate a Charliecloud image from an existing Docker image​

Generate a Docker image from an existing Charliecloud image​

Run a Charliecloud container and use it in jobs​