Skip to content

Configure Python Pipeline

Configuration

The Python Pipeline can be configured through environment variables for further customization. The available parameters inside your .env file are:

Resources

Parameter Description Default
PYTHON_PROCESS_TIMEOUT The timeout for a single script (in seconds) 60
PYTHON_TRANSFER_DIRECTORY_PATH The directory on the host where python-starter creates a folder for each individual script job /tmp
PYTHON_RUNNER_CONTAINER_CPUS The number of CPUs available to each script container 1
PYTHON_RUNNER_CONTAINER_MEMORY The amount of memory available to each script container 1g
PYTHON_RUNNER_READ_ONLY_FILESYSTEM Whether the root filesystem should be mounted as read-only (true or false) true
PYTHON_RUNNER_TMPFS_MOUNT_SIZE_IN_BYTES Maximum size of the tmpfs mount (mounted at /tmp inside the container) for each script container (in bytes) 104857600 (100MB)
PYTHON_RUNNER_DROPPED_CAPABILITIES Comma-separated list of capabilities that should be removed from the container. Please refer to the Docker documentation for more details CAP_NET_RAW
PYTHON_RUNNER_NO_NEW_PRIVILEGES Whether container processes should be prevented from gaining additional privileges true

Logging

Parameter Description Default
PYTHON_SCHEDULER_LOG_LEVEL The log level for the python-scheduler (DEBUG, INFO, WARNING, ERROR or CRITICAL) WARNING
PYTHON_STARTER_LOG_LEVEL The log level for the python-starter (DEBUG, INFO, WARNING, ERROR or CRITICAL) WARNING

Limiting Volume Size

By default, any script container can use up all of the available storage resources on your disk.

You can use the following instructions to set a limit for the directory that contains all volumes. These commands should be executed on your host.

# Create an empty file
touch python-pipeline-volume

# Resize the file (e.g. 2GB)
truncate -s 2G python-pipeline-volume

# Create a new ext4 filesystem
mke2fs -t ext4 -F python-pipeline-volume

# Create a new directory which will serve as the data transfer directory
mkdir /opt/python-pipeline-transfer

# Mount the filesystem
mount python-pipeline-volume /opt/python-pipeline-transfer

# Validate your changes
df -h /opt/python-pipeline-transfer

Afterwards, you should update your .env file and restart the python-starter by running docker compose up -d:

PYTHON_TRANSFER_DIRECTORY_PATH='/opt/python-pipeline-transfer'