Configure Python Pipeline
Configuration¶
The Python Pipeline can be configured through environment variables for further customization. The available parameters inside your .env
file are:
Resources¶
Parameter | Description | Default |
---|---|---|
PYTHON_PROCESS_TIMEOUT | The timeout for a single script (in seconds) | 60 |
PYTHON_TRANSFER_DIRECTORY_PATH | The directory on the host where python-starter creates a folder for each individual script job | /tmp |
PYTHON_RUNNER_CONTAINER_CPUS | The number of CPUs available to each script container | 1 |
PYTHON_RUNNER_CONTAINER_MEMORY | The amount of memory available to each script container | 1g |
PYTHON_RUNNER_READ_ONLY_FILESYSTEM | Whether the root filesystem should be mounted as read-only (true or false ) | true |
PYTHON_RUNNER_TMPFS_MOUNT_SIZE_IN_BYTES | Maximum size of the tmpfs mount (mounted at /tmp inside the container) for each script container (in bytes) | 104857600 (100MB) |
PYTHON_RUNNER_DROPPED_CAPABILITIES | Comma-separated list of capabilities that should be removed from the container. Please refer to the Docker documentation for more details | CAP_NET_RAW |
PYTHON_RUNNER_NO_NEW_PRIVILEGES | Whether container processes should be prevented from gaining additional privileges | true |
Logging¶
Parameter | Description | Default |
---|---|---|
PYTHON_SCHEDULER_LOG_LEVEL | The log level for the python-scheduler (DEBUG , INFO , WARNING , ERROR or CRITICAL ) | WARNING |
PYTHON_STARTER_LOG_LEVEL | The log level for the python-starter (DEBUG , INFO , WARNING , ERROR or CRITICAL ) | WARNING |
Limiting Volume Size¶
By default, any script container can use up all of the available storage resources on your disk.
You can use the following instructions to set a limit for the directory that contains all volumes. These commands should be executed on your host.
# Create an empty file
touch python-pipeline-volume
# Resize the file (e.g. 2GB)
truncate -s 2G python-pipeline-volume
# Create a new ext4 filesystem
mke2fs -t ext4 -F python-pipeline-volume
# Create a new directory which will serve as the data transfer directory
mkdir /opt/python-pipeline-transfer
# Mount the filesystem
mount python-pipeline-volume /opt/python-pipeline-transfer
# Validate your changes
df -h /opt/python-pipeline-transfer
Afterwards, you should update your .env
file and restart the python-starter by running docker compose up -d
:
PYTHON_TRANSFER_DIRECTORY_PATH='/opt/python-pipeline-transfer'