TaskVineExecutor#

class coffea.processor.TaskVineExecutor(status: bool = True, unit: str = 'items', desc: str = 'Processing', compression: int | None = 1, function_name: str | None = None, port: int = 9123, manager_name: str | None = None, status_display_interval: int = 5, ssl: bool = False, filepath: str = '/tmp', extra_input_files: list | None = None, x509_proxy: str | None = None, environment_file: str | None = None, verbose: bool = False, print_stdout: bool = False, password_file: str | None = None, treereduction: int = 20, cores: int = 1, memory: int | None = None, disk: int | None = None, gpus: int | None = None, replicas: int = 1, disable_worker_transfers: bool = False, resource_monitor: str = 'off', resources_mode: str = 'fixed', fast_terminate_workers: int | None = None, retries: int = 3, split_on_exhaustion: bool = True, checkpoint_proportion: float = 0.1, concurrent_reads: int = 2, custom_init: Callable | None = None)[source]#

Bases: ExecutorBase

Execute using TaskVine distributed computing framework

Parameters:

  • port (int, optional) – Port to listen on for workers (default: 9123)

  • manager_name (str, optional) – Name for the manager (default: None)

  • status_display_interval (int, optional) – Interval for status display updates (default: 5)

  • ssl (bool, optional) – Enable SSL for connections (default: False)

  • filepath (str, optional) – Path for temporary files (default: “/tmp”)

  • extra_input_files (list, optional) – List of extra input files to stage (default: [])

  • x509_proxy (str, optional) – Path to X509 proxy file (default: None)

  • environment_file (str, optional) – Path to environment file for workers (default: None)

  • status (bool, optional) – Enable status display (default: True)

  • verbose (bool, optional) – Enable verbose output (default: False)

  • print_stdout (bool, optional) – Print stdout from tasks (default: False)

  • password_file (str, optional) – Path to password file (default: None)

  • treereduction (int, optional) – Tree reduction factor for output accumulators (default: 20)

  • cores (int, optional) – Number of cores per task (default: 1)

  • memory (int, optional) – Memory requirement per task in MB (default: None)

  • disk (int, optional) – Disk requirement per task in MB (default: None)

  • gpus (int, optional) – Number of GPUs per task (default: None)

  • replicas (int, optional) – Number of replicas for tasks (default: 1)

  • disable_worker_transfers (bool, optional) – Disable worker-to-worker transfers (default: False)

  • resource_monitor (str, optional) – Resource monitoring mode: ‘off’, ‘measure’, or ‘watchdog’ (default: ‘off’)

  • resources_mode (str, optional) – Resource allocation mode: ‘fixed’, ‘max’, or ‘max-throughput’ (default: ‘fixed’)

  • fast_terminate_workers (int, optional) – Fast termination threshold (default: None)

  • retries (int, optional) – Number of retries for failed tasks (default: 3)

  • split_on_exhaustion (bool, optional) – Split tasks on resource exhaustion (default: True)

  • checkpoint_proportion (float, optional) – Proportion of tasks to checkpoint (default: 0.1)

  • concurrent_reads (int, optional) – Number of concurrent file reads (default: 2)

  • custom_init (callable, optional) – Custom initialization function (default: None)

  • compression (int, optional) – Compression level for data transfer (default: 1)

  • unit (str, optional) – Unit for progress display (default: “events”)

  • desc (str, optional) – Description for progress display (default: “Processing”)

  • function_name (str, optional) – Name of the function being executed (default: None)

Attributes Summary

checkpoint_proportion

compression

concurrent_reads

cores

custom_init

desc

disable_worker_transfers

disk

environment_file

extra_input_files

fast_terminate_workers

filepath

function_name

gpus

manager_name

memory

password_file

port

print_stdout

replicas

resource_monitor

resources_mode

retries

split_on_exhaustion

ssl

status

status_display_interval

treereduction

unit

verbose

x509_proxy

Methods Summary

__call__(items, function, accumulator)

Execute the function on items using TaskVine

Attributes Documentation

checkpoint_proportion: float = 0.1#
compression: int | None = 1#
concurrent_reads: int = 2#
cores: int = 1#
custom_init: Callable | None = None#
desc: str = 'Processing'#
disable_worker_transfers: bool = False#
disk: int | None = None#
environment_file: str | None = None#
extra_input_files: list | None = None#
fast_terminate_workers: int | None = None#
filepath: str = '/tmp'#
function_name: str | None = None#
gpus: int | None = None#
manager_name: str | None = None#
memory: int | None = None#
password_file: str | None = None#
port: int = 9123#
print_stdout: bool = False#
replicas: int = 1#
resource_monitor: str = 'off'#
resources_mode: str = 'fixed'#
retries: int = 3#
split_on_exhaustion: bool = True#
ssl: bool = False#
status: bool = True#
status_display_interval: int = 5#
treereduction: int = 20#
unit: str = 'items'#
verbose: bool = False#
x509_proxy: str | None = None#

Methods Documentation

__call__(items: Iterable, function: Callable, accumulator: Addable | MutableSet | MutableMapping)[source]#

Execute the function on items using TaskVine

Parameters:

  • items (Iterable) – Items to process

  • function (Callable) – Function to apply to each item

  • accumulator (Accumulatable) – Initial accumulator for results