FuturesExecutor

class coffea.processor.FuturesExecutor(status: bool = True, unit: str = 'items', desc: str = 'Processing', compression: int | None = 1, function_name: str | None = None, pool: ~collections.abc.Callable[[...], ~concurrent.futures._base.Executor] | ~concurrent.futures._base.Executor = <class 'loky.process_executor.ProcessPoolExecutor'>, mergepool: None | ~collections.abc.Callable[[...], ~concurrent.futures._base.Executor] | ~concurrent.futures._base.Executor | bool = None, recoverable: bool = False, merging: bool | tuple[int, int, int] = False, workers: int = 1, tailtimeout: int | None = None, retries: int = 3)

Bases: ExecutorBase

Execute using multiple local cores using python futures

Parameters:

• items (list) – List of input arguments

• function (callable) – A function to be called on each input, which returns an accumulator instance

• accumulator (Accumulatable) – An accumulator to collect the output of the function

• pool (concurrent.futures.Executor class or instance, optional) – The type of futures executor to use, defaults to loky.ProcessPoolExecutor. You can pass an instance instead of a class to reuse an executor

• workers (int, optional) – Number of parallel processes for futures (default 1)

• status (bool, optional) – If true (default), enable progress bar

• desc (str, optional) – Label of progress description (default: ‘Processing’)

• unit (str, optional) – Label of progress bar bar unit (default: ‘items’)

• compression (int, optional) – Compress accumulator outputs in flight with LZ4, at level specified (default 1) Set to None for no compression.

• recoverable (bool, optional) – Instead of raising Exception right away, the exception is captured and returned up for custom parsing. Already completed items will be returned as well.

• checkpoints (bool) – To do

• merging (bool | tuple(int, int, int), optional) – Enables submitting intermediate merge jobs to the executor. Format is (n_batches, min_batch_size, max_batch_size). Passing True will use default: (5, 4, 100), aka as they are returned try to split completed jobs into 5 batches, but of at least 4 and at most 100 items. Default is False - results get merged as they finish in the main process.

• nparts (int, optional) – Number of merge jobs to create at a time. Also pass via merging(X, ..., ...)

• minred (int, optional) – Minimum number of items to merge in one job. Also pass via merging(..., X, ...)

• maxred (int, optional) – maximum number of items to merge in one job. Also pass via merging(..., ..., X)

• mergepool (concurrent.futures.Executor class or instance | int, optional) – Supply an additional executor to process merge jobs independently. An int will be interpreted as ProcessPoolExecutor(max_workers=int).

• tailtimeout (int, optional) – Timeout requirement on job tails. Cancel all remaining jobs if none have finished in the timeout window.

• retries (int, optional) – Number of retries for failed tasks (default: 3)

Returns:

out – (accumulator, 0) after all items have been merged, or (partial_accumulator, exception) when recoverable=True and an exception was captured.

Return type:

tuple(Accumulatable, int | BaseException)

Attributes Summary

compression
desc
function_name
mergepool
merging
recoverable
retries
status
tailtimeout
unit
workers

Methods Summary

__call__(items, function, accumulator)

Call self as a function.

Attributes Documentation

compression: int | None = 1

desc: str = 'Processing'

function_name: str | None = None

mergepool: None | Callable[[...], Executor] | Executor | bool = None

merging: bool | tuple[int, int, int] = False

recoverable: bool = False

retries: int = 3

status: bool = True

tailtimeout: int | None = None

unit: str = 'items'

workers: int = 1

Methods Documentation

__call__(items: Iterable, function: Callable, accumulator: Addable | MutableSet | MutableMapping)

Call self as a function.