preprocess_parquet#

coffea.dataset_tools.preprocess_parquet(datagroupspec: ~coffea.dataset_tools.filespec.DataGroupSpec, step_size: None | int = None, use_row_groups: bool = False, recalculate_steps: bool = False, files_per_batch: int = 1, skip_bad_files: bool = False, file_exceptions: Exception | Warning | tuple[Exception | Warning] = (<class 'OSError'>, ), save_form: bool = True, scheduler: None | ~collections.abc.Callable | str = None, parquet_options: dict = {}, step_size_safety_factor: float = 0.5, allow_empty_datasets: bool = False) → tuple[DataGroupSpec, DataGroupSpec][source]#

Given a list of normalized files, determine the form, steps, and add the metadata for each file according to the supplied processing options.

Parameters:

  • datagroupspec (DataGroupSpec) – The set of datasets whose files will be preprocessed.

  • step_size (int | None, default None) – If specified, the size of the steps to make when analyzing the input files.

  • use_row_groups (bool, default False) – Use the row groups in the parquet files to determine the steps.

  • recalculate_steps (bool, default False) – If steps are present in the input normed files, force the recalculation of those steps, instead of only recalculating the steps if the uuid has changed.

  • skip_bad_files (bool, default False) – Instead of failing, catch exceptions specified by file_exceptions and return null data.

  • file_exceptions (Exception | Warning | tuple[Exception | Warning], default (OSError,)) – What exceptions to catch when skipping bad files.

  • save_form (bool, default True) – Extract the form of each file in each dataset, creating the union of the forms over the dataset.

  • scheduler (None | Callable | str, default None) – Specifies the scheduler that dask should use to execute the preprocessing task graph.

  • parquet_options (dict, default {}) – Options to pass to get_parquet_form_uuid_steps for opening files

  • step_size_safety_factor (float, default 0.5) – When using use_row_groups, if a resulting step is larger than step_size by this factor warn the user that the resulting steps may be highly irregular.

  • allow_empty_datasets (bool, default False) – When a dataset query comes back completely empty, this is normally considered a processing error. Toggle this argument to True to change this to warnings and allow incomplete returned filesets.

Returns:

  • out_available (DataGroupSpec) – The subset of files in each dataset that were successfully preprocessed, organized by dataset.

  • out_updated (DataGroupSpec) – The original set of datasets including files that were not accessible, updated to include the result of preprocessing where available.