DataDiscoveryCLI

class coffea.dataset_tools.dataset_query.DataDiscoveryCLI

Bases: object

Simplifies dataset query, replicas, filters, and uproot preprocessing with Dask. It can be accessed in a Python script or interpreter via this class, or from the command line (as in python -m coffea.dataset_tools.dataset_query --help).

Attributes Summary

as_dict

Methods Summary

do_allowlist_sites([sites])	Restrict the grid sites available for replicas query only to the requested list
do_blocklist_sites([sites])	Exclude grid sites from the available sites for replicas query
do_list_replicas()	Print the selected files replicas for the selected dataset
do_list_selected()	Print a list of the selected datasets
do_login([proxy])	Login to the rucio client.
do_preprocess([output_file, step_size, ...])	Perform preprocessing for concrete fileset extraction into a file, compressed with gzip.
do_query([query])	Look for datasets with * wildcards (like in DAS)
do_query_results()	List the results of the last dataset query
do_regex_sites([regex])	Select sites with a regex for replica queries: e.g. "T[123]_(FR|IT|BE|CH|DE)_w+".
do_replicas([mode, selection])	Query Rucio for replicas.
do_save([filename])	Save the replica information in yaml format
do_select([selection, metadata])	Selected the datasets from the list of query results.
do_sites_filters([ask_clear])	Show the active sites filters (allowed, disallowed, and regex) and ask to clear them
do_whoami()
load_dataset_definition(dataset_definition)	Initialize the DataDiscoverCLI by querying a set of datasets defined in dataset_definitions and selected results and replicas following the options.
start_cli()

Attributes Documentation

as_dict

Methods Documentation

do_allowlist_sites(sites=None)

Restrict the grid sites available for replicas query only to the requested list

Parameters:

sites (list[str] or None, default None) – The sites to allow the replicas query to look at. If passing in a list, elements of the list are sites. If passing in None, the prompt requires a single string containing a comma-separated listing.

do_blocklist_sites(sites=None)

Exclude grid sites from the available sites for replicas query

Parameters:

sites (list[str] or None, default None) – The sites to prevent the replicas query from looking at. If passing in a list elements of the list are sites. If passing in None, the prompt requires a single string containing a comma-separated listing.

do_list_replicas()

Print the selected files replicas for the selected dataset

do_list_selected()

Print a list of the selected datasets

do_login(proxy=None)

Login to the rucio client. Optionally a specific proxy file can be passed to the command. If the proxy file is not specified, voms-proxy-info is used

do_preprocess(output_file=None, step_size=None, align_to_clusters=None, scheduler_url=None, recalculate_steps=None, files_per_batch=None, file_exceptions=(<class 'OSError'>, ), save_form=None, uproot_options={}, step_size_safety_factor=0.5, allow_empty_datasets=False)

Perform preprocessing for concrete fileset extraction into a file, compressed with gzip.

Parameters:

• output_file (str or None, default None) – Target prefix for the generated *_available.json.gz and *_all.json.gz files.

• step_size (int or None, default None) – Chunk size (number of events) to process per step.

• align_to_clusters (bool or None, default None) – Whether to align step boundaries to ROOT cluster boundaries. Mirrors the align_clusters argument of coffea.dataset_tools.preprocess.

• scheduler_url (str or None, default None) – Dask scheduler URL on which to run preprocessing.

• recalculate_steps (bool or None, default None) – Recompute step definitions even if cached values are present.

• files_per_batch (int or None, default None) – Number of files to send to each preprocessing task.

• file_exceptions (tuple[type[BaseException], ], default (OSError,)) – Exceptions that should trigger file skipping instead of aborting.

• save_form (bool or None, default None) – Persist the Awkward form extracted during preprocessing alongside the output.

• uproot_options (dict, default {}) – Keyword arguments forwarded to uproot when opening files.

• step_size_safety_factor (float, default 0.5) – Multiplicative safety factor applied when estimating step sizes.

• allow_empty_datasets (bool, default False) – Whether to keep datasets that produce zero valid chunks.

do_query(query=None)

Look for datasets with * wildcards (like in DAS)

Parameters:

query (str or None, default None) – The query to pass to rucio. If None, will prompt the user for an input.

do_query_results()

List the results of the last dataset query

do_regex_sites(regex=None)

Select sites with a regex for replica queries: e.g. “T[123]_(FR|IT|BE|CH|DE)_w+”

Parameters:

regex (str or None, default None) – Sites to use for replica queries, described with a regex string.

do_replicas(mode=None, selection=None)

Query Rucio for replicas.

Parameters:

• mode (str or None, default None) –

  Selection strategy for preferred sites. Options:

  • None: ask the user about the mode

  • round-robin (take files randomly from available sites),

  • choose: ask the user to choose from a list of sites

  • first: take the first site from the rucio query

• selection (str or None, default None) – Indices (or the literal "all") identifying datasets on which to run the replica query.

do_save(filename=None)

Save the replica information in yaml format

Parameters:

filename (str or None, default None) – The name of the file to save the information into.

do_select(selection=None, metadata=None)

Selected the datasets from the list of query results. Input a list of indices also with range 4-6 or “all”.

Parameters:

• selection (str or None, default None) – Space-delimited indices corresponding to selected datasets. Can include ranges (like "4-6") or the literal "all".

• metadata (dict[Hashable, Any] or None, default None) – Metadata to store in associated with selected datasets.

do_sites_filters(ask_clear=True)

Show the active sites filters (allowed, disallowed, and regex) and ask to clear them

Parameters:

ask_clear (bool, default True) – If True, ask the user via prompt if allow, disallow, and regex filters should be cleared.

do_whoami()

load_dataset_definition(dataset_definition, query_results_strategy='all', replicas_strategy='round-robin')

Initialize the DataDiscoverCLI by querying a set of datasets defined in dataset_definitions and selected results and replicas following the options.

Parameters:

• dataset_definition (dict[str, dict[Hashable, Any]]) – Mapping from dataset query string to metadata to attach to the selection.

• query_results_strategy (str, default "all") – How to decide which datasets to select. If “manual”, user will be prompted for selection

• replicas_strategy (str, default "round-robin") –

  Options are:

  • ”round-robin”: select randomly from the available sites for each file

  • ”choose”: filter the sites with a list of indices for all the files

  • ”first”: take the first result returned by rucio

  • ”manual”: to be prompt for manual decision dataset by dataset

Returns:

out_replicas – An uproot-readable fileset. At this point, the fileset is not fully preprocessed, but this can be done with do_preprocess().

Return type:

FilesetSpecOptional

start_cli()