get_dataset_files_replicas#

coffea.dataset_tools.rucio_utils.get_dataset_files_replicas(dataset, allowlist_sites=None, blocklist_sites=None, regex_sites=None, mode='full', partial_allowed=False, client=None, scope='cms')[source]#

This function queries the Rucio server to get information about the location of all the replicas of the files in a CMS dataset.

The sites can be filtered in 3 different ways: - allowlist_sites: list of sites to select from. If the file is not found there, raise an Exception. - blocklist_sites: list of sites to avoid. If the file has no left site, raise an Exception - regex_sites: regex expression to restrict the list of sites.

The fileset returned by the function is controlled by the mode parameter: - “full”: returns the full set of replicas and sites (passing the filtering parameters) - “first”: returns the first replica found for each file - “best”: to be implemented (ServiceX..) - “roundrobin”: try to distribute the replicas over different sites

Parameters:

  • dataset (str) – The dataset to search for.

  • allowlist_sites (list or None) – List of sites to select from. If the file is not found there, raise an Exception.

  • blocklist_sites (list or None) – List of sites to avoid. If the file has no left site, raise an Exception.

  • regex_sites (list or None) – Regex expression to restrict the list of sites.

  • mode (str, default "full") – One of “full”, “first”, “best”, or “roundrobin”. Behavior of each described above.

  • client (rucio.client.Client or None, optional) – The rucio client to use. If not provided, one will be generated for you.

  • partial_allowed (bool, default False) – If False, throws an exception if any file in the dataset cannot be found. If True, will find as many files from the dataset as it can.

  • scope (str, default "cms") – The scope for rucio to search through.

Returns:

  • files (list) – Depending on mode. For "full" this is the list of replicas per file, with one (possibly empty) entry per file in the dataset; for "first" it contains only the first replica per file, and with partial_allowed=True files without a viable replica are omitted, so it can be shorter than the dataset’s file list.

  • sites (list) – Depending on mode. For "full" this is the list of sites where each file replica is available; for "first" it contains the site of the first replica. It is aligned with files and follows the same length caveat.

  • sites_counts (dict) – Metadata counting the coverage of the dataset by site.