Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions _episodes/03-data-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,8 @@ export SAM_EXPERIMENT=dune
-->
## Getting file locations using Rucio

### What is Rucio? <a name="Rucio"></a>
### What is Rucio?
<!-- <a name="Rucio"></a> -->
Rucio is the next-generation Data Replica service and is part of DUNE's new Distributed Data Management (DDM) system that is currently in deployment.
Rucio has two functions:
1. A rule-based system to get files to Rucio Storage Elements around the world and keep them there.
Expand All @@ -451,7 +452,7 @@ As of the date of the 2025 tutorial:
- The Rucio client is available in CVMFS and Spack
- Most DUNE users are now enabled to use it. New users may not automatically be added.

### You will need to authenticate to use read files
### You will need to authenticate to read files

> #### For SL7 use justin to get a token
{:.callout}
Expand Down
2 changes: 1 addition & 1 deletion _includes/02-storage-spaces.toc.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
- [Tape-backed dCache](#tape-backed-dcache)
- [Rucio Storage Elements](#rucio-storage-elements)
- [CVMFS](#cvmfs)
- [What is my quota?](#what-is-my-quota?)
- [What is my quota?](#what-is-my-quota)
- [Your home area at FNAL](#your-home-area-at-fnal)
- [Your home area at CERN](#your-home-area-at-cern)
- [The /app/ and /data/ areas at FNAL](#the-/app/-and-/data/-areas-at-fnal)
Expand Down
16 changes: 8 additions & 8 deletions _includes/03-data-management.toc.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,33 +3,33 @@
- [Live Notes](#live-notes)
- [Introduction](#introduction)
- [What we need to do to produce accurate physics results](#what-we-need-to-do-to-produce-accurate-physics-results)
- [How we do it ?](#how-we-do-it-?)
- [How we do it ?](#how-we-do-it-)
- [How do I use this.](#how-do-i-use-this.)
- [How to find and access official data](#how-to-find-and-access-official-data)
- [Official datasets](#official-datasets)
- [Official datasets <a name="Official_Datasets"></a>](#Official_Datasets)
- [Fast web catalog queries](#fast-web-catalog-queries)
- [Command line tools and advanced queries](#command-line-tools-and-advanced-queries)
- [metacat web interface](#metacat-web-interface)
- [Example of finding reconstructed Monte Carlo](#example-of-finding-reconstructed-monte-carlo)
- [you can use the web data catalog to do advanced searches](#you-can-use-the-web-data-catalog-to-do-advanced-searches)
- [What is metacat?](#what-is-metacat?)
- [What is metacat?](#what-is-metacat)
- [Find a file in metacat](#find-a-file-in-metacat)
- [Example of doing a metacat search](#example-of-doing-a-metacat-search)
- [then do queries to find particular groups of files](#then-do-queries-to-find-particular-groups-of-files)
- [What do those fields mean?](#what-do-those-fields-mean?)
- [What do those fields mean?](#what-do-those-fields-mean)
- [find out how much raw data there is in a run using the summary option](#find-out-how-much-raw-data-there-is-in-a-run-using-the-summary-option)
- [Fast web catalog queries](#fast-web-catalog-queries)
- [Command line tools and advanced queries](#command-line-tools-and-advanced-queries)
- [metacat web interface](#metacat-web-interface)
- [Example of finding reconstructed Monte Carlo](#example-of-finding-reconstructed-monte-carlo)
- [you can use the web data catalog to do advanced searches](#you-can-use-the-web-data-catalog-to-do-advanced-searches)
- [find out how much data there is in a dataset](#find-out-how-much-data-there-is-in-a-dataset)
- [What describes a dataset?](#what-describes-a-dataset?)
- [What files are in that dataset and how do I use them?](#what-files-are-in-that-dataset-and-how-do-i-use-them?)
- [What describes a dataset?](#what-describes-a-dataset)
- [What files are in that dataset and how do I use them?](#what-files-are-in-that-dataset-and-how-do-i-use-them)
- [Finding those files on disk](#finding-those-files-on-disk)
- [Getting file locations using Rucio](#getting-file-locations-using-rucio)
- [What is Rucio?](#what-is-rucio?)
- [You will need to authenticate to use read files](#you-will-need-to-authenticate-to-use-read-files)
- [What is Rucio?](#what-is-rucio)
- [You will need to authenticate to read files](#you-will-need-to-authenticate-to-read-files)
- [Accessing `rucio` and `justin` require a bit more](#accessing-`rucio`-and-`justin`-require-a-bit-more)
- [getting a token for xroot access in AL9](#getting-a-token-for-xroot-access-in-al9)
- [finding a file](#finding-a-file)
Expand Down
2 changes: 1 addition & 1 deletion _includes/03.2-UPS.toc.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
## Table of Contents
- [What is UPS and why do we need it?](#what-is-ups-and-why-do-we-need-it?)
- [What is UPS and why do we need it?](#what-is-ups-and-why-do-we-need-it)
- [UPS basic commands](#ups-basic-commands)
2 changes: 1 addition & 1 deletion _includes/04-Spack.toc.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
## Table of Contents
- [What is Spack and why do we need it?](#what-is-spack-and-why-do-we-need-it?)
- [What is Spack and why do we need it?](#what-is-spack-and-why-do-we-need-it)
- [Minimal spack for root analysis and file access](#minimal-spack-for-root-analysis-and-file-access)
- [A more flexible environment with more packages but you have to make choices of versions](#a-more-flexible-environment-with-more-packages-but-you-have-to-make-choices-of-versions)
- [Spack basic commands](#spack-basic-commands)
12 changes: 9 additions & 3 deletions code/tocgen.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ def dealwithline(input,toclines):
dealwithline(newlines,toclines)
continue
# remove any labels
line = line.split('<a')[0]
# if "<a" in line:
# line = line.split(' <a')[0]
level = 1
name = ""
indent = ''
Expand All @@ -26,10 +27,15 @@ def dealwithline(input,toclines):
name = line[2:].strip()
else:
continue
print (level,indent,name)
# if labeled get the label name
if "<a" in line:
linkname = line.split(' <a')[1].strip().split("name=\"")[1].split('"')[0]
else:
linkname=name.lower().strip().replace('?','').replace(' ',"-")
print (level,indent,name,linkname)


newline = '%s- [%s](#%s)' % (indent,name,name.lower().replace(' ',"-"))
newline = '%s- [%s](#%s)' % (indent,name,linkname)
toclines.append(newline)
print (newline)
return toclines
Expand Down