add in table-of-contents capability

_episodes/01-introduction.md

@@ -12,6 +12,8 @@ keypoints:
 - This tutorial is brought to you by the DUNE Computing Consortium.
 - The goals are to give you the computing basis to work on DUNE.
 ---
+
+{% include 01-introduction.toc.md %}
 ## DUNE Computing Consortium
 
 The DUNE Computing Consortium works to establish a global computing network that will handle the massive data streams produced by distributing these across the computing grid.

_episodes/02-storage-spaces.md

@@ -17,6 +17,7 @@ keypoints:
 - The tool suites idfh and XRootD allow for accessing data with appropriate transfer method and in a scalable way.
 ---
 
+{% include 02-storage-spaces.toc.md %}
 ## This is an updated version of the 2023 training
 
 <!-- #### Session Video
@@ -160,7 +161,7 @@ CVMFS is the  CERN Virtual Machine File System is a centrally managed storage ar
 
 See [CVMFS]({{ site.baseurl }}/02.3-cvmfs) for more information.
 
-## What's my quota?
+### What is my quota?
 
 We use multiple systems so there are multiple ways for checking your disk quota.
 

_episodes/03-data-management.md

@@ -13,7 +13,9 @@ keypoints:
 - Xrootd allows user to stream data files. 
 ---
 
-#### Session Video
+{% include 03-data-management.toc.md %}
+
+## Session Video
 
 <!--The session will be captured on video a placed here after the workshop for asynchronous study.-->
 

_includes/01-introduction.md.toc.md

@@ -0,0 +1,7 @@
+## Table of Contents
+- [DUNE Computing Consortium](#dune computing consortium)
+- [Schedule](#schedule)
+   - [Workshop Introduction Video from December 2024](#workshop introduction video from december 2024)
+- [Basic setup reminder](#basic setup reminder)
+- [Instructional Crew](#instructional crew)
+- [Support](#support)

_includes/01-introduction.toc.md

@@ -0,0 +1,7 @@
+## Table of Contents
+- [DUNE Computing Consortium](#dune-computing-consortium)
+- [Schedule](#schedule)
+   - [Workshop Introduction Video from December 2024](#workshop-introduction-video-from-december-2024)
+- [Basic setup reminder](#basic-setup-reminder)
+- [Instructional Crew](#instructional-crew)
+- [Support](#support)

_includes/01.5-documentation.toc.md

@@ -0,0 +1,9 @@
+## Table of Contents
+- [Documentation access](#documentation-access)
+- [DUNE tools list](#dune-tools-list)
+- [Docdb (Requires FNAL SSO)](#docdb-(requires-fnal-sso))
+- [DUNE wiki (Requires FNAL SSO)](#dune-wiki-(requires-fnal-sso))
+   - [Tutorials list on the wiki](#tutorials-list-on-the-wiki)
+- [CERN EDMS](#cern-edms)
+- [Github repositories](#github-repositories)
+- [DUNE Computing FAQ](#dune-computing-faq)

_includes/02-storage-spaces.toc.md

@@ -0,0 +1,35 @@
+## Table of Contents
+- [This is an updated version of the 2023 training](#this-is-an-updated-version-of-the-2023-training)
+      - [Live Notes](#live-notes)
+   - [Workshop Storage Spaces Video from December 2024](#workshop-storage-spaces-video-from-december-2024)
+- [Introduction](#introduction)
+- [Vocabulary](#vocabulary)
+- [Interactive storage volumes (mounted on dunegpvmXX.fnal.gov or lxplus.cern.ch)](#interactive-storage-volumes-(mounted-on-dunegpvmxx.fnal.gov-or-lxplus.cern.ch))
+   - [Your home area](#your-home-area)
+      - [at Fermilab](#at-fermilab)
+      - [at CERN](#at-cern)
+   - [Locally mounted volumes](#locally-mounted-volumes)
+   - [Network Attached Storage (NAS)](#network-attached-storage-(nas))
+      - [At Fermilab](#at-fermilab)
+      - [At CERN](#at-cern)
+   - [Grid-accessible storage volumes](#grid-accessible-storage-volumes)
+      - [Persistent dCache](#persistent-dcache)
+      - [Scratch dCache](#scratch-dcache)
+      - [Tape-backed dCache](#tape-backed-dcache)
+      - [Rucio Storage Elements](#rucio-storage-elements)
+   - [CVMFS](#cvmfs)
+   - [What is my quota?](#what-is-my-quota?)
+      - [Your home area at FNAL](#your-home-area-at-fnal)
+      - [Your home area at CERN](#your-home-area-at-cern)
+      - [The /app/ and /data/ areas at FNAL](#the-/app/-and-/data/-areas-at-fnal)
+      - [EOS at CERN](#eos-at-cern)
+      - [Fermilab dCache](#fermilab-dcache)
+- [Summary on storage spaces](#summary-on-storage-spaces)
+- [Monitoring and Usage](#monitoring-and-usage)
+- [Commands and tools](#commands-and-tools)
+   - [ifdh](#ifdh)
+   - [xrootd](#xrootd)
+   - [What is the right xroot path for a file.](#what-is-the-right-xroot-path-for-a-file.)
+   - [The df command](#the-df-command)
+- [Quiz](#quiz)
+- [Useful links to bookmark](#useful-links-to-bookmark)

_includes/02.3-cvmfs.toc.md

@@ -0,0 +1,4 @@
+## Table of Contents
+- [CVMFS](#cvmfs)
+- [Restrictions](#restrictions)
+- [Useful links to bookmark](#useful-links-to-bookmark)

_includes/03-data-management.toc.md

@@ -0,0 +1,39 @@
+## Table of Contents
+- [Session Video](#session-video)
+      - [Live Notes](#live-notes)
+- [Introduction](#introduction)
+   - [What we need to do to produce accurate physics results](#what-we-need-to-do-to-produce-accurate-physics-results)
+   - [How we do it ?](#how-we-do-it-?)
+   - [How do I use this.](#how-do-i-use-this.)
+- [How to find and access official data](#how-to-find-and-access-official-data)
+- [Official datasets](#official-datasets)
+   - [Fast web catalog queries](#fast-web-catalog-queries)
+   - [Command line tools and advanced queries](#command-line-tools-and-advanced-queries)
+   - [metacat web interface](#metacat-web-interface)
+   - [Example of finding reconstructed Monte Carlo](#example-of-finding-reconstructed-monte-carlo)
+   - [you can use the web data catalog to do advanced searches](#you-can-use-the-web-data-catalog-to-do-advanced-searches)
+- [What is metacat?](#what-is-metacat?)
+   - [Find a file in metacat](#find-a-file-in-metacat)
+   - [Example of doing a metacat search](#example-of-doing-a-metacat-search)
+   - [then do queries to find particular groups of files](#then-do-queries-to-find-particular-groups-of-files)
+   - [What do those fields mean?](#what-do-those-fields-mean?)
+   - [find out how much raw data there is in a run using the summary option](#find-out-how-much-raw-data-there-is-in-a-run-using-the-summary-option)
+   - [Fast web catalog queries](#fast-web-catalog-queries)
+   - [Command line tools and advanced queries](#command-line-tools-and-advanced-queries)
+   - [metacat web interface](#metacat-web-interface)
+   - [Example of finding reconstructed Monte Carlo](#example-of-finding-reconstructed-monte-carlo)
+   - [you can use the web data catalog to do advanced searches](#you-can-use-the-web-data-catalog-to-do-advanced-searches)
+   - [find out how much data there is in a dataset](#find-out-how-much-data-there-is-in-a-dataset)
+   - [What describes a dataset?](#what-describes-a-dataset?)
+   - [What files are in that dataset and how do I use them?](#what-files-are-in-that-dataset-and-how-do-i-use-them?)
+- [Finding those files on disk](#finding-those-files-on-disk)
+- [Getting file locations using Rucio](#getting-file-locations-using-rucio)
+   - [What is Rucio?](#what-is-rucio?)
+   - [You will need to authenticate to use read files](#you-will-need-to-authenticate-to-use-read-files)
+- [Accessing `rucio` and `justin` require a bit more](#accessing-`rucio`-and-`justin`-require-a-bit-more)
+   - [getting a token for xroot access in AL9](#getting-a-token-for-xroot-access-in-al9)
+   - [finding a file](#finding-a-file)
+- [More finding files by characteristics using metacat](#more-finding-files-by-characteristics-using-metacat)
+- [Accessing data for use in your analysis](#accessing-data-for-use-in-your-analysis)
+- [Quiz](#quiz)
+- [Useful links to bookmark](#useful-links-to-bookmark)

_includes/03.2-UPS.toc.md

@@ -0,0 +1,3 @@
+## Table of Contents
+- [What is UPS and why do we need it?](#what-is-ups-and-why-do-we-need-it?)
+   - [UPS basic commands](#ups-basic-commands)

_includes/04-Spack.toc.md

@@ -0,0 +1,5 @@
+## Table of Contents
+- [What is Spack and why do we need it?](#what-is-spack-and-why-do-we-need-it?)
+- [Minimal spack for root analysis and file access](#minimal-spack-for-root-analysis-and-file-access)
+- [A more flexible environment with more packages but you have to make choices of versions](#a-more-flexible-environment-with-more-packages-but-you-have-to-make-choices-of-versions)
+   - [Spack basic commands](#spack-basic-commands)

_includes/05-end-of-basics.toc.md

@@ -0,0 +1,3 @@
+## Table of Contents
+- [You can ask questions here:](#you-can-ask-questions-here:)
+- [You can continue on with these additional modules.](#you-can-continue-on-with-these-additional-modules.)

_includes/05.1-improve-code-efficiency.toc.md

@@ -0,0 +1,11 @@
+## Table of Contents
+      - [Session Video](#session-video)
+      - [Live Notes](#live-notes)
+   - [Code Make-over](#code-make-over)
+   - [CPU optimization:](#cpu-optimization:)
+- [Memory optimization:](#memory-optimization:)
+- [I/O optimization:](#i/o-optimization:)
+- [Build time optimization:](#build-time-optimization:)
+- [Workflow optimization:](#workflow-optimization:)
+- [Software readability and maintainability:](#software-readability-and-maintainability:)
+- [Coding for Thread Safety](#coding-for-thread-safety)

_includes/10-closing-remarks.toc.md

@@ -0,0 +1,6 @@
+## Table of Contents
+- [Video Session](#video-session)
+- [Two Days of Training](#two-days-of-training)
+- [Survey time!](#survey-time!)
+- [Next Steps](#next-steps)
+- [Long Term Support](#long-term-support)

_includes/setup.toc.md

@@ -0,0 +1,33 @@
+## Table of Contents
+- [Objectives](#objectives)
+- [Requirements](#requirements)
+- [Step 1: DUNE membership](#step 1: dune membership)
+- [Step 2: Getting accounts](#step 2: getting accounts)
+   - [With FNAL](#with fnal)
+   - [With CERN](#with cern)
+- [Step 3: Mission setup (rest of this page)](#step 3: mission setup (rest of this page))
+- [0. Basic setup on your computer.](#0. basic setup on your computer.)
+- [1. Kerberos business](#1. kerberos business)
+- [2. ssh-in](#2. ssh-in)
+- [3. Get a clean shell](#3. get a clean shell)
+- [Software setup <a name="software_setup"></a>](#software setup <a name="software_setup"></a>)
+   - [4.1 Setting up DUNE software - Scientific Linux 7 version <a name="SL7_setup"></a>](#4.1 setting up dune software - scientific linux 7 version <a name="sl7_setup"></a>)
+   - [Caveats for later](#caveats for later)
+   - [4.2 Setting up DUNE software - Alma9 version <a name="AL9_setup"></a>](#4.2 setting up dune software - alma9 version <a name="al9_setup"></a>)
+   - [Caveats for later](#caveats for later)
+- [4.2 Setting up DUNE software - Alma9 version](#4.2 setting up dune software - alma9 version)
+   - [Caveats](#caveats)
+- [5. Exercise! (For SL7 - it's easy)](#5. exercise! (for sl7 - it's easy))
+- [5. Exercise! (For AL9 - it's easy)](#5. exercise! (for al9 - it's easy))
+- [6. Getting setup for streaming and grid access](#6. getting setup for streaming and grid access)
+   - [Tokens method <a name="tokens"></a>](#tokens method <a name="tokens"></a>)
+      - [1. Get and store your token](#1. get and store your token)
+      - [2. Tell the system where your token is](#2. tell the system where your token is)
+- [Set up on CERN machines <a name="setup_CERN"></a>](#set up on cern machines <a name="setup_cern"></a>)
+   - [1. Setup in Alma9](#1. setup in alma9)
+   - [2. For SL7](#2. for sl7)
+      - [Source the DUNE environment SL7 setup script](#source the dune environment sl7 setup script)
+   - [3. Getting authentication for data access](#3. getting authentication for data access)
+   - [4. Access tutorial datasets](#4. access tutorial datasets)
+   - [5. Notify us](#5. notify us)
+   - [Useful Links](#useful links)

code/tocgen.py

@@ -0,0 +1,46 @@
+import os,sys,string
+
+def dealwithline(input,toclines):
+    for line in input:
+        if line.startswith('{% include'): 
+            print ("dealing with include",line)
+            includename = line.split('%')[1].strip().split(' ')[1]
+            newfile = os.path.join("_includes/",includename)
+            newlines = open(newfile,'r').readlines()
+            dealwithline(newlines,toclines)
+            continue
+        # remove any labels
+        line = line.split('<a')[0]
+        level = 1
+        name = ""
+        indent = ''
+        if line.startswith('####'):
+            level = 3
+            name = line[4:].strip()
+            indent = '   '*2
+        elif line.startswith('###'):
+            indent = '   '*1
+            name = line[3:].strip()
+        elif line.startswith('##'):
+            level = 1
+            name = line[2:].strip()
+        else:
+            continue
+        print (level,indent,name)
+
+
+        newline = '%s- [%s](#%s)' % (indent,name,name.lower().replace(' ',"-"))
+        toclines.append(newline)
+        print (newline)
+    return toclines
+mdname= sys.argv[1]
+inputmd = open(mdname,'r')
+newname = os.path.join("_includes/",os.path.basename(mdname[:-3])+'.toc.md')
+newtoc = open(newname,'w')
+input = inputmd.readlines()
+inputmd.close()
+toclines = []
+toclines.append('## Table of Contents')
+dealwithline(input,toclines)
+newtoc.write('\n'.join(toclines))
+newtoc.close()

gitadd.sh

@@ -1,5 +1,6 @@
 git add *.md
 git add _episodes/*.md
+git add code/*
 #git add _episodes/01-introduction.md
 #git add _episodes/02-storage-spaces.md
 #git add _episodes/03-data-management.md
@@ -19,3 +20,4 @@ git add *.yml
 git add _extras/*.md
 git add AUTHORS CITATION 
 git add gitadd.sh
+

setup.md

@@ -24,12 +24,13 @@ keypoints:
 - Do an exercise to help us check if all is good
 - Get streaming and grid access
 
-
-
+
 > ## If you run into problems now or later, check out the [Common Error Messages]({{ site.baseurl }}/ErrorMessages) page and the [FAQ page](https://github.com/orgs/DUNE/projects/19/)
 > if that doesn't help, use [DUNE Slack](https://dunescience.slack.com/archives/C02TJDHUQPR) channel `#computing-training-basics` to ask us about the problem - there is always a new one cropping up.
 {: .challenge} 
 
+{%include setup.toc.md%}
+
 ## Requirements
 
 
@@ -750,7 +751,7 @@ If you are experiencing issues, please contact us as soon as possible. Be sure t
 > If you have issues here, please go to the [#computing-training-basics](https://dunescience.slack.com/archives/C02TJDHUQPR)Slack channel to get support. Please note that you are on a CERN machine in your message. Thanks!
 {: .discussion}
 
-### Useful Links
+## Useful Links
 
 The [DUNE FAQ][DUNE FAQ] on GitHub.
 

setup.toc.md

@@ -0,0 +1,32 @@
+- [Objectives](#objectives)
+- [Requirements](#requirements)
+- [Step 1: DUNE membership](#step 1: dune membership)
+- [Step 2: Getting accounts](#step 2: getting accounts)
+   - [With FNAL](#with fnal)
+   - [With CERN](#with cern)
+- [Step 3: Mission setup (rest of this page)](#step 3: mission setup (rest of this page))
+- [0. Basic setup on your computer.](#0. basic setup on your computer.)
+- [1. Kerberos business](#1. kerberos business)
+- [2. ssh-in](#2. ssh-in)
+- [3. Get a clean shell](#3. get a clean shell)
+- [Software setup <a name="software_setup"></a>](#software setup <a name="software_setup"></a>)
+   - [4.1 Setting up DUNE software - Scientific Linux 7 version <a name="SL7_setup"></a>](#4.1 setting up dune software - scientific linux 7 version <a name="sl7_setup"></a>)
+   - [Caveats for later](#caveats for later)
+   - [4.2 Setting up DUNE software - Alma9 version <a name="AL9_setup"></a>](#4.2 setting up dune software - alma9 version <a name="al9_setup"></a>)
+   - [Caveats for later](#caveats for later)
+- [4.2 Setting up DUNE software - Alma9 version](#4.2 setting up dune software - alma9 version)
+   - [Caveats](#caveats)
+- [5. Exercise! (For SL7 - it's easy)](#5. exercise! (for sl7 - it's easy))
+- [5. Exercise! (For AL9 - it's easy)](#5. exercise! (for al9 - it's easy))
+- [6. Getting setup for streaming and grid access](#6. getting setup for streaming and grid access)
+   - [Tokens method <a name="tokens"></a>](#tokens method <a name="tokens"></a>)
+      - [1. Get and store your token](#1. get and store your token)
+      - [2. Tell the system where your token is](#2. tell the system where your token is)
+- [Set up on CERN machines <a name="setup_CERN"></a>](#set up on cern machines <a name="setup_cern"></a>)
+   - [1. Setup in Alma9](#1. setup in alma9)
+   - [2. For SL7](#2. for sl7)
+      - [Source the DUNE environment SL7 setup script](#source the dune environment sl7 setup script)
+   - [3. Getting authentication for data access](#3. getting authentication for data access)
+   - [4. Access tutorial datasets](#4. access tutorial datasets)
+   - [5. Notify us](#5. notify us)
+   - [Useful Links](#useful links)