Refactor MC part of AOD producers

The MC part of the AOD producer workflow
`o2::aodproducer::AODProducerWorkflowDPL` and
`o2::aodmcproducer::AODMcProducerWorkflowDPL` is refactored to use
functions from namespace `o2::aodmchelpers`.   The helpers are

- `updateMCCollisions` which takes in the `MCEventHeader` and writes to
  the `o2::aod::McCollisions` table.
- `updateHepMCXSection` which takes the `MCEventHeader` and writes to
  the `o2::aodHepMCSections` table.  This uses the predefined key
  constants as defined in `o2::dataformats::MCInfoKeys`
- `updateHepMCPdfInfo` similar to `updateHepMCXSection` above
- `updateHepMCHeavyIon` similar to `updateHepMCXSection` above
- `updateParticle` uses information from an `o2::MCTrack` and writes it
  to the `o2::aod::McParticles` table
- `updateParticles` loops over `o2::MCTrack` objects and calls
  `updateParticle`.

These functions, in particular `updateHepMC...` uses the functions
- `hasKeys` which checks if the `MCEventHeader` has any or all of the
  keys queried.
- `getEventInfo` gets auxiliary information from the `MCEventHeader` or
  a default value.

For the `o2::aod::HepMC...` tables: Depending on the policy parameter
passed to the `updateHepMC...` functons, these tables may or may not be
updated.
- If the policy is `HepMCUpdate::never` then the tables are never
  updated.
- If the policy is `HepMCUpdate::always` then the tables are _always_
  updated, possibly with default values.
- If the policy is `HepMCUpdate::anyKey` (default) or `HepMCUpdate::allKeys`, then
  the decision of what to do is taken on the first event seen.
  - If the policy is `HepMCUpdate::anyKey`, then if _any_ of the needed
    keys are present, then updating will be enabled for this and _all_
    subsequent events.
  - If the policy is `HepMCUpdate::allKeys`, then if _all_ of the needed
    keys are present, then updating will be enabled for this and _all_
    subsequent events.
  Note that the availability of keys is _not_ checked after the first
  event.

  That means, if the criteria isn't met on the first event, then
  the tables will _never_ be update (as if the policy was
  `HepMCUpdate::never`).

  On the other hand, if the criteria was met, than the tables _will_ be
  update an all events (as if the policy was `HepMCUpdate::always`).

Note the slightly tricky template `TableCursor` which allows us to
define a type that correponds to a table curser (which is really a
lambda).   This template could be moved to `AnalysisDataFormats.h` or
the like.

The applications `o2-aod-producer-workflow` and
`o2-aod-mc-producer-workflow` have been updated (via their respective
implementation classes) to use these tools, thus unifying how the MC
information is propagated to AODs.

The utility `o2-sim-mctracks-to-aod` (`run/o2sim_mctracks_to_aod.cxx`)
has _also_ been updated to use these common tools.

Both `o2-aod-mc-producer-workflow` and `o2-sim-mctracks-to-aod` has been
tested extensively.  `o2-aod-producer-workflow` has _not_ been tested
since it is not clear to me how to set-up such a test with limited
resources.  However, since the changes _only_ effect the MC part, and
that part is now common between the two `o2-aod-{,mc-}producer-workflow`
applications, I believe there is no reason to think that it wouldn't
work.

Author: cholmcc

CCDB/include/CCDB/BasicCCDBManager.h

@@ -48,6 +48,8 @@ class CCDBManagerInstance
     std::string uuid;
     long startvalidity = 0;
     long endvalidity = -1;
+    size_t minSize = -1ULL;
+    size_t maxSize = 0;
     int queries = 0;
     int fetches = 0;
     int failures = 0;
@@ -176,6 +178,10 @@ class CCDBManagerInstance
 
   std::string getSummaryString() const;
 
+  size_t getFetchedSize() const { return mFetchedSize; }
+
+  void report(bool longrep = false);
+
   void endOfStream();
 
  private:
@@ -190,10 +196,11 @@ class CCDBManagerInstance
   bool mCanDefault = false;                             // whether default is ok --> useful for testing purposes done standalone/isolation
   bool mCachingEnabled = true;                          // whether caching is enabled
   bool mCheckObjValidityEnabled = false;                // wether the validity of cached object is checked before proceeding to a CCDB API query
+  bool mFatalWhenNull = true;                           // if nullptr blob replies should be treated as fatal (can be set by user)
   long mCreatedNotAfter = 0;                            // upper limit for object creation timestamp (TimeMachine mode) - If-Not-After HTTP header
   long mCreatedNotBefore = 0;                           // lower limit for object creation timestamp (TimeMachine mode) - If-Not-Before HTTP header
   long mTimerMS = 0;                                    // timer for queries
-  bool mFatalWhenNull = true;                           // if nullptr blob replies should be treated as fatal (can be set by user)
+  size_t mFetchedSize = 0;                              // total fetched size
   int mQueries = 0;                                     // total number of object queries
   int mFetches = 0;                                     // total number of succesful fetches from CCDB
   int mFailures = 0;                                    // total number of failed fetches
@@ -218,11 +225,16 @@ T* CCDBManagerInstance::getForTimeStamp(std::string const& path, long timestamp)
       mFailures++;
     } else {
       mFetches++;
+      auto sh = mHeaders.find("fileSize");
+      if (sh != mHeaders.end()) {
+        size_t s = atol(sh->second.c_str());
+        mFetchedSize += s;
+      }
     }
   } else {
     auto& cached = mCache[path];
+    cached.queries++;
     if (mCheckObjValidityEnabled && cached.isValid(timestamp)) {
-      cached.queries++;
       return reinterpret_cast<T*>(cached.noCleanupPtr ? cached.noCleanupPtr : cached.objPtr.get());
     }
     ptr = mCCDBAccessor.retrieveFromTFileAny<T>(path, mMetaData, timestamp, &mHeaders, cached.uuid,
@@ -254,6 +266,13 @@ T* CCDBManagerInstance::getForTimeStamp(std::string const& path, long timestamp)
       } catch (std::exception const& e) {
         reportFatal("Failed to read validity from CCDB response (Valid-From :  " + mHeaders["Valid-From"] + std::string(" Valid-Until: ") + mHeaders["Valid-Until"] + std::string(")"));
       }
+      auto sh = mHeaders.find("fileSize");
+      if (sh != mHeaders.end()) {
+        size_t s = atol(sh->second.c_str());
+        mFetchedSize += s;
+        cached.minSize = std::min(s, cached.minSize);
+        cached.maxSize = std::max(s, cached.minSize);
+      }
     } else if (mHeaders.count("Error")) { // in case of errors the pointer is 0 and headers["Error"] should be set
       cached.failures++;
       cached.clear(); // in case of any error clear cache for this object

CCDB/include/CCDB/CcdbApi.h

@@ -542,7 +542,7 @@ class CcdbApi //: public DatabaseInterface
    * @param tcl The TClass object describing the serialized type
    * @return raw pointer to created object
    */
-  void* downloadFilesystemContent(std::string const& fullUrl, std::type_info const& tinfo) const;
+  void* downloadFilesystemContent(std::string const& fullUrl, std::type_info const& tinfo, std::map<string, string>* headers) const;
 
   // initialize the TGrid (Alien connection)
   bool initTGrid() const;

CCDB/src/BasicCCDBManager.cxx

@@ -54,7 +54,7 @@ std::pair<int64_t, int64_t> CCDBManagerInstance::getRunDuration(int runnumber, b
 
 std::string CCDBManagerInstance::getSummaryString() const
 {
-  std::string res = fmt::format("{} queries", mQueries);
+  std::string res = fmt::format("{} queries, {} bytes", mQueries, fmt::group_digits(mFetchedSize));
   if (mCachingEnabled) {
     res += fmt::format(" for {} objects", mCache.size());
   }
@@ -72,9 +72,20 @@ std::string CCDBManagerInstance::getSummaryString() const
   return res;
 }
 
-void CCDBManagerInstance::endOfStream()
+void CCDBManagerInstance::report(bool longrep)
 {
   LOG(info) << "CCDBManager summary: " << getSummaryString();
+  if (longrep && mCachingEnabled) {
+    LOGP(info, "CCDB cache miss/hit/failures");
+    for (const auto& obj : mCache) {
+      LOGP(info, "  {}: {}/{}/{} ({}-{} bytes)", obj.first, obj.second.fetches, obj.second.queries - obj.second.fetches - obj.second.failures, obj.second.failures, obj.second.minSize, obj.second.maxSize);
+    }
+  }
+}
+
+void CCDBManagerInstance::endOfStream()
+{
+  report(true);
 }
 
 } // namespace ccdb

CCDB/src/CCDBDownloader.cxx

@@ -529,6 +529,10 @@ void CCDBDownloader::transferFinished(CURL* easy_handle, CURLcode curlCode)
             (*requestData->headers)["Error"] = "An error occurred during retrieval";
           }
           LOGP(alarm, "Curl request to {}, response code: {}", url, httpCode);
+        } else {
+          if (requestData->headers && requestData->headers->find("fileSize") == requestData->headers->end()) {
+            (*requestData->headers)["fileSize"] = fmt::format("{}", requestData->hoPair.object ? requestData->hoPair.object->size() : 0);
+          }
         }
         --(*performData->requestsLeft);
         delete requestData;

CCDB/src/CcdbApi.cxx

@@ -832,6 +832,9 @@ void* CcdbApi::extractFromLocalFile(std::string const& filename, std::type_info
     if ((isSnapshotMode() || mPreferSnapshotCache) && headers->find("ETag") == headers->end()) { // generate dummy ETag to profit from the caching
       (*headers)["ETag"] = filename;
     }
+    if (headers->find("fileSize") == headers->end()) {
+      (*headers)["fileSize"] = fmt::format("{}", f.GetEND());
+    }
   }
   return extractFromTFile(f, tcl);
 }
@@ -857,7 +860,7 @@ bool CcdbApi::initTGrid() const
   return mAlienInstance != nullptr;
 }
 
-void* CcdbApi::downloadFilesystemContent(std::string const& url, std::type_info const& tinfo) const
+void* CcdbApi::downloadFilesystemContent(std::string const& url, std::type_info const& tinfo, std::map<string, string>* headers) const
 {
   if ((url.find("alien:/", 0) != std::string::npos) && !initTGrid()) {
     return nullptr;
@@ -867,6 +870,9 @@ void* CcdbApi::downloadFilesystemContent(std::string const& url, std::type_info
   if (memfile) {
     auto cl = tinfo2TClass(tinfo);
     auto content = extractFromTFile(*memfile, cl);
+    if (headers && headers->find("fileSize") == headers->end()) {
+      (*headers)["fileSize"] = fmt::format("{}", memfile->GetEND());
+    }
     delete memfile;
     return content;
   }
@@ -902,7 +908,7 @@ void* CcdbApi::navigateURLsAndRetrieveContent(CURL* curl_handle, std::string con
 
   // let's see first of all if the url is something specific that curl cannot handle
   if ((url.find("alien:/", 0) != std::string::npos) || (url.find("file:/", 0) != std::string::npos)) {
-    return downloadFilesystemContent(url, tinfo);
+    return downloadFilesystemContent(url, tinfo, headers);
   }
   // add other final cases here
   // example root://
@@ -933,6 +939,9 @@ void* CcdbApi::navigateURLsAndRetrieveContent(CURL* curl_handle, std::string con
     if (200 <= response_code && response_code < 300) {
       // good response and the content is directly provided and should have been dumped into "chunk"
       content = interpretAsTMemFileAndExtract(chunk.memory, chunk.size, tinfo);
+      if (headers && headers->find("fileSize") == headers->end()) {
+        (*headers)["fileSize"] = fmt::format("{}", chunk.size);
+      }
     } else if (response_code == 304) {
       // this means the object exist but I am not serving
       // it since it's already in your possession
@@ -1689,9 +1698,9 @@ void CcdbApi::vectoredLoadFileToMemory(std::vector<RequestContext>& requestConte
   // Save snapshots
   for (int i = 0; i < requestContexts.size(); i++) {
     auto& requestContext = requestContexts.at(i);
-    logReading(requestContext.path, requestContext.timestamp, &requestContext.headers,
-               fmt::format("{}{}", requestContext.considerSnapshot ? "load to memory" : "retrieve", fromSnapshots.at(i) ? " from snapshot" : ""));
     if (!requestContext.dest.empty()) {
+      logReading(requestContext.path, requestContext.timestamp, &requestContext.headers,
+                 fmt::format("{}{}", requestContext.considerSnapshot ? "load to memory" : "retrieve", fromSnapshots.at(i) ? " from snapshot" : ""));
       if (requestContext.considerSnapshot && fromSnapshots.at(i) != 2) {
         saveSnapshot(requestContext);
       }
@@ -1773,6 +1782,9 @@ void CcdbApi::loadFileToMemory(o2::pmr::vector<char>& dest, const std::string& p
     if ((isSnapshotMode() || mPreferSnapshotCache) && localHeaders->find("ETag") == localHeaders->end()) { // generate dummy ETag to profit from the caching
       (*localHeaders)["ETag"] = path;
     }
+    if (localHeaders->find("fileSize") == localHeaders->end()) {
+      (*localHeaders)["fileSize"] = fmt::format("{}", memFile.GetEND());
+    }
   }
   return;
 }

CMakeLists.txt

@@ -11,7 +11,7 @@
 
 # Preamble
 
-cmake_minimum_required(VERSION 3.21.1 FATAL_ERROR)
+cmake_minimum_required(VERSION 3.27.1 FATAL_ERROR)
 
 # it's important to specify accurately the list of languages. for instance C and
 # C++ as we _do_ have some C files to compile explicitely as C (e.g. gl3w.c)
@@ -140,4 +140,3 @@ if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
 endif()
 
 set_root_pcm_dependencies()
-

CODEOWNERS

@@ -35,7 +35,7 @@
 /DataFormats/Detectors/GlobalTrackingWorkflow  @shahor02
 /DataFormats/Detectors/HMPID                   @gvolpe79
 /DataFormats/Detectors/ITSMFT                  @mcoquet642 @mconcas @shahor02
-/DataFormats/Detectors/MUON                    @AliceO2Group/muon-experts
+/DataFormats/Detectors/MUON                    @AliceO2Group/muon-experts @shahor02
 /DataFormats/Detectors/PHOS                    @peressounko @kharlov
 /DataFormats/Detectors/Passive                 @sawenzel @benedikt-voelkel
 /DataFormats/Detectors/TOF                     @noferini
@@ -66,7 +66,7 @@
 /Detectors/GlobalTrackingWorkflow  @shahor02
 /Detectors/HMPID                   @gvolpe79
 /Detectors/ITSMFT                  @mcoquet642 @mconcas @shahor02
-/Detectors/MUON                    @AliceO2Group/muon-experts
+/Detectors/MUON                    @AliceO2Group/muon-experts @shahor02
 /Detectors/PHOS                    @peressounko @kharlov
 /Detectors/Passive                 @sawenzel @benedikt-voelkel
 /Detectors/TOF                     @noferini

Common/Constants/include/CommonConstants/PhysicsConstants.h

@@ -67,32 +67,32 @@ enum Pdg {
 };
 
 /// \brief Declarations of masses for additional particles
-constexpr double MassB0 = 5.27953;
-constexpr double MassB0Bar = 5.27953;
-constexpr double MassBPlus = 5.27915;
-constexpr double MassBS = 5.3663;
-constexpr double MassBSBar = 5.3663;
+constexpr double MassB0 = 5.27966;
+constexpr double MassB0Bar = 5.27966;
+constexpr double MassBPlus = 5.27934;
+constexpr double MassBS = 5.36692;
+constexpr double MassBSBar = 5.36692;
 constexpr double MassD0 = 1.86484;
 constexpr double MassD0Bar = 1.86484;
-constexpr double MassDMinus = 1.86962;
-constexpr double MassDPlus = 1.86962;
-constexpr double MassDS = 1.9685;
-constexpr double MassDSBar = 1.9685;
-constexpr double MassDStar = 2.01027;
-constexpr double MassChiC1 = 3.51066;
-constexpr double MassJPsi = 3.096916;
-constexpr double MassLambdaB0 = 5.6202;
+constexpr double MassDMinus = 1.86966;
+constexpr double MassDPlus = 1.86966;
+constexpr double MassDS = 1.96835;
+constexpr double MassDSBar = 1.96835;
+constexpr double MassDStar = 2.01026;
+constexpr double MassChiC1 = 3.51067;
+constexpr double MassJPsi = 3.0969;
+constexpr double MassLambdaB0 = 5.6196;
 constexpr double MassLambdaCPlus = 2.28646;
-constexpr double MassOmegaC0 = 2.6975;
-constexpr double MassPhi = 1.019455;
-constexpr double MassSigmaC0 = 2.45376;
-constexpr double MassSigmaCPlusPlus = 2.45402;
+constexpr double MassOmegaC0 = 2.6952;
+constexpr double MassPhi = 1.019461;
+constexpr double MassSigmaC0 = 2.45375;
+constexpr double MassSigmaCPlusPlus = 2.45397;
 constexpr double MassX3872 = 3.87165;
 constexpr double MassXi0 = 1.31486;
-constexpr double MassXiB0 = 5.7924;
-constexpr double MassXiCCPlusPlus = 3.59798;
-constexpr double MassXiCPlus = 2.4679;
-constexpr double MassXiC0 = 2.471;
+constexpr double MassXiB0 = 5.7919;
+constexpr double MassXiCCPlusPlus = 3.62155;
+constexpr double MassXiCPlus = 2.46771;
+constexpr double MassXiC0 = 2.47044;
 constexpr double MassDeuteron = 1.87561294257;
 constexpr double MassTriton = 2.80892113298;
 constexpr double MassHelium3 = 2.80839160743;
@@ -102,53 +102,53 @@ constexpr double MassHyperHydrogen4 = 3.9226;
 constexpr double MassHyperHelium4 = 3.9217;
 
 /// \brief Declarations of masses for particles in ROOT PDG_t
-constexpr double MassDown = 0.0048;
-constexpr double MassDownBar = 0.0048;
-constexpr double MassUp = 0.0024;
-constexpr double MassUpBar = 0.0024;
-constexpr double MassStrange = 0.104;
-constexpr double MassStrangeBar = 0.104;
+constexpr double MassDown = 0.00467;
+constexpr double MassDownBar = 0.00467;
+constexpr double MassUp = 0.00216;
+constexpr double MassUpBar = 0.00216;
+constexpr double MassStrange = 0.0934;
+constexpr double MassStrangeBar = 0.0934;
 constexpr double MassCharm = 1.27;
 constexpr double MassCharmBar = 1.27;
-constexpr double MassBottom = 4.68;
-constexpr double MassBottomBar = 4.68;
-constexpr double MassTop = 171.2;
-constexpr double MassTopBar = 171.2;
+constexpr double MassBottom = 4.18;
+constexpr double MassBottomBar = 4.18;
+constexpr double MassTop = 172.5;
+constexpr double MassTopBar = 172.5;
 constexpr double MassGluon = 0.0;
-constexpr double MassElectron = 0.00051099891;
-constexpr double MassPositron = 0.00051099891;
+constexpr double MassElectron = 0.000510999;
+constexpr double MassPositron = 0.000510999;
 constexpr double MassNuE = 0.0;
 constexpr double MassNuEBar = 0.0;
-constexpr double MassMuonMinus = 0.105658;
-constexpr double MassMuonPlus = 0.105658;
+constexpr double MassMuonMinus = 0.1056584;
+constexpr double MassMuonPlus = 0.1056584;
 constexpr double MassNuMu = 0.0;
 constexpr double MassNuMuBar = 0.0;
-constexpr double MassTauMinus = 1.77684;
-constexpr double MassTauPlus = 1.77684;
+constexpr double MassTauMinus = 1.77686;
+constexpr double MassTauPlus = 1.77686;
 constexpr double MassNuTau = 0.0;
 constexpr double MassNuTauBar = 0.0;
 constexpr double MassGamma = 0.0;
-constexpr double MassZ0 = 91.187;
-constexpr double MassWPlus = 80.398;
-constexpr double MassWMinus = 80.398;
-constexpr double MassPi0 = 0.134977;
-constexpr double MassK0Long = 0.497614;
-constexpr double MassPiPlus = 0.13957;
-constexpr double MassPiMinus = 0.13957;
-constexpr double MassProton = 0.938272;
-constexpr double MassProtonBar = 0.938272;
-constexpr double MassNeutron = 0.939565;
-constexpr double MassNeutronBar = 0.939565;
-constexpr double MassK0Short = 0.497614;
-constexpr double MassK0 = 0.497614;
-constexpr double MassK0Bar = 0.497614;
+constexpr double MassZ0 = 91.1876;
+constexpr double MassWPlus = 80.377;
+constexpr double MassWMinus = 80.377;
+constexpr double MassPi0 = 0.1349768;
+constexpr double MassK0Long = 0.497611;
+constexpr double MassPiPlus = 0.1395704;
+constexpr double MassPiMinus = 0.1395704;
+constexpr double MassProton = 0.9382721;
+constexpr double MassProtonBar = 0.9382721;
+constexpr double MassNeutron = 0.9395654;
+constexpr double MassNeutronBar = 0.9395654;
+constexpr double MassK0Short = 0.497611;
+constexpr double MassK0 = 0.497611;
+constexpr double MassK0Bar = 0.497611;
 constexpr double MassKPlus = 0.493677;
 constexpr double MassKMinus = 0.493677;
-constexpr double MassLambda0 = 1.11568;
-constexpr double MassLambda0Bar = 1.11568;
-constexpr double MassLambda1520 = 1.5195;
-constexpr double MassSigmaMinus = 1.19744;
-constexpr double MassSigmaBarPlus = 1.19744;
+constexpr double MassLambda0 = 1.115683;
+constexpr double MassLambda0Bar = 1.115683;
+constexpr double MassLambda1520 = 1.519;
+constexpr double MassSigmaMinus = 1.197449;
+constexpr double MassSigmaBarPlus = 1.197449;
 constexpr double MassSigmaPlus = 1.18937;
 constexpr double MassSigmaBarMinus = 1.18937;
 constexpr double MassSigma0 = 1.192642;

DataFormats/Detectors/CTP/include/DataFormatsCTP/Scalers.h

@@ -33,6 +33,7 @@ struct errorCounters {
   void printStream(std::ostream& stream) const;
   uint32_t lmB = 0, l0B = 0, l1B = 0, lmA = 0, l0A = 0, l1A = 0;       // decreasing counters
   uint32_t lmBlmA = 0, lmAl0B = 0, l0Bl0A = 0, l0Al1B = 0, l1Bl1A = 0; // between levels countres
+  uint32_t lmBlmAd1 = 0, lmAl0Bd1 = 0, l0Bl0Ad1 = 0, l0Al1Bd1 = 0, l1Bl1Ad1 = 0; // between levels countres - diff =1 - just warning
   uint32_t MAXPRINT = 3;
 };
 struct CTPScalerRaw {
@@ -115,7 +116,8 @@ class CTPRunScalers
   // v1
   // static constexpr uint32_t NCOUNTERS = 1070;
   // v2 - orbitid added at the end
-  static constexpr uint32_t NCOUNTERS = 1071;
+  static constexpr uint32_t NCOUNTERSv2 = 1071;
+  static constexpr uint32_t NCOUNTERS = 1085;
   static std::vector<std::string> scalerNames;
 
   void printLMBRateVsT() const; // prints LMB interaction rate vs time for debugging