Extending the CCDB API to optionally get the headers from first HTTP request (#14709)

* Extend CCDB functions with optional headers

* Remove the header filtering & add more tests

* Fix code snippets and enhance metadata retrieval documentation

Updated code snippets in README for header retrieval and added optional parameter documentation details.
Also fixed some CI formatting errors.

* Change unit test with to have correct includes

Author: mrtineide

CCDB/CMakeLists.txt

@@ -92,6 +92,12 @@ o2_add_test(CcdbDownloader
             PUBLIC_LINK_LIBRARIES O2::CCDB
             LABELS ccdb)
 
+o2_add_test(CcdbApi-Headers
+            SOURCES test/testCcdbApiHeaders.cxx
+            COMPONENT_NAME ccdb
+            PUBLIC_LINK_LIBRARIES O2::CCDB
+            LABELS ccdb)
+
 # extra CcdbApi test which dispatches to CCDBDownloader (tmp until full move done)
 #o2_add_test_command(NAME CcdbApi-MultiHandle
 #                    WORKING_DIRECTORY ${SIMTESTDIR}

CCDB/README.md

@@ -13,7 +13,7 @@ in circumstances of reduced or no network connectivity.
 
 There are currently 2 different kinds of store/retrieve functions, which we expect to unify in the immediate future:
 2. `storeAsTFile/retrieveFromTFile` API serializing a `TObject` in a ROOT `TFile`.
-3. A strongly-typed `storeAsTFileAny<T>/retrieveFromTFileAny<T>` API allowing to handle any type T 
+3. A strongly-typed `storeAsTFileAny<T>/retrieveFromTFileAny<T>` API allowing to handle any type T
    having a ROOT dictionary. We encourage to use this API by default.
 
 ## Central and local instances of the CCDB
@@ -37,12 +37,12 @@ api.init("http://ccdb-test.cern.ch:8080"); // or http://localhost:8080 for a loc
 auto deadpixels = new o2::FOO::DeadPixelMap();
 api.storeAsTFileAny(deadpixels, "FOO/DeadPixels", metadata);
 // read like this (you have to specify the type)
-auto deadpixelsback = api.retrieveFromTFileAny<o2::FOO::DeadPixelMap>("FOO/DeadPixels", metadata); 
-// read like this to get the headers as well, and thus the metadata attached to the object 
+auto deadpixelsback = api.retrieveFromTFileAny<o2::FOO::DeadPixelMap>("FOO/DeadPixels", metadata);
+// read like this to get the headers as well, and thus the metadata attached to the object
 std::map<std::string, std::string> headers;
-auto deadpixelsback = api.retrieveFromTFileAny<o2::FOO::DeadPixelMap>("FOO/DeadPixels", metadata /* constraint the objects retrieved to those matching the metadata */, -1 /* timestamp */, &headers /* the headers attached to the returned object */); 
+auto deadpixelsback = api.retrieveFromTFileAny<o2::FOO::DeadPixelMap>("FOO/DeadPixels", metadata /* constraint the objects retrieved to those matching the metadata */, -1 /* timestamp */, &headers /* the headers attached to the returned object */);
 // finally, use this method to retrieve only the headers (and thus the metadata)
-std::map<std::string, std::string> headers = f.api.retrieveHeaders("FOO/DeadPixels", f.metadata); 
+std::map<std::string, std::string> headers = api.retrieveHeaders("FOO/DeadPixels", metadata);
 ```
 
 * creating a local snapshot and fetching objects therefrom
@@ -85,7 +85,7 @@ user code. This class
 The class was written for the use-case of transport MC simulation. Typical usage should be like
 
 ```c++
-// setup manager once (at start of processing) 
+// setup manager once (at start of processing)
 auto& mgr = o2::ccdb::BasicCCDBManager::instance();
 mgr.setURL("http://ourccdbserverver.cern.ch");
 mgr.setTimestamp(timestamp_which_we_want_to_anchor_to);
@@ -111,6 +111,12 @@ This feature is useful to avoid using newer objects if the CCDB is updated in pa
 
 In cached mode, the manager can check that local objects are still valid by requiring `mgr.setLocalObjectValidityChecking(true)`, in this case a CCDB query is performed only if the cached object is no longer valid.
 
+If you want the headers/metadata for the object retrieved from the CCDB there is an optional paramater to `BasicCCDBManager::getForTimeStamp`. These headers are also cached (when caching is enabled) and is updated when a CCDB query is sent.
+```c++
+std::map<std::string,std::string> headers;
+mgr.getForTimeStamp(path, timstamp, metadata, &headers);
+```
+
 ## Future ideas / todo:
 
 - [ ] offer improved error handling / exceptions
@@ -129,26 +135,26 @@ A few prototypic command line tools are offered. These can be used in scriptable
 and facilitate the following tasks:
 
   1. Upload and annotate a generic C++ object serialized in a ROOT file
-  
+
      ```bash
      o2-ccdb-upload -f myRootFile.root --key histogram1 --path /Detector1/QA/ --meta "Description=Foo;Author=Person1;Uploader=Person2"
      ```
      This will upload the object serialized in `myRootFile.root` under the key `histogram1`. Object will be put to the CCDB path `/Detector1/QA`.
      For full list of options see `o2-ccdb-upload --help`.
-  
+
   2. Download a CCDB object to a local ROOT file (including its meta information)
-  
+
      ```bash
      o2-ccdb-downloadccdbfile --path /Detector1/QA/ --dest /tmp/CCDB --timestamp xxx
      ```
      This will download the CCDB object under path given by `--path` to a directory given by `--dest` on the disc.
      (The final filename will be `/tmp/CCDB/Detector1/QA/snapshot.root` for the moment).
      All meta-information as well as the information associated to this query will be appended to the file.
-     
+
      For full list of options see `o2-ccdb-downloadccdbfile --help`.
-  
+
   3. Inspect the content of a ROOT file and print summary about type of contained (CCDB) objects and its meta information
-  
+
      ```bash
      o2-ccdb-inspectccdbfile filename
      ```

CCDB/include/CCDB/BasicCCDBManager.h

@@ -23,6 +23,7 @@
 #include <string>
 #include <chrono>
 #include <map>
+#include <string_view>
 #include <unordered_map>
 #include <memory>
 #include <cstdlib>
@@ -57,6 +58,7 @@ class CCDBManagerInstance
     int queries = 0;
     int fetches = 0;
     int failures = 0;
+    std::map<std::string, std::string> cacheOfHeaders;
     bool isValid(long ts) { return ts < endvalidity && ts >= startvalidity; }
     bool isCacheValid(long ts)
     {
@@ -70,6 +72,7 @@ class CCDBManagerInstance
       uuid = "";
       startvalidity = 0;
       endvalidity = -1;
+      cacheOfHeaders.clear();
     }
   };
 
@@ -98,9 +101,9 @@ class CCDBManagerInstance
   /// query timestamp
   long getTimestamp() const { return mTimestamp; }
 
-  /// retrieve an object of type T from CCDB as stored under path and timestamp
+  /// retrieve an object of type T from CCDB as stored under path and timestamp. Optional to get the headers.
   template <typename T>
-  T* getForTimeStamp(std::string const& path, long timestamp);
+  T* getForTimeStamp(std::string const& path, long timestamp, std::map<std::string, std::string>* headers = nullptr);
 
   /// retrieve an object of type T from CCDB as stored under path and using the timestamp in the middle of the run
   template <typename T>
@@ -112,10 +115,7 @@ class CCDBManagerInstance
   {
     // TODO: add some error info/handling when failing
     mMetaData = metaData;
-    auto obj = getForTimeStamp<T>(path, timestamp);
-    if (headers) {
-      *headers = mHeaders;
-    }
+    auto obj = getForTimeStamp<T>(path, timestamp, headers);
     return obj;
   }
 
@@ -235,7 +235,7 @@ class CCDBManagerInstance
 };
 
 template <typename T>
-T* CCDBManagerInstance::getForTimeStamp(std::string const& path, long timestamp)
+T* CCDBManagerInstance::getForTimeStamp(std::string const& path, long timestamp, std::map<std::string, std::string>* headers)
 {
   mHeaders.clear(); // we clear at the beginning; to allow to retrieve the header information in a subsequent call
   T* ptr = nullptr;
@@ -258,15 +258,32 @@ T* CCDBManagerInstance::getForTimeStamp(std::string const& path, long timestamp)
         mFetchedSize += s;
       }
     }
+
+    if (headers) {
+      *headers = mHeaders;
+    }
   } else {
     auto& cached = mCache[path];
     cached.queries++;
     if ((!isOnline() && cached.isCacheValid(timestamp)) || (mCheckObjValidityEnabled && cached.isValid(timestamp))) {
+      // Give back the cached/saved headers
+      if (headers) {
+        *headers = cached.cacheOfHeaders;
+      }
       return reinterpret_cast<T*>(cached.noCleanupPtr ? cached.noCleanupPtr : cached.objPtr.get());
     }
     ptr = mCCDBAccessor.retrieveFromTFileAny<T>(path, mMetaData, timestamp, &mHeaders, cached.uuid,
                                                 mCreatedNotAfter ? std::to_string(mCreatedNotAfter) : "",
                                                 mCreatedNotBefore ? std::to_string(mCreatedNotBefore) : "");
+    // update the cached headers
+    for (auto const& h : mHeaders) {
+      cached.cacheOfHeaders[h.first] = h.second;
+    }
+    // return the cached headers
+    if (headers) {
+      *headers = cached.cacheOfHeaders;
+    }
+
     if (ptr) { // new object was shipped, old one (if any) is not valid anymore
       cached.fetches++;
       mFetches++;

CCDB/include/CCDB/CcdbApi.h

@@ -281,7 +281,7 @@ class CcdbApi //: public DatabaseInterface
    *  @return: True in case operation successful or false if there was a failure/problem.
    */
   bool retrieveBlob(std::string const& path, std::string const& targetdir, std::map<std::string, std::string> const& metadata, long timestamp,
-                    bool preservePathStructure = true, std::string const& localFileName = "snapshot.root", std::string const& createdNotAfter = "", std::string const& createdNotBefore = "") const;
+                    bool preservePathStructure = true, std::string const& localFileName = "snapshot.root", std::string const& createdNotAfter = "", std::string const& createdNotBefore = "", std::map<std::string, std::string>* headers = nullptr) const;
 
   /**
    * Retrieve the headers of a CCDB entry, if it exists.

CCDB/src/CcdbApi.cxx

@@ -835,7 +835,7 @@ TObject* CcdbApi::retrieveFromTFile(std::string const& path, std::map<std::strin
 }
 
 bool CcdbApi::retrieveBlob(std::string const& path, std::string const& targetdir, std::map<std::string, std::string> const& metadata,
-                           long timestamp, bool preservePath, std::string const& localFileName, std::string const& createdNotAfter, std::string const& createdNotBefore) const
+                           long timestamp, bool preservePath, std::string const& localFileName, std::string const& createdNotAfter, std::string const& createdNotBefore, std::map<std::string, std::string>* outHeaders) const
 {
 
   // we setup the target path for this blob
@@ -883,6 +883,9 @@ bool CcdbApi::retrieveBlob(std::string const& path, std::string const& targetdir
   CCDBQuery querysummary(path, metadata, timestamp);
 
   updateMetaInformationInLocalFile(targetpath.c_str(), &headers, &querysummary);
+  if (outHeaders) {
+    *outHeaders = std::move(headers);
+  }
   return true;
 }