Skip to content

cms-data/DQM-Integration

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

39 Commits
run362321
run362321
 
 
run381594
run381594
 
 
run392642
run392642
 
 
run397813
run397813
 
 
README.md
README.md
 
 

Repository files navigation

DQM-Integration

Repository to store input files for the CMSSW DQM/Integration package.

Streamer Files

For the moment this repository is used to store streamer files that are needed as input for the unitTest.

Currently the unitTest that make use of input streamer files are:

  • the onlinebeammonitor_dqm_sourceclient_cfg.py client, it reads the streamDQM (prepared such that only the hltOnlineBeamSpot is present in the file) streamer files regenerated from run 381594 (from 2024E pp run OMS link): 
    run381594/run381594_ls1000_streamDQM_pid1388480.dat
run381594/run381594_ls1000_streamDQM_pid1388480.jsn
  • the beamhlt_dqm_sourceclient-live_cfg.py client, it reads the streamDQMOnlineBeamspot streamer files regenerated from run 381594 (from 2024E pp run OMS link): 
    run381594/run381594_ls1000_streamDQMOnlineBeamspot_pid1752643.dat
run381594/run381594_ls1000_streamDQMOnlineBeamspot_pid1752643.jsn
  • the ecalgpu_dqm_sourceclient-live_cfg.py, hcalgpu_dqm_sourceclient-live_cfg.py, pixelgpu_dqm_sourceclient-live_cfg.py and pfgpu_dqm_sourceclient-live_cfg.py read the streamDQMGPUvsCPU streamer files regenerated from run 397813 (from Run2025G pp run OMS link): 
    run397813/run397813_ls0131_streamDQMGPUvsCPU.dat
run397813/run397813_ls0131_streamDQMGPUvsCPU.jsn
  • the sistrip_approx_dqm_sourceclient-live_cfg.py reads the streamDQM streamer files regenerated from run 362321 (from 2022 HI run, OMS link, though they have been re-HLT'ed, see for more details at CMSHLT-2884): 
    run362321/run362321_ls0231_streamHIDQM_pid276864.dat
run362321/run362321_ls0231_streamHIDQM_pid276864.jsn
  • the scouting_dqm_sourceclient-live_cfg.py reads the streamDQMOnlineScouting streamer files generated from run 392642 (from Run2025C pp run, OMS link): 
    run392642/run392642_ls0174_streamDQMOnlineScouting_pid1152594.dat
run392642/run392642_ls0174_streamDQMOnlineScouting_pid1152594.jsn

Recipe to regenerate Streamer files (when streamer layout gets broken)

In repsonse to issue cms-sw/cmssw#45224, streamer files have been regenerated in a release that contains cms-sw/cmssw#44978 (in this case CMSSW_14_0_9_MULTIARCHS. This was done using the following script for the pp data:

#!/bin/bash -ex

RUNNUMBER=381594
LUMISECTION=1000

# cmsrel CMSSW_14_0_9_MULTIARCHS
# cd CMSSW_14_0_9_MULTIARCHS/src
# cmsenv
# scram b

INPUTFILE=root://eoscms.cern.ch//store/express/Run2024E/ExpressPhysics/FEVT/Express-v1/000/381/594/00000/1e2c895f-a250-45be-a7ff-ee95e636a6e9.root
rm -rf run${RUNNUMBER}*

# run on 300 events of LS 1000, with 300 events per input file
convertToRaw -f 300 -l 300 -r ${RUNNUMBER}:${LUMISECTION} -o . -- "${INPUTFILE}"

tmpfile=$(mktemp)
hltConfigFromDB --runNumber "${RUNNUMBER}" > "${tmpfile}"
cat <<@EOF >> "${tmpfile}"
process.load("run${RUNNUMBER}_cff")
process.hltOnlineBeamSpotESProducer.timeThreshold = int(1e6)

# to run without any HLT prescales
del process.PrescaleService
del process.MessageLogger
process.load('FWCore.MessageLogger.MessageLogger_cfi')

process.options.numberOfThreads = 32
process.options.numberOfStreams = 32

process.options.wantSummary = True
# # to run using the same HLT prescales as used online in LS 1000
# process.PrescaleService.forceDefault = True
@EOF
edmConfigDump "${tmpfile}" > hlt.py

cmsRun hlt.py &> hlt.log

While the following script for the HIon data:

#!/bin/bash -ex

# cmsrel CMSSW_14_0_9_MULTIARCHS
# cd CMSSW_14_0_9_MULTIARCHS/src
# cmsenv
# scram b

# run 362321, LSs 231-232
INPUTFILE=root://eoscms.cern.ch//eos/cms/store/user/cmsbuild//store/hidata/HIRun2022A/HITestRaw0/RAW/v1/000/362/321/00000/f467ee64-fc64-47a6-9d8a-7ca73ebca2bd.root

HLTMENU=/dev/CMSSW_14_0_0/HIon/V141

rm -rf run362321*

# run on 100 events of LS 231, with 100 events per input file
convertToRaw -f 100 -l 100 -r 362321:231 -s rawDataRepacker -o . -- "${INPUTFILE}"

tmpfile=$(mktemp)
hltConfigFromDB --configName "${HLTMENU}" > "${tmpfile}"
sed -i 's|process = cms.Process( "HLT" )|from Configuration.Eras.Era_Run3_cff import Run3\nprocess = cms.Process( "HLT", Run3 )|g' "${tmpfile}"
cat <<@EOF >> "${tmpfile}"
process.load('run362321_cff')
process.hltOnlineBeamSpotESProducer.timeThreshold = int(1e6)

# override the GlobalTag, connection string and pfnPrefix
from Configuration.AlCa.GlobalTag import GlobalTag as customiseGlobalTag
process.GlobalTag = customiseGlobalTag(
    process.GlobalTag,
    globaltag = "140X_dataRun3_HLT_v3",
    conditions = "L1Menu_CollisionsHeavyIons2023_v1_1_5_xml,L1TUtmTriggerMenuRcd,frontier://FrontierProd/CMS_CONDITIONS,,9999-12-31 23:59:59.000"
)

# run the Full L1T emulator, then repack the data into a new RAW collection, to be used by the HLT
from HLTrigger.Configuration.CustomConfigs import L1REPACK
process = L1REPACK(process, "uGT")

# to run without any HLT prescales
del process.PrescaleService

# # to run using the same HLT prescales as used online in LS 231
# process.PrescaleService.forceDefault = True
@EOF
edmConfigDump "${tmpfile}" > hlt.py

cmsRun hlt.py &> hlt.log

# remove input files to save space
rm -f run362321/run362321_ls0*_index*.*

The streamer files for the streamDQMOnlineScouting were prepared using the scouting specific menu:

#!/bin/bash -ex

# cmsrel CMSSW_15_0_4_patch1 
# cd CMSSW_15_0_4_patch1/src
# cmsenv
# scram b

RUNNUMBER=392642 # 2025 EphemeralHLTPhysics
LUMISECTION=174

# LS 174 
INPUTFILES="\
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics0/RAW/v1/000/392/642/00000/06ac627c-f97b-40ed-a279-16fdcf990c2c.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics1/RAW/v1/000/392/642/00000/8ec096b9-1293-4ee7-8e27-369889a521ff.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics2/RAW/v1/000/392/642/00000/e39b2b26-9b68-40e7-8e2d-c259637ea5d3.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics3/RAW/v1/000/392/642/00000/5f25f30f-1372-4875-ac25-0f437f2590be.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics4/RAW/v1/000/392/642/00000/ddcb337e-3536-4361-b69a-02ba755b9eb0.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics5/RAW/v1/000/392/642/00000/e7512b2b-d00a-4d71-924d-9563c2b36c18.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics6/RAW/v1/000/392/642/00000/7dd68eaf-bb9b-456e-9318-961c3903c84a.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics7/RAW/v1/000/392/642/00000/a7b5720c-aa47-4783-b838-dce20b7130ee.root"
rm -rf run${RUNNUMBER}*

# run on 5000 events of given LS, with 1000 event limits per input file
convertToRaw -l 5000 -f 1000 -r ${RUNNUMBER}:${LUMISECTION} -o . -- ${INPUTFILES}

tmpfile=$(mktemp)

hltConfigFromDB --configName /users/jprendi/ScoutingOnlineDQM/Test0/HLT/V4 > dump.py


cat <<@EOF >> dump.py
process.load("run${RUNNUMBER}_cff")
del process.PrescaleService
del process.MessageLogger
process.load('FWCore.MessageLogger.MessageLogger_cfi')

process.GlobalTag.globaltag = cms.string('150X_dataRun3_HLT_v1')


process.options.numberOfThreads = 32
process.options.numberOfStreams = 32

process.options.wantSummary = True
@EOF

edmConfigDump dump.py > hlt.py


bash -c 'echo $$ > cmsrun.pid; exec cmsRun hlt.py &> hlt.log'
job_pid=$(cat cmsrun.pid)
echo "cmsRun is running with PID: $job_pid"

# remove input files to save space
rm -f run392642/run392642_ls0*_index*.*

# prepare the files by concatenating the .ini and .dat files
mkdir -p prepared
cat run392642/run392642_ls0000_streamDQMOnlineScouting_pid${job_pid}.ini run392642/run392642_ls0174_streamDQMOnlineScouting_pid${job_pid}.dat > prepared/run392642_ls0174_streamDQMOnlineScouting_pid${job_pid}.dat

The streamer files for the streamDQMGPUVsCPU were prepared using the following script:

#!/bin/bash -ex
RUNNUMBER=397813
LUMISECTION=131

# cmsrel CMSSW_16_0_X_2025-11-23-2300
# cd CMSSW_16_0_X_2025-11-23-2300/src/
# cmsenv
# git cms-addpkg DataFormats/TrackingRecHitSoA
# git remote add fwyzard git@github.com:fwyzard/cmssw.git; git fetch fwyzard
# git cherry-pick 9f90f7e952c86dd527fc5d373973ce93ca78217f
# cmsenv
# scram b

INPUTFILE=root://eoscms.cern.ch//eos/cms/store/express/Run2025F/ExpressPhysics/FEVT/Express-v2/000/397/813/00000/f95a79f9-18ef-48f1-951a-2ae78c1a107f.root
rm -rf run${RUNNUMBER}*

# run on 100 events of LS 131, with 100 events per input file
convertToRaw -f 100 -l 100 -r ${RUNNUMBER}:${LUMISECTION} -o . -- "${INPUTFILE}"

tmpfile=$(mktemp)
hltConfigFromDB --configName /users/musich/tests/dev/CMSSW_15_1_0/CMSHLT-3147/GRun > "${tmpfile}"
cat <<@EOF >> "${tmpfile}"
process.load("run${RUNNUMBER}_cff")

# to run without any HLT prescales
del process.PrescaleService
del process.MessageLogger
process.load('FWCore.MessageLogger.MessageLogger_cfi')

process.options.numberOfThreads = 32
process.options.numberOfStreams = 32

process.options.wantSummary = True
process.GlobalTag.globaltag = cms.string( "150X_dataRun3_HLT_v1" )
# # to run using the same HLT prescales as used online in LS 1000
# process.PrescaleService.forceDefault = True

# customization for the menu
from HLTrigger.Configuration.customizeHLTforCMSSW import *
process = customizeHLTfor48921(process)

## just output the GPU vs CPU output
streamPaths = [foo for foo in process.endpaths_() if foo.endswith('Output')]
streamPaths.remove('DQMGPUvsCPUOutput')
for foo in streamPaths:
    process.__delattr__(foo)
@EOF
edmConfigDump "${tmpfile}" > hlt.py

cmsRun hlt.py &> hlt.log

bash -c 'echo $$ > cmsrun.pid; exec cmsRun hlt.py &> hlt.log'
job_pid=$(cat cmsrun.pid)
echo "cmsRun is running with PID: $job_pid"

# remove input files to save space
rm -f run${RUNNUMBER}/run${RUNNUMBER}_ls0*_index*.*

# prepare the files by concatenating the .ini and .dat files
mkdir -p prepared

cat run${RUNNUMBER}/run${RUNNUMBER}_ls0000_streamDQMGPUvsCPU_pid${job_pid}.ini run${RUNNUMBER}/run${RUNNUMBER}_ls0${LUMISECTION}_streamDQMGPUvsCPU_pid${job_pid}.dat > prepared/run${RUNNUMBER}_ls0${LUMISECTION}_streamDQMGPUvsCPU_pid${job_pid}.dat
cp run${RUNNUMBER}/run${RUNNUMBER}_ls0${LUMISECTION}_streamDQMGPUvsCPU_pid${job_pid}.jsn prepared/run${RUNNUMBER}_ls0${LUMISECTION}_streamDQMGPUvsCPU_pid${job_pid}_prep.jsn

# now remove the extra 0

input="prepared/run${RUNNUMBER}_ls0${LUMISECTION}_streamDQMGPUvsCPU_pid${job_pid}_prep.jsn"
output="prepared/run${RUNNUMBER}_ls0${LUMISECTION}_streamDQMGPUvsCPU_pid${job_pid}.jsn"

jq '
  .data as $d |
  .data = (
    reduce range(0; $d|length) as $i ([];
      if ($i > 0 and .[-1] == "0" and $d[$i] == "0")
      then .
      else . + [$d[$i]]
      end
    )
  )
' "$input" > "$output"

Possible extenstions

There are two more clients for which the unitTets could be activated in the future, namely:

 <!-- streamDQMCalibration is required -->
 <!-- <test name="TestDQMOnlineClient-ecalcalib_dqm_sourceclient" command="runtest.sh ecalcalib_dqm_sourceclient-live_cfg.py" /> -->
 <!-- streamDQMCalibration is required -->
 <!-- <test name="TestDQMOnlineClient-hcalcalib_dqm_sourceclient" command="runtest.sh hcalcalib_dqm_sourceclient-live_cfg.py" /> -->

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

7 watching

Forks

4 forks
Report repository

Releases

13 tags

Packages

No packages published

Contributors 6