Skip to content

Commit 6520bbf

Browse files
author
miranov25
committed
ATO-648 - porting the code from the TPC gitlab to the O2DPG tools
1 parent 738007c commit 6520bbf

File tree

1 file changed

+138
-0
lines changed

1 file changed

+138
-0
lines changed

‎UTILS/Parsers/workflowToJSON.sh‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Source path for the script
2+
# source $NOTES/JIRA/ATO-648/workflowToJSON.sh
3+
4+
# Description:
5+
# This script converts workflow configuration logs into a structured JSON format for enhanced data analysis and readability.
6+
7+
# Usage:
8+
# Example: source $O2DPG/UTILS/Parsers/workflowToJSON.sh
9+
10+
alias helpCat=cat
11+
[[ -x "$(command -v pygmentize)" ]] && alias helpCat="pygmentize -O style=borland,linenos=1 -l bash"
12+
13+
function helpCat0() {
14+
local language="$1"
15+
shift
16+
[[ -x "$(command -v pygmentize)" ]] && pygmentize -O style=monokai,linenos=1 -l "$language" | cat - "$@"
17+
[[ ! -x "$(command -v pygmentize)" ]] && cat - "$@"
18+
}
19+
20+
init648() {
21+
cat <<HELP_USAGE | helpCat
22+
Function Overview: init648
23+
24+
This function initializes the script environment and provides access to a series of utility commands designed to assist in handling and transforming workflow logs.
25+
26+
Available Commands:
27+
- \[init648]: Initializes the necessary environment settings for script execution. Use this before running other related functions to ensure all configurations are correctly set.
28+
- \[description]: Provides a comprehensive explanation of the workflow processing, detailing each step and its purpose within the system.
29+
- \[makeParse]: Executes the log parsing process, transforming verbose workflow logs into a structured JSON format, facilitating easier data manipulation and analysis.
30+
- \[makeDiffExample]: Demonstrates how to compare two JSON files derived from workflow logs, highlighting differences.
31+
32+
Usage:
33+
To learn more about each command, type the command followed by 'help'. This will display detailed information about the command's function and usage examples.
34+
35+
Example:
36+
\$ init648 help # Displays help information for the init648 command.
37+
38+
Note: Tests were conducted in the directory:
39+
HELP_USAGE
40+
}
41+
42+
description() {
43+
cat <<HELP_USAGE | helpCat0 bash
44+
Description:
45+
This script parses 'workflowconfig.log' files, which contain lines of commands complete with switches and configuration settings. It is designed to transform these log entries into a structured JSON format, making the data easier to manipulate and read.
46+
47+
Purpose:
48+
- To convert workflow configuration logs into a JSON structure where each command, along with its switches and key values, is represented as an object within an array. This facilitates easier data manipulation and readability.
49+
50+
Structure of Log Entries:
51+
- Each line in the log file represents a single command with its options, structured as follows:
52+
<commandName> <switches> --configKeyValues
53+
54+
Example Command:
55+
A typical command in the log might appear like this:
56+
o2-ctf-reader-workflow --session default_1304519_3825 --severity info --shm-segment-id 0 --shm-segment-size 64000000000 --resources-monitoring 50 --resources-monitoring-dump-interval 50 --early-forward-policy noraw --fairmq-rate-logging 0 --timeframes-rate-limit 2 --timeframes-rate-limit-ipcid 0 --ans-version compat --delay 1 --loop 0 --max-tf 2 --ctf-input list.list --onlyDet ITS,TPC,TOF,FV0,FT0,FDD,TRD,CTP --pipeline tpc-entropy-decoder:1 --allow-missing-detectors --its-digits --mft-digits --configKeyValues "keyval.input_dir=/tmp/tmp.rgwfzmuG63;keyval.output_dir=/dev/null;;"
57+
58+
Transformation:
59+
This script processes each command line from the log, turning them into JSON objects. This structural change not only organizes the data but also enhances accessibility for programmatic queries and analysis.
60+
HELP_USAGE
61+
}
62+
63+
64+
makeParse() {
65+
# Use heredoc to send help text through helpCat alias, which will apply syntax highlighting if pygmentize is available.
66+
if [[ -z "$1" ]]; then
67+
# Use heredoc to send help text through helpCat alias, which will apply syntax highlighting if pygmentize is available.
68+
cat <<'HELP_USAGE' | helpCat0 bash
69+
makeParse: Parse the workflow log and create an output.json file.
70+
Usage:
71+
makeParse <workflowconfig.log>
72+
73+
Example usage:
74+
#makeParse workflowconfig.log > ~/output.json # To parse a specific log file.
75+
makeParse /lustre/alice/tpcdata/Run3/SCDprodTests/fullRec/PbPb_Streamers_Tune_ClusterErrors-merge-streamer/avgCharge_fullTPC_sampling_TimeBins16-Average0_rejectEdgeCl-Seed0-Track0-margin0/LHC23zzh.b5p/544116.38kHz/0110/workflowconfig.log > workflow.json
76+
cat workflow.json | jq '.[] | select(.command | test("^o2-dpl"))' # Filter DPL workflows.
77+
jq '.[] | select(.command | test("^o2-gpu"))' workflow.json # Filter GPU related commands.
78+
79+
HELP_USAGE
80+
return # Exit the function if no parameters provided
81+
fi
82+
#
83+
log_file=$1
84+
jq -Rn '
85+
[inputs | split("\n")[] | select(length > 0 and startswith("o2-")) |
86+
{
87+
command: (split(" ")[0]),
88+
switches: (split(" ") | .[1:-1] |
89+
reduce .[] as $item ({};
90+
if $item | startswith("--") then
91+
if $item | contains("=") then
92+
. + ({($item | ltrimstr("--") | split("=")[0]): ($item | split("=")[1])})
93+
else
94+
. + ({($item | ltrimstr("--")): true})
95+
end
96+
else
97+
.[keys_unsorted[-1]] = $item
98+
end
99+
)),
100+
configKeyValues: (if (contains("--configKeyValues")) then
101+
(split("--configKeyValues")[1] | split("|")[0] | gsub("^\\s+\"|\"\\s+;"; "") | split(";") |
102+
map(select(. != "" and contains("="))) |
103+
map(split("=") | select(length == 2)) |
104+
map({(.[0]): .[1]})) | add
105+
else
106+
{}
107+
end
108+
)
109+
}
110+
]' "$log_file"
111+
}
112+
113+
makeDiffExample(){
114+
cat <<HELP_USAGE | helpCat
115+
Description:
116+
117+
This function provides examples of how to parse workflow configuration logs into JSON format and then compare these JSON files using \`diff\`. The comparison can be done directly using \`jq\` to filter and sort the JSON data, which helps in identifying the differences more clearly.
118+
119+
Examples:
120+
121+
1. Parse workflow configuration logs into JSON format:
122+
\`makeParse /lustre/alice/tpcdata/Run3/SCDprodTests/fullRec/PbPb_Streamers_Tune_ClusterErrors-merge-streamer/avgCharge_fullTPC_sampling_TimeBins16-Average0_rejectEdgeCl-Seed0-Track0-margin0/LHC23zzh.b5p/544116.38kHz/0110/workflowconfig.log > workflow0.json\`
123+
124+
\`makeParse /lustre/alice/tpcdata/Run3/SCDprodTests/fullRec/PbPb_Streamers_Tune_ClusterErrors-merge-streamer/avgCharge_fullTPC_sampling_TimeBins16-Average0_rejectEdgeCl-Seed0-Track0-margin0-ref/LHC23zzh.b5p/544116.38kHz/0110/workflowconfig.log > workflow1.json\`
125+
126+
2. Compare two JSON files using \`jq\` and \`diff\` directly without temporary files:
127+
\diff <(jq --sort-keys . workflow1.json) <(jq --sort-keys . workflow0.json)\
128+
129+
3. Use \`diff\` with side-by-side view and color using ANSI color codes:
130+
\`diff --side-by-side --left-column --color=always <(jq --sort-keys . workflow1.json) <(jq --sort-keys . workflow0.json) | less -R\`
131+
132+
4. Compare JSON files focusing only on commands starting with "o2-gpu":
133+
Filtering the entries where the command starts with "o2-gpu" and then comparing:
134+
\ diff --side-by-side --left-column --color=always <(jq '.[] | select(.command | test("^o2-gpu"))' workflow1.json | jq --sort-keys .) <(jq '.[] | select(.command | test("^o2-gpu"))' workflow0.json | jq --sort-keys .) | less -R
135+
HELP_USAGE
136+
}
137+
138+
init648

0 commit comments

Comments
 (0)