You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* In Hyperloop, an estimate of the CO2eq produced by your trains is shown in order to give an idea of the environmental impact. We need to run analysis to achieve our scientific goals but we can optimize the code and sometimes work efficiently even with less trains. The displayed value should help educate the decision if a train is needed.
14
-
The value is shown before trains have been run (an estimate based using the wagon test) and then when a train run has finished.
14
+
The value is shown before trains have been run (an estimate based using the wagon test) and then when a train run has finished.
15
15
* The estimate is visible in:
16
16
* Wagon test view and train submission view (an estimate based on the wagon test)
17
17
* Train test view (an estimate based on the train test. So, this estimate accounts for all wagons in the train)
18
-
* In the 'General' tab of the train view, when a train is in the state 'Done'. This estimate directly uses the CPU usage of the train run.
18
+
* In the 'General' tab of the train view, when a train is in the state 'Done'. This estimate directly uses the CPU usage of the train run.
19
19
20
20
## <aname="calculations"></a> How is it calculated?
21
21
22
22
Principle:
23
+
23
24
* There are a few studies for estimating CO2eq of computing, and there are many factors to consider such as the efficiency of the machines used and the power grid which the machines are on.
24
25
* Furthermore, grey energy (the energy to produce the machines) and the transfers of the data could be considered.
25
26
* It would be difficult to give justice to all these details and check for each train where exactly the jobs were running.
26
27
* Therefore, in Hyperloop, estimates of CO2eq are directly derived from CPU usage using an average conversion factor.
27
28
28
29
Calculation input:
29
-
* Power per computing core: We use an optimistic value of 10 W/core for pure power consumption and assume a 50/50 split between the carbon from power and carbon from embodied / embedded emissions, therefore working effectively with 20 W/core. Some literature estimates up to 53 W/core for power alone in high-performance computing (see https://www.nature.com/articles/s41550-020-1169-1).
30
-
* Electricity emission factors (kgCO2eq per kWh) vary depending how electricity is produced. However, carbon is not everything and we do not want to enter a debate on how energy should be produced here. Therefore we use an average value from https://arxiv.org/pdf/2011.02839.pdf of 0.301 kgCO2eq / kWh, noting that even for the same country the estimates vary significantly depending on the source (e.g. comparing to https://arxiv.org/pdf/2101.02049).
31
-
* Hyperloop estimations do not account for the power consumption of data transfer, central infrastructure, or power for storage. The paper 'electricity intensity of internet data transmission', https://onlinelibrary.wiley.com/doi/pdf/10.1111/jiec.12630, estimates 0.06kWh/GB in data transfer. This would mean that a petabyte of data transfer needs 18 tCO2eq. Additionally, the carbon produced by storing the data would not be negligible. For simplicity, we do not account for these aspects, so that our estimates are more directly linked to individual train runs, and not the wider Grid infrastructure.
30
+
31
+
* Power per computing core: We use an optimistic value of 10 W/core for pure power consumption and assume a 50/50 split between the carbon from power and carbon from embodied / embedded emissions, therefore working effectively with 20 W/core. Some literature estimates up to 53 W/core for power alone in high-performance computing (see <https://www.nature.com/articles/s41550-020-1169-1>).
32
+
* Electricity emission factors (kgCO2eq per kWh) vary depending how electricity is produced. However, carbon is not everything and we do not want to enter a debate on how energy should be produced here. Therefore we use an average value from <https://arxiv.org/pdf/2011.02839.pdf> of 0.301 kgCO2eq / kWh, noting that even for the same country the estimates vary significantly depending on the source (e.g. comparing to <https://arxiv.org/pdf/2101.02049>).
33
+
* Hyperloop estimations do not account for the power consumption of data transfer, central infrastructure, or power for storage. The paper 'electricity intensity of internet data transmission', <https://onlinelibrary.wiley.com/doi/pdf/10.1111/jiec.12630>, estimates 0.06kWh/GB in data transfer. This would mean that a petabyte of data transfer needs 18 tCO2eq. Additionally, the carbon produced by storing the data would not be negligible. For simplicity, we do not account for these aspects, so that our estimates are more directly linked to individual train runs, and not the wider Grid infrastructure.
32
34
* At 20W per core, and 0.301 kgCO2eq / kWh, this gives us: **6t CO2eq per 1MCPUh** or **1 CPU year = 53.3 kgCO2eq**
33
-
* In order to compare these emissions to something we know, we use CO2eq produced by flights based on curb6.com
35
+
* In order to compare these emissions to something we know, we use CO2eq produced by flights based on curb6.com
Copy file name to clipboardExpand all lines: docs/hyperloop/hyperlooppolicy.md
+14-14Lines changed: 14 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,38 +7,38 @@ title: Fair usage policy
7
7
8
8
The very large amount of data that will be collected in Run 3 represents a challenge for analysis, for both the CPU needs and the read data from storage, and therefore a resource usage policy has been put in place to ensure proper use of computing resources. The policy has been openly discussed in multiple meetings, including ALICE weeks, and is subject to adjustments as necessary and as the collaboration gains experience with the Run 3 analysis. If you have questions or doubts, please first refer to your PWG convener who will then bring up the case with the analysis coordinator.
<imgsrc="../images/hyperlooppolicy.png"width="80%"alt="Screenshot of hyperlooppolicy">
14
14
</div>
15
15
16
-
In general, four categories of trains exist:
16
+
In general, four categories of trains exist:
17
17
18
-
* Trains below 30 TB and taking more than 2.0y of CPU time (red shaded area) are very strongly discouraged. In those cases, please resort to very small trains (where throughputs of even 100 KB/s are allowed with autosubmission) to run.
18
+
* Trains below 30 TB and taking more than 2.0y of CPU time (red shaded area) are very strongly discouraged. In those cases, please resort to very small trains (where throughputs of even 100 KB/s are allowed with autosubmission) to run.
19
19
* Trains that are lower than 2y in CPU usage and loop over less than 200 TB are free to execute and can be executed on Hyperloop via autosubmission. In a certain region between 30-200 TB, slightly more than 2y in CPU time is allowed (see sketch).
20
-
* Trains that loop over more than 200 TB and less than 800 TB are dealt with as follows:
21
-
* if they require less than 10 years of CPU time, they need only PWG convener approval.
22
-
* if they require more than 10 years of CPU time but less than 200 years, they need Analysis and Physics Coordinator approval to run.
23
-
* if they require over 200 years of CPU, they need excplicit PB approval.
24
-
* Heavy trains looping over datasets bigger than 800 TB are dealt with as follows:
25
-
* if they require less than 20 years of CPU time, they need only PWG approval.
26
-
* if they require between 20 to 200y of CPU, they can be approved offline by Analysis and Physics Coordination.
27
-
* if they require over 200 years of CPU, they need explicit PB approval.
20
+
* Trains that loop over more than 200 TB and less than 800 TB are dealt with as follows:
21
+
* if they require less than 10 years of CPU time, they need only PWG convener approval.
22
+
* if they require more than 10 years of CPU time but less than 200 years, they need Analysis and Physics Coordinator approval to run.
23
+
* if they require over 200 years of CPU, they need excplicit PB approval.
24
+
* Heavy trains looping over datasets bigger than 800 TB are dealt with as follows:
25
+
* if they require less than 20 years of CPU time, they need only PWG approval.
26
+
* if they require between 20 to 200y of CPU, they can be approved offline by Analysis and Physics Coordination.
27
+
* if they require over 200 years of CPU, they need explicit PB approval.
28
28
29
29
## <aname="implementation"></a>Implementation in Hyperloop datasets
30
30
31
31
In practice the chart above is mapped on a number of distinct resource groups which determine the limits assigned to each dataset:
32
32
33
33
<divalign="center">
34
-
<imgsrc="../images/resourcetable.png"width=800>
34
+
<imgsrc="../images/resourcetable.png"width=800alt="Screenshot of resourcetable">
35
35
</div>
36
36
37
37
The smaller the dataset size, the more often it is automatically submitted per week and the more often you are allowed to run on it per week. Manual requests to datasets above 50 TB are only fulfilled at the automatical submission times defined. This is in order to allow grouping of wagons to large trains.
38
38
39
39
## <aname="deriveddata"></a>Derived data
40
40
41
-
Derived datasets can be created on Hyperloop which are by construction much smaller than the original datasets. Those are advantagous because steps which are identical in each analysis train run (e.g. event selection and centrality calculation, secondary-vertex finding) are only executed once which saves CPU. Furthermore, as the size is smaller such trains cause less load on the storages.
41
+
Derived datasets can be created on Hyperloop which are by construction much smaller than the original datasets. Those are advantagous because steps which are identical in each analysis train run (e.g. event selection and centrality calculation, secondary-vertex finding) are only executed once which saves CPU. Furthermore, as the size is smaller such trains cause less load on the storages.
42
42
43
43
As an example, you can imagine that you run a derived data train on a dataset of 500 TB where you need explicit approval. Say you have a reduction factor of 100, then your output derived data is about 5 TB. You will be allowed to run on that dataset much more frequent, see the table above.
Copy file name to clipboardExpand all lines: docs/hyperloop/legoexpert.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,7 +17,7 @@ title: For the Run 2 LEGO train expert. What has changed?
17
17
* There is a history feature for wagons and datasets. You can access it by clicking on the button `📜` available inside of a wagon/dataset view. A detailed view of what has been created/updated/removed from the wagon/dataset is shown, as well as the username and the time when the change was made.
<imgsrc="../images/datasetHistory.png"width="100%"alt="Screenshot of dataset history">
21
21
</div>
22
22
23
23
* There are automated notifications. These notifications are created per user, and display changes made to tools, like _Datasets_, that are being used by the user. They are displayed per _Analysis_ in the _My Analyses_ page, or globally in the button `🔔` which can be found on the top menu.
@@ -31,5 +31,5 @@ title: For the Run 2 LEGO train expert. What has changed?
31
31
***Performance Graphs** page allows the user to upload his own local metrics file, and then generate the test graphs specific to that file. You produce a local _performanceMetrics.json_ by running the o2 workflow with the argument _--resources-monitoring 2_ which, in this example, produces monitoring information every 2 seconds. These are the same type of graphs produced in the _Test Graphs_ tab of the train run. This page can be accessed at: <https://alimonitor.cern.ch/hyperloop/performance-graphs>.
<img src="../images/notificationsMyAnalyses.png" width="90%" alt="Screenshot of notifications in my analyses">
13
13
</div>
14
14
15
15
* The user can click the `✖️` button to remove a notification. In order to remove all the notifications, go to the Notifications page, and click the `❌`_Dismiss all_ button.
<imgsrc="../images/runlistUpdated.png"width="90%"alt="Screenshot of runlist updated">
67
67
</div>
68
68
69
69
## <aname="mergelistUpdate"></a>Mergelist updated
70
70
71
71
* The mergelist defines which runs are merged into one file at the end of the train running. The user is informed when a mergelist has been modified, added to or removed from the dataset production.
<imgsrc="../images/mergelistUpdate.png"width="90%"alt="Screenshot of mergelist update">
75
75
</div>
76
76
77
77
## <aname="linkedDataset"></a>Short datasets
@@ -89,55 +89,53 @@ Informs the user when a wagon has been disabled in different circumstances:
89
89
* Local tests are cleaned if the wagons are not submitted in a period of 4 weeks. The user is notified that the respective wagons are automatically disabled.
90
90
91
91
<divalign="center">
92
-
<imgsrc="../images/testCleaned.png"width="90%">
92
+
<imgsrc="../images/testCleaned.png"width="90%"alt="Screenshot of test cleaned">
93
93
</div>
94
94
95
95
* When a wagon with derived data output is enabled, the test cannot start if the wagon and its dependencies share the same workflow. As a result, the wagon is disabled and the user is notified about the wagons which share the same task.
96
96
97
97
* The notification format is: The wagon _"wagon_name"_ was disabled in _"dataset_name"_. There is derived data. The following wagons have the same workflows {_wagon1_, _wagon2_: _common_workflow_},...,{_wagonX_, _wagonY_: _common_workflow_}
<imgsrc="../images/wagonDisabled1.png"width="90%"alt="Screenshot of wagon disabled 1">
101
101
</div>
102
102
103
103
* If among the wagon and its dependencies there are identical derived data outputs, the test cannot start, and the wagon is disabled.
104
104
105
105
* The notification format is: The wagon _"wagon_name"_ was disabled in _"dataset_name"_. The following wagons have the same derived data outputs {_wagon1_, _wagon2_: _common_derived_data_},...,{_wagonX_, _wagonY_: _common_derived_data_}
106
106
107
107
<divalign="center">
108
-
<imgsrc="../images/wagonDisabled.png"width="90%">
108
+
<imgsrc="../images/wagonDisabled.png"width="90%"alt="Screenshot of wagon disabled">
109
109
</div>
110
-
110
+
111
111
* The wagon is disabled if the workflow name has been changed in the meantime. This is fixed by updating the workflow name in the wagon configuration.
<imgsrc="../images/notificationWorkflow.png"width="90%"alt="Screenshot of notification workflow">
115
115
</div>
116
116
117
117
* The wagon is disabled if one of the user defined dependencies of the wagon is considered identical to a service wagon. In order to most efficiently make use of the Grid and the analysis factilities, copies of core services are not permitted as it prevents combining several users into one train.
<imgsrc="../images/notificationIdenticalWagon.png"width="90%"alt="Screenshot of notification of identical wagon">
121
121
</div>
122
122
123
123
A service wagon is considered identical to a user wagon if it shares the same activated output tables, the same workflow, and it has matching configurables. To fix this error, please use the listed service wagon as a dependency instead of the copy.
* Hyperloop makes a comparison between the wagon configuration and the configuration defined in O2 for the package tag selected for the wagon. If they do not coincide, the user will be informed about the mismatch. The comparison is case sensitive, therefore a Configurable will not match if its name does not contain the identical lowercase / uppercase combination.
130
128
131
129
* The user is notified if there is a configurable present in the wagon configuration that is not defined in O2 for the selected package tag tag. Likewise, it informs the user when the wagon configuration misses one or more of the Configurables defined in O2 for the specific tag.
<imgsrc="../images/inconsistentParameters2.png"width="90%"alt="Screenshot of inconsistent parameters 2">
135
133
</div>
136
134
137
135
* If the **wagon configuration is old**, and the wagon is enabled with the latest package tag, the user is advised to sync the wagon in order to get the present configuration. Following this, the test will start automatically. Likewise, the test is reset whenever there is a change in the database, such as updating or syncing the wagon configuration or its dependencies.
<imgsrc="../images/inconsistentParameters.png"width="90%"alt="Screenshot of inconsistent parameters">
141
139
</div>
142
140
143
141
* If the **wagon is enabled with an older tag**, the configuration might not match (hence the notification). If the old tag is needed, then syncing is not an option because this will set the package to the latest one. Therefore, the wagon configuration has to be modified as needed. The user can take as a reference _full_config.json_ in the test output, which shows the configuration the test is being run with, and compare it to the wagon configuration.
0 commit comments