Save hash and path grid informations when generating FK Tables by kamillaurent · Pull Request #247 · NNPDF/pineko

kamillaurent · 2026-04-28T11:54:47Z

When a FK Table is generated, the hash information and path of the pineappl grid used for it are saved in the metadata of the FK Table.

These information can be read using the pinealppl-cli, for example:

pineappl read --show ATLAS_WZ_TOT_13TEV-ATLASWZTOT13TEV81PB_Z_tot.pineappl.lz4

outputs, between the other informations, this lines:

grid_files: {"ATLAS_WZ_TOT_13TEV-ATLASWZTOT13TEV81PB_Z_tot.pineappl": {"hash": "f1110fd0bb8ccddf28bf722f9f9c3dd5", "path": "/data/theorie/klaurent/FKTables/pineko/data/grids/41000000/ATLAS_WZ_TOT_13TEV-ATLASWZTOT13TEV81PB_Z_tot.pineappl.lz4"}}

This PR is related to issue #225

scarlehoff

LGTM.
(haven't tested it, but I trust the OP is a real-life test)

scarlehoff · 2026-05-01T05:45:53Z

+    if grid_path is not None:
+        grid_path_obj = pathlib.Path(grid_path)
+        grid_hash = hashlib.md5(grid_path_obj.read_bytes()).hexdigest()
+        grid_files = {grid_path_obj.stem: {"hash": grid_hash, "path": str(grid_path_obj.resolve())}}


Suggested change

grid_files = {grid_path_obj.stem: {"hash": grid_hash, "path": str(grid_path_obj.resolve())}}

grid_files = {grid_path_obj.stem: {"hash": grid_hash, "theory_folder": grid_path_obj.parent.name}}

Out of privacy concerns, I would just store the path starting from the theory_slim folder.

My suggestion is, since you have the grid name as the key, save just the theory folder name.

agreed on the privacy, but I think I would keep the resolve(), because I think this way we can resolve symbolic links, right? i.e. .resolve().parent (or similar)

scarlehoff · 2026-05-12T09:44:49Z

Hi @kamillaurent please do the small change requested and fix the issues found by pre-commit (usually just means you didn't run pre-commit, if you do the changes then git add the evolve.py file and run pre-commit it will be fixed.

felixhekhorn

Remember the other things we discussed:

split in two simple keys "grid_path" and "grid_hash"
address FONLL magic

felixhekhorn · 2026-05-12T11:53:15Z

+    if grid_path is not None:
+        grid_path_obj = pathlib.Path(grid_path)
+        grid_hash = hashlib.md5(grid_path_obj.read_bytes()).hexdigest()
+        grid_files = {grid_path_obj.stem: {"hash": grid_hash, "path": str(grid_path_obj.resolve())}}


agreed on the privacy, but I think I would keep the resolve(), because I think this way we can resolve symbolic links, right? i.e. .resolve().parent (or similar)

scarlehoff · 2026-05-19T07:29:54Z

@kamillaurent please address the changes and run pre-commit so that we can merge this

scarlehoff · 2026-05-25T09:53:16Z

There's a few conflicts due to having merge @Radonirinaunimi's PR but they are in the documentation of the functions so it should be quick to fix. It would be good to have and use this PR for the next batch of fk tables so we have this information in 😅

…etadata, keeping path to grids from pineko

kamillaurent · 2026-05-26T10:17:16Z

Now I made the informations in the metadata more readable. For example, using the command
pineappl read --show ATLAS_WZ_TOT_13TEV-ATLASWZTOT13TEV81PB_Z_tot.pineappl.lz4 one can find the informations:

...
grid_hash: f1110fd0bb8ccddf28bf722f9f9c3dd5
grid_path: /data/grids/41000000/ATLAS_WZ_TOT_13TEV-ATLASWZTOT13TEV81PB_Z_tot.pineappl.lz4
grid_theory: 41000000
...

When we use the FONLL procedure, in the last step many FK-tables get merged into one. I am working on retaining the info about hash and theory of the original FKs into the merged one.

felixhekhorn · 2026-05-26T11:51:28Z

+    if grid_path is not None:
+        grid_path_obj = pathlib.Path(grid_path).resolve()
+        grid_hash = hashlib.md5(grid_path_obj.read_bytes()).hexdigest()
+        grid_path_parts = grid_path_obj.parts
+        if "pineko" in grid_path_parts:
+            pineko_idx = grid_path_parts.index("pineko")
+            display_path = str(pathlib.Path("/", *grid_path_parts[pineko_idx + 1 :]))
+        elif "data" in grid_path_parts:
+            data_idx = grid_path_parts.index("data")
+            display_path = str(pathlib.Path("/", *grid_path_parts[data_idx:]))
+        else:
+            display_path = grid_path_obj.name
+        fktable.set_metadata("grid_hash", grid_hash)
+        fktable.set_metadata("grid_theory", grid_path_obj.parent.name)
+        fktable.set_metadata("grid_path", display_path)


This is far too much code duplication - this needs to be encapsulated

do we really need grid_path? this looks fairly complicated ...

Yes, the grids path part is a bit more complicated than needed, data/grids are not important.

Once you have the grid path, the "path of the grid itself" is grid_path.name and the theory folder grid_path.parent.name. You don't need anything else.

I tried to simplify the metadata, using a function and calling it when the FK table is constructed (so there is no code duplication):

def set_fktable_metadata(fktable, theory_meta, grid_path=None): """Store the common FK metadata and, if available, grid provenance.""" fktable.set_metadata("pineko_version", version.__version__) fktable.set_metadata("theory_card", json.dumps(theory_meta)) if grid_path is None: return grid_path_obj = pathlib.Path(grid_path).resolve() grid_hash = hashlib.md5(grid_path_obj.read_bytes()).hexdigest() fktable.set_metadata("grid_hash", grid_hash) fktable.set_metadata("grid_theory", grid_path_obj.parent.name) fktable.set_metadata("grid_name", grid_path_obj.name)

For consistency I moved into the function also pineko_version and theory_card. I removed the path since the relevant information is the name of the folder, which is also the theory ID.

kamillaurent · 2026-05-27T15:55:03Z

@scarlehoff regarding the FONLL procedure, fonll.py merges all the FK-Tables into the $THEORY_ID_00 and retains informations only from this one. For example, pineappl read --show 41000000/HERA_NC_225GEV_EP_SIGMARED.pineappl.lz4 gives

...
eko_version: 0.15.3
fk_assumptions: Nf4Ind
fktable_id: HERACOMBNCEP460
grid_hash: 0a34218e5e6b254475243264d1ced40e
grid_name: HERA_NC_225GEV_EP_SIGMARED.pineappl.lz4
grid_theory: 4100000000
hepdata: 10.17182/hepdata.68951.v1/t4
nnpdf_id: HERACOMBNCEP460
...

Is this sufficient or we want informations about all the merged grids? In the case we want all the merged grids, the grid_name will be the same for all. The grid_hash and grid_theory will be different, should we store them into an array and store the array in the metadata of the merged FK Table?

scarlehoff · 2026-05-27T15:58:56Z

I would say that grid theory should strip the extra 0s (so it should say just 41000000, since that covers all).
Grid name is fine.

And my preference for the hash would be simply a string of

00-0a34218e5e6b254475243264d1ced40e 01-hash 02-hash 03-...

felixhekhorn · 2026-05-28T07:16:29Z

Are you sure multiple plain keys wouldn't be better? Like grid_name_00?

In that case we should also mark the standard field grid_name: /FONLL or something that can never be a hash.

I'm thinking about what would be easiest to parse afterwards...

scarlehoff · 2026-05-28T07:21:45Z

In my view, the important part about the hash is that it is unique in the comparison so making sure that all fktables have the key grid_hash and that the string is unique is sufficient and preferable.
The fact that one of the hashes is made by multiple hashes is irrelevant.

And, for the cases in which the hashes are not identical, then one still has the information about the partial grids but most of the time we want a yes/no question to the "are these coming from the same grids" question. It is enough for one to be different.

kamillaurent · 2026-05-28T08:46:18Z

Done! Now for a merged FK-Table the metadata contains the main theory and the hash of all the original (merged) FKs. For example:
pineappl read --show 41000000/HERA_NC_225GEV_EP_SIGMARED.pineappl.lz4 gives

...
fktable_id: HERACOMBNCEP460
grid_hash: 00-0a34218e5e6b254475243264d1ced40e 01-efd1748bb08c25c43a22971cfe8699a9 02-6d0e6e92977898c4f052c05b4880eb51 03-bf95481ca46d9859e57476a0c3ded8db 04-fe3403683c505c439ef9e0dbfbdb74a7 05-df3dc24b51036ff6f2a97d93c38f445d 06-a70df1c6de56f8f5acb0229f18faa250
grid_name: HERA_NC_225GEV_EP_SIGMARED.pineappl.lz4
grid_theory: 41000000
...

scarlehoff · 2026-05-29T11:56:19Z

@kamillaurent make sure to run pre-commit on the changed files so that the CI doesn't complain!

add hash and parth information when generating FK Tables

3e9a620

kamillaurent mentioned this pull request Apr 28, 2026

Save the hash and theory directory of the grid #225

Open

kamillaurent requested a review from scarlehoff April 28, 2026 11:58

scarlehoff approved these changes May 1, 2026

View reviewed changes

felixhekhorn requested changes May 12, 2026

View reviewed changes

jekoorn mentioned this pull request May 21, 2026

Log a hash of every fktable upon vp-setupfit NNPDF/nnpdf#2397

Open

Kamil Laurent added 2 commits May 26, 2026 09:57

resolved conflicts

e12865f

made info in the metadata more readable; removed absolute path from m…

0a184f6

…etadata, keeping path to grids from pineko

felixhekhorn reviewed May 26, 2026

View reviewed changes

simplified metadata

6f6f402

Addressing metadata of merged FK Tables

bbf62d2

	grid_files = {grid_path_obj.stem: {"hash": grid_hash, "path": str(grid_path_obj.resolve())}}
	grid_files = {grid_path_obj.stem: {"hash": grid_hash, "theory_folder": grid_path_obj.parent.name}}

Conversation

kamillaurent commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scarlehoff left a comment

Choose a reason for hiding this comment

Uh oh!

scarlehoff May 1, 2026

Choose a reason for hiding this comment

Uh oh!

felixhekhorn May 12, 2026

Choose a reason for hiding this comment

Uh oh!

scarlehoff commented May 12, 2026

Uh oh!

felixhekhorn left a comment

Choose a reason for hiding this comment

Uh oh!

felixhekhorn May 12, 2026

Choose a reason for hiding this comment

Uh oh!

scarlehoff commented May 19, 2026

Uh oh!

scarlehoff commented May 25, 2026

Uh oh!

kamillaurent commented May 26, 2026

Uh oh!

felixhekhorn May 26, 2026

Choose a reason for hiding this comment

Uh oh!

scarlehoff May 26, 2026

Choose a reason for hiding this comment

Uh oh!

kamillaurent May 27, 2026

Choose a reason for hiding this comment

Uh oh!

kamillaurent commented May 27, 2026

Uh oh!

scarlehoff commented May 27, 2026

Uh oh!

felixhekhorn commented May 28, 2026

Uh oh!

scarlehoff commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kamillaurent commented May 28, 2026

Uh oh!

scarlehoff commented May 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kamillaurent commented Apr 28, 2026 •

edited

Loading

scarlehoff commented May 28, 2026 •

edited

Loading