Adding support for .jpk-qi-data and .bin files by ahobbs7 · Pull Request #190 · AFM-SPM/AFMReader

ahobbs7 · 2026-05-03T16:58:46Z

Adds support for .jpk-qi-data files, allowing the loading of the image for the selected channel. It also allows for the raw force-distance curve data as well as metadata for each curve to be loaded efficiently, using a lazy data structure approach that loads each section of the dataset into memory only when it is required.
The updates also allow for the conversion of .jpk-qi-data files into a HDF5 file format (.h5-jpk) for much faster mass analysis. Hence, the h5-jpk loading code has also been updated to support the loading of these converted files.

The update also allows .bin with different binary formats to be loaded.

closes #174 closes #173

…igger point or by contact point

… a function to general loader

… format

…make the save an adjustable option

… once, then extracting separately

…larity

…files and removing the ability to run curve analysis directly in the reader

…ore robust coordinate access

…o eagerly load lazy loaded data

…me efficient method

…g to memory then saving in one go when duplicating data to h5

…sampled curves then are streamed directly into h5

…much more memory efficient

codecov-commenter · 2026-05-04T12:55:24Z

Codecov Report

❌ Patch coverage is 22.79910% with 684 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.60%. Comparing base (d65d297) to head (0146542).
⚠️ Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
AFMReader/jpk_qi.py	10.07%	491 Missing ⚠️
AFMReader/h5_jpk.py	22.65%	99 Missing ⚠️
AFMReader/lazy_data_classes.py	38.77%	30 Missing ⚠️
AFMReader/general_loader.py	50.84%	29 Missing ⚠️
AFMReader/raw_bin.py	21.21%	26 Missing ⚠️
AFMReader/asd.py	72.72%	3 Missing ⚠️
AFMReader/jpk.py	89.28%	3 Missing ⚠️
AFMReader/spm.py	76.92%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #190       +/-   ##
===========================================
- Coverage   79.84%   51.60%   -28.25%     
===========================================
  Files          12       15        +3     
  Lines         928     1804      +876     
===========================================
+ Hits          741      931      +190     
- Misses        187      873      +686

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

SylviaWhittle

Small partial review.

I'd like to call with you to talk about the interesting lazy loading of data if you have time?

SylviaWhittle · 2026-05-22T15:02:36Z

        return frames, pixel_to_nanometre_scaling_factor, header_dict


+def get_asd_channels(file_path: Path):


I don't have time to check this manually - does it work? There isn't a test but frankly we don't have the dev time to move slowly. If you say it works, this is fine with me.

The channel fetching seems to work though I'm happy to quickly make some tests for them as that should be pretty quick.

SylviaWhittle · 2026-05-22T15:13:33Z

+                elif len(h5_returned) == 4:
+                    image, pixel_to_nanometre_scaling_factor, _, curve_data = h5_returned  # type: ignore[misc]
+                    self.loaded_curves = True
+                    print(


Suggested change

print(

logger.info(

or just remove it.

Yes, I'll just remove them, they are some accidentally left in prints for debugging.

SylviaWhittle · 2026-05-22T15:13:43Z

+                        f"Loaded image with shape {image.shape} and pixel to nanometre "
+                        f"scaling factor {pixel_to_nanometre_scaling_factor}"
+                    )
+                    print(f"Image has max value {image.max()} and min value {image.min()}")


SylviaWhittle · 2026-05-22T15:14:32Z

                image, pixel_to_nanometre_scaling_factor = spm.load_spm(self.filepath, self.channel)
            elif self.suffix == ".h5-jpk":
-                image, pixel_to_nanometre_scaling_factor, _ = h5_jpk.load_h5jpk(self.filepath, self.channel)
+                h5_returned = h5_jpk.load_h5jpk(self.filepath, self.channel, load_curves=not self.loaded_curves)


Could this entire block be absorbed into the h5_jpk.load_h5jpk function call? It's a little messy here.

I'll have a go too.

Yes, is definitely a bit messy. My thinking was preventing the need to make changes to topostats by not changing what each load function returns if it isn't reading the curve data I'm adding support for. Though I think h5jpk is not used in TopoStats anyway? And it might be necessary to make changes to topostats reading anyway due to the changes I have been working on for returning the z unit for the read files?

SylviaWhittle · 2026-05-22T15:15:20Z

+            raise e

-    # scope for a "check what channels are available" function similar to above.
+    def get_available_channels(self):  # noqa: C901


Guessing this is for the napari feature to list the channels?

Well done if this is all working, that's a lot of work.

SylviaWhittle · 2026-05-22T15:25:09Z

+            A proxy object for the specified row.
+        """
+
+        class RowProxy:


Is the reason RowProxy is defined within the scope of LazyQiData encapsulation? It'll get redefined each time __getitem__ runs, though this is likely a negligible and unimportant cost.

SylviaWhittle · 2026-05-22T15:41:59Z

+                self.parent = parent
+                self.y = y
+
+            def __getitem__(self, x: int):


I'm going to need a bit of an explanation on how this is intended to work - could we set up a call to talk about it? There's a lot going on here. Plus, frankly I'm just curious.

Yes, definitely!

SylviaWhittle · 2026-05-22T15:43:05Z

 # Set the format to have blue time, green file, module, function and line, and white message
 logger.add(
-    sys.stderr,
+    lambda msg: sys.stderr.write(msg),  # pylint: disable=unnecessary-lambda


What necessitated this change?

It seems like the general_loader and some of the spm tests which required were failing due to not being able to correctly reading the logged output (instead always reading an empty string), they seem to be failing when I run even the main branch locally for that reason. The lamda appears to be necessary due to a quirk of loguru where lamda makes sys.stderr be dynamically retrieved at the time of logging rather than once at import time.

SylviaWhittle · 2026-05-27T12:02:59Z

+    return final_multiplier, final_offset, unit
+
+
+class jpk_qi_loader:


Capitalise this please :)

SylviaWhittle · 2026-05-27T12:18:54Z

+        # Open the ZIP archive once and keep it open for the duration of the loading process
+        self.qi_archive = zipfile.ZipFile(self.filepath, "r")  # pylint: disable=consider-using-with
+        logger.info(f"Opened JPK QI archive at {self.filepath}")
+        self.namelist = self.qi_archive.namelist()


Maybe call this list_of_all_paths or something - since namelist could be anything? Just so future people (including ourselves) can tell at a glance what this is.

Maybe also add a comment above as to what it's used for? :)

SylviaWhittle · 2026-05-27T12:20:20Z

+        self.qi_archive = zipfile.ZipFile(self.filepath, "r")  # pylint: disable=consider-using-with
+        logger.info(f"Opened JPK QI archive at {self.filepath}")
+        self.namelist = self.qi_archive.namelist()
+        # Set path to the .jpk-qi-image file within the archive for later use


Suggested change

# Set path to the .jpk-qi-image file within the archive for later use

# For holding the reference to where the actual .jqk-qi image is (not the metadata).

SylviaWhittle · 2026-05-27T12:39:31Z

+        if self.save_as_h5:
+            self.save_lite_data()
+
+        return (self.image, self.px2nm, (self.curve_data, self.channels_units, self.full_metadata))


Possibly bundle these objects into one metadata dataclass, JPKQiCurveData? Gentle suggestion.

ahobbs7 added 30 commits February 12, 2026 17:57

Added jpk-qi-data loading functionality. Two possible channels: by tr…

d25abaf

…igger point or by contact point

Adding get available channels function to each file format as well as…

9d28dec

… a function to general loader

Making load jpk qi data function use zipfile instead of afmformats

4bd6bb3

Fixing scaling

28f7633

Updating general_loader and dependencies

a67b11f

Adjusting h5_jpk so it works for different shaped images

37e1ae2

Making the jpk-qi-data processing save the h5 jpk file in the correct…

16a827c

… format

Add the ability to save the metadata to h5 from jpk qi. Additionally …

c780de5

…make the save an adjustable option

Adding ability to load force curves from h5 file

a764ca6

Improving speed of loading qi curve data by loading all the curves at…

405a95d

… once, then extracting separately

Making loading jpk-qi-data return all the curves

3198567

Adding .bin files support

c25fb75

Adding returning of metadata

085b252

Refactoring jpk-qi-data reader to use a loader class for greater modu…

06ec0de

…larity

Starting to implement lazy loading

78c54a7

Made force curves lazy loaded for h5-jpk and jpk-qi-data

3c327e9

Implementing caching of heavy data objects/ references to large open …

e776c66

…files and removing the ability to run curve analysis directly in the reader

Adjusting curve data access method to work more like a 2D array for m…

3e0b8b1

…ore robust coordinate access

Separating saving functionality for jpk-qi-data and adding function t…

b8ec43a

…o eagerly load lazy loaded data

Fixing duplicated converting to nm bug

10c1050

Minor changes to fix double scaling on current channel as well

b510141

Fixing minor error

afc65cc

Added timing for testing and started converting to more memory and ti…

b1ec527

…me efficient method

Making jpk-qi-data loading stream data into h5 file rather than savin…

1fd1eed

…g to memory then saving in one go when duplicating data to h5

Changing metdata data saving so 'changing keys' are assumed based on …

4e90188

…sampled curves then are streamed directly into h5

Pre-sizing the curve data to make loading faster (using a best guess)

66f388a

Improving performance by removing javaproperties reliance in loop

f286e57

Making the saving to h5 save in sections using a buffer

5391179

Removing redundant functions and fixing minor index bugs

f0235ad

removing possibility of size 1 image stack

62481da

ahobbs7 and others added 12 commits April 3, 2026 19:24

adding comments and formatting

588c725

Ensuring that the h5-jpk copy of the jpk-qi-data doesn't get overwritten

4c7ea88

Improving __iter__ function so loading of all curves for analysis is …

62637c2

…much more memory efficient

Fixing errors with tests caused by logging problems

b9104cc

Updating tests for jpk-qi-data and h5-jpk with curve data

0a1f459

Updating documentation

58a3a0f

Reformatting to match pre-commit conditions and make more robust

c0a2924

Minor formatting changes on tests

e646205

Indices spelling correction

a6388fd

Skipping tests requiring large test files which cannot be added to repo

b88ee26

[pre-commit.ci] Fixing issues with pre-commit

be2840e

Fixing pre-commit problems

a1764fe

ahobbs7 marked this pull request as ready for review May 5, 2026 07:54

ahobbs7 requested a review from SylviaWhittle May 5, 2026 07:54

SylviaWhittle requested changes May 22, 2026

View reviewed changes

ahobbs7 added 3 commits May 22, 2026 18:12

chore: removing print statements

603d7bf

tests: adding tests for get channel functions

43640ac

fix: stopped unnecessary redefining of nested row proxy classes

0146542

SylviaWhittle requested changes May 27, 2026

View reviewed changes

		return frames, pixel_to_nanometre_scaling_factor, header_dict


		def get_asd_channels(file_path: Path):

		return final_multiplier, final_offset, unit


		class jpk_qi_loader:

	# Set path to the .jpk-qi-image file within the archive for later use
	# For holding the reference to where the actual .jqk-qi image is (not the metadata).

Conversation

ahobbs7 commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

SylviaWhittle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ahobbs7 commented May 3, 2026 •

edited

Loading

codecov-commenter commented May 4, 2026 •

edited

Loading