Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions Readme.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# netchdf
_last updated: 7/25/2025_
_last updated: 7/26/2025_

This is a rewrite in Kotlin of parts of the devcdm and netcdf-java libraries.

Expand Down Expand Up @@ -133,15 +133,15 @@
are about 2X slower than native code. Unless the deflate libraries get better, there's not much gain in trying to make
other parts of the code faster.

We are seeing 10x speedup on data reading. see https://github.com/JohnLCaron/netchdf/issues/189.
We are seeing 10x speedup on data reading using coroutines for chunked data. see https://github.com/JohnLCaron/netchdf/issues/189.


### Goals and scope

Our goal is to give read access to all the content in NetCDF, HDF5, HDF4, and HDF-EOS files.

The library will be thread-safe for reading multiple files concurrently. We are also exploring concurrent reading within
the same file.
The library will be thread-safe for reading multiple files concurrently. We also have implemented concurrent reading within
the same file for HDF5 chunked data.

We are focussing on earth science data, and don't plan to support other uses except as a byproduct.

Expand Down Expand Up @@ -171,7 +171,8 @@
1. Unit testing that doesn't require reading files.
2. Testing with files in core/commonTest/data. These are fast and are run in a Github Action.
3. Testing with files in TestFiles.testData in module testfiles. These are medium fast (< 11 min wallclock).
4. Testing with files in TestFiles.testData in module testclibs. These are slow.
4. Testing with files in TestFiles.testData in module testclibs. These are slow, mainly because the tests
are run serially because the C libraries are not thread safe.

Currently we have 1500+ test files in the core and testdata modules:

Expand Down Expand Up @@ -229,8 +230,8 @@

Currently using
* HDF5 library version: 1.14.6.
* netcdf-c library version 4.10.0-development of May 23 2025
* HDF-4 library version: ???
* Netcdf-c library version: 4.10.0-development of May 23 2025
* HDF-4 library version: HDF Version 4.2 Release 17-1, March 8, 2023

In order to run, you must install the C libraries on your computer and ad them to the LD_LIBRARY_PATH.

Expand Down Expand Up @@ -262,7 +263,7 @@
return data as ArrayUByte.
* Netcdf-4 encodes CHAR values as HDF5 string type with elemSize = 1, so we use that convention to detect
legacy CHAR variables in HDF5 format. (NC_CHAR should not be used in new Netcdf-4 files, use NC_UBYTE or NC_STRING.)
Variables of type CHAR return data as STRING, since users can use UBYTE if thats what they intend.

Check failure on line 266 in Readme.md

View workflow job for this annotation

GitHub Actions / Check for spelling errors

thats ==> that's
* Netcdf-4/HDF5 String variables may be fixed or variable length. For fixed Strings, we set the size of Datatype.STRING to
the fixed size. For both fixed and variable length Strings, the string will be truncated at the first zero byte, if any.
* HDF4 does not have a STRING type, but does have signed and unsigned CHAR, and signed and unsigned BYTE.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,12 @@ import com.sunya.netchdf.hdf4.H4builder.Companion.tagid
import com.sunya.netchdf.hdf4.H4builder.Companion.tagidName
import com.sunya.netchdf.hdf4.H4builder.Companion.tagidNameR
import com.sunya.netchdf.mfhdfClib.ffm.mfhdf_h.*
import io.github.oshai.kotlinlogging.KotlinLogging
import java.io.IOException
import java.lang.foreign.*
import java.lang.foreign.ValueLayout.JAVA_BYTE
import java.util.*

private val showLibraryVersion = false
private val debugVGroup = false
private val debugVGroupDetails = false
private val debugVSdata = false
Expand Down Expand Up @@ -54,6 +55,29 @@ class HCheader(val filename: String) {
this.read_access_mode = session.allocateUtf8String("r")
val rootBuilder = build(session)
this.rootGroup = rootBuilder.build(null)
checkLibraryVersion(session)
}
}

fun checkLibraryVersion(session: Arena) {
val major_p = session.allocate(C_INT, 0)
val minor_p = session.allocate(C_INT, 0)
val release_p = session.allocate(C_INT, 0)
val info_p: MemorySegment = session.allocate(MAX_NAME) // char string[] ??

// Hgetlibversion(uint32 *majorv, uint32 *minorv, uint32 *release, char string[])
Hgetlibversion(major_p, minor_p, release_p, info_p)

val major = major_p[C_INT, 0]
val minor = minor_p[C_INT, 0]
val release = release_p[C_INT, 0]
val info = info_p.getUtf8String(0)

if (showLibraryVersion) println("Hgetlibversion major = $major minor = $minor release = $release info = $info")

val fromHfile = "HDF Version 4.2 Release 17-1, March 8, 2023" // from testclibs/src/main/java/com/sunya/netchdf/mfhdfClib/include/hfile.h
if (info != fromHfile) {
logger.warn{"HDF4 library version mismatch. Expected '$fromHfile' but got '$info'" }
}
}

Expand Down Expand Up @@ -1012,6 +1036,10 @@ class HCheader(val filename: String) {
gb.addAttributeIfNotExists(attr)
}
}

companion object {
val logger = KotlinLogging.logger("HCheader")
}
}

///////////////////////////////////////////////////////////////////////////////////////////
Expand Down