ISTP Validation Errors on Logical Source

[vineetbansal]

I'm opening an issue that may serve as the discussion-point for the ISTP validation errors that @davidt0x observed. I'll let him chip in, but at a high-level the problem seems to be that around 1/3 of the cdfs are failing with errors of the form:

Logical_file_id should be 'imap_swe_l2_sci_20260427_v001'.  It is 'imap_swe_l2_sci_20260428_v001'.

The issue seems to be that the first epoch has to fall in the date pattern specified by the logical source.

To reproduce one such (arbitrarily chosen) error:

To install the required tools on a *nix machine:

wget https://spdf.gsfc.nasa.gov/skteditor/skteditor_1.3.12-1_amd64.deb
dpkg-deb -x skteditor_1.3.12-1_amd64.deb skteditor

To run and observe the error:

imap-data-access download imap_swe_l2_sci_20260428_v001.cdf

java -Djava.library.path="./skteditor/opt/skteditor/lib/app" -cp "./skteditor/opt/skteditor/lib/app/spdfjavaClasses.jar:./skteditor/opt/skteditor/lib/app/cdfjava.jar" gsfc.spdf.istp.tools.CDFCheck $IMAP_DATA_DIR/imap/swe/l2/2026/04/imap_swe_l2_sci_20260428_v001.cdf

If we run a script (below) to determine the first timestamp in the epoch for this l2 file, we get:

l2

uv run --with cdflib --with spacepy python check_epoch_dates.py $IMAP_DATA_DIR/imap/swe/l2/2026/04/imap_swe_l2_sci_20260428_v001.cdf 

...
  First: 2026-04-27T23:59:46.761652352
  Last:  2026-04-28T23:59:59.976963456

Tracing back from l2 -> l1b -> l1a, the progression of starting epoch seems to be:

l1b

uv run --with cdflib --with spacepy python check_epoch_dates.py $IMAP_DATA_DIR/imap/swe/l2/2026/04/imap_swe_l1b_sci_20260428_v001.cdf
  First: 2026-04-27T23:59:46.761652352
  Last:  2026-04-28T23:59:59.976963456

l1a (I couldn't find a v001 but I think it shouldn't matter?)

uv run --with cdflib --with spacepy python check_epoch_dates.py $IMAP_DATA_DIR/imap/swe/l2/2026/04/imap_swe_l1a_sci_20260428_v002.cdf
  First: 2026-04-27T23:59:00.128774784
  Last:  2026-04-29T00:00:46.053757440

l0

python check_packet_time.py $IMAP_DATA_DIR/imap/swe/l0/2026/04/imap_swe_l0_raw_20260428_v001.pkts
...
First packet UTC: 2026-04-27T23:59:20.375214+00:00

The scripts themselves (that AI wrote), are:

# check_epoch_dates.py
import sys
import cdflib
import spacepy.pycdf

CDF_FILE = sys.argv[1] 
with cdflib.CDF(CDF_FILE) as cdf:
    info = cdf.cdf_info()
    print(f"Logical_file_id: {cdf.globalattsget().get('Logical_file_id', ['<not set>'])[0]}")

    epoch_vars = [v for v in info.zVariables if "epoch" in v.lower()]
    print(f"Epoch variables: {epoch_vars}\n")

    for var in epoch_vars:
        raw = cdf.varget(var)
        if raw is None or len(raw) == 0:
            print(f"{var}: no data")
            continue

        times = cdflib.cdfepoch.to_datetime(raw)
        print(f"  First: {times[0]}")
        print(f"  Last:  {times[-1]}")

# check_packet_time.py
import sys
import struct
import datetime

IMAP_EPOCH = datetime.datetime(2010, 1, 1, tzinfo=datetime.timezone.utc)

PKT_FILE = sys.argv[1]

with open(PKT_FILE, "rb") as f:
    data = f.read(12)

apid = ((data[0] & 0x07) << 8) | data[1]
coarse = struct.unpack(">I", data[6:10])[0]
fine = struct.unpack(">H", data[10:12])[0]

t = IMAP_EPOCH + datetime.timedelta(seconds=coarse, microseconds=fine / 65536 * 1e6)

print(f"APID:        {apid} (0x{apid:03x})")
print(f"Coarse MET:  {coarse} s")
print(f"Fine:        {fine} ({fine/65536:.6f} s)")
print(f"First packet UTC: {t.isoformat()}")
print(f"UTC date:    {t.date()}")

[mmshaw]

@davidt0x we need more detail here. Which instruments are affected? @tech3371 already identified HIT, MAG, SWAPI, and SWE. But I thought there were others...

[mmshaw]

One related issue is #3062 per @jtniehof

[davidt0x]

Hi @mmshaw,

Sorry for the delayed response on this, I found this last week and spoke with Vineet and Margaret about it. I got sick on Sunday though and was out until today. Thanks @vineetbansal for starting the issue.

I split the Logical_file_id failures into a few separate classes so we are not lumping unrelated problems together.

For the specific issue described here, where SPDF expects the Logical_file_id date to be one day earlier than the file’s actual Logical_file_id date, I see 973 affected files in the latest L2 data. I have scanned all 3912 L2 files, only latest versions.

This class is defined as:

• expected and actual IDs have the same product stem
• expected and actual IDs have the same version
• no repoint suffix difference
• expected date is exactly one day before the actual date

Affected files by instrument:

Instrument	Affected files
CoDICE	612
SWAPI	170
SWE	140
HIT	51
Total	973

Breakdown by product/descriptor:

Instrument	Descriptor	Affected files
CoDICE	hi-omni	145
CoDICE	hi-sectored	89
CoDICE	lo-nsw-angular	87
CoDICE	lo-sw-angular	85
CoDICE	lo-nsw-species	84
CoDICE	lo-sw-species	82
HIT	macropixel-intensity	51
CoDICE	hi-direct-events	20
CoDICE	lo-direct-events	20
SWAPI	sci	170
SWE	sci	140

I do not currently see MAG in this previous-day Logical_file_id class.

There are two other Logical_file_id classes I have found, which I think should be tracked separately from this issue:

Other class	Affected files	Instruments
Repoint suffix mismatch: SPDF expects the -repointNNNNN suffix to be omitted	224	GLOWS, HI
Aggregate/map epoch-date mismatch: product date differs from SPDF’s epoch-derived date by something other than the one-day boundary case	100	GLOWS, HI, IDEX, LO, Ultra

So for this issue, I would scope the root-cause discussion to CoDICE, HIT, SWAPI, and SWE daily products where the first epoch falls on the previous UTC day.

It seems to me that each of these issues will need a different fix. The fix for this previous day issue seems to be stricter windowing of the data since it seems the filename is largely correct. The others I think make more sense to just change the filenames? There is also the possibility of just writing the expected logical_file_id regardless of the filename. The checker doesn't seem to check if they actually match, but that is probably against the spec from what I can tell.

Lastly, I have attached a vibe coded static HTML dashboard to collate and organize all these various ISTP compliance issues if anyone else finds it useful. If you want to see these Logical_file_id issues you can type that in the search bar and click on each one to get a more detailed breakdown.

imap-cdf-spdf-validation-dashboard-2026-05-08.html

[tech3371]

Thank you Vineet and David for these summary!

This list and finding lines up with instruments who had buffer added to their daily files, [SWAPI, SWE, HIT, MAG]. Looks like MAG is filtering their data out correctly in this case.

CoDICE was a unique case where their data acquisition time was beyond day boundary while their packet creation time is within day boundary. But CoDICE team does want to add buffer to their L0 data now. So they will be added to above list. Thank you for finding these details. It makes sense!