ENH - Filter GLOWS L1A Histogram data points which are empty.

[maxinelasp]

Summary

Detailed description in file. Basically, there are extra data points in the L1A Histogram file that need to be filtered. Any data points with no data should be removed.

validation_investigation_summary.md

Requirements

The more specific the better.

• Remove data points with no data in L1A histograms

Related tickets

No response

[tech3371]

What needs to be done for this ticket?
In l1a code file, we need to filter python object that has no data in it. all those object contains data per packet.

Why are some of those empty? we don't know yet. may be because of validation data. Validation data have timestamp that doesn't have actual data.

Did test data have those empty timestamp? yes but their code that produces validation data drops those and ours keeps them since it's not expected in actual pipeline.

[tech3371]

look at this line of code - https://github.com/IMAP-Science-Operations-Center/imap_processing/blob/dev/imap_processing/glows/l1a/glows_l1a.py#L309. Filter at first line of this function.

[tech3371]

right now, we validate through script but could add function to test filtering empty object with timestamp in this function test_generate_histogram_dataset.

[tech3371]

I ran copilot code locally and validate with GLOWS validation data and we match. Here is output from validation script provided by Maxine.

(imap-processing-py3.12) tech3371@MacL4670 imap_processing % python validate_l1a_histograms.py --validation-dir /Users/tech3371/Documents/IMAP/glows/data_products_cbk_implementation_2025-07-25_validation_set_1/data_l1a_histograms/ --cdf-dir /Users/tech3371/git/imap_processing/data/imap/glows/l1a/2026/04/ --version v002 --repoint repoint00005
Loading validation data from /Users/tech3371/Documents/IMAP/glows/data_products_cbk_implementation_2025-07-25_validation_set_1/data_l1a_histograms/...
Found 1 repoints in validation data

Loading CDF data from /Users/tech3371/git/imap_processing/data/imap/glows/l1a/2026/04/...
Found 1 repoints in CDF data

Comparing 1 repoints...
  Processing repoint00005...

Generating report to validation_report_l1a_histograms.txt...
======================================================================
=== GLOWS L1A Histogram Validation Report ===
======================================================================
Date: 2026-02-24 13:51:20

SUMMARY:
  Total repoints processed: 1
  Passed: 1
  Failed: 0

----------------------------------------------------------------------
REPOINT00005: PASS
  Validation records: 705
  CDF records: 705
  Matched: 705
  Mismatched: 0
  All records match exactly.

======================================================================
=== OVERALL RESULT: PASS ===
======================================================================
(imap-processing-py3.12) tech3371@MacL4670 imap_processing % python validate_l1a_histograms.py --validation-dir /Users/tech3371/Documents/IMAP/glows/data_products_cbk_implementation_2025-07-25_validation_set_1/data_l1a_histograms/ --cdf-dir /Users/tech3371/git/imap_processing/data/imap/glows/l1a/2026/04/ --version v002 --repoint repoint00004
Loading validation data from /Users/tech3371/Documents/IMAP/glows/data_products_cbk_implementation_2025-07-25_validation_set_1/data_l1a_histograms/...
Found 1 repoints in validation data

Loading CDF data from /Users/tech3371/git/imap_processing/data/imap/glows/l1a/2026/04/...
Found 1 repoints in CDF data

Comparing 1 repoints...
  Processing repoint00004...

Generating report to validation_report_l1a_histograms.txt...
======================================================================
=== GLOWS L1A Histogram Validation Report ===
======================================================================
Date: 2026-02-24 13:51:34

SUMMARY:
  Total repoints processed: 1
  Passed: 1
  Failed: 0

----------------------------------------------------------------------
REPOINT00004: PASS
  Validation records: 725
  CDF records: 725
  Matched: 725
  Mismatched: 0
  All records match exactly.

======================================================================
=== OVERALL RESULT: PASS ===
======================================================================
(imap-processing-py3.12) tech3371@MacL4670 imap_processing % python validate_l1a_histograms.py --validation-dir /Users/tech3371/Documents/IMAP/glows/data_products_cbk_implementation_2025-07-25_validation_set_1/data_l1a_histograms/ --cdf-dir /Users/tech3371/git/imap_processing/data/imap/glows/l1a/2026/04/ --version v002 --repoint repoint00002
Loading validation data from /Users/tech3371/Documents/IMAP/glows/data_products_cbk_implementation_2025-07-25_validation_set_1/data_l1a_histograms/...
Found 1 repoints in validation data

Loading CDF data from /Users/tech3371/git/imap_processing/data/imap/glows/l1a/2026/04/...
Found 1 repoints in CDF data

Comparing 1 repoints...
  Processing repoint00002...

Generating report to validation_report_l1a_histograms.txt...
======================================================================
=== GLOWS L1A Histogram Validation Report ===
======================================================================
Date: 2026-02-24 13:51:48

SUMMARY:
  Total repoints processed: 1
  Passed: 1
  Failed: 0

----------------------------------------------------------------------
REPOINT00002: PASS
  Validation records: 360
  CDF records: 360
  Matched: 360
  Mismatched: 0
  All records match exactly.

======================================================================
=== OVERALL RESULT: PASS ===
======================================================================