The main and most important issue they need to address is the fact that IDEX packets are stamped with shcoarse at the time of packet creation which is at transmit time. So basically the actual events in the packet do not line up with shcoarse and therefore do not line up with the l0-l2 cdf filenames.
This will cause multiple problems:
- Events do not align with the packet shcoarse and filenames. If they request a retransmit, there will be CDFs named with the date of that transmit but containing events from potentially weeks/months ago.
- This will cause duplicate data. Shcoarse will NOT be duplicated but the data getting transmitted may have overlap. A byproduct problem of this is that events will also get double counted in later l2 products for the rates and counts maps.
The IDEX team has proposed the following plan to sort the events into 10 day files:
To implement this, Tenzin and I came up with the following plan
- At an IDEX l0 file ingest, populate an IDEX specific database with the packet file name, event start date, and event end date
- Trigger batch starter as usual. From the given l0 file and using the event start and end date, determine which l1a 10 day jobs we need to run. For each of those jobs, query the IDEX table for all l0 packets that have overlap and run the job. L1a will filter for events that are not in the correct range.
- l1b -> end works the same.
The main and most important issue they need to address is the fact that IDEX packets are stamped with shcoarse at the time of packet creation which is at transmit time. So basically the actual events in the packet do not line up with shcoarse and therefore do not line up with the l0-l2 cdf filenames.
This will cause multiple problems:
The IDEX team has proposed the following plan to sort the events into 10 day files:
To implement this, Tenzin and I came up with the following plan