Duplicate entries in Best track

[SorooshMani-NOAA]

Some of the storms have adjusted best tracks (e.g. Ian 2020, Dorian 2019). These adjusted tracks have duplicate for the adjusted entries. After discussion with @FariborzDaneshvar-NOAA and @WPringle one solution is to get the last entry from the file based on duplicate detection from the timetag and isotach.

This might need to be addressed in PaHM by @pvelissariou1 as well, since the BEST track downloaded from the NHC itself has this duplication.

[SorooshMani-NOAA]

@pvelissariou1 I just realized the "duplicate entries" in the best track is actually not duplicates. Some entries in BEST track have minute field (TECHNUM/MIN) and the actual full time tags don't collide, e.g.:

AL, 05, 2019090612,   , BEST,   0, 351N,  757W,  85,  956, HU,  34, NEQ,  190,  190,  150,   80, 1014,  300,  25, 100,   0,   L,   0,    ,  44,   7,     DORIAN,,,,,,,,
AL, 05, 2019090612,   , BEST,   0, 351N,  757W,  85,  956, HU,  50, NEQ,  110,  120,   80,   70, 1014,  300,  25, 100,   0,   L,   0,    ,  44,   7,     DORIAN,,,,,,,,
AL, 05, 2019090612,   , BEST,   0, 351N,  757W,  85,  956, HU,  64, NEQ,   50,   60,   50,   40, 1014,  300,  25, 100,   0,   L,   0,    ,  44,   7,     DORIAN,,,,,,,,
AL, 05, 2019090612, 30, BEST,   0, 352N,  756W,  85,  956, HU,  34, NEQ,  190,  190,  150,   80, 1014,  300,  25,   0,   0,    ,   0,    ,  39,   8,     DORIAN,,,,,,,,
AL, 05, 2019090612, 30, BEST,   0, 352N,  756W,  85,  956, HU,  50, NEQ,  110,  120,   80,   70, 1014,  300,  25,   0,   0,    ,   0,    ,  39,   8,     DORIAN,,,,,,,,
AL, 05, 2019090612, 30, BEST,   0, 352N,  756W,  85,  956, HU,  64, NEQ,   50,   60,   50,   40, 1014,  300,  25,   0,   0,    ,   0,    ,  39,   8,     DORIAN,,,,,,,,

Ideally this should be handled by PaHM. I can filter it in the client script, but maybe it doesn't make sense to automatically filter it in stormevents

What do you think @WPringle?

[WPringle]

@SorooshMani-NOAA I realized that it was different actually in terms of adding minutes but it seems like a few of the fields are missing but maybe they are not important ones? If could be handled by PaHM it does make sense, just need make sure the fields are all filled in on StormEvents side.

[SorooshMani-NOAA]

@WPringle it seems that among the two set of 3 isotach lines above, the missing info is the subregion code (L) and the gust speed. I don't think they matter for PaHM. Am I right @pvelissariou1?

Before PaHM can support these minute-different rows, I can drop the 0 minute entries when their day/hour field is duplicated like above and keep the corresponding 30 min entry as we discussed, and for now don't change anything related to the missing fields.

[pvelissariou1]

Yes, Soroosh that is right. I am keeping track of the issues, I'll be working on updating/fixing the issues in PaHM. I'll keep you posted.

[SorooshMani-NOAA]

This issue was being discussed in #81. I'll close the older ticket, but refer to it for prior discussions on normalization