Detect if the time has jumped too far

[opatut]

Apparently some files start counting at unix epoch 0 and already have a timestamp, then jump to actual GPS time once that is available.

Filter out timestamps that are not feasible, e.g.

• No OBS recorded data in 1970. That's probably epoch related. Maybe filter out everything prior to 2000.
• If a jump happens in the date/time file that does not match the internal time counter of the recording device, assume a correction in the system time. Maybe calculate the amount the clock has jumped and retroactively correct the previous timestamps by that difference.

Not sure what the smartest way is to handle this. But we can't trust a 01.01.1970 00:04:23 ;)

[amandel]

The millisecond timer is always linear and not jumping. If you get the correct time anywhere later in the file you can calculate backward.
Even if the UNIX time is set via Date and Time - it should only be used as reference for the Millis value. All time related calculations should be based on the Millis value. Note that lines might appear multiple times if more than 1 pass event had been confirmed during that second.

[opatut]

That's what I meant with internal time counter.

Problem is that the internal counter is not as accurate as the clocks in the GPS sattelites :D So if you have 99.9% of true speed on the internal timer, that is very much accurate enough to compute velocity, measure distances, and whatnot, but if you drive for 2 hours then your skew is already 7.2 seconds. That's bad if you e.g. want to correlate events across two tracks.

I have no idea how accurate the run-time clock is. But I wouldn't bet on too much accuracy, since we're using the cheapest hardware. Probably no external XTAL mounted, so possibly up to 5%?

That's why I thought we believe the measured timestamps, and use Millis to detect jumps, but then shift one of the sides to match the other. Basically we only take the difference in Millis between before and after the jump, which is very low (~1s), so it doesn't matter how much the drift is. Then if before was e.g. network time and after was e.g. GPS time, both are globally referenced and don't experience time drift, even if they are off by some offset. Of course this doens't help if one of the times was just referenced off the internal timer or even the RTC during deep sleep, but ¯\_(ツ)_/¯.

All time related calculations should be based on the Millis value.

Well, we do want absolute dates for some parts, e.g. the minimum timestamp, which is stored as recordedAt in the database (to distinguish it from the upload date, which is createdAt, by convention).

[amandel]

Be aware that a new csv line is not created exactly after 1000 millis because we wait at some points for the tof sensor results and other stuff which might delay the switch to a new line. This will get closer with openbikesensor/OpenBikeSensorFirmware#221 but still there will be differences.
One more internal that might be affecting this. The time in the CSV is the system time of the ESP. It is set once when GPS information is flagged as accurate and then corrected every 4 minutes if there is GPS reception. Since we do not have accuracy in the sub second area there might be jumps up to around 150 millis for the system time at that moment.

[opatut]

Also apparently sometimes for invalid dates 00.00.2000 is used.

Date;Time;Latitude;Longitude;Course;Speed;Right;Left;Confirmed;insidePrivacyArea
00.00.2000;00:00:00;0.000000;0.000000;0.000;0.0000;255;255;0;0

[FlorusCiphersmith]

Incorrect dates (00.00.XXXX)and times should already be marked as invalid. In addition I pushed a change which invalidates all dates before January 1, 2018.

Concerning the jumps in time: usually, when time is off, also coordinates were incorrect. So is it really worth the effort reconstructing invalid timestamps? Let me know if you have a use case.

[biker314]

In my case, coordinates were pretty correct for about 12:45 Minutes before GPS-time was detected. 6 confirmed overtakers were deleted as a consequence, all coordinates of these confirmed events seem pretty accurate. I can provide the complete CSV if necessary.

Date	Time	Millis	Comment	Latitude	Longitude	Altitude	Course	Speed	HDOP	Satellites	BatteryLevel	Left	Right	Confirmed	Marked	Invalid	InsidePrivacyArea	Factor	Measurements	Tms1	Lus1	Rus1	Tms2	Lus2	Rus2	Tms3	Lus3	Rus3	Tms4	Lus4	Rus4	Tms5	Lus5	Rus5	Tms6	Lus6	Rus6	Tms7	Lus7	Rus7	Tms8	Lus8	Rus8	Tms9	Lus9	Rus9	Tms10	Lus10	Rus10	Tms11	Lus11	Rus11	Tms12	Lus12	Rus12	Tms13	Lus13	Rus13	Tms14	Lus14	Rus14	Tms15	Lus15	Rus15	Tms16	Lus16	Rus16	Tms17	Lus17	Rus17	Tms18	Lus18	Rus18	Tms19	Lus19	Rus19	Tms20	Lus20	Rus20	Tms21	Lus21	Rus21	Tms22	Lus22	Rus22	Tms23	Lus23	Rus23	Tms24	Lus24	Rus24	Tms25	Lus25	Rus25	Tms26	Lus26	Rus26	Tms27	Lus27	Rus27	Tms28	Lus28	Rus28	Tms29	Lus29	Rus29	Tms30	Lus30	Rus30	Tms31	Lus31	Rus31	Tms32	Lus32	Rus32	Tms33	Lus33	Rus33	Tms34	Lus34	Rus34	Tms35	Lus35	Rus35	Tms36	Lus36	Rus36	Tms37	Lus37	Rus37	Tms38	Lus38	Rus38	Tms39	Lus39	Rus39	Tms40	Lus40	Rus40	Tms41	Lus41	Rus41	Tms42	Lus42	Rus42	Tms43	Lus43	Rus43	Tms44	Lus44	Rus44	Tms45	Lus45	Rus45	Tms46	Lus46	Rus46	Tms47	Lus47	Rus47	Tms48	Lus48	Rus48	Tms49	Lus49	Rus49	Tms50	Lus50	Rus50	Tms51	Lus51	Rus51	Tms52	Lus52	Rus52	Tms53	Lus53	Rus53	Tms54	Lus54	Rus54	Tms55	Lus55	Rus55	Tms56	Lus56	Rus56	Tms57	Lus57	Rus57	Tms58	Lus58	Rus58	Tms59	Lus59	Rus59	Tms60	Lus60	Rus60
01.01.1970	00:03:48	228019		48.1187692	11.5675412	507.34	40.37590	10.4	1.01	9	3.98	98	73	29		0	0	58	56	0	8320		9		6827	35	8291		44			70	8226		80		6318	105	8187		116		6255	141	8155		151			176	8130		186			220	8087		229			255	8055		265			290	8028		301			326	7998		336			361	7976		371			396	7946		406			432	7898		442			467	7724		477			502	7697		512			541	7717		550			576	7988		587			612	7737		622			647	7670		657			682	7884		692			718	7976		728			753	8040		763			788	8201		798			824	8398		834			859	8669		869			894			913			933			952			971			991
01.01.1970	00:09:21	561032		48.1256545	11.5788828	564.08	331.98525	9.1	1.23	8	3.98	84	77	9		0	0	58	53	0	7962		14		6962	36	8536		50		6949	71	6921		85		6941	106	8684		120		6907	141	2998		155		6858	177			196		6830	212	17812		231		6810	247			267		6795	283			302		6781	318			337		6748	353			373		6770	389			408		7383	424			443			470			490		6548	505			525		6535	541			560		6548	576			595			615			634			662			682			701			721			741			761			781			800			819			839			858			877			897			916			935			955			974
01.01.1970	00:09:23	563069		48.1256982	11.5788405	564.40	323.87735	9.9	1.23	8	3.98	188		27		0	0	58	51	0			22			42			61			80			100			119			138			158			177			196			216			235			254			274			293			312			332			351			370			389			409			428			447			467	12939		483			502	12932		525			544	13022		560			579	13104		595			615	13395		631			650	13302		666			685	13373		701			722	13310		737			757	13426		772			793	13545		807			828			847			867	14229		883			902			921			941
01.01.1970	00:10:54	654041		48.1281623	11.5760421	529.00	319.15321	17.8	0.89	10	3.97	146		25		0	0	58	52	0			20			39			58			78			97			116			136			155			174			194			213			232	14239		248			268	10996		288			308			327			347			366			385	16206		402			422	10542		437			457	5946		473			502			521			540	14560		556			576			595			614			634			653			672			692	16856		709			729	10766		744			765			784			804			823			842			862			881	11741		897			916	13483		932			952			971
01.01.1970	00:11:01	661033	DEV: Mem: 233k Buffer: 2k last write time: 41	48.1283926	11.5757852	531.84	322.04536	14.8	1.01	9	3.97	185		33		0	0	58	51	0			20			39			58			78			97			116	12885		132			151			171			190	13370		206			235			255			274			293			313	12980		329			348	12840		364			391	12839		404			429	12903		443			464			484			503	12905		519			538	12837		554			573	12783		589			609	12796		625			649	12764		663			685	12886		699			720	12764		734			755	12824		769			791	12772		804			826	12787		840			861	12753		875			896	12857		910			932	12765
01.01.1970	00:11:24	684024		48.1290756	11.5746277	551.10	313.29643	17.2	1.04	8	3.97	67		53		0	0	58	55	0	9916		11			35	9870		46			71	9843		81			106	9824		117			141	9796		152			177	9789		187			212	9763		223			247	9751		258			283	9717		293			318	9732		329			353	9741		364			389	9730		399			424	9747		435			459	9791		470			495	9802		505			530	9862		540			565	9910		576			600	9950		611			636	9950		646			671	8574		682			706	9483		717			746	10302		757			781	10329		792			816	10294		827			852	10433		863			887	4898		898			922	5941		933			962	17601
14.06.2021	18:26:33	792088		48.1322782	11.5717896	564.20	42.77491	16.2	1.01	9	3.97	128	134	27		0	0	58	51	0			20			40			59			78	10608		94			113	13687		137			157			176			195			215		10082	231			250			269	11320		285		10028	305	9707		321		9993	350	9632		361		9991	386	9537		396		9924	421	9514		431		9894	459	9494		469		9838	494	9481		504		9846	530			549		9846	565	9549		585		9802	600	9521		620			639	9512		655			683	9500		695			719	9505		730			754	9542		765			789	9511		801			825	9515		836			860	9486		871			895	9679		906			931

My suggestion would be to add date/time reconstruction as an option for the annotation step. We could mark this corrected data in the comment-column (and overwrite the other data, but this only affects annotated files). An additional (obs-firmware related) idea would be to correct the GPS-time-info more often, if the OBS is capable of doing so.

[FlorusCiphersmith]

That's indeed a case where we could try to reconstruct Date/Time. Thank you, @biker314

However, I'm still puzzled about the data we see here:

My understanding of GPS was that determining the data/time is a much easier task than determining position etc.. Thus, I'm surprised to see that for the fields Latitude, Longitude, Altitude, Course, Speed, reasonable values are provided at any time. Also there are at least 8 satellites and HDOP is at a good level while the time is obviously off.

@amandel Is it possible that writing time and date goes wrong in the OBS?

[amandel]

We also have a discussion on this in slack. The information how many leap seconds need to be considered to calculated the correct UTC based on the GPS time needs some minutes. The GPS time is as you correctly state available even before the fix. My proposal is to write GPS time from the OBS device to the csv and let others take care of the leap seconds if needed. Please comment at https://openbikesensor.slack.com/archives/CRTNL8RQF/p1623613950039900 with your opinion.

[amandel]

Also apparently sometimes for invalid dates 00.00.2000 is used.

Date;Time;Latitude;Longitude;Course;Speed;Right;Left;Confirmed;insidePrivacyArea
00.00.2000;00:00:00;0.000000;0.000000;0.000;0.0000;255;255;0;0

@opatut , can you check if this was a development version? Or which OBS device version this was?

[opatut]

What I can recover about this:

• I've found 101 track files by 7 users in the old database which contain the string "00.00.2000"
• They are definitely all OBSCSV1 format.
• Only 16 of those tracks end with a line that has a different date than "00.00.2000", so 85 never "recover" from this issue.
• Those that do recover were recorded between August and November 2020. The others I don't know, because they don't have a proper timestamp ;)
• The 101 files have varying sizes between 260 bytes and 1.2MB.