fix: Convert to schedule timezone on activity retry

[tomasfarias]

Problem

We have a function adjust_bound_datetime_to_schedule_time_zone intended to resolve the problem of a backfill input not matching a schedule's timezone. In this case, we always prefer the schedule's timezone, assuming that as the source of truth.

Consider the way we resolve inputs for daily backfills: We simply replace the timezone for the schedule's timezone. This is fine for when the backfill is just starting: Since backfills are always at midnight, and we assume the input is midnight in some timezone, we can just replace that for the schedule's timezone and move on. For example an input of "2025-01-01T00:00:00+00:00" for a schedule in CET timezone would become "2025-01-01T00:00:00+01:00".

Continuing, later in the code we convert this input back into UTC. Continuing with the example, "2025-01-01T00:00:00+01:00" becomes "2024-12-31T23:00:00+00:00". Since these times are equivalent, so far so good.

The problem comes when the backfill activity fails: We send the converted timestamp as temporal heartbeat details and use that to recover from a failure. In the retry, we once again run adjust_bound_datetime_to_schedule_time_zone to adjust the timezone, but this timestamp was already converted UTC, so if the schedule is not in UTC, we will end up "double correcting": The "2024-12-31T23:00:00+00:00" in the details is now replaced to "2024-12-31T23:00:00+01:00" which is then converted to UTC and drifted back one hour to "2024-12-31T22:00:00+01:00".

Changes

When receiving a timestamp from activity details, cast it to the schedule's timezone, thus undoing the UTC cast, and letting everything proceed as it should.

How did you test this code?

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Changelog: (features only) Is this feature complete?

[greptile-apps]

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[rossgray]

Sounds like this should work. Were you able to reproduce and test that it fixes things locally? If not, maybe we can test in prod once this is live just to confirm it does fix the issue

[tomasfarias]

@rossgray
I unfortunately wasn't able to reproduce a failure of the backfill activity