Compatibility with latest timely+differential

[brabeji]

Trying to use this tool as-is with timely 0.24.0 and differential 0.17.0, my program never connects to the diagnostics process. Assuming this can be related to version incompatibility with newer timely versions, I've tried to update it in this project. There are some incompatibilities and I don't have enough experience to pull it off reasonably :(

Is this project still a go-to solution for perf profiling? Should I look elsewhere or try some more to fix this? Thanks!

[frankmcsherry]

I think the repo hasn't been maintained for quite a while. I'm not current on it, but happy to take responsibility for the lack of good observability tools. Potentially the right thing to do (assuming no one leaps in and wants to speak for the repo) is collect some of the requirements you have, features the repo has, and see if there is a better supportable way to provide them. For example, if nothing else folding a solution in to timely's CI.

Timely's logging shouldn't have had any regressions (though, it's possible there have been breaking changes in types), but it's possible that e.g. the logic that automatically listens on some port has been removed (at least, I hope it has but don't remember if/when that happened). Potentially the right thing to do is ship timely with an easy way to prep such a connection for remote log collection, rather than do it by default.

Let me try and page some of this in over the next few days, but also if you have the ability to unpack what you need for us, that would be great. Roughly, it's relatively easy to capture the logged event streams from any timely instance and provide that as data for another timely instance, and then harder to figure out what you want to do with them (and I can imagine that the repo here has some pre-baked thoughts that are useful, and we'd have to hose off and scratch our heads about).

[frankmcsherry]

Yup, there is at least a mismatch between the current docs for this repo, which mention setting TIMELY_WORKER_LOG_ADDR, and the timely repo which has no mention of such a variable. I believe this was the setting that prompted a worker to establish a socket on which it would transmit logged events.

This is something that can be done in user code, and was previously done in the paired examples/{logging-send.rs, logging-recv.rs}, but looking at the examples their "pairing" has been broken; the latter still uses a network connection for events, but the former doesn't provide them that way (and just logs them to stdout instead).

I'll take a peek at trying to sort this out, first by bringing the examples closer to in sync, which would better reveal how one can open some sockets and instruct the logging infrastructure to exfiltrate the logged events. Separate Q about is that the best architecture for all of this, or would any capture/replay mechanism work (including an in-process version, that wouldn't risk version skew).

[brabeji]

Finally getting around to respond. I have a quite simple program where the execution time grows faster than input size so I just wanted to give this tool a shot, and see what I can get.

No specific requirements from me at this time but I guess #3 and #18 might be valid? I've yet to see the diagnostic output so not sure. I think just updating to latest timely+dd and fixing the bug you've mentioned to bring this to a functioning state is all I can ask for rn. I'll make sure to post any new feature I might be able to add myself, if that comes up! Just started using Rust so some experience is still missing.

[brabeji]

Anyways, found the source of the slowdown! Will post in timely to ask about it.