feat: add benchmark automation bot [WIP]

[andygrove]

Closes #3556

Summary

I have been testing a version of this code for several weeks and it seems to work fairly well now, so I would like to get the code into OSS for transparency, and allow others to help make improvements.

It is just a first step. The benchmarks do run in k8s in a constrained environment, which is good, but the tests run in Spark local mode. It would be better to deploy as a real cluster in k8s later on.

There is currently an assumption that TPC-H 100GB data already exists on the k8s nodes. It would be better to generate the data using tpchgen-cli directly in the containers. It would also be nice to support different scale factors.

There are likely many other improvements that can be made in the future.

Changes

• Adds a GitHub bot (cometbot) that monitors PR comments for slash commands (/run tpch, /run micro, /help) and automatically runs benchmarks in Kubernetes, posting results back as PR comments
• Includes a Click CLI for manual benchmark runs, Docker image build/push, K8s job management, and deployment tooling
• Adds contributor guide documentation explaining how to trigger benchmarks and how the bot works

Details

The bot lives in dev/benchmarking-bot/ and includes:

• Bot (src/cometbot/bot.py): Polls GitHub for slash commands on open Comet PRs
• K8s (src/cometbot/k8s.py): Builds Docker images, creates/manages Kubernetes jobs
• CLI (src/cometbot/cli.py): Click-based CLI for manual benchmark runs and bot management
• Dockerfile: Container with JDK 17, Rust, Maven, and Spark 3.5 for running benchmarks
• K8s templates (k8s/): Job and deployment manifests
• Deploy script (deploy/deploy.sh): Automated deployment to a remote host via SSH

All configuration is via COMETBOT_* environment variables (registry, GitHub token, deploy host, etc.).

[andygrove]

@Shekharrajak fyi

[milenkovicm]

will try to review it as soon as possible, it might be a bit too big for review.
if it works similarly to

[at]sqlbenchmark tpch --iterations 5 --scale-factor 10

I would suggest one minor change, when bot comments it would be good if it mentions person who's triggered the bench in the comment. In my case i have github app notifications disabled for comments but still have notifications for mentions. would it make sense ?

[andygrove]

will try to review it as soon as possible, it might be a bit too big for review. if it works similarly to

[at]sqlbenchmark tpch --iterations 5 --scale-factor 10

I would suggest one minor change, when bot comments it would be good if it mentions person who's triggered the bench in the comment. In my case i have github app notifications disabled for comments but still have notifications for mentions. would it make sense ?

Thanks. I was thinking that we could create a similar PR in Ballista once we get this through review.

[milenkovicm]

I think that makes sense @andygrove

I guess main comment I have is how to update authorised users, can't really tell from PR and no documentation section about it

[milenkovicm]

Was this repo retired or is going to be?

[milenkovicm]

If the data is pre generated would it make sense to be able to retrieve that info as part of help or some other way?

[milenkovicm]

Mentioning user who trigger benchmark would make sense in my opinion

[milenkovicm]

If the user list is updated, how does the change get propagated to cluster ?

Its a bit big pr to trace it, but I hope pr author can't abuse file and add itself to it and run benchmarks

[milenkovicm]

Should this script be run when list of authorised users is updated?

[milenkovicm]

Maybe cli could be used to propagate authorised users update?