extensions: authenticate allocation endpoint with requestheader client CA by adilburaksen · Pull Request #4582 · agones-dev/agones

adilburaksen · 2026-05-23T00:40:04Z

Summary

This PR fixes an authorization bypass in Agones' extension API server where any in-cluster workload could call the allocation endpoint directly, bypassing Kubernetes RBAC entirely.

Problem

The Agones extension API server (port 8082) serves /apis/allocation.agones.dev/v1/namespaces/…/gameserverallocations via its own HTTP mux. When a client calls this endpoint through the Kubernetes aggregation layer, kube-apiserver proxies the request and includes a requestheader client certificate signed by the requestheader CA (from kube-system/extension-apiserver-authentication).

Previously, the server did not verify that certificate. Any pod with network access to port 8082 could send a raw HTTP request and receive a valid response — no Kubernetes ServiceAccount token, no RBAC check, nothing.

Fix

Three files changed:

pkg/util/https/server.go

Set ClientAuth: tls.RequestClientCert on the TLS config.
This asks callers to present a certificate but does not require one, so webhook callers that do not present a cert are unaffected.
The cert becomes available in r.TLS.PeerCertificates for the auth middleware.

pkg/util/apiserver/apiserver.go

Added SetRequestHeaderCA(*x509.CertPool) — called at startup to supply the CA.
Added authenticatedHandler — wraps any ErrorHandlerFunc to verify the first peer certificate against the CA pool with ExtKeyUsageClientAuth. Mirrors the logic in k8s.io/apiserver/pkg/authentication/request/x509.
If no CA is configured (unit tests, local dev), the handler passes through unchanged.
Discovery (/apis/allocation.agones.dev/v1) and OpenAPI handlers are left unauthenticated — they carry no sensitive data and are called by tools that do not present certs.
Only the /apis/…/namespaces/ resource handler (i.e. the allocation endpoint) requires the cert.

cmd/extensions/main.go

Added loadRequestHeaderCA — reads kube-system/extension-apiserver-authentication ConfigMap, parses requestheader-client-ca-file PEM, returns an *x509.CertPool.
Wired in after NewAPIServer; if the ConfigMap is absent or empty (e.g. clusters that don't use the aggregation layer), the server starts with a warning and no auth enforcement, preserving backward compatibility.

Testing

Existing unit tests pass (go test ./pkg/util/apiserver/...). The authenticatedHandler nil-CA path ensures existing tests continue to work without TLS setup.

Integration / e2e: the TLS RequestClientCert flag is transparent to callers that don't present a cert (webhook callers), and kube-apiserver always presents the requestheader cert when proxying aggregation-layer requests.

References

Kubernetes requestheader authentication: https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/
Upstream x509 authenticator: https://github.com/kubernetes/apiserver/blob/master/pkg/authentication/request/x509/x509.go#L138
extension-apiserver-authentication ConfigMap: https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/#contacting-the-extension-apiserver

markmandel · 2026-05-23T04:11:22Z

/gcbrun

agones-bot · 2026-05-23T05:04:50Z

Build Failed 😭

Build Id: c818caef-47eb-44ed-90a9-b5a07b23a94f

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

adilburaksen · 2026-05-23T18:40:26Z

These failures appear to be pre-existing infrastructure issues unrelated to this PR's changes:

Step #26 (upgrade-test): The build fails with code=404, Not found: projects/agones-images/locations/us-east1/clusters/standard-upgrade-test-cluster-1-31. The GKE cluster does not exist — this is an infrastructure issue.

Step #28 (e2e-test): Of the four clusters tested, three pass and one fails:

generic-1.33: SUCCESS
gke-autopilot-1.34: SUCCESS
generic-1.35: SUCCESS
gke-autopilot-1.35: FAILURE

Since generic-1.35 and gke-autopilot-1.35 run the same Kubernetes version and our change (adding RequestClientCert to the TLS config and the authenticatedHandler middleware) would affect all clusters equally, the gke-autopilot-1.35-specific failure looks like a pre-existing flaky test or autopilot-specific infrastructure issue rather than a regression introduced here.

Happy to re-trigger (/gcbrun) if that would be helpful to confirm.

markmandel · 2026-05-23T19:43:25Z

/gcbrun

It's a flake we see on autopilot sometimes.

VERBOSE: time="2026-05-23 05:04:19.565" level=info msg="2026/05/23 05:04:18 Starting TCP server, listening on port 7654" options="&PodLogOptions{Container:game-server,Follow:false,Previous:false,SinceSeconds:nil,SinceTime:<nil>,Timestamps:false,TailLines:nil,LimitBytes:nil,InsecureSkipTLSVerifyBackend:false,Stream:nil,}" test=TestGameServerTcpProtocol
VERBOSE: time="2026-05-23 05:04:19.565" level=info msg="2026/05/23 05:04:18 Marking this server as ready" options="&PodLogOptions{Container:game-server,Follow:false,Previous:false,SinceSeconds:nil,SinceTime:<nil>,Timestamps:false,TailLines:nil,LimitBytes:nil,InsecureSkipTLSVerifyBackend:false,Stream:nil,}" test=TestGameServerTcpProtocol
VERBOSE: time="2026-05-23 05:04:19.565" level=info msg="---End of container logs---" options="&PodLogOptions{Container:game-server,Follow:false,Previous:false,SinceSeconds:nil,SinceTime:<nil>,Timestamps:false,TailLines:nil,LimitBytes:nil,InsecureSkipTLSVerifyBackend:false,Stream:nil,}" test=TestGameServerTcpProtocol
VERBOSE: time="2026-05-23 05:04:19.623" level=warning msg="Error opening log stream for container" error="previous terminated container \"game-server\" in pod \"game-serverxzq28\" not found" options="&PodLogOptions{Container:game-server,Follow:false,Previous:true,SinceSeconds:nil,SinceTime:<nil>,Timestamps:false,TailLines:nil,LimitBytes:nil,InsecureSkipTLSVerifyBackend:false,Stream:nil,}" test=TestGameServerTcpProtocol
VERBOSE:     gameserver_test.go:1068: 
VERBOSE:         	Error Trace:	/go/src/agones.dev/agones/test/e2e/gameserver_test.go:1068
VERBOSE:         	Error:      	Received unexpected error:
VERBOSE:         	            	dial tcp 35.227.71.13:7855: connect: connection refused
VERBOSE:         	Test:       	TestGameServerTcpProtocol
VERBOSE: --- FAIL: TestGameServerTcpProtocol (2.64s)

Tempted to add a retry in there for CI, but it shouldn't be needed -- but I digress.

markmandel · 2026-05-23T19:43:40Z

Also noting, you will need to sign your commits per DCO.

…t CA The extensions HTTPS server exposes the allocation resource handler (/apis/allocation.agones.dev/v1/namespaces/...) to any in-cluster workload that can reach agones-controller-service:443 directly, bypassing Kubernetes RBAC for the aggregated resource. Fix using the standard Kubernetes aggregation-layer authentication protocol (https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/): * pkg/util/https/server.go — add ClientAuth: RequestClientCert so the TLS handshake forwards the client certificate (from the kube-apiserver proxy) into r.TLS.PeerCertificates without requiring callers that do not present a cert (e.g. webhooks) to do so. * pkg/util/apiserver/apiserver.go — add SetRequestHeaderCA(*x509.CertPool) and an authenticatedHandler wrapper. The wrapper verifies the leaf certificate in r.TLS.PeerCertificates against the CA pool with ExtKeyUsageClientAuth, mirroring the approach in k8s.io/apiserver/pkg/authentication/request/x509. Discovery and OpenAPI handlers are left unauthenticated. If no CA is configured (e.g. in unit tests) the handler is called directly. * cmd/extensions/main.go — load requestheader-client-ca-file from the kube-system/extension-apiserver-authentication ConfigMap (populated by Kubernetes for every aggregated API server) and wire it into the APIServer via SetRequestHeaderCA. A log warning is emitted if the ConfigMap is unavailable; auth is then effectively disabled for that start-up, preserving the previous behaviour while making the failure visible. Fixes: agones-dev#4572 Signed-off-by: adilburaksen <adilburaksen@gmail.com>

agones-bot · 2026-05-23T20:42:33Z

Build Succeeded 🥳

Build Id: 5c38641c-8e08-4493-bc28-2d26dea689a5

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.58.0-dev-08a3dfe
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.58.0-dev-08a3dfe
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.58.0-dev-08a3dfe-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.58.0-dev-08a3dfe
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.58.0-dev-08a3dfe
image: us-docker.pkg.dev/agones-images/ci/agones-processor:1.58.0-dev-08a3dfe
Linux C++ SDK (build): agonessdk-1.58.0-dev-08a3dfe-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.58.0-dev-08a3dfe.zip

A preview of the website (the last 30 builds are retained):

https://08a3dfe-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4582/head:pr_4582 && git checkout pr_4582
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.58.0-dev-08a3dfe

markmandel

So I'm going to preface this with a small warning.

Going by the current engagement, I'm pretty sure you are just copy pasting whatever your LLM is giving you, and not actually taking the time to look at what you are generating, and then making changes or improvements accordingly. If you continue to do so, I'll shut this PR down as it's wasting maintainer time reviewing code that hasn't been review by you, the author before being sent out.

If not, then I apologise, but comments and code look very LLM generated, and going from the initial PR as well, there are lots of issues we need to work through with this code.

Either way, hoping to make this a learning opportunity.

Some extra review notes:

Did you test this locally at all before sending this to us for review? If not, do so. We have a contributing and development guide that shows you how to do that locally with minikube.
Let's get some unit tests going. No tests, no merge 👍🏻

…aderCA Add five table-driven tests in pkg/util/apiserver covering all branches of authenticatedHandler: passthrough when no CA is set, 401 when TLS is absent, 401 when no peer certificate is present, 200 for a cert signed by the expected CA, and 401 for a cert from an untrusted CA. Add four subtests in cmd/extensions for loadRequestHeaderCA: valid CA PEM in the ConfigMap, missing ConfigMap, missing key, and invalid PEM data. Signed-off-by: Adil Burak ŞEN <adilburaksen@gmail.com>

adilburaksen · 2026-05-24T21:55:02Z

Fair enough on the review notes. I went back through the code more carefully.

Tests are up: 5 cases for authenticatedHandler (no CA passthrough, nil TLS, empty PeerCertificates, valid cert from expected CA, cert from untrusted CA) and 4 subtests for loadRequestHeaderCA (valid PEM, missing ConfigMap, missing key, bad PEM). DCO is now clean on the new commit.

I haven't run this against a real cluster yet — I'll set up minikube and work through the dev guide before asking for another review pass.

Separately: you mentioned "lots of issues" — I want to make sure I'm addressing the right things rather than guessing. Is the main concern around the approach (TLS client cert verification for requestheader auth), the startup behavior (CA loaded once with a soft warn on failure), or something else in the implementation? Happy to rework whichever pieces are off.

markmandel · 2026-05-27T01:27:43Z

/gcbrun

There you are 😄 glad to meet you.

Separately: you mentioned "lots of issues" — I want to make sure I'm addressing the right things rather than guessing.

My comments outline the issues - the cert refresh issue is the one I'm most concerned about. Interrupting of allocation would be a big problem.

I look forward to seeing your contributions!

agones-bot · 2026-05-27T02:33:39Z

Build Succeeded 🥳

Build Id: 1f8dae26-16df-46c5-a9f9-622e155c097e

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.59.0-dev-b11d769
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.59.0-dev-b11d769
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.59.0-dev-b11d769-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.59.0-dev-b11d769
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.59.0-dev-b11d769
image: us-docker.pkg.dev/agones-images/ci/agones-processor:1.59.0-dev-b11d769
Linux C++ SDK (build): agonessdk-1.59.0-dev-b11d769-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.59.0-dev-b11d769.zip

A preview of the website (the last 30 builds are retained):

https://b11d769-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4582/head:pr_4582 && git checkout pr_4582
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.59.0-dev-b11d769

github-actions Bot added kind/other size/S labels May 23, 2026

adilburaksen force-pushed the fix/allocation-rbac-requestheader-auth branch from 08a3dfe to 8bc0208 Compare May 23, 2026 19:45

Merge branch 'main' into fix/allocation-rbac-requestheader-auth

9be83de

markmandel reviewed May 24, 2026

View reviewed changes

Comment thread cmd/extensions/main.go

Comment thread cmd/extensions/main.go

Comment thread cmd/extensions/main.go

Comment thread cmd/extensions/main.go

github-actions Bot added the size/M label May 24, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

extensions: authenticate allocation endpoint with requestheader client CA#4582

extensions: authenticate allocation endpoint with requestheader client CA#4582
adilburaksen wants to merge 3 commits into
agones-dev:mainfrom
adilburaksen:fix/allocation-rbac-requestheader-auth

adilburaksen commented May 23, 2026 •

edited

Loading

Uh oh!

markmandel commented May 23, 2026

Uh oh!

agones-bot commented May 23, 2026

Uh oh!

adilburaksen commented May 23, 2026

Uh oh!

markmandel commented May 23, 2026

Uh oh!

markmandel commented May 23, 2026

Uh oh!

agones-bot commented May 23, 2026

Uh oh!

markmandel left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adilburaksen commented May 24, 2026

Uh oh!

markmandel commented May 27, 2026

Uh oh!

agones-bot commented May 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

adilburaksen commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Fix

Testing

References

Uh oh!

markmandel commented May 23, 2026

Uh oh!

agones-bot commented May 23, 2026

Uh oh!

adilburaksen commented May 23, 2026

Uh oh!

markmandel commented May 23, 2026

Uh oh!

markmandel commented May 23, 2026

Uh oh!

agones-bot commented May 23, 2026

Uh oh!

markmandel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adilburaksen commented May 24, 2026

Uh oh!

markmandel commented May 27, 2026

Uh oh!

agones-bot commented May 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adilburaksen commented May 23, 2026 •

edited

Loading