Skip to content

Conversation

@ujjwalkalia
Copy link

@ujjwalkalia ujjwalkalia commented Jun 4, 2025

Type of change

Select the type of your PR

  • Enhancement / new feature

Description

This is a first step for multiple requests raised on Kroxylicious. ( see #2166 and #1637), this adds support of getting the client principal and making it available for filters to consume.

Additional Context

Why are you making this pull request?
This will make the downstream client certificate principal, which is received during SSL handshake if mTLS is enabled on the proxy, available to the filter context, so that filters can be added which can do authentication swapping/ audit logging for the received principal.

Checklist

Please go through this checklist and make sure all applicable tasks have been done

  • PR raised from a fork of this repository and made from a branch rather than main.
  • Write tests
  • Update documentation
  • Make sure all unit/integration tests pass
  • Make sure all Sonarcloud warnings are addressed or are justifiably ignored.
  • If applicable to the change, trigger the system test suite. Make sure tests pass.
  • If applicable to the change, trigger the performance test suite. Ensure that any degradations to performance numbers are understood and justified.
  • Ensure the PR references relevant issue(s) so they are closed on merging.
  • For user facing changes, update CHANGELOG.md (remember to include changes affecting the API of the test artefacts too).

NOTE: You must be a member of @kroxylicious/developers to trigger the system test and performance test suites. If you are not part of this group, comment on the PR requesting a trigger, tagging @kroxylicious/developers.

@ujjwalkalia ujjwalkalia requested a review from a team as a code owner June 4, 2025 18:32
Copy link
Member

@SamBarker SamBarker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HI @ujjwalkalia thanks for the PR. You are quite right its certainly a request thats had a fair bit of interest so great to see some progress on it.

As it touches the public API we tend to be quite careful about the API choices as we are aiming to provide a good level of stability there. So while this change as it stands is a simple addition we want to make sure its the right long term option.

Could you please fix the DCO sign off? I'll kick of a test run now.

}
catch (SSLPeerUnverifiedException e) {
LOGGER.debug("No client principal received, setting principal as ANONYMOUS");
downstreamCertificatePrincipal = ANONYMOUS;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What makes this case different to the default of null ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when only TLS is enabled we will get ANONYMOUS, but when the channel is unencrypted then the value will be null.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but what is the advantage of returning null vs ANONYMOUS ? Shouldn't it just be ANONYMOUS unless there is a client cert?

Or maybe the default should be UNAUTHENTICATED?

* @return the client certificate principal as a string
*/
@Nullable
default String downstreamCertificatePrincipal() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is a String really the appropriate choice for the API?

Would there be any downsides to having this return the actual java.security.Principal ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one was rumbling through my head all weekend, java.security.Principal is probably not type we want for the interface as we'd really like this to work for other auth mechanisms. On that basis we'd probably also want to change the method name to remove Certificate. I'm not keen on the idea of adding a different method on the context for each auth mechanism.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this makes sense, one use case that I see is there can be a filter which wants both the certificate principal as well as the principal from other auth mechanism e.g SASL to make some decisions.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah something like an AuditLoggingFilter would want the principle from any source.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also have a look at the possibility of using KafkaPrincipal

if (sslHandshakeCompletionEvent.isSuccess()) {
SslHandler sslHandler = getSslHandler(ctx);
try {
downstreamCertificatePrincipal = sslHandler.engine().getSession().getPeerPrincipal().toString();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is toString the right choice here? Principal#getName looks like a more appropriate choice on my naive reading of the Javadoc.

@SamBarker
Copy link
Member

@ujjwalkalia could you please run the maven build locally to fix the formatting errors. CI will detect the problem but wont push fixes.

@ujjwalkalia
Copy link
Author

HI @ujjwalkalia thanks for the PR. You are quite right its certainly a request thats had a fair bit of interest so great to see some progress on it.

As it touches the public API we tend to be quite careful about the API choices as we are aiming to provide a good level of stability there. So while this change as it stands is a simple addition we want to make sure its the right long term option.

Could you please fix the DCO sign off? I'll kick of a test run now.

Thanks for the review, working on fixing the DCO sign off and formatting errors.

@dmariassy
Copy link

Sorry for the potentially dumb question but I don't fully understand how this would help with auth swapping. By the time a filter is invoked the upstream connection is already established, no?

I agree that having the client's auth context in the filters would useful but I see that as a separate problem from #1637.

My team was exploring these issues last week, and created a separate implementations for the 2 problems in our private forks. They are described here:

We have adopted an approach that allows for customizing the behavior of these interfaces using kroxylicious' extensible @Plugin stack

@tombentley
Copy link
Member

@dmariassy I've been prevaricating about bringing that up, but I've been working a bit on that and have opened kroxylicious/design#70 to discuss one idea (still a WIP). You might want to take a look. @ujjwalkalia you also might be interested.

@ujjwalkalia
Copy link
Author

Sorry for the potentially dumb question but I don't fully understand how this would help with auth swapping. By the time a filter is invoked the upstream connection is already established, no?

I agree that having the client's auth context in the filters would useful but I see that as a separate problem from #1637.

My team was exploring these issues last week, and created a separate implementations for the 2 problems in our private forks. They are described here:

We have adopted an approach that allows for customizing the behavior of these interfaces using kroxylicious' extensible @Plugin stack

@dmariassy , yes you are right, the connection is already established to the broker by the time filter is executed, but this change helps in the case we are during mTLS to SASL swap, so although SSL connection is established with the broker the channel will still be unauthenticated as SASL handshake is required, which the filter will be able to do.

ujjwalkalia and others added 23 commits June 9, 2025 02:47
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Ujjwal <ukalia@confluent.io>
…nal (kroxylicious#2248)

Signed-off-by: k-wall <18440250+k-wall@users.noreply.github.com>
Co-authored-by: k-wall <18440250+k-wall@users.noreply.github.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
…2.6.9.Fi…"

This reverts commit 6021132.

Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.12.2 to 5.13.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](junit-team/junit-framework@r5.12.2...r5.13.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-version: 5.13.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Bumps [org.apache.maven.plugins:maven-clean-plugin](https://github.com/apache/maven-clean-plugin) from 3.4.1 to 3.5.0.
- [Release notes](https://github.com/apache/maven-clean-plugin/releases)
- [Commits](apache/maven-clean-plugin@maven-clean-plugin-3.4.1...maven-clean-plugin-3.5.0)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-clean-plugin
  dependency-version: 3.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Bumps [dawidd6/action-download-artifact](https://github.com/dawidd6/action-download-artifact) from 8 to 10.
- [Release notes](https://github.com/dawidd6/action-download-artifact/releases)
- [Commits](dawidd6/action-download-artifact@v8...v10)

---
updated-dependencies:
- dependency-name: dawidd6/action-download-artifact
  dependency-version: '10'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Co-authored-by: Keith Wall <kwall@redhat.com>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Ujjwal <ukalia@confluent.io>
…etrics

* Frame now carries api key id/version
* pushed responsibility for emitting metrics to dedicated handlers.

Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
…ponse

Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Co-authored-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
…f API

Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
SamBarker and others added 22 commits June 9, 2025 02:48
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Formatter didn't fix it as it was in a massive block of formatter off.

Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Ujjwal <ukalia@confluent.io>
…ls to resolve to virtual cluster.

Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: ShubhamRwt <shubhamrwt02@gmail.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: k-wall <18440250+k-wall@users.noreply.github.com>
Co-authored-by: k-wall <18440250+k-wall@users.noreply.github.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Keith Wall <kwall@apache.org>
Signed-off-by: Ujjwal <ukalia@confluent.io>
For TLS clusterIP ingress we manifest:
1. a bootstrap service named ${virtualClusterName}-${ingressName}, the
   same naming we use for the service carrying the TCP bootstrap port.
2. a service for each upstream node named like
   ${virtualClusterName}-${ingressName}-${upstreamNodeId}.
3. we configure the proxy virtual cluster gateway like:
   sniHostIdentifiesNode:
     bootstrapAddress: "${virtualClusterName}-${ingressName}.${proxyNamespace}.svc.cluster.local:9191"
     advertisedBrokerAddressPattern: "${virtualClusterName}-${ingressName}-$(nodeId).${proxyNamespace}.svc.cluster.local:9292"

The user must ensure their downstream TLS certificates contain SANs for the
bootstrap and each upstream node.

Why:
Previously we limited the user to a single VKC+ingress using identifying
ports due to the risks of using port for identification (during
reconciliation, the identity of a port can change, so clients could
connect to an unexpected vcluster/node).

By changing to SNI as the mechanism, the identity is unambiguous as
VKCs/gateways are added and removed, as the hostnames should be stable
and uniquely address a gateway bootstrap or node. So we can safely allow
numerous TLS clusterIP ingresses to co-exist within a cluster without
implementing a safer port allocation strategy for identifying ports.

Signed-off-by: Robert Young <robeyoun@redhat.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
kubectl exited non-zero when we try to delete some resources, when k8s
has no knowledge of those CRDs. This command is intended to clean out
existing resources and CRDs from an established minikube, as well as
install into a fresh minikube. So we can tactically ignore those
failures.

Signed-off-by: Robert Young <robeyoun@redhat.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Why:
We want to be able to suffix the proxy name to create a shared
LoadBalancer service. Limiting the KafkaProxy name gives us some wiggle
room since we can only create Services with names up to 63 characters.

Signed-off-by: Robert Young <robeyoun@redhat.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Ujjwal <ukalia@confluent.io>
Fixes: kroxylicious#1852
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Ujjwal <ukalia@confluent.io>
* first try

Signed-off-by: Francisco Vila <fvila@redhat.com>

* make olm installation work

Signed-off-by: Francisco Vila <fvila@redhat.com>

* added some default values

Signed-off-by: Francisco Vila <fvila@redhat.com>

* fix installation method

Signed-off-by: Francisco Vila <fvila@redhat.com>

* add olm deployment name as env variable

Signed-off-by: Francisco Vila <fvila@redhat.com>

* replace install type env variable

Signed-off-by: Francisco Vila <fvila@redhat.com>

* fix format

Signed-off-by: Francisco Vila <fvila@redhat.com>

* added undeclared dependencies

Signed-off-by: Francisco Vila <fvila@redhat.com>

* set new default value for olm deployement

Signed-off-by: Francisco Vila <fvila@redhat.com>

* small nits from Keith

Signed-off-by: Francisco Vila <fvila@redhat.com>

---------

Signed-off-by: Francisco Vila <fvila@redhat.com>
Signed-off-by: Ujjwal <ukalia@confluent.io>
@ujjwalkalia ujjwalkalia force-pushed the ssl_principal_propagation branch from 037b011 to b0e4f40 Compare June 8, 2025 21:22
@ujjwalkalia ujjwalkalia changed the base branch from main to SamBarker-patch-1 June 8, 2025 21:38
@ujjwalkalia ujjwalkalia changed the base branch from SamBarker-patch-1 to main June 8, 2025 21:39
@ujjwalkalia
Copy link
Author

HI @ujjwalkalia thanks for the PR. You are quite right its certainly a request thats had a fair bit of interest so great to see some progress on it.
As it touches the public API we tend to be quite careful about the API choices as we are aiming to provide a good level of stability there. So while this change as it stands is a simple addition we want to make sure its the right long term option.
Could you please fix the DCO sign off? I'll kick of a test run now.

Thanks for the review, working on fixing the DCO sign off and formatting errors.

@SamBarker the DCO fix seems to have messed up the commit history, I will close this and open a new PR referencing this.

@SamBarker
Copy link
Member

HI @ujjwalkalia thanks for the PR. You are quite right its certainly a request thats had a fair bit of interest so great to see some progress on it.
As it touches the public API we tend to be quite careful about the API choices as we are aiming to provide a good level of stability there. So while this change as it stands is a simple addition we want to make sure its the right long term option.
Could you please fix the DCO sign off? I'll kick of a test run now.

Thanks for the review, working on fixing the DCO sign off and formatting errors.

@SamBarker the DCO fix seems to have messed up the commit history, I will close this and open a new PR referencing this.

Thanks, there are days when I hate git and then I remember how painful CVS and SVN branching as settle for terrible UX...

@ujjwalkalia
Copy link
Author

Closing this PR in favour of #2294.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants