Session cookies never expires

[abompard]

According to the GssapiUseSessions documentation, the session cookies should expire according to the lifetime of the GSSAPI session established at authentication. I don't see the expiration beeing set in the cookie header:

$ curl -v -u : --negotiate https://fasjson.fedoraproject.org/v1/me/
[...]
< HTTP/2 200 
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< www-authenticate: Negotiate [...]
< set-cookie: ipa_session=MagBearerToken=UXmr[...]Gwo%3d;path=/;httponly;secure;
< set-cookie: 258ec7ac3fe42ca4f3a9165f864d24b3=50374418bc7687d83d82fe30a6c36ce4; path=/; HttpOnly; Secure; SameSite=None
< apptime: D=319790
< 
{"result": {"dn": "uid=abompard,cn=users,cn=accounts,dc=fedoraproject,dc=org", "username": "abompard", "service": null, "uri": "https://fasjson.fedoraproject.org/v1/users/abompard/"}}

My config file includes:

GssapiUseSessions On
Session On
SessionCookieName ipa_session path=/;httponly;secure;
SessionHeader IPASESSION
GssapiSessionKey file:/httpdir/run/session.key

If I look at the ipa_session cookie header sent back to curl, I don't see any Expires attribute.
I think that may be why my long-running http client end up getting 401's: they keep the session cookie around when they should drop it.

[simo5]

On a 401 the client should re-attempt authentication and get a new cookie, not sure why that would not be working.
Adding an expiration on the cookie can probably be done, but would rather complicate matters and risk other sync issue.

[abompard]

I am currently testing another hypothesis, I had set the GssapiSessionKey to a file that does not exist on startup, so according to the docs mod_auth_gssapi will create a new one on startup. But it's in a temp storage, so if the pod is restarted, the file will change and clients with long-running sessions will suddenly see their session invalidated. I think that's why I'm getting 401s with Credential lifetime has expired (I've tested the credentials, they aren't expired).
The client is in Python, based on requests-gssapi.

[abompard]

I was wrong again, the cause of the 401 is that the credentials made available in GssapiDelegCcacheDir on the server have expired:

$ klist FILE:/httpdir/run/ccaches/bugzilla2fedmsg~os-control01.iad2.fedoraproject.org\@FEDORAPROJECT.ORG 
Ticket cache: FILE:/httpdir/run/ccaches/bugzilla2fedmsg~os-control01.iad2.fedoraproject.org@FEDORAPROJECT.ORG
Default principal: bugzilla2fedmsg/os-control01.iad2.fedoraproject.org@FEDORAPROJECT.ORG

Valid starting     Expires            Service principal
09/16/24 23:58:01  09/17/24 23:58:01  krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
        for client HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG
09/17/24 08:49:46  09/18/24 08:47:13  HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG
        renew until 09/24/24 08:47:13
09/17/24 08:49:46  09/17/24 23:58:01  ldap/ipa02.iad2.fedoraproject.org@
        Ticket server: ldap/ipa02.iad2.fedoraproject.org@FEDORAPROJECT.ORG
09/17/24 08:49:46  09/17/24 23:58:01  ldap/ipa03.iad2.fedoraproject.org@
        Ticket server: ldap/ipa03.iad2.fedoraproject.org@FEDORAPROJECT.ORG
09/17/24 08:49:46  09/17/24 23:58:01  ldap/ipa01.iad2.fedoraproject.org@
        Ticket server: ldap/ipa01.iad2.fedoraproject.org@FEDORAPROJECT.ORG

Is there something my application should do to renew them? The credentials on the client still had a lifetime of 59536s when the delegated credential on the server was found expired and the 401 was returned. Could it be that using sessions prevents the client from renewing the delegated credential on the server? Or are sessions unrelated? I'm not sure what's going on here, and what I'm missing.

[simo5]

Ideally you have a session key has a shorter expiration than credentials, when using sessions the client will just try to send a session cookie, however upon receiving a 401 the client should simply try a new gssapi authentication, did the client simply send back the session cookie and not try to perform an actual authentication ?
Or was auth attmpted and something went wrong ?

The web server log may have some good pointers if you enable the debug level.

[abompard]

The server code checks the ticket lifetime of the credentials that mod_auth_gssapi makes available in KRB5CCACHE (in /httpdir/run/ccaches), find that they are expired by looking at their lifetime attribute. If it's expired, it'll return a 401 response, but it's a regular 401 response without any "Negotiate" header, maybe the client expects it to retry authentication (at least that's what I understand from requests-gssapi's code)?

Is there a way for my server python code to trigger a proper 401 response from mod_auth_gssapi with the Negotiate header when the delegated credentials are expired?

[simo5]

I think 401 with Negotiate header is the regular way to ask a client to authenticate, why is the client not trying is the question ...

[abompard]

Hmm no I'm not sending a Negotiate header to the client when I do a 401. I "manually" do a 401 from the Python code when I see that the delegated credentials have expired. And I can't send the Negotiate header because this is all handled at the mod_auth_gssapi level. I it normal that the delegated credentials that mod_auth_gssapi provides my application with can be expired? Shouldn't it see that they are expired and ask the client to re-authenticate? Or am I missing something else?

[simo5]

If mod_auth_gssapi sees that the creds expiration has been reached in mag_check_session() then it will not proceed to the application at all and will return a 401 negotiate.

However if you return 401 directly from the application then mod_auth_gssapi is not involved anymore and you will have to return the Negotiate header yourself, alternatively you could return a Redirect error to the client where you also tell the client to invalidate the cookie. This will caus an additional rountrip in the client but will allow mod_auth_gssapi to send the 401 itself with the correct negotiate headers.

What is odd is that it seem there is a disagreement between mod_auth_gssapi and your application on the expiration time of the creds??

[abompard]

alternatively you could return a Redirect error to the client where you also tell the client to invalidate the cookie.

Oh good idea, I'll try that.

What is odd is that it seem there is a disagreement between mod_auth_gssapi and your application on the expiration time of the creds??

Indeed! Here's the delegated credentials that mod_auth_gssapi provides my application with:

$ klist -c /httpdir/run/ccaches/bugzilla2fedmsg~os-control01.iad2.fedoraproject.org\@FEDORAPROJECT.ORG
Ticket cache: FILE:/httpdir/run/ccaches/bugzilla2fedmsg~os-control01.iad2.fedoraproject.org@FEDORAPROJECT.ORG
Default principal: bugzilla2fedmsg/os-control01.iad2.fedoraproject.org@FEDORAPROJECT.ORG

Valid starting     Expires            Service principal
09/23/24 07:56:02  09/24/24 07:56:02  krbtgt/FEDORAPROJECT.ORG@FEDORAPROJECT.ORG
        for client HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG
09/23/24 09:06:17  09/24/24 09:06:16  HTTP/fasjson.fedoraproject.org@FEDORAPROJECT.ORG
        renew until 09/30/24 09:06:16
09/23/24 09:06:17  09/24/24 07:56:02  ldap/ipa02.iad2.fedoraproject.org@
        Ticket server: ldap/ipa02.iad2.fedoraproject.org@FEDORAPROJECT.ORG
09/23/24 09:06:17  09/24/24 07:56:02  ldap/ipa03.iad2.fedoraproject.org@
        Ticket server: ldap/ipa03.iad2.fedoraproject.org@FEDORAPROJECT.ORG
09/23/24 09:06:18  09/24/24 07:56:02  ldap/ipa01.iad2.fedoraproject.org@
        Ticket server: ldap/ipa01.iad2.fedoraproject.org@FEDORAPROJECT.ORG

I had to restart the client this morning because the authentication was expired again, but if it unfolds as before this time around again, I can tell that at 09/24/24 07:56:02, the session will still be valid, mod_auth_gssapi will not return 401, but my application will end up with an expired delegated credential and will not be able to talk to IPA over LDAP. Unfortunately I don't know C so I can't help with the code, but I'm happy to provide all the necessary info or run tests.

[abompard]

OK I did manage to workaround the issue by having the server return a 302 to the same address with the HTTP header that I had set in SessionHeader set to "MagBearerToken=". This invalidates the session, and the client just has to follow the redirect.
Unfortunately, our client is based on a library that interprets the Swagger/OpenAPI spec very strictly, and does not follow redirects. There's (what I think is) a bug in there too that makes it not trivial to tell it to follow redirects. Changing that client-side code is possible but not easy to deploy as it would mean updating all the apps that use it, which all have their own lifecycles. At the moment, if the server sends them a 302, they'll just traceback, which is not ideal. So I'm interested in trying to fix this at the mod_auth_gssapi level, if we agree there's a bug in there about the session lifetime.

[simo5]

Perhaps I need to cross check the cache liftime with the lifetime claimed in the cookie.
I expect there may be cases when the client can get confused and send a cookie for a lifetime that differs from the latest cached credentials ... I am not sure how that could happen, and I do not see any special security considerations to it. I am just not sure I can easily check for creds lifetime in the session handling code though. And I have no idea how I would test it ...

[simo5]

Ah but I thik I know how something like that can happen now that I think about your situation.
Are you, by chance, sharing the same krb principal among multiple different clients?

That could cause a client that is configured to obtain a shorter lived credentials to overwrite the cache creds with ones that are shorter lived than the creds another client originally used to obtain their cookie.

Like:

Client A fetches krb ticket with 24h expiration time
Client A auths to your server and obtain session cookie with exp timestamp 24h in the future
Server stores cached creds valid 24h
... one hour later...
Client B fetches krb tikcet with 10h expiration time
Client A auths to your server and obtain session cookie with exp timestamp 10h in the future
Server stores cached creds valid 10h (overwriting the previous ones valid 24h)
... 10 hours later ...
ClientA contacts server with its session cookie that says there are still 13h of validity to spare .. but the creds are actually expired due to substitution.

Can you confirm if this is a scenario that may be happening in your environment?

[abompard]

Perhaps I need to cross check the cache liftime with the lifetime claimed in the cookie. I expect there may be cases when the client can get confused and send a cookie for a lifetime that differs from the latest cached credentials ...

When I looked at the cookie that the client receives, I didn't see any expiration date set. This is the received header:

'set-cookie': 'ipa_session=MagBearerToken=b1qD[...snip...]6sa8fw%3d%3d;path=/;httponly;secure;'

Ah but I thik I know how something like that can happen now that I think about your situation. Are you, by chance, sharing the same krb principal among multiple different clients?

I don't think so, the credential comes from a keytab that only this app is using, and there's only one concurrent pod running.

I've managed to reproduce it on my local dev env and I can easily test it by setting ticket_lifetime = 10s on the server's krb5.conf, so I'm happy to test any patch.
The client is in Python, based on requests-gssapi. I create a client session, have it make a server call every 2s and surely enough after the 5th call it's getting a 401. I can try to make a minimal reproducing environment if that would help.