[Bug]: Proxy temp files accumulate in `/tmp` for long-running Site Directors

[aldbr]

Search before creating an issue

• I have searched existing issues and confirmed this is not a duplicate

Bug Description

Follows up on #8415

SiteDirectors still rely mostly on proxies to authenticate to Computing Elements and submit pilots.
ComputingElement._prepareProxy() calls gProxyManager.dumpProxyToFile() without specifying a destinationFile, so a new temp file is created in /tmp each time the cache doesn't have a hit for the given proxy chain.

While dumpProxyToFile cache files to avoid duplicates, a cache miss (and therefore a new file) occurs whenever the proxy chain hash changes.

The previous temp files are only cleaned up on process exit. For a long-running SiteDirector with many CEs, this leads to an accumulation of orphaned proxy files in /tmp.

Steps to Reproduce

• Run a SiteDirector for a very long time.
• See an increasing number of proxies appearing in /tmp.

Expected Behavior

One temp proxy file per CE/VO.

Actual Behavior

Many temp proxy files with random names.

Environment

No response

Relevant Log Output

Additional Context

An elegant fix would consist in passing a deterministic destinationFile based on the CE and VO (extracted from proxy?) names, so the file is overwritten in place rather than creating a new temp file each time.

This would eliminate the accumulation entirely: one file per CE/VO, overwritten in place.

[aldbr]

From the ddev meeting:

• One could have different identities per VO (not recommended though but could be a use case)
• Always risky to overwrite the proxy file, can be source of issues

@marianne013 could we have further details about your issue here please?
Wouldn't it be simpler to decrease the number of cycles?

[marianne013]

The MaxCycles on the test (v9) CE is 40, on the prod it's 50, that's not it.
On the test (v9) CE, without any queuing/running jobs I see 12 proxies generated every 2 min cycle. Simon thinks it might be the ARCs, because the test CE has 3 ARCs with 4 queues and that's the only way I can get 12 ;-)
I have another v8 test CE, which currently has no jobs queued and which doesn't produce any proxies.
The rate of proxies accumulating on the v8 production service is also much much slower, despite the fact that it has more CEs, VOs and jobs. I just checked an the test (v9) CE made 8796 proxies yesterday and the prod (v8) 946.
So something is going haywire, but I can't pinpoint where.

[sfayer]

Hi,

I've been looking at this and it appears a change of behaviour in v9 has created a new edge case here. This pull request changed the proxy renewal logic in v9 quite significantly:
7f2df73

The SiteDirector will check the proxy for each queue it has to submit to (there is meant to be some sharing of CE objects between the queues, which also appears to be broken but we won't worry about that here), so it renews the proxy:

DIRAC/src/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py

Line 929 in eb2e18b

result = gProxyManager.getPilotProxyFromDIRACGroup(self.pilotDN, pilotGroup, proxyMinimumRequiredValidity)

The requested lifetime is 96 hours on our instance, which appears to be the default (3 day max queue length + 1 day leeway). The VOMS servers for a least ones of our VOs is capped at 48 hours, so the returned proxy is significantly shorter than expected, but this isn't checked (maybe this should generate a warning in the log?).

One cycle time (2 minutes) later, the code goes around again and checks the lifetime:

DIRAC/src/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py

Lines 918 to 923 in eb2e18b

	result = ce.proxy.getRemainingSecs()
	if not result["OK"]:
	return result
	if result["Value"] < proxyMinimumRequiredValidity:
	getNewProxy = True

This has no margin in the check time, the proxy will nominally be 95.96 hours long which is less than 96 hours, so the proxy will be renewed. This will happen every time, resulting in an accumulation of proxy files in /tmp.

I think ideally we'd like this to use the same 1 day safety margin from here when it checks if the proxy needs to be renewed or not:

DIRAC/src/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py

Lines 318 to 319 in eb2e18b

	cpuTime = queueCPUTime + 86400
	result = self._setCredentials(ce, cpuTime)

The proxy cache doesn't help with the above situation because the proxy is different each time, it misses the cache and just gets added as a new entry. I think the proxy cache has a few other problems: I can't see any call to purgeExpired, so I don't think old proxies will ever be tidied up unless explicitly asked for and expired? The lifetime of the SiteDirector process is maybe a few hours (when it reaches its MaxCycles and restarts) but the proxy lifetime is days, so the cache dictionary gets "lost" during the restart and it never gets to the point of deleting the expired proxies anyway.

A quick summary of the above, what we'd like is:

• Check returned proxy lifetime in _setCredentials is actually equal or more than what was requested, log a warning if not.
• Add a time margin so that _setCredentials doesn't renew the proxy every time.
• Maybe add some kind of destructor/shutdown call to ProxyManagerClient's cache so that it deletes proxies when the cache is destroyed (or ideally don't rely on that behaviour in the first place and tidy old proxy files up as soon as they're no longer needed although knowing when that is seems more complicated).

Regards,
Simon

[aldbr]

ddev meeting:
this issue should only target point 1 and 2
we can have another issue targeting specifically the 3rd point: that would be much more work

• need to investigate if the issue happens everywhere or not
• if any issue, fix the issue at the low level (DictCache)