[9.0] Optimising PilotWrapper size?

[sfayer]

Hi,

While looking into why v9 pilots fail on the cloud with @marianne013, one of the issues we're running into is the cloud user_data sometimes overflows the default 64k limit of OpenStack (unfortunately there is a lot of base64-inside-base64 on the cloud that's somewhat unavoidable). Now that the PilotWrapper includes its own proxy, there is an outer proxy that can probably be removed at the cloud-init level, which we'll look at doing soon.

The runner up on "pilot wrapper space use" is this construct:

DIRAC/src/DIRAC/WorkloadManagementSystem/Utilities/PilotWrapper.py

Lines 180 to 183 in 4634fbc

	if sys.version_info < (3,):
	fd.write(bz2.decompress(base64.b64decode(\"\"\"%(encodedPf)s\"\"\")))
	else:
	fd.write(bz2.decompress(base64.b64decode(b'%(encodedPf)s')))

The encodedPf variable expands to around a ~8k string so having two copies is somewhat suboptimal (the entire pilot wrapper is 25k, 16k of that is the two proxy copies). Do you think there is some trick we can do to support py2 & py3 with only a single copy of the actual long string? (Alternately I guess dropping py2 support here at this point would also be an easy fix, but I'm not sure what people think about that...)

Regards,
Simon