QoS Turns Files PRECIOUS again after Flush to Tape

[christianvoss]

Hi all,

we've observed a strange behaviour recently. Files with QoS disk+tape have been made PRECIOUS again even after a flush and remain so even after a manual verify in the admin interface or even after a migration move.

We see the flush of a particular file:

{"instance": "cms", "host_type": "pool", "host": "dcache-cms239", "Domain": "dcache-cms239-04Domain", "date": "2025-12-08T02:21:33.273Z", "message": "[] Changing state of 0000F2EC85D5CEA041BCA85143245D2DE524 from PRECIOUS to CACHED: file sucessfully flushed", "cellName": "dcache-cms239-04", "pid": 708037}

while shortly afterwards, QoS sets the file again to PRECIOUS

{"instance": "cms", "host_type": "pool", "host": "dcache-cms239", "Domain": "dcache-cms239-04Domain", "date": "2025-12-08T02:21:45.143Z", "message": "[QOS-56d41442-ba26-4196-a89d-0f686bd80846 qos-adjuster PoolMigrationCopyReplica 0000F2EC85D5CEA041BCA85143245D2DE524] Changing state of 0000F2EC85D5CEA041BCA85143245D2DE524 from CACHED to PRECIOUS: migration request from qos-adjuster@dcache-cms-qos01_QoS-adjusterDomain", "cellName": "dcache-cms239-04", "pid": 708037}

For another file we tried to trigger a manual check:

[dcache-cms-core01] (local) vossc > \s qos-verifier verify ls 00007729AA5E141442FEB5625818A7FFCC09
[dcache-cms-core01] (local) vossc > \sl 00007729AA5E141442FEB5625818A7FFCC09 rep ls 00007729AA5E141442FEB5625818A7FFCC09
dcache-cms237-02:
    00007729AA5E141442FEB5625818A7FFCC09 <-P------X--L(0)[1]> 250950793 si={cms:user}

[dcache-cms-core01] (local) vossc > \sn storageinfoof 00007729AA5E141442FEB5625818A7FFCC09
size=250950793;new=false;stored=true;sClass=cms:user;cClass=-;hsm=cta;accessLatency=ONLINE;retentionPolicy=CUSTODIAL;links=0000926291B0C0FC44A797A93DD3BE6BF693 18ParkingBPH5_RunD_2021UL_5832.root;cta://cta/00007729AA5E141442FEB5625818A7FFCC09?archiveid=74645683;store=cms;group=user;bfid=<Unknown>;

As can be seen, the QoS would be disk+tape but the file itself has a valid tape location. Still, the verification finished and the file remains precious. A migration move does not change it either.

[dcache-cms-core01] (dcache-cms237-02@dcache-cms237-02Domain) vossc > \sl 00007729AA5E141442FEB5625818A7FFCC09 rep ls 00007729AA5E141442FEB5625818A7FFCC09
dcache-cms278-02:
    00007729AA5E141442FEB5625818A7FFCC09 <-P------X--L(0)[1]> 250950793 si={cms:user}

We're running

[root@dcache-cms-qos01 ~]# dcache version
10.2.18

for the QoS service and pools.

Cheers,
Christian

[khys95]

Hi Christian,
I'm having trouble recreating this. So far I am able to see expected results locally using version 10.2.18:

[p200300e01f101d42ff7668632d3f0eb4] (qos-verifier@dCacheDomain) admin > \s qos-verifier verify ls 00002565E8BCA86C481881B1FBB29982115A
[p200300e01f101d42ff7668632d3f0eb4] (qos-verifier@dCacheDomain) admin > \sl 00002565E8BCA86C481881B1FBB29982115A rep ls 00002565E8BCA86C481881B1FBB29982115A
pool_write:
    00002565E8BCA86C481881B1FBB29982115A <C-------X--L(0)[0]> 2100 si={test:public}

[p200300e01f101d42ff7668632d3f0eb4] (qos-verifier@dCacheDomain) admin > \sn storageinfoof 00002565E8BCA86C481881B1FBB29982115A
size=2100;new=false;stored=true;sClass=test:public;cClass=-;hsm=osm;accessLatency=ONLINE;retentionPolicy=CUSTODIAL;links=00008D83EAD22F184E5DACF9F0B328D96903 atlas-8.txt;osm://osm/00002565E8BCA86C481881B1FBB29982115A;store=test;group=public;bfid=<Unknown>;

My next step is to connect to the dot instance and try some stuff there.
I'm guessing you're running the pnfs_manage_bulkQoS.py script when chaning the QoS to disk + tape ?