Cannot Reproduce the results

[jieWANGforwork]

Hi, thanks for you work and contribution.
I cannot reproduce the results of TFR-CVR shown in the paper.
The results are highly lower than the paper reported.
TFR-CVR (Ours) ✓ ✓ ✓ Captioning 14.1 39.5 54.4
But I got the results 9.8 0.31.9 49.6
I am wandering if any setting I did is wrong.
So could you please specify the setting of TFR-CVR in Table 1,
E.g., the 1 and 2 stages's video model used specifically, cause I didn't find the illusration of either EgoVLPv2 or Languagebind is used for which stage.

Thanks a lot.

[hummelth]

Hi, I am not able to reproduce your issue. Can you specify which command you used for evaluation?
For TFR-CVR, the correct evaluation command for the global setting is

python egocvr_retrieval.py --evaluation global --model languagebind egovlpv2 --modalities visual text --text tfcvr

giving the following results: Recall@1,5,10: 0.141 & 0.395 & 0.544

TFR-CVR uses the video model of LanguageBind for the ranking in the first stage, followed by text-to-video retrieval on the modified captions/instructions on the filtered gallery from the first stage using EgoVLPv2.