[Performance Inquiry] Potential gap in replicating 67.7% F1 on ChangeVPR benchmark

[06PingGong]

Hello @1124jaewookim,
Thank you for your excellent work on GeSCF. I am working on reproducing the results for the ChangeVPR benchmark but am currently hitting a performance plateau at ~64.6% F1 (weighted average).
Configuration used:
• SAM Backbone: ViT-H
• Input resolution: 512x512
• Thresholds: pred_iou_thresh=0.7, stability_score_thresh=0.7
• Current Results: SF-XL (71.2%), Nordland (59.0%), St Lucia (62.1%)
Question on Spatial Consistency:
In the public framework.py, we observed that masks_t1 are not regenerated for the ChangeVPR dataset after registration (unlike for ChangeSim). For datasets with significant seasonal shifts like Nordland, would regenerating masks for aligned images help bridge this performance gap?
Are there any other specific hyperparameters (like cosine_thr) used specifically for the final reported evaluation?
Best regards,
[Student Researcher:Mia]

[1124jaewookim]

Hello,

Thank you for your interest in our work.

Your reproduction results are accurate. In our paper, we reported an F1 score of 64.1% on the ChangeVPR benchmark, and your result (~64.6%) is consistent with ours. The higher score of 67.7% corresponds to standard SCD datasets (e.g., VL-CMU-CD, Tsunami, ChangeSim), not ChangeVPR.

Regarding framework.py, your understanding is correct. For datasets such as ChangeSim, image pairs are often significantly misaligned, and the temporal consistency (TC) assumption relies on sufficiently accurate alignment. Therefore, we include additional processing steps (e.g., mask regeneration after alignment) to mitigate errors in severe cases.

In contrast, datasets like ChangeVPR are relatively well-aligned during collection (<10m & <30 degrees) or preprocessing. As a result, we do not apply additional alignment-based operations (such as mask regeneration) in this case.

Importantly, our method is designed as a modular framework, consisting of a coarse alignment stage followed by the GeSCF model, which performs change detection on the coarsely-aligned inputs.

If you aim to further improve performance, we suggest exploring the following directions (as well as tuning hyperparameters):

• Improving alignment quality to enhance pixel-wise correlation fidelity in GeSCF
• Making the skewness module learnable
• Applying feature engineering to better capture subtle or meaningful changes that SAM may miss

Thank you again for your interest in our work:)