Question about instance-level prediction

[JosonChan1998]

Thanks for your nice work.

I have a question about the method:
How should the query be defined for multi-person human pose estimation and human parsing?
Does it need an extra detector to crop the human instance from the original image?

[orashi]

For UniHCP as described in the paper and implemented in the repo, a Top-down approach would be needed: First
detects humans via Pedestrian detection queries, cropping out each identified individual by the bounding box, and then do pose estimation separately. There might ways to incorporate bottom-up to UniHCP during training/inferencing but they are not discussed in this version.

[JosonChan1998]

Thanks for your reply.

[JosonChan1998]

Sorry for reopening the issue.
Recently, I have a idea for creating a unified instance-level human-centric perception model, including human parsing, pose estimation and part detection. In other words, it is an end-to-end paradigm without the extra person detector and support multi-task dataset training.
I would like to sincerely ask you what you think about this idea? Looking forward to your reply!

[orashi]

That's a cool idea and something we thought of the same but were unable to easily achieve under the current UniHCP structure, our thoughts were: Fine-grained instance level semantics identification would make it hard to let semantics self-contained in queries, e.g., either we have a lot of semantic-self-contained queries like what we do for detection (but more kind of things to detect, with separate set of queries), which is computation-inefficient; or a mask2former style bipartisan matching for this purpose, which would require classification heads on top of decoder, and that classification head is dataset-dependent (class definition varies across different datasets) thus intruducing unsharable parameters.

[JosonChan1998]

Thanks for your reply!
As you said, the most difficult thing is how to unify instance-aware and part-aware queries. I am still exploring it now.
Thanks again.