{"id":776,"date":"2022-12-02T20:11:00","date_gmt":"2022-12-02T12:11:00","guid":{"rendered":"https:\/\/511cvlab.sinkers.cn\/?p=776"},"modified":"2025-10-17T17:09:13","modified_gmt":"2025-10-17T09:09:13","slug":"mmi","status":"publish","type":"post","link":"https:\/\/cv.nirc.top\/zh\/2022\/mmi\/","title":{"rendered":"Mining Multi-View Information: A Strong Self-Supervised Framework for Depth-Based 3D Hand Pose and Mesh Estimation"},"content":{"rendered":"
\n
\n \n
\n \n
\n \n \n <\/path>\n <\/svg>\n <\/div>\n
Paper<\/div>\n <\/a>\n <\/div>\n\n
\n \n
\n \n \n <\/path>\n <\/svg>\n <\/div>\n
Code<\/div>\n <\/a>\n <\/div>\n\n\n <\/div>\n<\/div>\n\n\n\n

Abstract<\/h2>\n\n\n\n

In this work, we study the cross-view information fusion problem in the task of self-supervised 3D hand pose estimation from the depth image. Previous methods usually adopt a hand-crafted rule to generate pseudo labels from multi-view estimations in order to supervise the network training in each view. However, these methods ignore the rich semantic information in each view and ignore the complex dependencies between different regions of different views. To solve these problems, we propose a cross-view fusion network to fully exploit and adaptively aggregate multi-view information. We encode diverse semantic information in each view into multiple compact nodes. Then, we introduce the graph convolution to model the complex dependencies between nodes and perform cross-view information interaction. Based on the cross-view fusion network, we propose a strong self-supervised framework for 3D hand pose and hand mesh estimation. Furthermore, we propose a pseudo multi-view training strategy to extend our framework to a more general scenario in which only single-view training data is used. Results on NYU dataset demonstrate that our method outperforms the previous self-supervised methods by 17.5% and 30.3% in multi-view and single-view scenarios. Meanwhile, our framework achieves comparable results to several strongly supervised methods.<\/p>\n\n\n\n

Overview<\/h2>\n\n\n\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
The
single-view estimation network<\/strong> (SEN) predicts 3D hand pose and 3D hand model in parallel. The cross-view fusion network<\/strong> (CFN) utilizes visual features and estimated hand poses from multiple views to generate more accurate results in the candidate view. Here, we choose view 1 as the candidate view.<\/figcaption><\/figure>\n\n\n\n

<\/p>\n\n\n\n

1\ufe0f\u20e3 Pre-training SEN on Single-view Data<\/strong>: First, we use synthetic data for pre-training. <\/p>\n\n\n\n

2\ufe0f\u20e3 Training CFN to Improve Multi-view Fusion<\/strong>: Then, the two networks adopt a single-view model-fitting loss<\/strong> Lsm<\/sub> and a cross-view model-fitting loss<\/strong> Lcm<\/sub> to perform self-supervised learning<\/strong> on unlabeled real data<\/strong>. <\/p>\n\n\n\n

3\ufe0f\u20e3 Self-distillation to Improve SEN<\/strong>: Meanwhile, the single-view estimation network is trained to produce results consistent with the cross-view fusion network, which is called multi-view self-distillation loss<\/strong> Lms<\/sub>. <\/p>\n\n\n\n

4\ufe0f\u20e3 During the inference, only the single-view estimation network is utilized.<\/p>\n\n\n\n

Qualitative Results<\/h2>\n\n\n\n

On ICVL and MSRA datasets, our method predicts more accurate 3D hand pose than ground-truth (GT)<\/strong>.<\/p>\n\n\n\n

\n