{"id":793,"date":"2022-12-01T21:10:16","date_gmt":"2022-12-01T13:10:16","guid":{"rendered":"https:\/\/511cvlab.sinkers.cn\/?p=793"},"modified":"2025-10-17T17:06:41","modified_gmt":"2025-10-17T09:06:41","slug":"dsf","status":"publish","type":"post","link":"https:\/\/cv.nirc.top\/zh\/2022\/dsf\/","title":{"rendered":"A Dual-Branch Self-Boosting Framework for Self-Supervised 3D Hand Pose Estimation"},"content":{"rendered":"
\n
\n \n
\n \n
\n \n \n <\/path>\n <\/svg>\n <\/div>\n
Paper<\/div>\n <\/a>\n <\/div>\n\n
\n \n
\n \n \n <\/path>\n <\/svg>\n <\/div>\n
Code<\/div>\n <\/a>\n <\/div>\n\n\n <\/div>\n<\/div>\n\n\n\n
\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
ICVL Dataset<\/figcaption><\/figure>\n\n\n\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
MSRA Dataset<\/figcaption><\/figure>\n<\/div>\n\n\n\n

For ICVL and MSRA, we show that DSF is able to generate more reasonable hand poses than annotations.
We use red circles to locate errors in annotations.<\/p>\n\n\n\n

Overview<\/h2>\n\n\n\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Overview of our proposed dual-branch self-boosting framework (DSF).<\/figcaption><\/figure>\n\n\n\n

Through image-to-image translation technology, our framework can make better use of synthetic data for pre-training. The dual-branch design allows our framework to adopt a part-aware model-fitting loss for self-supervised learning on unlabeled real data while achieving high-accuracy pose estimation and flexible and robust hand model estimation simultaneously.<\/p>\n\n\n\n

Qualitative Results<\/h2>\n\n\n\n
\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Qualitative results for NYU dataset.<\/figcaption><\/figure>\n\n\n\n

\u2705 DSF can maintain the rationality of hand pose when fingers contact intensively<\/strong>, such as clenching <\/strong>(3361) or when fingers are pressed together<\/strong> (3993).<\/p>\n\n\n\n

\u2705 DSF is robust to extreme perspective<\/strong> (609, 4606) and the depth holes<\/strong> (1194, 4329). Although DSF\u2019s predictions in missing regions do not completely consistent with the ground truth, they remain structurally reasonable<\/strong>.<\/p>\n\n\n\n

\u2705 DSF can predict complex hand poses<\/strong> (5027, 7122, 6817) with high accuracy.<\/p>\n\n\n\n

\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Qualitative results for testing in real scenarios on different subjects with different hand sizes and hand poses. For each subject, the first line is the depth images captured by the KinectV2; the second line is the cropped hand depth images and the estimated hand pose; the third line is the estimated hand mesh.<\/figcaption><\/figure>\n\n\n\n

Comparison to SOTA<\/h2>\n\n\n\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Comparison of the aje (mm) with state-of-the-art methods on NYU dataset.<\/figcaption><\/figure>\n\n\n\n

Downstream task: Skeleton-based Action Recognition<\/h2>\n\n\n\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
The comparison of the recognition accuracy (%) with the SOTA approaches on SHREC and DHG datasets. The upward arrows represent the improvement compared to DG-STA.<\/figcaption><\/figure>\n\n\n\n

Using the 3D skeleton generated by DSF can greatly improve the accuracy of the skeleton-based action recognition, which shows that our method has strong application potential in downstream tasks.<\/p>\n\n\n\n

Conclusion<\/h2>\n\n\n\n

In this paper, we propose a Dual-branch Self-boosting Framework (DSF) to achieve accurate, robust and flexible 3D hand pose and model estimation without using any labeled real data.<\/p>\n\n\n\n

1\ufe0f\u20e3 Firstly, we propose an image-to-image translation technology<\/strong> to reduce the domain gap between synthetic data and real data, which significantly improves the effect of network pre-training.<\/p>\n\n\n\n

2\ufe0f\u20e3 Secondly, we propose a dual-branch self-boosting network that can maintain the accuracy of 3D pose estimation and the robustness and flexibility of 3D hand model estimation simultaneously.<\/p>\n\n\n\n

3\ufe0f\u20e3 Through an inter-branch loss<\/strong> and a part-aware model-fitting loss<\/strong>, we fully explore the advantages of the dual-branch structure. The two branches can promote each other continuously during the self-supervised training on unlabeled real data.<\/p>\n\n\n\n

\u2705 Our method achieves comparable results to state-of-the-art fully supervised methods and shows better generalization performance.<\/p>\n\n\n\n

\u2705 Our method outperforms previous self-supervised methods without using paired multi-view images.<\/p>\n\n\n\n

\u2705 In addition, our method greatly improves the accuracy of skeleton-based gesture recognition, which shows that our method has strong application potential in downstream tasks\u200b.<\/p>\n\n\n\n

\n

Supplementary Videos<\/h2>\n\n\n\n
This video shows the performance of our method on the test set of the NYU dataset.<\/summary>\n
\n