{"id":796,"date":"2021-12-19T21:12:27","date_gmt":"2021-12-19T13:12:27","guid":{"rendered":"https:\/\/511cvlab.sinkers.cn\/?p=796"},"modified":"2025-10-17T17:04:25","modified_gmt":"2025-10-17T09:04:25","slug":"sar","status":"publish","type":"post","link":"https:\/\/cv.nirc.top\/zh\/2021\/sar\/","title":{"rendered":"SAR: Spatial-Aware Regression for 3D Hand Pose and Mesh Reconstruction from a Monocular RGB Image"},"content":{"rendered":"
\n
\n \n
\n \n
\n \n \n <\/path>\n <\/svg>\n <\/div>\n
Paper<\/div>\n <\/a>\n <\/div>\n\n
\n \n
\n \n \n <\/path>\n <\/svg>\n <\/div>\n
Code<\/div>\n <\/a>\n <\/div>\n\n\n <\/div>\n<\/div>\n\n\n\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n

Illustration of 3D hand reconstruction from a monocular RGB image input. From the camera input (left), we reconstruct 3D hand mesh (upper right) and 3D hand pose (lower right). Because of our good balance of accuracy and efficiency, our method has more potential for real-world applications in VR\/AR scenarios.<\/p>\n\n\n\n

Abstract<\/h2>\n\n\n\n

3D hand reconstruction is a popular research topic in recent years, which has great potential for VR\/AR applications. However, due to the limited computational resource of VR\/AR equipment, the reconstruction algorithm must balance accuracy and efficiency to make the users have a good experience. Nevertheless, current methods are not doing well in balancing accuracy and efficiency. Therefore, this paper proposes a novel framework that can achieve a fast and accurate 3D hand reconstruction. Our framework relies on three essential modules, including spatial-aware initial graph building (SAIGB), graph convolutional network (GCN) based belief maps regression (GBBMR), and pose-guided refinement (PGR). At first, given image feature maps extracted by convolutional neural networks, SAIGB builds a spatial-aware and compact initial feature graph. Each node in this graph represents a vertex of the mesh and has vertex-specific spatial information that is helpful for accurate and efficient regression. After that, GBBMR first utilizes adaptive-GCN to introduce interactions between vertices to capture short-range and long-range dependencies between vertices efficiently and flexibly. Then, it maps vertices\u2019 features to belief maps that can model the uncertainty of predictions for more accurate predictions. Finally, we apply PGR to compress the redundant vertices\u2019 belief maps to compact joints\u2019 belief maps with the pose guidance and use these joints\u2019 belief maps to refine previous predictions better to obtain more accurate and robust reconstruction results. Our method achieves state-of-the-art performance on four public benchmarks, FreiHAND, HO-3D, RHD, and STB. Moreover, our method can run at a speed of two to three times that of previous state-of-the-art methods.<\/p>\n\n\n\n

Overview<\/h2>\n\n\n\n
\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Illustration of our framework.<\/figcaption><\/figure>\n\n\n\n

1\ufe0f\u20e3 We first utilize convolutional neural networks to extract feature maps from the input image. <\/p>\n\n\n\n

2\ufe0f\u20e3 Then, we use SAIGB<\/strong> module to build a mesh initial feature graph<\/strong> with every vertex contains several feature maps. <\/p>\n\n\n\n

3\ufe0f\u20e3 After that, we use GBBMR<\/strong> module to map the initial vertex feature to vertex belief maps with Adaptive-GCN<\/strong>. Here, we can obtain coarse coordinates with those coarse belief maps. <\/p>\n\n\n\n

4\ufe0f\u20e3 Subsequently, we employ PGR<\/strong> to refine those coarse belief maps to obtain fine ones. More specifically, PGR first obtains pose joints\u2019 belief maps from mesh vertices\u2019 belief maps and then concatenates pose belief maps and the last stage features for refinement. <\/p>\n\n\n\n

5\ufe0f\u20e3 Finally, we use these refined coordinates as our output.<\/p>\n\n\n\n

Qualitative Results<\/h2>\n\n\n\n
\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Visual results on FreiHAND, HO-3D, RHD, STB, and real-world images. We show the input image, predicted 3D pose mapped in the input image, predicted 3D mesh mapped in the input image, and two different views of 3D mesh in order.<\/figcaption><\/figure>\n\n\n\n

1\ufe0f\u20e3 On FreiHAND, we can see that our method can solve unusual viewpoints and complex poses. <\/p>\n\n\n\n

2\ufe0f\u20e3 On HO-3D, our method can predict reasonable results under the occlusions caused by hand-object interaction.<\/p>\n\n\n\n

3\ufe0f\u20e3 Results on RHD and STB datasets show that our method can generalize well for other datasets.<\/p>\n\n\n\n

4\ufe0f\u20e3 Results in real-world prove our method has the cross-domain generalization ability.<\/p>\n\n\n\n

These results prove our method has the cross-domain generalization ability and great potential for VR\/AR applications.<\/p>\n\n\n\n

Comparisons with SOTA<\/h2>\n\n\n\n
\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Comparisons with state-of-the-arts on FreiHAND.<\/figcaption><\/figure>\n\n\n\n

FreiHAND<\/strong>: Our method outperforms other methods<\/strong> in terms of all the aforementioned metrics. Moreover, we compare the inference speed of our method with other methods using the same computer. Our method has a much faster inference speed<\/strong> due to the simplicity of our method.<\/p>\n\n\n\n

\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Comparisons with state-of-the-arts on HO-3D. + denotes training with weights pre-trained on FreiHAND.<\/figcaption><\/figure>\n\n\n\n

FreiHAND<\/strong>: We find that the model training on this dataset tends to overfit. Therefore, we utilize weights pre-trained on FreiHAND for regularization. In this way, our method outperforms four recent works in a large margin<\/strong>. When directly training on this dataset, our method can still achieve state-of-the-art performance<\/strong>.<\/p>\n\n\n\n

\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n
\n
\n

Comparisons of 3D PCK on RHD dataset.<\/p>\n<\/div>\n\n\n\n

\n

Comparisons of 3D PCK on STB dataset.<\/p>\n<\/div>\n<\/div>\n\n\n\n

RHD and STB<\/strong>: Our method is a general framework<\/strong> that can adapt to pose estimation task. our method outperforms all other methods on RHD dataset<\/strong> and achieve comparable performance on STB dataset<\/strong> with other methods. Because STB is relatively easier than other datasets, recent methods all perform similarly on this dataset.<\/p>\n\n\n\n

Conclusion<\/h2>\n\n\n\n

In this paper, we propose a simple and effective framework for 3D hand reconstruction from a monocular RGB image. Our method introduces three impressive ideas of detection-based methods to regression-based methods framework<\/strong>, including preserving spatial information<\/strong>, modeling uncertainty<\/strong>, and introducing refinement strategy<\/strong>. We achieve these three ideas with our proposed SAIGB, GBBMR, and PGR, respectively. Extensive experiments on four public benchmarks demonstrate that our method achieves state-of-the-art performance<\/strong> as well as the highest inference speed<\/strong>. Because of this good balance of accuracy and efficiency, our method has more potential for real-world applications in VR\/AR scenarios.<\/p>\n\n\n\n

Bibtex<\/h2>\n\n\n\n
@inproceedings{zheng2021sar,\n  title={Sar: Spatial-aware regression for 3d hand pose and mesh reconstruction from a monocular rgb image},\n  author={Zheng, Xiaozheng and Ren, Pengfei and Sun, Haifeng and Wang, Jingyu and Qi, Qi and Liao, Jianxin},\n  booktitle={2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)},\n  pages={99--108},\n  year={2021},\n  organization={IEEE}\n}<\/code><\/pre>\n\n\n\n
<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"
Paper Code Illustration of 3D hand reconstruction from a monocular RGB image input. From the camera input (left), we reconstruct 3D hand mesh (upper right) and 3D hand pose (lower right). Because of our good balance of accuracy and efficiency, our method has more potential for real-world applications in VR\/AR scenarios. Abstract 3D hand reconstruction […]<\/p>\n","protected":false},"author":1,"featured_media":797,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":"","_links_to":"","_links_to_target":""},"categories":[17],"tags":[],"class_list":["post-796","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-rgb-based-3d-hand-pose-estimation"],"acf":{"writer":{"simple_value_formatted":"This data type is not supported! Please contact the author for help.<\/em><\/code>","value_formatted":[{"writer_link":{"simple_value_formatted":"Xiaozheng Zheng<\/a>","value_formatted":{"title":"Xiaozheng Zheng","url":"https:\/\/scholar.google.com\/citations?user=3hSD41oAAAAJ","target":"_blank"},"value":{"title":"Xiaozheng Zheng","url":"https:\/\/scholar.google.com\/citations?user=3hSD41oAAAAJ","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Pengfei Ren<\/a>","value_formatted":{"title":"Pengfei Ren","url":"https:\/\/pengfeiren96.github.io\/","target":"_blank"},"value":{"title":"Pengfei Ren","url":"https:\/\/pengfeiren96.github.io\/","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Haifeng Sun<\/a>","value_formatted":{"title":"Haifeng Sun","url":"https:\/\/scholar.google.com\/citations?user=dwhbTsEAAAAJ&hl=en","target":"_blank"},"value":{"title":"Haifeng Sun","url":"https:\/\/scholar.google.com\/citations?user=dwhbTsEAAAAJ&hl=en","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Jingyu Wang<\/a>","value_formatted":{"title":"Jingyu Wang","url":"https:\/\/scholar.google.com\/citations?user=H441DjwAAAAJ","target":"_blank"},"value":{"title":"Jingyu Wang","url":"https:\/\/scholar.google.com\/citations?user=H441DjwAAAAJ","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Qi Qi<\/a>","value_formatted":{"title":"Qi Qi","url":"https:\/\/scholar.google.com\/citations?user=2W2h0SwAAAAJ","target":"_blank"},"value":{"title":"Qi Qi","url":"https:\/\/scholar.google.com\/citations?user=2W2h0SwAAAAJ","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Jianxin Liao<\/a>","value_formatted":{"title":"Jianxin Liao","url":"https:\/\/dblp.org\/pid\/60\/4951.html","target":"_blank"},"value":{"title":"Jianxin Liao","url":"https:\/\/dblp.org\/pid\/60\/4951.html","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}}],"value":[{"field_687f0c15f3394":{"title":"Xiaozheng Zheng","url":"https:\/\/scholar.google.com\/citations?user=3hSD41oAAAAJ","target":"_blank"}},{"field_687f0c15f3394":{"title":"Pengfei Ren","url":"https:\/\/pengfeiren96.github.io\/","target":"_blank"}},{"field_687f0c15f3394":{"title":"Haifeng Sun","url":"https:\/\/scholar.google.com\/citations?user=dwhbTsEAAAAJ&hl=en","target":"_blank"}},{"field_687f0c15f3394":{"title":"Jingyu Wang","url":"https:\/\/scholar.google.com\/citations?user=H441DjwAAAAJ","target":"_blank"}},{"field_687f0c15f3394":{"title":"Qi Qi","url":"https:\/\/scholar.google.com\/citations?user=2W2h0SwAAAAJ","target":"_blank"}},{"field_687f0c15f3394":{"title":"Jianxin Liao","url":"https:\/\/dblp.org\/pid\/60\/4951.html","target":"_blank"}}],"field":{"ID":2366,"key":"field_687f08dfb7e07","label":"\u4f5c\u8005","name":"writer","aria-label":"","prefix":"acf","type":"repeater","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"acfe_repeater_stylised_button":0,"layout":"row","pagination":0,"min":0,"max":0,"collapsed":"","button_label":"Add Row","rows_per_page":20,"_name":"writer","_valid":1,"sub_fields":[{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}]}},"\u4f1a\u8bae\u540d\u79f0":{"simple_value_formatted":"ISMAR","value_formatted":"ISMAR","value":"ISMAR","field":{"ID":53,"key":"field_6759c5b33fdb3","label":"\u4f1a\u8bae\u540d\u79f0","name":"\u4f1a\u8bae\u540d\u79f0","aria-label":"","prefix":"acf","type":"text","value":null,"menu_order":1,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"default_value":"\u586b\u5199\u4f1a\u8bae","maxlength":"","allow_in_bindings":1,"placeholder":"","prepend":"","append":"","_name":"\u4f1a\u8bae\u540d\u79f0","_valid":1}},"\u5e74":{"simple_value_formatted":"2021","value_formatted":"2021","value":"2021","field":{"ID":254,"key":"field_675b036a7706e","label":"\u5e74","name":"\u5e74","aria-label":"","prefix":"acf","type":"text","value":null,"menu_order":2,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"default_value":2024,"maxlength":"","allow_in_bindings":1,"placeholder":"","prepend":"","append":"","_name":"\u5e74","_valid":1}},"code":{"simple_value_formatted":"Code<\/a>","value_formatted":{"title":"Code","url":"https:\/\/github.com\/zxz267\/SAR","target":"_blank"},"value":{"title":"Code","url":"https:\/\/github.com\/zxz267\/SAR","target":"_blank"},"field":{"ID":54,"key":"field_6759c5dc3fdb4","label":"code","name":"code","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":3,"instructions":"","required":0,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"code","_valid":1}},"arxiv":{"simple_value_formatted":"","value_formatted":"","value":"","field":{"ID":55,"key":"field_6759c5f83fdb5","label":"arXiv","name":"arxiv","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":4,"instructions":"","required":0,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":0,"_name":"arxiv","_valid":1}},"pdf":{"simple_value_formatted":"PDF<\/a>","value_formatted":{"title":"PDF","url":"https:\/\/ieeexplore.ieee.org\/document\/9583792","target":"_blank"},"value":{"title":"PDF","url":"https:\/\/ieeexplore.ieee.org\/document\/9583792","target":"_blank"},"field":{"ID":56,"key":"field_6759c6b83fdb6","label":"pdf","name":"pdf","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":5,"instructions":"","required":0,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":0,"_name":"pdf","_valid":1}},"rank":{"simple_value_formatted":"CCF-B","value_formatted":"CCF-B","value":"CCF-B","field":{"ID":2316,"key":"field_686b28a2069eb","label":"\u4f1a\u8bae\/\u671f\u520a\u7ea7\u522b","name":"rank","aria-label":"","prefix":"acf","type":"text","value":null,"menu_order":6,"instructions":"","required":0,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"default_value":"CCF-A","maxlength":"","allow_in_bindings":0,"placeholder":"","prepend":"","append":"","_name":"rank","_valid":1}}},"_links":{"self":[{"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/posts\/796","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/comments?post=796"}],"version-history":[{"count":30,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/posts\/796\/revisions"}],"predecessor-version":[{"id":2543,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/posts\/796\/revisions\/2543"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/media\/797"}],"wp:attachment":[{"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/media?parent=796"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/categories?post=796"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/tags?post=796"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}