{"id":785,"date":"2023-12-01T20:04:56","date_gmt":"2023-12-01T12:04:56","guid":{"rendered":"https:\/\/511cvlab.sinkers.cn\/?p=785"},"modified":"2025-10-17T17:14:31","modified_gmt":"2025-10-17T09:14:31","slug":"sa-fusion","status":"publish","type":"post","link":"https:\/\/cv.nirc.top\/zh\/2023\/sa-fusion\/","title":{"rendered":"SA-Fusion: Multimodal Fusion Approach for Web-based Human-Computer Interaction in the Wild"},"content":{"rendered":"
\n

1<\/sup>Beijing University of Posts and Telecommunications, 2<\/sup>State Key Laboratory of Networking and Switching Technology, 3<\/sup>China Mobile Research Institute<\/p>\n\n\n\n

\n \n
\n \n
\n \n \n <\/path>\n <\/svg>\n <\/div>\n
Paper<\/div>\n <\/a>\n <\/div>\n\n\n\n\n <\/div>\n<\/div>\n\n\n\n

Abstract<\/h2>\n\n\n\n

Web-based AR technology has broadened human-computer interaction scenes from traditional mechanical devices and flat screens to the real world, resulting in unconstrained environmental challenges such as complex backgrounds, extreme illumination, depth range differences, and hand-object interaction. The previous hand detection and 3D hand pose estimation methods are usually based on single modality such as RGB or depth data, which are not available in some scenarios in unconstrained environments due to the differences between the two modalities. To address this problem, we propose a multimodal fusion approach, named Scene-Adapt Fusion (SA-Fusion), which can fully utilize the complementarity of RGB and depth modalities in web-based HCI tasks. SA-Fusion can be applied in existing hand detection and 3D hand pose estimation frameworks to boost their performance, and can be further integrated into the prototyping AR system to construct a web-based interactive AR application for unconstrained environments. To evaluate the proposed multimodal fusion method, we conduct two user studies on CUG Hand and DexYCB dataset, to demonstrate its effectiveness in terms of accurately detecting hand and estimating 3D hand pose in unconstrained environments and hand-object interaction.<\/p>\n\n\n\n

Overview<\/h2>\n\n\n\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
The design of
web-based human-computer interactive AR applications<\/strong> for unconstrained environments. The multimodal hand detection framework as well as the 3D hand pose estimation framework adopts the two parallel CNN streams with multiple SA-Fusion modules.<\/figcaption><\/figure>\n\n\n\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
The design of integrating the proposed SA-Fusion module into two parallel CNNs streams.<\/figcaption><\/figure>\n\n\n\n

Qualitative results<\/h2>\n\n\n\n
\n
\n
\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n
\n
\n

(a) AWR<\/p>\n<\/div>\n\n\n\n

\n

(b) Ours<\/p>\n<\/div>\n\n\n\n

\n

(c) GT<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\"\"\/\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n\n\n
\n
\n

(a) AWR<\/p>\n<\/div>\n\n\n\n

\n

(b) Ours<\/p>\n<\/div>\n\n\n\n

\n

(c) GT<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

Qualitative results of the single-modal AWR and our multimodal method integrated with SA-Fusion on DexYCB test dataset.<\/p>\n\n\n\n

The above results illustrate that the proposed SA-Fusion can efectively improve the performance of existing 3D hand pose estimation methods in hand-object interaction scenarios.<\/p>\n\n\n\n

Comparison with SOTA<\/h2>\n\n\n\n

User Study 1: Hand Detection<\/strong><\/h3>\n\n\n\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Comparison of AP
50<\/sub> with other RGB-D fusion methods in the back-light case<\/strong>, dark case<\/strong>, and overall cases<\/strong> of the CUG Hand test set<\/strong>. In addition to the RGB-D detection method based on the yolov5s framework, the results of YOLOv5s with only RGB and only depth images as input are also listed for reference. Our model achieves leading AP50<\/sub> and AP50:95<\/sub>, reaching 82.1% and 46.1% respectively.<\/figcaption><\/figure>\n\n\n\n

User Study 2: 3D Hand Pose Estimation<\/strong><\/h3>\n\n\n\n
\"\"\n\t\t\t\n\t\t\t\t\n\t\t\t<\/svg>\n\t\t<\/button>
Comparison with other SOTA methods of the MPJPE (mm) on
DexYCB<\/strong> dataset. By integrating SA-Fusion into the backbone of AWR, our model achieves the lowest Mean Per Joint Position Error (MPJPE).<\/figcaption><\/figure>\n\n\n\n

Conclusion<\/h2>\n\n\n\n

\u2705 We propose Scenes-Adapt Fusion<\/strong>, a multimodal fusion approach that can fully utilize the complementarity of RGB and depth modalities in unconstrained environments, to solve the problem of poor applicability of previous single-modal HCI methods in real-world scenarios. By applying the proposed SA-Fusion module to existing hand detection and 3D hand pose estimation frameworks, their performance has been significantly improved.<\/p>\n\n\n\n

\u2705 Furthermore, we build a web-based interactive AR application<\/strong> by integrating the proposed multimodal HCI frameworks into the prototyping AR system.<\/p>\n\n\n\n

Bibtex<\/h2>\n\n\n\n
@inproceedings{liu2023sa,\n  title={SA-Fusion: Multimodal Fusion Approach for Web-based Human-Computer Interaction in the Wild},\n  author={Liu, Xingyu and Ren, Pengfei and Chen, Yuchen and Liu, Cong and Wang, Jing and Sun, Haifeng and Qi, Qi and Wang, Jingyu},\n  booktitle={Proceedings of the ACM Web Conference 2023},\n  pages={3883--3891},\n  year={2023}\n}<\/code><\/pre>\n<\/div>\n\n\n\n
<\/p>","protected":false},"excerpt":{"rendered":"
1Beijing University of Posts and Telecommunications, 2State Key Laboratory of Networking and Switching Technology, 3China Mobile Research Institute Paper Abstract Web-based AR technology has broadened human-computer interaction scenes from traditional mechanical devices and flat screens to the real world, resulting in unconstrained environmental challenges such as complex backgrounds, extreme illumination, depth range differences, and hand-object […]<\/p>\n","protected":false},"author":4,"featured_media":786,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":"","_links_to":"","_links_to_target":""},"categories":[16],"tags":[],"class_list":["post-785","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-depth-based-3d-hand-pose-estimation"],"acf":{"writer":{"simple_value_formatted":"This data type is not supported! Please contact the author for help.<\/em><\/code>","value_formatted":[{"writer_link":{"simple_value_formatted":"Xingyu Liu<\/a>","value_formatted":{"title":"Xingyu Liu","url":"https:\/\/scholar.google.com\/citations?hl=zh-CN&user=xTd_jR0AAAAJ","target":"_blank"},"value":{"title":"Xingyu Liu","url":"https:\/\/scholar.google.com\/citations?hl=zh-CN&user=xTd_jR0AAAAJ","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Pengfei Ren<\/a>","value_formatted":{"title":"Pengfei Ren","url":"https:\/\/pengfeiren96.github.io\/","target":"_blank"},"value":{"title":"Pengfei Ren","url":"https:\/\/pengfeiren96.github.io\/","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Yuchen Chen<\/a>","value_formatted":{"title":"Yuchen Chen","url":"https:\/\/ieeexplore.ieee.org\/author\/37089349401","target":"_blank"},"value":{"title":"Yuchen Chen","url":"https:\/\/ieeexplore.ieee.org\/author\/37089349401","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Cong Liu<\/a>","value_formatted":{"title":"Cong Liu","url":"http:\/\/Cong Liu","target":"_blank"},"value":{"title":"Cong Liu","url":"http:\/\/Cong Liu","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Jing Wang<\/a>","value_formatted":{"title":"Jing Wang","url":"http:\/\/Jing Wang","target":"_blank"},"value":{"title":"Jing Wang","url":"http:\/\/Jing Wang","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Haifeng Sun<\/a>","value_formatted":{"title":"Haifeng Sun","url":"https:\/\/scholar.google.com\/citations?user=dwhbTsEAAAAJ","target":"_blank"},"value":{"title":"Haifeng Sun","url":"https:\/\/scholar.google.com\/citations?user=dwhbTsEAAAAJ","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Qi Qi<\/a>","value_formatted":{"title":"Qi Qi","url":"https:\/\/scholar.google.com\/citations?user=2W2h0SwAAAAJ","target":"_blank"},"value":{"title":"Qi Qi","url":"https:\/\/scholar.google.com\/citations?user=2W2h0SwAAAAJ","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}},{"writer_link":{"simple_value_formatted":"Jingyu Wang<\/a>","value_formatted":{"title":"Jingyu Wang","url":"https:\/\/scholar.google.com\/citations?user=H441DjwAAAAJ","target":"_blank"},"value":{"title":"Jingyu Wang","url":"https:\/\/scholar.google.com\/citations?user=H441DjwAAAAJ","target":"_blank"},"field":{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}}}],"value":[{"field_687f0c15f3394":{"title":"Xingyu Liu","url":"https:\/\/scholar.google.com\/citations?hl=zh-CN&user=xTd_jR0AAAAJ","target":"_blank"}},{"field_687f0c15f3394":{"title":"Pengfei Ren","url":"https:\/\/pengfeiren96.github.io\/","target":"_blank"}},{"field_687f0c15f3394":{"title":"Yuchen Chen","url":"https:\/\/ieeexplore.ieee.org\/author\/37089349401","target":"_blank"}},{"field_687f0c15f3394":{"title":"Cong Liu","url":"http:\/\/Cong Liu","target":"_blank"}},{"field_687f0c15f3394":{"title":"Jing Wang","url":"http:\/\/Jing Wang","target":"_blank"}},{"field_687f0c15f3394":{"title":"Haifeng Sun","url":"https:\/\/scholar.google.com\/citations?user=dwhbTsEAAAAJ","target":"_blank"}},{"field_687f0c15f3394":{"title":"Qi Qi","url":"https:\/\/scholar.google.com\/citations?user=2W2h0SwAAAAJ","target":"_blank"}},{"field_687f0c15f3394":{"title":"Jingyu Wang","url":"https:\/\/scholar.google.com\/citations?user=H441DjwAAAAJ","target":"_blank"}}],"field":{"ID":2366,"key":"field_687f08dfb7e07","label":"\u4f5c\u8005","name":"writer","aria-label":"","prefix":"acf","type":"repeater","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"acfe_repeater_stylised_button":0,"layout":"row","pagination":0,"min":0,"max":0,"collapsed":"","button_label":"Add Row","rows_per_page":20,"_name":"writer","_valid":1,"sub_fields":[{"ID":2368,"key":"field_687f0c15f3394","label":"\u4f5c\u8005\u4e0e\u4f5c\u8005\u4e3b\u9875","name":"writer_link","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":0,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":2366,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"writer_link","_valid":1,"parent_repeater":"field_687f08dfb7e07"}]}},"\u4f1a\u8bae\u540d\u79f0":{"simple_value_formatted":"WWW","value_formatted":"WWW","value":"WWW","field":{"ID":53,"key":"field_6759c5b33fdb3","label":"\u4f1a\u8bae\u540d\u79f0","name":"\u4f1a\u8bae\u540d\u79f0","aria-label":"","prefix":"acf","type":"text","value":null,"menu_order":1,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"default_value":"\u586b\u5199\u4f1a\u8bae","maxlength":"","allow_in_bindings":1,"placeholder":"","prepend":"","append":"","_name":"\u4f1a\u8bae\u540d\u79f0","_valid":1}},"\u5e74":{"simple_value_formatted":"2023","value_formatted":"2023","value":"2023","field":{"ID":254,"key":"field_675b036a7706e","label":"\u5e74","name":"\u5e74","aria-label":"","prefix":"acf","type":"text","value":null,"menu_order":2,"instructions":"","required":1,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"default_value":2024,"maxlength":"","allow_in_bindings":1,"placeholder":"","prepend":"","append":"","_name":"\u5e74","_valid":1}},"code":{"simple_value_formatted":"","value_formatted":"","value":"","field":{"ID":54,"key":"field_6759c5dc3fdb4","label":"code","name":"code","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":3,"instructions":"","required":0,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":1,"_name":"code","_valid":1}},"arxiv":{"simple_value_formatted":"","value_formatted":"","value":"","field":{"ID":55,"key":"field_6759c5f83fdb5","label":"arXiv","name":"arxiv","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":4,"instructions":"","required":0,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":0,"_name":"arxiv","_valid":1}},"pdf":{"simple_value_formatted":"PDF<\/a>","value_formatted":{"title":"PDF","url":"https:\/\/dl.acm.org\/doi\/abs\/10.1145\/3543507.3587429","target":"_blank"},"value":{"title":"PDF","url":"https:\/\/dl.acm.org\/doi\/abs\/10.1145\/3543507.3587429","target":"_blank"},"field":{"ID":56,"key":"field_6759c6b83fdb6","label":"pdf","name":"pdf","aria-label":"","prefix":"acf","type":"link","value":null,"menu_order":5,"instructions":"","required":0,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"return_format":"array","allow_in_bindings":0,"_name":"pdf","_valid":1}},"rank":{"simple_value_formatted":"CCF-A","value_formatted":"CCF-A","value":"CCF-A","field":{"ID":2316,"key":"field_686b28a2069eb","label":"\u4f1a\u8bae\/\u671f\u520a\u7ea7\u522b","name":"rank","aria-label":"","prefix":"acf","type":"text","value":null,"menu_order":6,"instructions":"","required":0,"id":"","class":"","conditional_logic":0,"parent":52,"wrapper":{"width":"","class":"","id":""},"default_value":"CCF-A","maxlength":"","allow_in_bindings":0,"placeholder":"","prepend":"","append":"","_name":"rank","_valid":1}}},"_links":{"self":[{"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/posts\/785","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/comments?post=785"}],"version-history":[{"count":26,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/posts\/785\/revisions"}],"predecessor-version":[{"id":2549,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/posts\/785\/revisions\/2549"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/media\/786"}],"wp:attachment":[{"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/media?parent=785"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/categories?post=785"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cv.nirc.top\/zh\/wp-json\/wp\/v2\/tags?post=785"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}