BUPT NIRC CV LAB

A3-Net: Calibration-Free Multi-View 3D Hand Reconstruction for Enhanced Musical Instrument Learning

IJCAI
2025

Geng Chen∗ , Xufeng Jian∗ , Yuchen Chen , Pengfei Ren† , Jingyu Wang† , Haifeng Sun , Qi Qi , Jing Wang and Jianxin Liao

State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications

Extrinsic-calibration-free multi-view 3D hand reconstruction for musical instrument playing in everyday scenario. Players can capture multi-view data using their daily devices, which can be either fixed or handheld.

Abstract

Precise 3D hand posture is essential for learning musical instruments. Reconstructing highly precise 3D hand gestures enables learners to correct and master proper techniques through 3D simulation and Extended Reality. However, exsiting methods typically rely on precisely calibrated multicamera systems, which are not easily deployable in everyday environments. In this paper, we focus on calibration-free multi-view 3D hand reconstruction in unconstrained scenarios. Establishing correspondences between multi-view images is particularly challenging without camera extrinsics. To address this, we propose A3-Net, a multi-level alignment framework that utilizes 3D structural representations with hierarchical geometric and explicit semantic information as alignment proxies, facilitating multi-view feature interaction in both 3D geometric space and 2D visual space. Specifically, we first perfrom global geometric alignment to map multi-view features into a canonical space. Subsequently, we aggregate information into predefined sparse and dense proxies to further integrate cross-view semantics through mutual interaction. Finnaly, we perfrom 2D alignment to align projected 2D visual features with 2D observations. Our method achieves state-of-the-art results in task of multi-view 3D hand reconstruction, demonstrating the effectiveness of the proposed framework.

Video

Overview

Comparison with other methods (a) and overview of the proposed alignment-proxy based A3-Net (b), which leverages predefined alignment proxies (meshes and joints) to build multi-view feature correspondence semantically and geometrically. It first leverages an encoder-decoder model for initial hand pose estimation and 2D visual feature extraction within each view. Geometric alignment are applied on meshes and joints to project them into the canonical space. Finally, it iteratively adopts multi-view feature alignment in both 3D point cloud space and 2D image space. For simplicity, only one iteration is shown.

Comparison with SOTA

BUPT NIRC CV LAB

Pengfei Ren, Jingyu Wang, Jianxin Liao

Contact Us