Vision-Based Three Dimensional Hand Interaction In Markerless Augmented Reality Environment
Loading...
Date
2015-05
Authors
Ng, Kah Pin
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
The advent of augmented reality (AR) enables virtual objects to be superimposed on the real world and provides a new way to interact with the virtual objects. AR system requires an indicator to determine for how the virtual objects aligned in the real world. The indicator must first be obtained to access to a particular AR system. It may be inconvenient to have the indicator in reach at all time. Human hand, which is part of the human body may be a solution for this. Besides, hand is also a promising tool for interaction with virtual objects in AR environment. This thesis presents a markerless Augmented Reality system which utilizes outstretched hand for registration of virtual objects in the real environment and enables the users to have three dimensional (3D) interaction with the augmented virtual objects. To employ the hand for registration and interaction in AR, hand postures and gestures that the user perform has to be recognized. Computer vision algorithms are used in detecting the bare hand without the assistance of markers or any other devices such as mechanical devices and magnetic devices. Stereo camera is used to capture video images, so that the depth information of the hand can be computed by using the stereovision approach. Skin color segmentation is employed to segment the hand region from the images and hand features are then extracted from the segmented region, so that the features can be used for the recognition of different hand postures. Instead of fiducial markers, the palm center and fingertips of the outstretched hand are tracked in real time. Incorporating the depth information computed, 3D positions of these hand features are then used to estimate the 6DOF camera pose with respect to the hand, which in turn allows the virtual objects to be augmented onto the palm accurately. A few simple gestures are mapped for the interaction with the virtual objects. The developed markerless AR system enables the users to inspect the virtual objects from different view angles by simply moving the hand and to select and translate the virtual objects in 3D space with the other hand intuitively. The system achieves an interactive frame rate of 12 fps. The hand posture recognition algorithm developed is able to recognize the hand postures accurately, with overall recognition rate of 97.0%. The accuracy of the camera pose estimation is not very high, but it is sufficient to be used for the registration of virtual objects on the hand.
Description
Keywords
Mathematics