A real-time reconstructed 3D environment augmented with virtual objects rendered with correct occlusion

Publication Title
IEEE Xplore
Publication/Creation Date
October 22 2014
Saad R. Khattak (contributor)
Brent Cowan (contributor)
Iuliia Chepurna (contributor)
Brent Cowan (creator)
Iuliia Chepurna (creator)
Andrew Hogue (creator)
Saad R. Khattak (creator)
Persuasive Intent
In this work, a novel framework is presented for the real-time interaction with 3D models in augmented virtual reality. This framework incorporates view-dependent stereoscopic rendering of the reconstructed environment together with user's hands and a virtual object, and high-precision gesture recognition to manipulate it. Proposed setup consists of a Creative RGB-D camera, Oculus Rift VR head mounted display (HMD), Leap Motion hands and fingers tracker and an AR marker. The system is capable of augmenting the user's hands in relation to their point of view (POV) using the depth sensor mounted on the HMD, and allows manipulation of the environment through the Leap Motion sensor. The AR marker is used to determine the location of the Leap Motion sensor to help with consolidation of transformations between the Oculus and the Leap Motion sensor. Combined with accurate information from the Oculus HMD, the system is able to track the user's head and fingers, with 6-DOF, to provide a spatially accurate augmentation of the user's virtual hands. Such an approach allows us to achieve high level of user immersion since the augmented objects occlude the user's hands properly; something which is not possible with conventional AR. They hypothesize that users of our system will be able to perform better object manipulation tasks in this particular augmented VR setup as compared to virtual reality (VR) where user's hands are not visible, or if visible, always occlude virtual objects.
HCI Platform
Discursive Type
Location on Body
Finger, Hand, Eye

Date archived
September 14 2015
Last edited
April 4 2018
How to cite this entry
Brent Cowan, Iuliia Chepurna, Andrew Hogue, Saad R. Khattak. (October 22 2014). "A real-time reconstructed 3D environment augmented with virtual objects rendered with correct occlusion". IEEE Xplore. IEEE. Fabric of Digital Life. https://fabricofdigitallife.com/index.php/Detail/objects/1207