Mobile Augmented Reality

Computer Vision & Geovisualization

Project Name: Research on Spatial Information Visualization and Multi-modal Interaction Mechanism Based on Outdoor Augmented Reality

Funding Source

National Natural Science Foundation of China
Project No. 41371427

My Role

Core Member
(October 2016 - June 2019)

My Contribution

Data Collection & Preprocessing
Deep Neural Network Model
Android Development
Paper Writing

Reference:

Rao, J.✉, Gao, S.*, Kang, Y. and Du, Q. (2020). Landmarks as Beacons: Pedestrian Navigation Based on Landmark Detection and Mobile Augmented Reality. AutoCarto 2020.

Rao, J., Qiao, Y., Ren, F., Wang, J. and Du, Q.*✉ (2017). A Mobile Outdoor Augmented Reality Method Combining Deep Learning Object Detection and Spatial Relationships for Geovisualization. Sensors, 17(9), 1951.

Qiao, Y., Rao, J., Wang, J., Du, Q.*✉ and Ren, F. (2017). Geographic Object Detection for Outdoor Augmented Reality. Geomatics World, 5, 011.

Introduction

The purpose of this study was to develop a robust, fast and markerless mobile augmented reality method for registration, geovisualization, and interaction in uncontrolled outdoor environments. We propose a lightweight deep-learning-based object detection approach for mobile or embedded devices; the vision-based detection results of this approach are combined with spatial relationships by means of the host device’s built-in Global Positioning System receiver, Inertial Measurement Unit, and magnetometer. Virtual objects generated based on geospatial information are precisely registered in the real world, and an interaction method based on touch gestures is implemented. The entire method is independent of the network to ensure robustness to poor signal conditions.

A mobile prototype system based on Android was developed and tested on the Wuhan University campus to evaluate the method and validate its results. The findings demonstrate that our method achieves a high detection accuracy, stable geovisualization results and interaction. Lately, we experimentally integrated a high-speed tracking algorithm (Kernelized Correlation Filters, KCF) in the latest system, and it can run at approximately 10 FPS currently.

UPDATE: Recently, we proposed an augmented reality pedestrian navigation method that uses landmarks as beacons to guide users’ wayfinding in urban areas. Specifically, landmarks are detected using vision-based models and then augmented by virtual arrows and panels with navigation information.