Hybrid Positioning Approach for Context-Aware Indoor Navigation of Visually Impaired People

It is difficult for blind persons to be comfortable in places where they cannot better perceive the environment due to the difficulty in understanding where they are, what is around them and how to get somewhere safely. Context-aware navigation considers the user's surroundings and locations to create a better experience for the navigator and provide better options. Due to the unavailability of global positioning systems (GPS) in indoor environments, there is an open problem in investigating an accurate indoor localization method for better context-aware navigation for visually impaired persons [1].

Vision-based approach [2] and the geolocation-based approach [3] are two main approaches to context-aware indoor blind navigation. The vision-based approach has limitations with processing power and lighting conditions. Therefore, this study is focused on the geolocation-based approach.

Geomagnetic field-based indoor navigation systems are one of the latest research areas in visually impaired indoor navigation coming under the geolocation-based approach. This approach is using magnetometers to measure magnetic field variations, which determine the position of a person or object. Geomagnetic fields are freely available. Many recent literature reports consist of magnetic field-based visually impaired navigation approaches using fingerprinting [4]. Riehle et al. [3] presented an indoor navigation system for visually impaired persons using geomagnetic fingerprinting consisting of a prototype of a wireless magnetometer placed at the users’ hip streaming magnetic readings to a smartphone. However, the geomagnetic field cannot be used alone due to less uniqueness between magnetic fingerprint readings of different locations.

There are many radio frequency-based indoor positioning technologies and methodologies for indoor localization, such as Wi-Fi, Bluetooth low energy (BLE), UWB (ultra wide band), and visible light communication. BLE-based positioning can be effectively used in visually impaired navigation systems. One of the most recent pragmatic moves for indoor navigation is the development of “indoor beacons.” These technologies can communicate with the smartphone in real-time and identify the current location of the mobile device holder. Sato et al. [5] have presented the smartphone-based indoor blind navigation called NavCog3, which uses smartphone-inbuilt inertial measurement unit (IMU) and BLE signals for absolute indoor positioning. Consequently, Cheraghi et al. [6] have presented an indoor wayfinding system called GuideBeacon for blind and visually impaired people. Nevertheless, BLE signals usually do not have stable RSS (Receive Signal Strength) values [7]. Moreover, using BLE signals alone causes less positioning accuracy. Measurements of beacons are variants with the time and the battery power of the beacons and fingerprinting databases need to update with those variations rapidly [7]. Therefore, there is an open problem in increasing indoor localization accuracy through a hybrid positioning method by combining a few positioning approaches.

Augmented reality (AR) improves the traveling experience in real-time using mobile devices. Navigation applications with location-based AR dramatically improve the effectiveness of navigation. AR combines real and virtual worlds through accurate 3D registration of virtual and real objects by establishing real-time interaction. Location-based AR is geo-based AR that relies on GPS, accelerometer, digital compass, and other technologies to identify a device's location and position. Location-based AR can be divided into outdoor and indoor. Outdoor AR uses GPS, while indoor recognizes the current location of a mobile device with other indoor positioning geo markers called POI (Point of Interest) to identify the device's location. Wikitude [8] is such location-based AR technology. Wikitude has been developing AR technology since 2008 and is responsible for launching the world's first pedestrian and car navigation system that integrated an AR display—eliminating the need for a map. The Wikitude AR SDK includes convenient features that make geo-referenced data easy to work with. Depending on the use case, the sensor-based location tracking will be activated via GPS, compass, accelerometer, network, or beacon.

Feedback on the current context should be given to the visually impaired navigator to identify the surrounding environment. Audio and tactile feedback methods are most common among visually impaired persons due to the limitations in vision. Among them, tactile feedback is preferred since it does not block the environmental sound cues. Martin Pielot et al. [9] have explored tactile sensation as a modality to present information for orientation and navigation. These tactile displays are located around the user's waist, which uses the body location of tactors to express directions to the visually impaired or blind. Straub et al. [10] experimented with a system that uses vibrotactile guidance cues to allow users to reach their destination.

Most researchers have shown that context-awareness is important to blind navigation [11]. However, literature corresponding to context awareness has limitations in accuracy due to poor indoor positioning [12]. Plikynas et al. mentioned that the indoor positioning accuracy can be increased by fusing the advantages of two or more (Hybrid) positioning technologies [12]. Therefore, this study focus on increasing indoor positioning accuracy for visually impaired navigation using a hybrid positioning approach.

The design science methodology [13] is selected as the research methodology for this research. Following are the assumptions of the proposed approach: (1) The prototype is evaluated within selected in-house areas with predefined POIs and routes. (2) Obstacle detection and avoidance are not considered. Therefore, the experimental area is prepared as an obstacle-free area. (3) Only a single floor in a building is considered. (4) Landmarks such as rooms, elevators, stairs, and doors are considered for context awareness.

The proposed approach consists of four main modules, as shown in Fig. 1: indoor localization, context detection, wayfinding, and feedback.

Visually impaired indoor localization based on BLE beacons and geomagnetic fields is considered in this research.

Then, the coordinates of BLE signal transmitters can be used to calculate the mobile device's location coordinates using the trilateration method. Finally, estimated location coordinates (x, y, and z) are converted into geolocation coordinates.

Geomagnetic Field-based Localization—Geomagnetic field-based localization consists of two phases: an offline phase and an online phase. Magnetic fingerprints collected during the offline phase are stored in a database. Moreover, the location coordinates obtained by the BLE beacons are inputted in the process of finding the location based on magnetic field fingerprinting as a filtering technique. This filtering process eliminates fingerprints that are far away from the target location. This approach also reduces the computational complexity in fingerprint matching during the online phase.

There are some noises when obtaining magnitude values from the smartphone magnetometer due to hardware issues. The moving average filter is used to further smoothen the fingerprint sequence to eliminate noises.

Then, the obtained smoothened magnetic field reading sequence (obtained during the online phase) and filtered pre-stored magnetic fingerprints (obtained during the offline phase) can be matched with the dynamic time wrapping (DTW) algorithm [14]. Then, the relevant location coordinates can be extracted by using the minimum distance of the fingerprint matching.

Nevertheless, the earlier version of the DTW algorithm cannot be used with real-time scenarios because there is no completed time series to match with the warping function of DTW. Therefore, open-end and open begin variations of the DTW algorithm are used to obtain optimal matching sub-sequence.

The Haversine [15] distance measurement is used to calculate the distance between the location points obtained during online and offline phases. By using the Haversine distance formula, a distance between two geo-locations can be calculated without any conversion.

Geo-positioning and AR is required to implement location-based AR applications. The exact position of the smartphone is to determine by identifying geo markers using the mobile application. When the app identifies a specific point of interest, it triggers AR elements which are placed on top of the natural environment.

Location coordinates obtained by the proposed hybrid positioning approach and the pre-stored point of interest database are processed with the smartphone inertial sensor readings.

The field of view (FOV) is a concept of filtering places according to the view of the camera. Nearby places and landmarks are filtered using the FOV concept in order to reduce the cognitive load associated with the information provided to visually impaired people. FOV can filter nearby places using users’ heading directions.

Once a user selects a destination place by a given set of POIs, the navigation and wayfinding engine processes the path from the current position to the destination. The navigation engine selects the shortest possible path, calculates the orientation and directions that the user should move according to the path drawn, and gives the routing directions to the feedback module.

The feedback method consists of both voice feedback and tactile feedback. According to the literature, tactile feedback is the most preferred way to bypass the information to blind subjects. Moreover, voice feedback is selected as a secondary feedback method. Output texts are passed to a text-to-speech engine, and it generates audio output.

It is required to create a fingerprinting map (database) containing magnetic field intensity values during the offline phase. An android application is developed to capture the indoor magnetic distortions using a smartphone-inbuilt magnetometer and implemented in the Samsung Galaxy Note 3 Android device as depicted in Fig. 2.

Calibration of the magnetometer sensor is performed to eliminate hard iron effects and soft iron effects from the measured magnetic field of the magnetometer sensor.

Location coordinates of each fingerprint value have been obtained using indoorAtlas Map Creator 2 tool [16]. The experiment map and the fingerprint points of the experimental area is shown in Fig. 3. Collected magnetic field magnitude fingerprints are preprocessed with the moving average mean filter, and the obtained fingerprint values are stored as JSON Objects in a JSON file.

BLE beacons are small electronic devices to obtain BLE signals. The iBK3S105 is selected for the POC prototype in this study. The selected beacon supports both iBeacon and Eddystone protocols [7]. Four beacons are placed in the experimental area, as depicted in Fig. 4. Location coordinates of each beacon were obtained through the indoorAtlasMapCreator tool. The iBKS beacon configuration tool provided by the manufacturer of the beacons can configure beacons. The iBeacon protocol is selected as the advertising protocol for this study. The advertising interval is selected as 850 ms, which is the time delay between two advertising packets.

For this study, maximum power +4 dBm is used since accuracy is a significant point. Then, the calibration power set is −58 dBm, which represents the BLE signal strength at the 1 m distance to the beacon. This value is used to calculate the distance between the beacon and the receiver. Then, the mac address, primary id, and minor id are set, respectively.

The proposed hybrid localization approach combines BLE signal-based localization and magnetic fingerprint-based localization.

Bluetooth Low Energy Signal-Based Localization—An open-source Android library called neXenio-BLE-Indoor Positioning [17] is used. The distance to the beacon method has been overridden by using the log-distance path loss model. The trilateration method is used to estimate the location coordinates using the measured distances and the location coordinates of beacons.

Magnetic Fingerprint Filtering and Matching—The computation complexity of the DTW algorithm is high since the fingerprint matching process is not feasible to be done on the smartphone due to low computation resources. Therefore, magnetic fingerprint-based localization is implemented on a python server using the Django rest framework based on python language. Collected magnetic field magnitudes for each location were stored as JSON objects. Nevertheless, the collected magnetic fingerprints suffered from noises due to hardware issues such as hard iron and soft iron issues. Moving average filter has been used to reduce such noises. Location points that are obtained from the BLE beacons are used to filter the magnetic fingerprints. This filtering has reduced the time complexity of the fingerprint matching algorithm. Then, the filtered magnetic fingerprints and the magnetic field magnitude reading sequence, coming from the smartphone magnetometer, are matched with filtered fingerprints using an open-ended DTW algorithm to find the minimum distance between two magnetic field value sequences. DTW-python package for python environment is to implement open-end and openly begin variations of DTW algorithm.

Context detection is achieved through geolocation-based AR. Wikitude geolocation-based AR platform is selected for the POC prototype implementation. Location coordinates obtained from the hybrid localization approach are fed into Wikitude SDK. Then, the Wikitude API gets the current location of the android device and the orientation using the inertial sensors of the smartphone. According to the mobile device location and the orientation, Wikitude API calculates the FOV and places the relevant landmarks retrieved from the pre-stored POIs database.

A ray is calculated from the user's viewpoint to the POI to load a place on the device screen. Also, the mobile device orientation is retrieved from the built-in gyroscope. The point on the screen where the mobile device screen intersects the ray is then calculated. This point is on the screen, and it is used to draw the text label of the POI.

Indoor wayfinding by drawing a path from the current location to the desired destination is based on IndoorAtlas API and the google maps platform. To use IndoorAtlas’ wayfinding API, it is required to upload the indoor map and draw the routes using the IndoorAtlas map creation tool. The destination place needs to be marked on the map using a marker. Then, the route from the user's current location to the destination is shown in the line segment as depicted in Fig. 5. After that, the wayfinding system guides the user according to the polyline created in blue color. The polyline creation is done by retrieving all the waypoints between two markers. It calculates the distance between waypoints and rotation from the current direction to the predefined route to guide the user.

POC prototype contains a wearable tactile belt consisting of five coin vibration motors, a NodeMCU board, a battery pack, and 40 signal lights. The front vibrator gives the command to go forward in one vibration period and gives the command to turn back in continuous two vibrate periods. Left and right vibrators are used to give commands to turn left and turn right, and two middle vibrators are used to give commands to turn a little left and turn a little right accordingly. Stop command is given to all five vibrators simultaneously.

The NodeMCU board is programmed using an Arduino IDE with the codes for activating and deactivating vibration motors. The inbuilt ESP8266 Wi-Fi module does the NodeMCU and the smartphone communication in the NodeMCU 41 board. The Voice feedback module is used to give the same feedback given by tactile feedback modules, but this is used to assure the feedback given. Clear short sentences are given to the user via the Android Text-to-Speech (TTS) engine in the vocale of the UK English language.

When the smartphone is placed on the feedback belt, heading directions are not performed well using the inbuilt inertial sensors. Therefore, the smartphone's rear camera is used and flipped the smart device to the opposite side, as shown in Fig. 6. In this research, multiple sensors of the smartphone are being utilized continuously. Therefore, the temperature of the device increased with time. Due to this temperature, smartphone sensor accuracy can be changed. Therefore, each sensor is calibrated and clears the random access memory of the device before each experiment.

The POC prototype is evaluated with three blindfolded subjects, including two male subjects and one female subject. All experiments were done in accordance with the declaration of Helsinki and its later amendments [18]. All participants had normal hearing abilities, and they do not have any other specific disabilities. Six different indoor locations are selected as experimental environments. Module-wise evaluation is conducted to identify the strengths and weaknesses of each module: localization, context awareness, wayfinding, and feedback.

BLE beacon-based localization approach has given an average 9.45 m error. BLE signals did not provide stable signal strength values with time. BLE signal and magnetic field distortion-based hybrid localization algorithm have given an average 2.72 m distance error.

Figure 7 shows the graphical representation of the distance errors of the proposed hybrid localization approach and the BLE beacon-based approach. Sometimes, the hybrid distance error becomes high because of weak geomagnetic field distortions near the location due to low metallic infrastructure in experimental areas. In such situations, more accurate results can be expected from modern buildings containing a considerable amount of metallic infrastructure.

Three blindfolded subjects are asked to select a destination using voice commands to evaluate the context detection module. If it identifies the destination and draws a navigation path to the correct destination, it is considered a hit and otherwise a miss. Each subject is asked to select destinations ten times among three destinations randomly. The context detection approach has given a 78.6% average hit rate and 21.4% average miss rate.

In the evaluation of the wayfinding module, each subject is asked to walk according to the feedback that is given by the navigation prototype. A third party is asked to correct incorrect turns if a subject turned in a wrong direction, and all the results are recorded as videos, and the device screen is monitored and recorded using android mirror screen functionality. The number of correct turnings and wrong turnings is recorded, and then hit rate and miss rates are calculated.

The average hit rate is 75.19%, and the miss rate is 24.81%. According to the observed results, the miss rate of the navigation approach has slightly increased with the number of turning points. The miss rate is increased due to the walking speed of different users and localization errors.

Voice and tactile feedback are evaluated by making the subject walk according to the commands in a predefined path. Moreover, when evaluating the feedback, the voice input commands and voice are designed with single-word commands, and voice feedback has been created with short sentences to reduce cognitive load and minimize the word recognition error rate. Both, the voice feedback module and the tactile feedback modules, obtained the highest scores. Nevertheless, due to the voice recognition issues, the performance of the voice input module is low.

This research aims to investigate a novel hybrid indoor localization approach for context-aware blind indoor navigation. The hybrid localization approach is based on the geomagnetic fields and Bluetooth Low Energy. BLE beacon-based localization has been performed using the trilateration method. The magnetometer embedded in the smartphone has been used to obtain magnetic fingerprints in geomagnetic field-based localization. In order to eliminate fingerprints that are far away from the target location are filtered by using location coordinates obtained by BLE beacons as an input during the process of finding magnetic fingerprints. DTW algorithm is used to match the measures obtained during the online phase against the fingerprint database. The Haversine distance measurement is used to find the distance between true and estimated location. The location coordinates obtained through the hybrid approach are input to perform context detection through geolocation-based AR. Contextual information such as nearby locations and landmarks are obtained using geolocation-based AR with a pre-deployed map and Point of Interest. The feedback method consists of both voice feedback and tactile feedback. POC prototype contains a wearable tactile belt consisting of sensors, feedback units, and processors. The average distance error of BLE beacon-based localization is estimated as 9.45 m. The average distance error of the hybrid localization algorithm has given an average 2.72 m distance error. Evaluation of the context detection module has given a 78.6% average hit rate and 21.4% average miss rate. Evaluation of the wayfinding module gives the average hit rate of 75.19%, and the miss rate of 24.81%. Usability evaluation results show that the proposed algorithm improves the context-awareness of the visually impaired indoor navigation system. As a future extension, the work can be extended for a whole building using barometer readings to detect multiple floor levels. Moreover, the usability of the proposed approach can be improved by personalizing the prototype according to each user's characteristics.

There are no conflicts of interest. All procedures performed for studies involving human participants were in accordance with the ethical standards stated in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Informed consent was obtained from all participants. Documentation was provided upon request. Informed consent is not applicable. This article does not include any research in which animal participants were involved.

The authors attest that all data for this study are included in the paper.