UbitourAR : A ubiquitous tourism model based on augmented reality

Nowadays, Brazil has an expected growth in the tourism sector due to the FIFA World Cup (which had a huge success) and the Olympics in Rio de Janeiro. However, there is a lack of tourism applications for Brazil, such as Electronic Tourism Guides (ETGs) for mobile devices. The existence of ETGs that takes into account real world environment is still very rare. By taking into account this scenario, this article presents a ubiquitous tourism model based on augmented reality. The proposed architecture, which is called UbitourAR, takes into account the context awareness focusing on localization-based services allied with user profile and restrictions. This article describes the development of UbitourAR enabling users to interact with the real world environment using augmented reality when visiting a place or point of interest (POI). In order to evaluate the model we had taken two approaches. Firstly, we evaluated the ontology proposed using scenarios. Secondly, we evaluated the usability in terms of efficiency, effectiveness and user satisfaction. For doing that, we developed a mobile application for the Caminhos de Pedra itinerary, in Bento Gonçalves, Brazil. The proposed model extends the previous work of Ubitour by adding augmented reality. The results of the tests showed a user acceptance higher than 80%, which encourages the research in augmented reality allied with ubiquitous tourism.


Introduction
With the expected tourism growth in Brazil, especially because the recently realized FIFA World Cup 2014, and the forthcoming Olympics, which will be held in 2016, in Rio de Janeiro, there is a need for applications that support this increase.By analyzing the tourism in Brazil, in general, we identified an issue: the lack of information about Points of Interests (POIs).
With the smartphones growth in the world, as well as with the great research progress in the ubiquitous and mobile computing area, handheld devices, such as tablets and smartphones, can be used to improve the tourist experience when visiting a POI (Moura et al., 2013;Fenza et al., 2011).Mobile devices applications can act as a help for searching information about a POI, by detecting contextual data about the place, such as the specific pinpoint of the user, and display information that might positively impact on their tourist experience when visiting a place.One of the techniques to be explored in the ubiquitous computing (Costa et al., 2008) is the augmented reality, which permits to merge images of the real world, captured by the device camera, with virtual objects.These objects can display information about the context of the POI environment, such as images, photos, videos, texts, and so on (Martin et al., 2012).
Applications that use the ubiquitous computing concept for tourism are commonly known as u-tourism or ubiquitous tourism (Moura et al., 2013).One possibility is to use augmented reality to assist the navigation and the knowledge of the place that is being visited, for instance, displaying graphical objects and POIs directions (Gervautz and Schmalstieg, 2012).In this perspective, this article introduces the development of an Electronic Silva, Costa, Righi, Costa | UbitourAR: A ubiquitous tourism model based on augmented reality Tourism Guide (ETG) based on augmented reality, named UbitourAR.The objective is to provide functionalities for mobile devices that enable the overlapping of data into the captured images from the user camera, using the available sensors (such as GPS, compass and accelerometer).
The proposed model is an extension of a previous work named Ubitour (Moura et al., 2013).One of the main features of the model is its focus on context awareness, using information such as location, user preferences and restrictions (such as time, means of transportation and resources), among others, like previous experiences, most preferred events, and so on.UbitourAR is based on ontologies to optimize the user tourism experience in a region and generate the information needed.This information is presented using augmented reality.
The main scientific contribution is to integrate the augmented reality with users' context, particularly their localization, identifying different states.These states should be pre-defined in the application and generate different data according to the user's context.The use of a state machine along with the representation of context information using an ontology constitute the main innovation when comparing to related works.
After proposing the model, we have created an application for the Caminhos de Pedra itinerary, in Bento Gonçalves, for evaluating Ubitour (Bruscato et al., 2013).We extended this prototype to evaluate UbitourAR.To evaluate the model, first, we assessed the ontology, creating rules to show how context information can be obtained for specific scenarios.This information was later presented in the interface of the App using augmented reality.The second evaluation was developed to answer whether the augmented reality can increase the tourism experience when visiting a place.Aiming at answering this question, we conducted a usability test, evaluating efficiency, effectiveness and user satisfaction.
This article is organized in seven sections.The next section describes the UbitourAR model.The third section describes the application developed and shows in detail the states defined for the evaluation, which are described in the fourth section.The fifth section describes the related works.And finally, the last section presents the conclusion of this work, as well as directions for future works.

UbitourAR model
UbitourAR is an ETG model based on the u-tourism concept, satisfying the following ubiquitous computing basic premises (Costa et al., 2008): (i) access to the computational environment anytime and anywhere; (ii) integrated with user's needs, adapting itself to the user profile; (iii) adapting to the environment context.In Figure 1, we can observe the proposed architecture for UbitourAR.The model is based on a client/server architecture, extending the original Ubitour (Moura et al., 2013).The server layer is composed of software components that provide u-tourism services to the client layer.These services were selected according to the proposed model premises.The client layer is composed by the application of the mobile device.In this layer, software component for integrative usage with the server (on-line mode) or off-line mode was provided.
UbitourAR extends the original proposal by providing a Controller, isolating all the interaction with the clients, and by the creation of a visualization component.Furthermore, the interaction with the ontology is divided in three components: Core Tourism (main functionalities of the model), Social Interaction (interaction among tourists and with social networks), Context Environment (obtaining information regarding place), Tourist Profile (obtaining information regarding user) and External Data (allowing integration with other sources of touristic data).Each component has a specific function, but, in this work, the focus will be on extending the ontology in order to generate the appropriate context information, and on proposing the Visualization component.
UbitourAR is used in order to decide which the POI best visualization strategy (how?) will be and which complementary data (what?)shall be displayed according to the environment context and the tourist's profile.The previous visualization strategy of Ubitour consisted of supporting the POIs exhibition in maps or in list, where the POIs were exhibited by best-rated POIs.The POIs information could be recorded in multimedia, such as videos or photos about a specific place and related information (websites and other documents).All the information that shall be displayed depends on the user's profile and on the environment context.

UbitourAR ontology
The main classes of the developed ontology can be seen in Figure 2. The figure shows two types of lines: dashed and continuous.The dashed lines represent the concept of object properties, i.e., a class that has a property as its object, thereby forming a relationship such as "has a".The continuous line describes the concept of inheritance, also known as relationship-type "is a".
The class Tourist represents an individual who does tourism.The tourism activity can be divided in three phases (Planning, Touring and Reminiscing) with which tourists can take advantage from information systems (Watson, 2004).The Planning Phase, which is before the tourist trip, corresponds to the knowledge of the travel destination, means of transport to be used, and the main attractions to be visited, among other information.The phase of Tourism Phase, comprising visits to POIs, i.e., places that someone finds it useful or interesting, and lots more.In Reminiscing Phase, tourists have finished their trip and need to remember it through pictures, videos and other information about their holiday.From this information stored in the ontology, UbitourAR creates a history of actions of a tourist, and then we can make several inferences to make the tourism experience more personalized.It is also possible for the tourist to share this information with his/her friends and other tourists to increase the interaction between them.
The Route class represents a custom itinerary.Each route consists of two or more POIs, which represent the available points of interest.Each POI, in turn, owns a Review, which represents the rates and comments made by tourists.The review information is used by the system to classify the best POIs to visit and to filter those that are not indicated according the profile of tourist.The ontology also has the classes Context, Location and Restriction representing, respectively, a context, a specific place and constrains that tourists may have, such as available amount of time or money.The users can extend the classes Restriction, Context, Profile and POI in accordance to the growth of the ontology.For example, Time, Money and Health are restrictions already enrolled, but the tourist can create others classes extending from the superclass Restriction, such as Temperature, or any other he/she wants.

Proposed augmented reality visualization
A general vision of the proposed visualization component is presented in Figure 3.The Context Analyzer subcomponent captures an event for each geographic coordinate change of the mobile device.This analyzer captures a localization context, analyzing the data retrieved by the sensor of the mobile device, the geographic coordinates of the GPS device and the directions of the compass.After this analysis, the state control, based on the context, determines which state the application is in.These states could have their own business logic and display a distinct interface.This interface is displayed above the image captured by the mobile device's camera.At the end, according to the selections mode, there will be calls executed through the input/output component together with the other components of UbitourAR.
The visualization of UbitourAR supports the following functionalities: •  Display map -the map shall be displayed using augmented reality and shall contain POI information; • POI selection -the POIs shall be displayed using augmented reality also when the mobile device is pointed to it, considering a 90 degree error (45 degrees to the left and 45 degrees to the right).Moreover, these point shall be selectable, and a brief description about this shall be displayed, as well as a button with a 'go to' description; • Show route -when a route is built, the system shall display the direction using augmented reality.This route shall be hidden only when the user gets to the POI perimeter; • Get POI information -when entering the POI's perimeter, users could have information about objects next to them through augmented reality.It could be texts, videos or tridimensional objects; • Translation to the user's native language -users shall have the possibility, at visualizing texts, to select the translation to their native language.

UbitourAR prototype
UbitourAR was implemented in Java using web services.The ontology was modeled in Protégé and accessed using the framework Apache Jena.All communications used REST (JAX-RS), using the RESTFul architecture.We defined three web services: List, Enroll and Infer.The List web service contains all the elements that can be searched, such as POIs, Tourist Profile and other information.The web service Enroll is in charge of storing information related to the user, either from a social network (such as Facebook or Twitter) or by the client App.Finally, the Infer web service is used for using the ontology.We employed, in this case, the SPARQL language and the RDF protocol.
Concerning the implementation of the visualization component, we have used a framework denoted Nineveh GL (http://nineveh.gl).This framework works as a layer above the OpenGL, which allows the focus on augmented reality.This framework was used in order to draw the 3D objects on the screen and modify its attributes, such as rotations and positions.Moreover, we proposed seven basic classes (Figure 4).
The Context class contains the current context information, such as latitudes, longitudes, and direction (0 to 360 degrees).The StateInterface works as interface for the states developed.The RAframework class works as an singleton and is called inside the states, in order to provide basic functionalities to developers, such as build routes, create POIs, get angles values and load videos to screen.Moreover, the developer has access to the framework basic object, such as route indication arrows and navigators.
The step class represents a step of a route, this route is built by using a functionality of Google Maps to generate routes, and each step has a latitude, a longitude and a step description.The AugmentedPOI class symbolizes a graphic frame that represents a POI and could be displayed when the used point in direction of the POI, it has a "go to" button, and its logic should be provided by the state.The Contex-tAnalyzed class implements the protocol Con-textAnalyzerInterface, which will determine the analysis and create methods for each state.The last class introduced in this work is the RA-ViewControl class, which is controlled by the augmented reality component and inherits the functionalities of the UIImagePicker, for camera usage, LocationManager for work with localization contexts, and the NGLViewdelegates (from Nineveh GL).This class triggers events when the mobile device changes the directions or positions.It creates the context instances and enquires to context analyzer which state shall be executed.
We used the Objective-C language to create a client for the iOS platform.Some iOS frameworks were used in this development, such as Location Manager (it works with the position and direction context) and UIImagePicker (it works with the camera) and the Nineveh GL framework (to work which OpenGL).To test the client, we developed an application for the Caminhos de Pedra Itinerary, located in Bento Gonçalves, Brazil.
We developed three states to be used in the Caminhos de Pedra iOS application, as exposed in Figure 5.The oufofCP state will be executed when the user is out of the perimeter of the tourist region.In this case, the application will display the route to the user; thought an indication arrow, it will display which route the user should follow in order to go to Caminhos de Pedra.In Figure 6a there is an illustration of state 1 being executed.This state uses augmented reality component tools such as angle calculation and route generation.The inCP state will be executed when the user is in the Caminhos de Pedras.In this case, the application will show description windows in the direction of the POIs, displaying that there is a POI in the direction towards which the device is pointing.Moreover, this state has also a "go to" functionality, which consists in providing the directions to the user to go to the POI by using the same logic of the outofCP state.In figure 6b there is an illustration of state 2. The inPOI state will be executed when the user arrives at the POI.When users are at the POI, the state will show information about that place, through videos and description texts.In Figure 6c there is an illustration of state 3 for Cantina Strapazzon POI.

Evaluation
The model was evaluated in two ways.First, we evaluated the proposed ontology by using scenarios.The aim is to show how the ontology could be used in different situation to provide the information that should be displayed to the user.The second evaluation regards the visualization model.A usability evaluation was conducted by using the prototype with augmented reality.We implemented the three states proposed in section 3 and evaluated the prototype with real users, all tourists.

Evaluation of scenarios
In order to evaluate the proposed project, we used an evaluation based on scenar-ios.The scientific community has been using this evaluation in order to assess projects on context awareness environment (Dey, 2001), and in ubiquitous computing environment (Satyanarayanan, 2011).In this subsection, we present some possible test case scenarios to evaluate the use of the ontology.These are used to check the main features that the system has to answer using the knowledge base.The evaluated scenarios are shown as follows: (1) Scenario 1 -Tour trip planning to Caminhos de Pedras The goal of this scenario is to demonstrate the use of the model in the planning of the tour trip (Watson et al., 2004) to the Caminhos de Pedra region, in the state of Rio Grande do Sul.In this task, we explore budget restrictions.Following, we describe a possible application scenario: Maria has an iPhone and wishes to travel to Caminhos de Pedras for two days.By consulting the iTunes Store, she finds an App for the region, called UbitourAR.When executing it for the first time she makes the log in using her Facebook account.Maria then defines a budget of 100 reais for lunch each day.Besides that, she sets a preference of paying by credit card.Maria selects the options to restrict the budget to 100 reais and the 2-day limit to the travel.After that,  Maria searches the POIs and the results will be restricted to her criteria.Maria has then planned her travel.
To execute this scenario, the UbiTourAR App creates a BudgetRestriction class instance and fills in the ontology the values, assigning the Instance of Maria.After Maria confirms the data, the client connects herself to the server thought the interactServer( ) method and save the information by using the saveData-OffLine( ) method of the class Configuration-AndManagementService.
The connection with the server is made thought the REST web service and then the ferBudgetrestriction( ) is executed.This method, from class Infer, delegates to a SPARQL query regarding budget restriction, as presented in Figure 7.Besides that, an instance of PlanningPhase class is created and saved in the poisToVisit attribute the POIs that Maria wishes to visit. (

2) Scenario 2 -Unexpected visit to Caminhos de Pedra
The goal of this scenario is to demonstrate the use of the ontology in an unplanned visit to Caminhos de Pedra.In that scenario, the App suggests restaurants that are opened in the moment of the trip, considering the credit card payment restriction.The description of the scenario is as follows: Alex has been working for the day in his office located in Porto Alegre.At the beginning of the afternoon, he receives an urgent call from his boss requesting his presence in the city of Bento Gonçalves in order to get some documents in the office there.After jogging, since Alex does not know the region, he decided to have dinner in Caminhos de Pedras.To do that, Alex used this app to find open restaurants.
To complete this task, the client Ubitou-rAR connects to the server thought the inter-actServer( ) method, gets the current date and time and executes the inferOppnedPOIs( ) method of the class Infer executing SPARQL, shown in Figure 8, which returns with available POIs at the moment. (

3) Scenario 3 -Customized travel to Caminhos de Pedra
The goal of this scenario is to demonstrate, in the tour phase (Watson et al., 2004), the use of the ontology when user's preferences are taken into account for the POIs suggestion.As an example, consider the scenario exposed as follows.
José has travelled to Caminhos de Pedra region.When arriving in the city, he executes the UbitourAR App in order to know the places that he might visit and enjoy.By log-  ging with his Facebook account, the system retrieves his preferences data by accessing his Facebook profile.José selects the recommended POIs list and defines a route to them.After that, José follows the route and visits the POIs, commenting them and providing feedbacks.
José logs in the App with his Facebook account.The system creates an instance of the FacebookProfile class and fills in the ontology with the values displayed in Table 1, assigning the instance of José profile.The client then connects himself to the server and saves the user profile data through saveDataOffLine( ) of the class ConfigurationAndManagement Service.By executinng the search for a POIs, the client connects himself through web services and executes the inferProfileContext( ) of the class Infer that executes the SPARQL query of context profile inference, as shown in Figure 9.When the search returns with the results, an instance of Route class is created to maintain the POIs in a route.At each new evaluation that José makes about a POI, an instance of VisitedPoi is created containing the visited POIs and evaluations made by José.

Evaluation of usability
We developed an evaluation of usability to determine the application of the proposed model.The ISO 9241-11 (International Organization for Standardization, 1998) defines usability as the extension to which an application is used by its users to achieve their objectives in an effective, efficient and satisfactory manner.By taking this concept into account, UbitourAR was evaluated in three attributes, as described below: • Effectiveness: if the users can finish a set of proposed tasks; • Efficiency: the time that users take to complete these tasks; and, • Satisfaction: users' attitudes after the use of the application.
In order to evaluate the prototype, we defined three tasks, one related to each augmented reality state.Ten different tourists executed these tasks.All tasks were timed and executed two times for each person.The goal of these tests was to determine the degree of effectiveness, efficiency and the learning of the application.
Task 1 was executed when the user was out of the bounds of the Caminhos de Pedras location.The system displayed the route, and then users had the task to explain the evaluator what their understanding of the system output was.Task 2 was executed in order to test the second state, when the user is in the bounds of Caminhos de Pedras.The goal of this task was to search for the Cantina Strapazzon POI and request a route for this POI.Finally, Task 3 consisted of evaluating state 3, what happens when the user is in the bounds of the Cantina Strapazzon POI.In this task, users had to check out videos and other related information.Considering that the proposed model triggers actions only when the direction or the location of the mobile device changes, for testing purposes, we provided 4 buttons on the screen: "not force location", "out of CDP location", "in CDP location" and " in POI location".These buttons were inserted to force the location in order to trigger the states.That adaption made the evaluation process feasible in a different location.The evaluation was made with 10 people.The choosing criteria were the personal interest in visiting the location of Caminhos de Pedras and in tourism in general.Moreover, the users were selected in order to make this sample as heterogeneous as possible.So, we selected 5 women and 5 men, 3 with ages ranging from 20 to 25 years old, 3 with ages ranging from 26 to 30, and 4 with older than 30 years old.All people considered in this study have at least finished High School.
To evaluate satisfaction, the App was submitted to a Technologic Acceptance Model (TAM) analysis (Marangunić and Granić, 2014;Yoon and Kim, 2007).This evaluation measured how much the App was useful and easy to use by the users.The idea is to answer if augmented reality can provide a better experience to tourists.The answers were made using Likert scale (Likert, 1932), which provides partial options (from 5, totally agree, to 1, totally disagree) for answering, helping in the understanding of the results.
The effectiveness average was measured according to whether the user could complete the proposed tasks.In Figure 10a the effectiveness rate is presented in a graphic bar.Analyzing the graphic, we can see that 100% of the users have finished tasks 2 and 3, while in task 1, only 90% of the user could finish it, which means that only one user could not finish it.By analyzing these numbers, we can conclude that the prototype is effective.Moreover, a feedback from the person that could not finish the task was taken.According to the user, the App should be more intuitive if some maps were displayed in the screen.
The App average efficiency consisted in how much time a user took to use the app compared to the time of an expert user.Each person executed 2 times each task and then an average between theses times was made.In Figure 10b we display a chart showing the results of the average efficiency.The chart demonstrates that the average efficiency increased at each task.Moreover, it is possible to observe that there is age efficiency dependence.People from 20 to 30 were more efficient than people older than 30.In Figure 10c, we can observe the performance of people aged from 20 to 25 were almost twice better when compared to people older than 30 years.
Once the users executed each task twice, it was possible to measure the learning average -in other words, the time reduction in the second execution.This average shows what the reduction percentage in the second execution of the task was. Figure 10d shows a learning average chart for each task.By analyzing the  chart, it is easy to notice that tasks 1 and 2 have better learning average, since people took too much time to do it in its first execution.Task 3 was more intuitive, so the time spent in the second execution did not vary too much from the first execution.
At last, we conducted an evaluation of satisfaction.We proposed 8 statements, 4 related to the ease of use and the other 4 related to utility perception.Table 2 lists the statements chosen and the percentage of answers according to the Likert scale.Regarding the ease of use, it is noticed that there is accordance with the proposed statements.When we asked about the functionalities, most people fully agreed that it is easy to use.Moreover, 80% of the people fully agreed with the statements proposed, while the remaining 20% partially agreed.
By taking into account the user perception, it is easy to notice an agreement that the App answered one of the main question raised in this work, because 100% of the people think that showing tourism information through augmented reality is interesting.Moreover, an average of 82,5% of the people fully agreed with the statements, 12,5% partially agreed, and only 5% are indifferent, which leads to conclude that in fact the App has potential.

Related work
When proposing UbitourAR we considered related works that use augmented reality.Unfortunately, we did not find a sufficient number of works related to tourism, so we considered some models that apply augmented reality for other areas.As the basis for finding related works, we combined one or more of these keywords: "augmented reality", "mobile computing", "tourism", "ubiquitous computing" and "context awareness".We used these keywords in search engines provided by ACM, IEEE, Google, and Springer.
ClayVision (Takeuchi and Perlin, 2007), is an App project for mobile devices that uses an augmented reality interface, developed for the area of urbanism.This work redesigns the city by using urbanism concepts in order to improve people's navigation.The Clayvision also uses an interface which makes the navigation more intuitive, once the objects are being displayed in the device screen, while other navigation systems uses dimension maps, which makes the navigation more complex.In this model, the user does the task of assigning the points displayed in the map with the real world object.

Measures and Statements
Fully agree Partially agree Indifferent Partially disagree

Fully agree
Ease of use The Augmented Reality Framework for Internal Environments (Zhang et al., 2010) was developed by a team from Intel in order to help workers in their daily tasks, such as make computers maintenance or discover a computer setup more easily.This framework allows ubiquitous augmented reality applications developers to design the states in which the application should run.These states are sub-applications that are monitored and controlled, which changes in the application layer, according to context data.This framework allows multiple states association in the application.To work with these states, the framework architecture was designed in 3 layers: the context generation layer, the context analyzed layer, and the application layer.
Another considered work was the model proposed by Karlekar et al. (2010).The model consists in a ubiquitous augmented reality application for open environments.This works stands out for the automated initiation and its by free movement tracking.The automated initiation consists in retrieving the current location, using the GPS device, and refining it using a component called shape context descriptor.After retrieving data from the camera, the system goes to the tracking task.The model stands out also for the movement's correction according to the accelerometer and modifications considering climatic changes.

Conclusion
This work considers the lack of exploring augmented reality and other ubiquitous computing concepts to aid the process of doing tourism in Brazil.In this context, we proposed an augmented reality model named Ubitou-rAR.The model is based on a state machine, where the state control chooses which action should be executed; this is done by analyzing the context environment.UbitourAR proposes an ontology to help users find the appropriated POIs to their needs.In order to evaluate the model, we developed an application for the Caminhos de Pedra itinerary, in Bento Gonçalves, Brazil.
The evaluation was twofold.The first assessment showed the feasibility of using an ontology for helping users in finding appropriate POIs.Three scenarios were created and the corresponding rules were developed.The second assessment is related to usability.The results show that the developed model is effective, efficient, especially for younger peo-ple, who tend to be more familiar with the smartphones and tables technology available in the market.Moreover, the users declared to be more than 80% satisfied.We also observed that presenting information thought augmented reality is a really good approach in order to improve tourist experience and it should be explored more deeply in the ubiquitous tourism area.
There are some opportunities for further works.We should implement some additional functionality, such as a navigation map, which has impacted the efficiency of the App, according to the evaluation.Moreover, it is important to improve the augmented reality component adding the support of using camera device frames.By supporting it, general graphical computation algorithms can be applied, such as tracking algorithms, which can provide the developer with a higher functionality API.Finally, the use of the ontology should be integrated in the App for providing suggestions of POIs in a real-time fashion.

Figure 4 .
Figure 4. Diagram of the visualization component class.

Figure 7 .
Figure 7. Query for exploring budget restrictions in the ontology.

Figure 8 .
Figure 8. Query for available POIS considering time restrictions.

Figure 9 .
Figure 9. Query for search POIS considering the user's profile.

Table 1 .
Profile attributes obtained from Facebook.

Table 2 .
Results of the evaluation of users' satisfaction.