A proposal of a classification model for the cognitive workload of human activity in a context-aware system

The skill level of a person in processing information, reacting to his/her surroundings and decision making for performing an activity is determined by the allocation of the mental resources demanded by such activity. When the allocation is inappropriate, there is a higher possibility for some accident to occur. Thus, one can notice that the cognitive workload spent by the person is an important variable that can take him to a risky situation. Since it is not possible to measure the cognitive workload spent by a person during the performance of an activity directly, we noticed the need to evaluate the level of his/her performance in order to be possible to infer the cognitive workload used. So, we propose the creation of a model to classify the cognitive workload based on the behavioral model skill-rule-knowledge and the relations of performance properties with the context surrounding the person. The evaluation of the model was made using a public dataset and the results showed a promising approach for the classification of human performances.


Introduction
The mental resources of a person must be allocated in a way to encompass the demands of an activity for the performance to be adequate (i.e., lower chance of happening an accident).However, humans have a limited mental capacity (Boff et al., 1994).So, the demands of the activity can surpass the available limits for the person.When this happens, the person's skill in processing information, reacting to his/her surroundings and making decisions are negatively affected, allowing for a greater chance of accidents to happen.
Physical injuries can bring different problems to people.In the case of recurrent falls in the elderly, as noted by Gasparotto et al. (2014), one of the problems is related to deficits in the length of the gait of the elder.This way, we can notice the need for a system that is able to detect possible risk situations in order to indicate and help people in their decision making process, thus, avoiding injuries.
Considering current technology, it is possible to develop a system capable of doing such task.The capture of information about the environment through sensors is a characteristic of context-aware systems.This work follows the definition of context introduced by Abowd et al. (1999): Context is any information that can be used to characterize the situation of an entity.An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves.
From this notion, one can notice that context-aware systems incorporate what is called omnipresence, where the computers become "invisible" to their users and perform their tasks in an autonomous way (with no interaction) using information gathered from the environment, allowing for the creation of ubiquitous systems (Weiser, 1991).For this work, we consider that it is necessary for the evaluation to be performed in a non-intrusive manner, i.e., the user should not indicate any kind of abnormal situation.In order to accomplish that, it is pertinent for the system to be developed based on the idea of ubiquity.
This paper is structured as follows: Section 2 gives a brief introduction to core concepts used in our work.Section 3 presents some related works.Section 4 describes the proposed model and Section 5 presents its evaluation.Section 6 draws our final considerations and future work.

Core concepts
Since a ubiquitous system needs to consider information related to the environment, in this work we consider the taxonomy of context presented by Mikalsen and Kofod-Petersen (2004) shown in Figure 1.The contexts that are considered in the evaluation of the risk level of an activity are: the environmental context (describes the surrounding where the activity is being performed); the spatio-temporal context; the task context; and the personal context (describes physiological and mental information of a person).This work considers that risk situations occur according to the conditions of the environment where the person is in and the performance (related to the physiological and mental information) of the activity by such person.The social context (roles that one can take in the society) can also influence the risk evaluation, however, it is not considered in this work.
Cognitive workload refers to the mental workload that a person uses during the performance of an activity (Craven et al., 2007), i.e., the quantity of mental resources allocated for its execution.It is the resources allocation that determines the ability of the person to process information, react to its surroundings and make decisions.If this allocation is not appropriate for the demands of the activity, such abilities are reduced in a way that the possibility for some accident to occur is raised.This work is based on the approach given to the cognitive workload in the behavioral model skill-rule-knowledge (SRK) by Rasmussen (1983), shown in Figure 2.This model classifies the human behavior in three levels, from the behavior that demands a lower cognitive workload to the one that demands higher cognitive workload: (1) skill-based behavior (SBB) -occurs in a known context, with the environment information being sensed as signals; (2) rule-based behavior (RBB) -occurs in familiar context, but with some differences, with the environment information being sensed as signs; (3) knowledge-based behavior (KBB) -occurs in unfamiliar contexts, with the environment information being sensed as symbols.Therefore, the way in which the information is sensed is what determines the level of cognitive workload used.Signals are related to spatio-temporal signals, implying that the person is performing a usual physical action, signs are perceived when the information sensed is used to modify pre-determined actions (i.e., situations where the activity is performed by convention or previous knowledge) and symbols are information used to predict or explain non-familiar situations.
This work associates the inference of cognitive workload used in an activity based on the SRK model and the taxonomy of context, and is a part of the development of the middleware for the inference of risk in activity presented in a previous work (Del Fabro et al., 2013).In the middleware (Figure 3) the risk is inferred by the component Risk Analyzer from environmental and user information received from the Activity Manager component and performance information received from the SRK Classifier.This paper proposes a conceptual  (Mikalsen and Kofod-Petersen, 2004).model for the latter based on the relationships of environmental and performance properties.
Even though there are neurophysiological sensors that can capture information such as pupil dilation and heart rate (Craven et al., 2007), this work considers that it is not possible to determine directly the cognitive workload used by a person and that there is "frequently a negative relationship between mental workload and performance" (Rantanen and Levinthal, 2005).This way, the higher the cognitive workload used in an activity, the worst its performance.This principle can be verified by realizing that experts in performing some activity tend to use efficient and reliable cognitive shortcuts without losing depth of processing (Loveday et al., 2013), i.e., experts use less cognitive workload while performing well their activities.The properties that could be used for the measure of a performance are: • Attention: the load of attention over a person can reflect directly the level of performance of some activity.It is noticeable in tasks that involve sensorial and motor skills, in a way that novices tend to consciously control each step of the execution of some ability, whereas experts do not need aid of the attention to perform fast and efficient control procedures (Gray, 2004).Besides, attention can reflect directly the possibility that some kind of error occurs during the performance of a task.For example, when operators need to perform multiple tasks, like the handling and moni-toring of controls, it is necessary that his attention is correctly allocated in an efficient and effective manner, even under the pressure of time, e.g., control operators of amusement rides (Woodcock, 2014).Related to attention, the anxiety is  a factor proven as prejudicial to the performance of a task, especially in complex tasks and that demand attention (Derakshan and Eysenck, 2009); • Effectiveness and efficiency: besides the effectiveness of the performance of a task, usually related to the accuracy of its result (Eysenck et al., 2007), the efficiency of the behavior must also be considered for the evaluation of a performance (Robert and Hockey, 1997).The efficiency of the performance of an activity lowers while the cognitive workload spent rises so that the task is effectively performed (Eysenck et al., 2007); • Mental workload: the performance of a task changes according to the mental workload used for its execution (the higher the demands of the task, the higher the mental workload needed).However, people have a limited capability of mental workload that can be spent in a task.If this task exceed such capability, people tend to lower their criteria of performance so that the mental workload also lowers.This mental workload can be related to several factors, such as pressure of time, uncertainty of how to proceed, data quality for the task and conflicting demands (Cook and Salvendy, 1999).An interesting point is when the cognitive workload is considered for the development of human-computer interfaces (HCI), where it is indicated that the designers use structures that explicit the cognitive levels with the more efficient processing (SBB and RBB), i.e., structures that require the lowest cognitive workload at the same time they preserve the applicability when the cognitive workload is higher (Vicente and Rasmussen, 1992).However, the performance of an activity can be negatively affected even when the cognitive workload is small, as a result of boredom (Lin et al., 2014); • Time: the task load can be defined as the relation of available and spent time for its execution.So, as the task load gets higher, the performance of the person is reduced, i.e., people change from pro-active mode to reactive as this task load gets higher -a pro-active mode results in a better performance and the reactive mode in a poor performance (Rantanen and Levinthal, 2005).
• Others: besides the already mentioned factors, the evaluation of the performance of an activity can be also related to the ability and experience of the executor (Vicente and Rasmussen, 1992); to the complexity, demand and quantity of tasks and sub-tasks being performedmore tasks mean less response time and, thus, worse the performance (Ujita et al., 1995) and; slips, mistakes and errors that may happen during the performance of the activity (related to effectiveness/ accuracy).Moreover, depending on the occurred problem (slip, mistake or error), it is possible to determine directly the cognitive workload (SRK) used at the moment of the activity's performance, e.g. as in the work of Woodcock (2014).
So, the measurement of the performance of an activity is important for the inference of the cognitive workload spent during its execution, since it is not possible to measure the cognitive workload in a direct manner.Thus, we can observe in works that use the SRK model a tendency of using factors for the measurement of human performances, like the mentioned, for the classification of the cognitive workload (SBB, RBB or KBB) of their users/operators during the performance of the specific tasks of their systems (Lin et al., 2014;Woodcock, 2014;Skalle et al., 2014).

Related work
With the goal of understanding the behavior of operators with different responsibilities in a main advanced control room, the work of Lin et al. (2014) classifies the tasks of each worker from the cognitive workload used based on the SRK model and analyzing properties such as time, frequency and mental workload.
In the case of controls of amusement rides, the analysis of the behavior of the ride operators was used in the work of Woodcock (2014) to determine guidelines in the design of the control interface of the rides with an approach for error prevention.The errors were classified according to the cognitive workload spent in each task.The analysis of such cognitive workloads was made considering properties such as time pressure and focus of attention of the operator.
In the work of Skalle et al. (2014), the SRK model was also used for the classification of human failures, however, the properties used are related to mistakes, classification errors and lack of experience.
Table 1 makes a comparison between these works.We can notice that in the cited works, but not limited to them -e.g.(Stahl et al., 2013), the relationships of certain tasks with the cognitive workload levels based on the SRK model are determined manually, where the researchers observe how the subjects of the study behave and evaluate their performance, thus, not allowing these tasks to have a change in their cognitive workload level, even if the subjects get better or worse in their performance after some time (contrary to our work).Other performance properties used by the works refer to time, frequency, lack of experience, etc.The proposed model in this paper accepts any kind of performance property capable of being inferred by a context-aware system.
During the related work research we could not find any works that evaluate the perfor-mance of human activities in real time by a context-aware system.So, we consider our work as a novel approach by using a cognitive workload framework to the area of computational contexts.The Relationship Analysis component verifies how usual the values for the current context and the performance properties of the current activity are.The analysis is based on historical information about the context and activities.

Proposed model
In order to verify the user's performance related to a performance property, we measure its probability with all context properties.If we consider the SRK model, this probability can be related to how the sensorial information about the environment is sensed, i.e., signals, signs or symbols.Since each performance property has its own probability, the environment is sensed differently when considering each property separately.For example, if the probability of the performance property duration is high in its relationship with the context property temperature, we can infer that, for the current activity, the property sensed the environment (i.e., the temperature) as a signal.The equation that does this is the following: In the equation ( 1), p p is the performance property being analyzed; p c is the context property; n is the number of context properties; and  is the probabilitsy function.This way, we can notice that the sensoring is the sum of how each context was perceived by the performance property.
In the Properties Analysis, the performance properties have a sensoring associated.This sensoring is then used by this component to infer the cognitive workload based on the SRK model.Since not every performance property has the same importance for some activity (e.g., the performance property attention may not be relevant for the activity brushing teeth), it is necessary to measure their relevance.It is measured by using data provided by the component Activities and Cognitive Workload by measuring how much the performance property changed for some activity and what impact it had in the cognitive workload inferred by the system.The following equation shows how it is done for the performance property p p : In the equation ( 2), p p p represents the performance property list of values; and W Activity is the list of values for the cognitive workload inferred for some activity.The covariance was used to measure the relationship between the performance property and cognitive workload.
The Property Analysis component infers the cognitive workload based on the following equation: The final result of this equation is the sum of the sensing for the performance properties multiplied by their relevance.Based on the SRK model, we can notice that this sum gives an idea of which level in the model the cognitive workload used is.The inferred cognitive workload is then used to feed the historical data.

Example scenario
Figure 5 shows how the model should work for a scenario related to the activity "showering".From the history of Context and Activities, composed by context information (e.g.humidity level) and activity information such as its performance properties, the component of the model Relationship Analysis is responsible for the analysis of such history in order to detect possible relationships between the performance and context properties.The relationships represent the way a context influences some performance property.In Figure 5, it is shown by the relationship of the properties duration and attention (the higher the duration of the activity showering, the higher the attention used), as well as the relationship of humidity and mental workload (the higher the humidity during the activity, the higher the mental workload required).
This way, after a filtering of the most relevant relationships is made according to the current activity and context, the model performs an analysis of the informations of the current activity and context with such relationships.For instance, if the current attention of the user is high, as well as the duration of the activity, the analysis would point out for a perception of a signal for such performance property, because it follows the pattern for existing relationships.Instead, when a performance property is not following a pattern, the sensory perception would tend to represent a high level of cognition (sign or symbol).The analysis of relationships performs the sensory inference of all performance properties for the current activity and context, which are ana-lyzed according to their relevance level in the component Properties Analysis.
The analysis of the properties verifies how all the inferred perceptions of the performance properties were and, considering each relevance level, performs the inference of the cognitive workload used.In Figure 5, we can notice that the result of the inference was a SBB, because the most relevant performance properties had an inferred sensory perception with low levels (i.e., signal or sign).In other words, the patterns found in the history were generally followed.

Model evaluation
For the evaluation of our model we used as performance property the effectiveness of an activity and as context property the temperature of the environment.In order to obtain the effectiveness of an activity we had to define an activity composed by sub-activities that are present in the dataset.The effectiveness of an activity is directly related to how many core sub-activities were performed during the time window of such activity.
So, activities are composed by a set of subactivities.There are two kinds of sub-activities: (i) core sub-activities and (ii) secondary subactivities.The former represents sub-activities that are essential to achieve the object of the activity, thus necessary for its recognition.Secondary sub-activities are those related to a given activity but not essential to it, they are useful to give additional meaning to the activity (e.g., the sub-activity of closing the window while taking a bath).
In order to detect an activity, every core subactivity related to it must take place in a predefined time window.The initial time window is determined by a preliminary phase during a certain time for system calibration.It depends on the frequency of each activity, so the time window can range according to the user's behavior.
An activity is started when a core sub-activity is detected during the time window.An  activity is finished with the detection of the last missing core sub-activity.The activity is considered not completed if not all of its core sub-activities were detected during the time window.It means that only some part of the activity was performed.This approach allows the detection of concurrent and interleaved activities.
T he evaluation was performed using a public dataset (Cook, 2010) and we considered the scenario depicted in Figure 6.It represents a timeline of detected sub-activities.Each rectangle represents the user context in a specific period of time.The first number inside the rectangles is the value of the temperature in degree Celsius and the a is the detected user sub-activity.
Have Meal is the activity being recognized.Activities are detected based on their core sub-activities, which must rely inside a time window (for the activity Have Meal, the time window is set as 1 hour).The core sub-activities composing the activity Have Meal are: meal preparation, eating and relax.It is important to notice that Have Meal was not defined in the dataset, we defined it by using related activities as its core sub-activities.This way, by looking at Figure 6, one can observe that Have Meal was successfully detected, because its core sub-activities were detected in the time periods 3, 4, 5 and 6 (giving an effectiveness of 100%).When t 4 was the last detected action, the activity Have Meal was also detected but with an effectiveness of only 66.6%, since only two of its core sub-activities were detected during the time window (a 1 and a 2 ), as seen in Table 2.The changes in temperature shown in Table 2 are related to the fact that the user was moving between different environments in the house, e.g., from the kitchen to the living room, each with its own climate, or because of some other factor, such as turning on the oven or opening the refrigerator in the kitchen.
The value for the performance is measured using equation 3. Since we are considering only one performance property (effectiveness), its relevance level is said to be maximum (100%).For example, considering the entry at time 13:42:10 (in Table 2), we can notice that the performance is very close to 1, i.e., the activity was really well performed (the probability for the value of the effectiveness happening with the current value of the temperature was high).

Conclusion and future works
The main contribution of this paper is the proposal of a model for the classification of the cognitive workload spent by a person during the performance of an activity in a contextaware system.We can notice the relevance of using performance properties for the inference of the cognitive workload, because they allow the system to adapt itself according to the changes in the behavior of the user.The fact that the model identify the relationships based on previous events and that the database of events is fed as new activities are being performed and their cognitive workloads are inferred, we can notice that, giving enough time, new relationships can be found and others can get weaker.Thus, it is visible the capability of the model to adapt to changes of user behavior, i.e., the model is capable of perceiving changes in the cognitive workload of the user during the performance of his activities, allowing the inference of the final cognitive workload to vary between the three levels of the SRK model.
In future works we intend to implement the proposed model linking the equations shown in Section 4 with some algorithm for data mining, such as pattern recognition (combining the performance level obtained with some known pattern could yield a more precise result).

Figure 4
Figure 4 shows the model proposed for the inference of the performance of a human activity.Such model is based on the performance properties and the SRK model.The Context and Activities and Activities and Cognitive Workload components are historical data for each activity that was performed.This data is composed by performance properties and context information gathered at the moment the activity was performed.Also, it stores the cognitive workload used by the person (inferred by the Cognitive Workload Inference component).

Figure 4 .
Figure 4. Proposed model for performance evaluation based on the SRK model and properties of context and performance.

Figure 5 .
Figure 5. Example of how the conceptual model of Figure 4 should work for the activity "showering".

Figure 6 .
Figure 6.Last entries in the dataset used for evaluation.

Table 2 .
Entries from the test dataset for the day 2011-06-11.