Dealing with Uncertainty in Activity Recognition

Get Complete Project Material File(s) Now! »

Ontological Reasoning with Uncertainty for Activity Recognition

Introduction

This chapter introduces the methodology of the ADL (later referred to as activity) recognition or dense sensing-based activity recognition. The approach is based on the aggregation of context information from diverse sources which contain some knowledge about the events in the environment. As such, it is assumed that activities can be recognized through the inference of user-object interaction and location of the user. Activity recognition in a smart environment still faces a number of challenges. Firstly, the activities performed by persons depend on their habits and lifestyle and hence they are carried out in different sequences and with different durations. Although there exist correlations among some activities, there is no strict pattern in a sequence of activities. Secondly, multimodal sensors embedded in smart home environment generate heterogeneous data that varies in terms of formats, sensing rates and semantics. Furthermore, a fusion of these sensor data is required to establish the context of the activity being carried out. Finally, uncertainties are always present in ambient intelligence environment For instance, sensor data are inherently noisy. This can be due to sensor errors (run out of batteries, imprecise outputs, missing activations etc.), communication failures and variability in human activities. These issues may significantly influence the accuracy of activity recognition.
Different approaches have been proposed by researchers for activity modeling and recognition. They can be classified into data-driven and knowledge-driven approaches [35], Data-driven approaches use learning-based techniques with robust activity models that extract specific features from sensor data. The main advantage of learning-based techniques is the ability to handle uncertainty and noise. Previous research works have shown that they are able to obtain high accuracy rate of activity recognition [56]–[58], [135]–[139]. Furthermore, learning-based techniques are applicable to different domains and achieve good results [32], However, data-driven approaches tend to suffer from the curse of dimensionality and require large amount of training data to train activity models. As users perform activities in various ways and orders, it is difficult to obtain sufficient and representative datasets [35], [50], [140], especially in smart home environment due to cost, privacy and ethical consideration. Moreover, collecting and manually annotating huge amount of sensor data is an extremely time-consuming task. Therefore, data-driven approaches suffer from scalability, applicability and adaptability [50], [72], [141].
Knowledge-driven approaches exploit prior knowledge to build semantic activity model by using knowledge engineering techniques (also called specification-based techniques), and then reason on it with input sensor data. The advantages of these approaches are interoperability and ability to adapt to different scenarios, which are essential for context-aware environment where the sensors are multimodal. Moreover, they provide a way to represent sensor data and contexts by a formal data structure with the aid of semantic descriptions, which makes them understandable to both human and machine. Consequently, they facilitate the development of semantic activity model and recognition process. Numerous knowledge-based techniques have been introduced for context modeling. Among the techniques, ontology-based models are preferable for managing and modeling context recently [12], [50]. Despite the advantages of ontology-based technique, there are still limitations that must be tackled: ontological reasoning is computationally expensive, support for modeling temporal information is minimal and they cannot deal with uncertainty.
In this thesis, we are focusing on the weakness of ontology-based techniques to deal with uncertainty, because it affects the accuracy of activity recognition [73]. There are three levels of uncertainty in decision making process: data uncertainty, comprehension uncertainty and projection uncertainty [74]. Data uncertainty is normally associated with errors in sensor’s measurements, which arise due to incompleteness (missing sensor data), imprecise, inaccurate, timeliness and incongruent [50], [74], [142]. This study is focusing on incompleteness which is the most common in smart home environments because sensors operate with certain degree of reliability or loss of data during transmission. Existing ontology-based activity recognition systems can only infer an activity when all the contextual information that defines the activity is asserted. The contextual information is captured by the sensors embedded in the environment. If one of the sensor data is missing, ontology will not be able to infer the activity that is being carried out, which is indicated by a total ignorance about the current situation in the environment. In Dempster-Shafer (DS) [143], [144] theory, total ignorance can be assigned with a weightage (called belief) and combined with other evidences with a series of mathematical functions. Furthermore, DS theory can also resolve conflicting data by combining the evidences and arriving at a degree of belief [50], [145] to facilitate the activity recognition process.
In this thesis, we propose a novel reasoning algorithm which integrates ontological reasoning based on Description Logic (DL) [146] with DS theory. The proposed algorithm preserves the advantages of ontological reasoning and has the ability to manage data uncertainties that occur during the activity recognition process. An activity is modeled as a sequence of actions separated by elapsed time between two actions and may be used to represent the real-life activity. The reasoning algorithm assigns degree of beliefs to actions based on their states: active, inactive or uncertain which are determined by using the actions’ temporal sequence and inference of the actions. Then, the algorithm aggregates the action contexts to produce a belief for the activity which supports the decision making of activity recognition process. In addition, we propose a four-layered activity ontology which systematically organizes the contextual information in accordance with the activity inference process. We also propose a methodology to incorporate the evidential parameters in the ontology for reasoning using DS theory. The new algorithm is applied on two datasets, one collected internally and one publicly available, and then compared with ontology-only based recognition approach and data-driven approach. It shows very good recognition accuracy compared with other approaches.

Related Works

Ontologies for Activity Recognition

A number of ontology-based systems have been developed for activity recognition. In [147], an DL-based reasoning engine is used to recognize coarse-grained and fine-grained activities. Bae [148] proposed a method for recognition of ADL. Ontology is used to model the activity while semantic reasoning and rule engines are used to recognize the activities. Okeyo et al.
proposed a novel sensor data segmentation approach for activity recognition. Activities are modeled using ontology and semantic reasoner is used to recognize the activities. Ye et al.
present a novel ontology-based approach for concurrent activity recognition. Semantic dissimilarity is used to segment a continuous sensor sequence into fragments, which corresponds to one ongoing activity. In [151], an ontology-based hybrid framework for activity recognition is proposed by combining the standard reasoning semantics of OWL 2 and the standard query language of the Semantic Web. The proposed framework allows the OWL 2 reasoning module to incorporate temporal correlations of complex activities which is essential in activity recognition. Khattak et al. [152] proposed an approach to improve the general health and life status of elderly peoples by monitoring the dietary intake and health activity information. The ontology is used to model the daily life activities and patient profile information, allowing the analysis of fine-grained situation for personalized service recommendations. Ahmadi-Karvigh et al. [153] proposed a novel ontology-based framework to allocate appliance-level electricity consumption to daily activities. In the framework, appliance usage data is separated into categories of activity events, which are next segmented into activity segments. Then, a classification model is used to classify the activity segments into activity classes. None of the aforementioned approaches address uncertainty in activity recognition scenarios. Riboni and Bettini [154] proposed a hybrid reasoning for activity recognition which combines data-driven and knowledge-driven approaches called COSAR. In COSAR, statistical classifier recognizes an activity which is then tested through consistency checking by ontological reasoning to verify the recognition. COSAR can deal with uncertainties since machine learning technique is used as the classifier. However, the approach suffers from data scarcity to train the activity model. In [155], a novel unsupervised approach that combines data-driven and knowledge-based techniques for mining activity recognition is proposed. The ontology is used to represent the domain knowledge for facilitating the unsupervised discovery of activity patterns. However, the system fall short in distinguishing semantically similar activities that are occurring close together. Furthermore, the system has limited ability to deal with uncertainty.

Reasoning under Uncertainty

A number of approaches have been proposed in the literature for reasoning with uncertainty. Probabilistic theory is a widely used method in dealing with uncertainty. It provides a mathematically sound representation for degrees of belief. Ranganathan et al. [156] use ontologies combined with probabilistic logic to infer on-going activity based on object and location contexts. Uncertainty is modeled by a confidence value specified to context predicates. Similar approach is found in [157] in which, probabilities are assigned to events to handle noisy and ambiguous observations. However, the approaches do not utilize ontological reasoning for inferring new context information. Helaoui et al. [158] combines log-linear models with DL to represent uncertainty in the ontology for activity recognition. However, log-linear DL do not support nominal and concrete domains to model concrete properties and values. Furthermore, the proposed approach does not support complex temporal modeling and reasoning. Several previous works deal with uncertainty by extending OWL through Bayesian Network. These approaches represent uncertain information by using probability and dependency annotation. For example, BayesOWL [159] extends OWL by a set of rules to transform the defined concepts in ontology into a Bayesian network. OntoBayes [160] improves BayesOWL by supporting OWL properties and multi-valued random variables. However, it lacks OWL’s class support and hence it is not possible to model relationship between concepts. Ausín [161] overcomes limitations of OntoBayes and BayesOWL and offers uncertainty information isolation to ease the reutilization of probabilistic ontologies. Although Bayesian model is capable of dealing with uncertainties due to inaccurate and contradicting sensor data, it is not capable of dealing with missing sensor data [50] which is the focus of this study. Using Bayesian theory a missing sensor data could be represented by a proposition of inactive sensor. However, such proposition is not always true because the system might not receive the data due to communication loss. Unlike Bayesian theory where each individual proposition is assigned a non-negative value (probability), DS theory distributes non-negative weights (masses) to any combination of propositions [144]. This means that the belief function can explicitly represent any ambiguity or ignorance about what has been observed such as missing sensor data.
DS theory has been used in activity recognition for handling uncertainty. Wu [162] proposed to combine sensor outputs using DS theory for context-aware computing. Hong et al. [163] introduced evidential-based activity model where DS theory is used for combining contextual information to infer activities. Zhang et al. [164] used similar model for activity recognition and presented evidence selection and conflict resolution techniques based on evidence theory. Directed acyclic graph-based activity model is introduced in [165], [166] for activity recognition. DS theory is extended by including temporal information when fusing contextual information to improve recognition accuracy. Sebbak et al. [167] proposed new conflict resolution and evidential mapping techniques to optimize decision making in activity recognition. Similar model found in [165] is used for modeling activity. Liao et al. [168] introduced new activity model in the form of three-layer lattice structure which allows historical data to be used as a priori knowledge, and DS theory is used to handle uncertainty derived from sensor errors. Finally, Chen et al. [169] proposed a framework to fuse activity classification obtained by processing signals from a depth camera and an accelerometer using DS theory. The framework resolved uncertainty due to differing modality sensors. Although all presented approaches address the uncertainty problem, ontology technique is not used, which we consider to be the most advantageous and convenient tool for activity modeling. Aloulou et al. [170] proposed an algorithm for handling uncertainty in sensor detection by modeling its reliability in terms of battery level, physical and operational behaviors. Ontology is used for representing uncertainty level in context information and DS theory is used to combine contexts acquired from multiple sources in order to obtain the consensual uncertainty value. Similar work is found in [171], in which a novel modeling approach based on DS theory is proposed to handle uncertainty in sensor data. The proposed approach handles uncertainty by modeling not only hardware characteristics of the sensor, but also the consensus of a group of sensors. However, the literature does not address the uncertainty due to missing sensor data in ontological reasoning which is the focus of this research. Uncertainty could be handled through a hybrid approach, in which the data-driven approach is used to enrich the activity ontology [140], [172]. However, incorrect activity definitions could be encoded and that may lead to incorrect classification. Furthermore, traditional ontological reasoning is used for activity recognition. Fuzzy logic has been used for handling uncertainty in ontology-based activity recognition [78]. However, fuzzy logic is dealing with the concept of vagueness in context information, not the occurrence of an activity. Dai et al. [173] proposed a missing data reconstruction approach based on similarity measure to improve human behavior prediction. However, the transfer learning algorithm requires a significant amount of data, in which it involves binary matrices of user behaviors where a matrix represents an activity data for a number of days.
Ontology offers several advantages over other specification-based techniques [35], [174]. Firstly, it is understandable, sharable and reusable by both human and machine, and hence allows non-technical users to encode domain knowledge. Secondly, it provides reasoning services to infer activity by fusing information through reasoning mechanism of DL such as subsumption, satisfiability and instance checking. Furthermore, ontological reasoning can detect possible inconsistencies in the definition of concepts and properties of an ontology. Consistency checking is crucial because it may lead to erroneous conclusions. Finally, user’s activity preferences and styles can be encoded easily and hence facilitate personalized and adaptive modeling process. All the above features make ontology the preferred approach for activity modeling and recognition.
In this study, we propose a new reasoning algorithm for recognition of activities. It features reasoning mechanism of DL and uncertainty management due to missing sensor data while combining contexts from different sensors and provides a degree of belief of the activities, supporting the decision making process in order to improve the reasoning performance. The proposed algorithm is evaluated using two datasets, an internally collected and the other public one. In addition, we have implemented the HMM-based approach that can handle uncertainties to compare with the proposed approach. This work is, to the best of our knowledge, the first to propose an integration of DS theory with the OWL-DL based reasoning to deal with missing sensor data.

Ontological Reasoning with Uncertainty

An activity has diverse contextual information in terms of spatial, object and temporal contexts. Spatial context contains location and area information such as rooms, household furniture and appliances. Object contexts refer to human-object interactions such as opening a door, using a burner etc. Temporal contexts represent the time and duration. The contextual information is captured by embedded sensors in an environment, which provide clues about the activity being performed. By capturing and modeling this information, it is possible to infer the corresponding object interaction and location contexts from the activation of the sensors, which is generally referred to as human action context. For example, a magnetic contact switch installed on a door of a kitchen cabinet and PIR sensors installed in the kitchen can indicate the action of opening and closing the cabinet door. A series of human actions form an activity. By fusing human action contexts, it is possible to infer an activity being performed by a person in the inferred location with the inferred object interactions. In summary, activity is an aggregation of contexts from diverse sources which contain some knowledge about the events in the environment.
In this research, we propose an activity ontology organized into four layers of concepts, in which each layer of concept is explicitly defined in the ontology. The rationale is that by explicitly defining the concepts, annotation property can be used to annotate the evidential parameters which are required when reasoning using DS theory. The details are explained in Section 5.4.2. Figure 5.1 illustrates the four-layered activity ontology. The object interaction and location concepts are the atomic concepts which are used to describe the action concepts, while action concepts are used to describe activity concepts. An action might be associated with multiple activities. Activity inference is a process whereby a lower layer concept is semantically interpreted by the higher layer concept. The sensor layer models sensor states and provides contextual information on object interaction and location in the environment. From here, the process goes further up to the action layer, in which object interaction and location concepts are propagated and fused to infer an action. In the end, several actions are combined to form a conclusion on what activity is being performed. The modeling of the activity ontology is described in Section 5.4.1.

Contents
Abstract 
Acknowledgements 
Contents 
Publications
Introduction
1.1 Aging Population
1.2 Motivation and Objectives
1.3 Contributions
1.4 Thesis Outline
Background Research and System Architecture 
2.1 Introduction
2.2 Smart Homes
2.3 Context-aware System for Elderly Healthcare
2.4 Human Activity Sensing
2.5 Activity Modeling and Recognition
2.6 Dealing with Uncertainty in Activity Recognition
2.7 System Architecture
2.8 Conclusions
Adaptive Sliding Window for Physical Activity Recognition 
3.1 Introduction and Problem Formulation (Physical Activity)
3.2 Related Works
3.3 Characterization of Activity Signals
3.4 System Overview
3.5 Adaptive Sliding Window
3.6 Experimental Setup for Physical Activity Recognition
3.7 Results and Discussion
3.8 Conclusions
Physical Activity Transition Model .
4.1 Introduction
4.2 Related Works
4.3 Integration of Activity Recognition with Transition Model
4.4 Activity Transition Diagram
4.5 Results and Discussion
4.6 Conclusions
Ontological Reasoning with Uncertainty for Activity Recognition
5.1 Introduction
5.2 Related Works
5.3 Ontological Reasoning with Uncertainty
5.4 Activity Model
5.5 Activity Recognition Algorithm
5.6 Scenario of Activity Recognition with Uncertainty
5.7 Experimental Setup for Activity Recognition
5.8 Results and Discussion
5.9 Conclusions
Ontology-based Sensor Fusion Activity Recognition
6.1 Introduction
6.2 Related Works
6.3 Ontology-based Sensor Fusion
6.4 Experimental Results
6.5 Conclusions
Conclusions 
7.1 Achievements and Contributions
7.2 Future Work
References
GET THE COMPLETE PROJECT
Context-aware Activity Recognition for Elderly Healthcare using Wearable and Sensors Embedded in Environment