Principles and de nitions about sensorimotor learning
In this section, we introduce learning theories, concepts and main results in sensorimotor control. We will expose basic concepts, that are widely used today, and sometimes referred to in our work. We rst de ne the cognitive and sensorimotor levels as generally considered in neuroscience. We will then brie y summarize the main concepts regarding sensory feedback, and depict principles of sensorimotor control and learning, with a particular focus on basic notions on adaptive processes. Finally, we will give an overview of the main theories of perception-action coupling, illustrating the various approaches.
During the interaction with an object, or while using a gestural interactive system, the user is involved in a loop where several levels of processes are involved: perception, information processing, decision making, action, etc.). A key feature of the physiological and functional characteristics of the biological sensors, actuators and of the nervous system involved, is the dynamic aspect of the interaction. Even in the case of voluntary actions, the system has very short time to take decision and perform actions. Brain structures responsible for planing, problem solving, decision making, cannot be activated continuously. Consequently, the separation between two levels of neural activity has been introduced in neuroscience, the sensorimotor level and the cognitive level (Paillard, 1985).
This distinction is partly based on functional roles of the neural processes. Kerlirzin et al. (Kerlirzin et al., 2009) de ne cognition as the mental processes that include acquiring, storing, transforming and using knowledge and expertise. They give the examples of perception, memorization, reasoning, information processing, problem solving and decision making as such processes. These cognitive processes are largely voluntary and conscious. The sensorimotor level refers to more low-level processes involved in transmitting to the brain signals coming from the sensory receptors, in the adjustment of muscle length and forces, and in the regulation of feedback loops involved in the control of movement. However, the two levels are not independent and have to collaborate during action-perception processes. As we explained in chapter 1, we focused our work on the sensorimotor system, and particularly on the function and adaptive properties of the sensorimotor loop (Wolpert and Ghahramani, 2000) regarding auditory inputs.
Feedback is de ned by Hartveld (Hartveld and Hegarty, 1996) as an \information received back by the appropriate control centers in the brain on the resultant movement ». This sensory (somatic) information is provided during the production of a movement, or after a task achievement. Any movement of the body generates intrinsic feedback, which is a sensory information resulting from the deformations of the sensory organs (skin, muscles and tendons receptors) during the movement itself. It can be for instance tactile, proprioceptive or kinesthetic. This one word encompasses a very wide range of perceptive variables. Proprioceptive feedback for instance is always active while we move and is omnipresent. The vestibular system gives the nervous system information that can be quali ed as intrinsic as it delivers data from inertial and inclination sensors in the inner ear. Exteroceptive feedback de nes information from the outcome of the movement through the senses, like the vision of the body moving. This information can come from an observer, like a coach or a physical therapist. Hearing the auditory outcome of our actions can be considered as exteroceptive feedback, either its related directly to our body (sound of steps) or to the object or system we manipulate (from opening a lock to driving a car). This type of information, coming from outside of the body is called extrinsic feedback and can supplement intrinsic feedback taking many forms (auditory, visual, verbal, etc.).
Extrinsic feedback can be arti cially created. In this case, it is called augmented feedback. It can take the same forms (for instance a visual cue or a verbal advice) but is generated by a speci c equipment like computer, body sensors, etc). It can be conveyed through any sensory modality. Feedback is often named after the source of information it gives: ‘biofeedback’ refers to providing quantitative biological information that are normally non accessible, like the expression of a muscle force or electrical activity. Augmented feedback is implemented to add supplementary information to an active sensory channel: for instance, it augments the visual channel when providing supplementary written or iconic information on a display when performing a task.
Motion capture and sensing technology allow to provide concurrent feedback while performing an action. The system usually acquires information from the action with sensors and computes and deliver an ‘online’ feedback synchronized with the action. It can still be discrete or continuous information, and represents either body-related or task-related information.
Main theories about action-perception coupling
Many theories allowing to interpret the coupling between perception and action, and its role in motor control, have been established since the advent of cognitive sciences. We describe here the most important of them and precise the speci cities of the auditory coupling that we consider. The large majority of the studies on action and perception have been carried out in the elds of vision and language development. More recently, the speci c case of expert musicians served as a particular context of complex auditory-motor interaction. The perception-action coupling has been less investigated in the case of interactive sonic systems. We draw the reader’s attention on some useful concepts that still need to be extended to interactive sonic systems.
From a cognitivist point of view, it is essential to constantly process data coming from our sensory system when performing a motor action. Perception enables us to modify and correct a trajectory (either on a single limb or a global trajectory as while driving a car). The trajectory must be predicted, simulated internally and evaluated, to achieve the best performance. This requires the notions of motor programs and internal representation in the CNS (central nervous system), which is considered as purely an information processing unit. A typical example is given by Kerlirzin et al. (Kerlirzin et al., 2009): a waiter when lifting a bottle from his tray will automatically adjust the force in his arm as the bottle leaves its surface. On the other hand, if the bottle is removed by anyone else, it is impossible for the waiter to anticipate and adapt. Even if he receives warning he will not keep the tray still. Although this approach has historically served many experimental work and models (Wolpert and Flanagan, 2001; Wolpert et al., 2005) for motor control and prediction, others approaches question the capability of the central nervous system to build these internal representations of movements, especially 1- because of the quantity of information and schemes that would need to be stored and processed during the action, and 2- because the metaphor of the CNS as an information processing and storage machine can be challenged (Varela, 1989).
Enaction and embodiment
Varela, Thompson and Rosch (Varela et al., 1991; Varela, 1989) suggested an alternative to the cognitivist approach of a pre-determined world that the brain would need to learn and retrieve. They proposed that cognition should be considered as a situated action. Two main points characterize the notion of enaction: perception is seen as an action, and cognitive structures arise from sensorimotor schemes (which guide action with perception) (Varela et al., 1991). The theory of enaction explores the way a subject, doted with perceptual abilities, can guide his actions within a local context. As the context is constantly changing, due to perceptual action from the subject, perception is not deciphering the pre-determined world, but determining the sensorimotor relationships describing how action can be guided. The mind and the body being functionally connected in both directions represents the embodiment of the human experience. This notion tends to blur the theoretical separation of cognitive and sensorimotor levels. It is also close to (and inspired by) the phenomenological approach by Merleau-Ponty (Merleau-Ponty, 1945). The concept of embodiment has been recently applied to many elds where interaction is central, like music cognition and production (Leman, 2008), communication, human-computer interaction, and gesture-sound interaction (Caramiaux, 2011). For an application of enactment and ‘re-enactment’ in sonic interactions see (Schnell, 2013). Implementing and testing models derived from the notion of enaction is still a challenging question, although some example with musical or sonic interactive systems can be found (see section 2.2).
Ecological and direct perception approaches
For Gibson, no pre-existing model or representation of the outer world is needed to actually perceive; in the ecological approach, the information is ‘given’ to us through the stimulation of the perceptual system (Gibson, 1986). The foundation for perception is ambient, ecologically available information. This theory aims at re-locating the subject in his environment, hence the term of ‘eco-logical’ or ecological. In the framework of the ecological theory, Gibson introduces the concept of \a ordance », which represents the possible actions that an object o ers in a particular context or environment. These a ordances can be interpreted as actions which are directly perceived and di er from the actual physical properties of the object.
Inspired by an ecological thinking, the general (tau) theory is an example of these principles. The body is considered as a whole entity, interacting with its environment from which it gets a continuous ow of information. The general tau theory, also referred to as theory of perceptual guidance, formalizes movement control as the closure of a spatial and/or force gap between the e ector and the goal of the movement (Lee et al., 1999). This closure is ensured by the regulation of the taus of the gap, which are the time-to-closure of the gap at the current closure-rate. Under this framework, movement coordination is modeled as keeping the taus of the gaps involved in a constant ratio, called coupling (Lee et al., 2001). This theory states that sensorimotor control is achieved by constantly sensing the taus of the gaps and keeping them coupled. It has been observed in both human (e.g. in sports or daily activities actions) and animals (like in bats using echolocation).
Also inspired by the ecological theory, the direct approach assumes that the central nervous system does not have to make calculus or computation in order to draw a link between sensations and actions. It only has to nd in the environment the appropriate signals that have to be properly associated with the correct motor response. Experimental results show that observation is linked to imagined actions to predict forthcoming action e ects, and tend to demonstrate the importance of self-perception to predict actions outcomes (Knoblich and Flach, 2001).
Motor theory of perception
In this framework, perception is an action that is simulated. E cient adaptive models have been proposed combining feedforward and feedback controllers (Wolpert, 1997; Wolpert and Kawato, 1998; Miall and Wolpert, 1996). Motor theory of perception considers the CNS as a simulator that uses internal models and predicts future perceptual states (resulting from the application of motor commands) in order to take decisions. Berthoz (Berthoz, 2003) formulated that the experience of the body (and memory) allows the CNS to anticipate the sensory and motor consequences of the action.
Table of contents :
1.1 General introduction
1.2 Research context
1.2.1 Gesture-sound interactive systems
1.2.2 Why auditory feedback?
1.3 Specic aims of the work
1.4 Experimental choices
1.4.1 Materials and methods
1.4.2 Sound synthesis
1.4.3 Proposed concepts
1.5 Structure of the manuscript
2 Basic concepts and literature results overview
2.1 Principles and denitions about sensorimotor learning
2.1.1 Sensorimotor and cognitive levels
2.1.2 Sensory feedback
2.1.3 Sensorimotor learning and integration
2.1.4 Main theories about action-perception coupling
2.2 State of the Art: auditory feedback of motion
2.2.1 Auditory feedback and sonication
2.2.2 Sound-movement relationship
2.2.3 Learning (with) auditory feedback
2.3 Connected elds and applications
2.3.1 Sonic Interaction Design
2.3.2 Music performance
2.3.3 Auditory feedback for sport and physical exercise
2.3.4 Auditory feedback for rehabilitation
2.4 General comments
3 ARTICLE j From ear to hand: the role of the auditory-motor loop in pointing to an auditory source
3.2 Materials and methods
3.2.2 Experimental setup
3.2.3 Experimental procedure
3.3 Data analysis
3.3.1 Level of performance
3.3.2 Movement analysis
3.4.1 Statistical analysis
3.4.2 Level of performance
3.4.3 Global kinematics
3.4.4 Movement dynamics and segmentation
3.4.5 Head movement analysis
3.5 Discussion and conclusion
4 Touching sounds: gestural interaction with a virtual sonied object
4.1 The sensory substitution paradigm
4.1.1 Neural basis: cerebral plasticity
4.1.2 Experimental tool and compensatory apparatus
4.2 Audio-based sensory substitution
4.3 A need for interaction and movement
4.4 The concept of Auditory Virtual Surface
4.6 Experiment: sensing a geometric property of a virtual sounding object
4.6.3 Data analysis
4.7 Conclusion and perspectives
4.7.1 Related questions
4.7.2 The virtual water bowl
5 ARTICLE j Continuous sonication in a two-dimensional visuo-manual tracking task
5.2 Materials and Methods
5.2.2 Experimental setup
5.2.3 Experimental procedure
5.3.1 Experiment 1
5.3.2 Experiment 2
5.3.3 Experiment 3
6 Sonication for eye movement control
6.2 Smooth pursuit eye movements
6.2.1 Eye movements
6.2.3 The \reverse-phi » illusion
6.2.4 Auditory feedback in oculomotor studies
6.3 Sonication for free smooth pursuit eye movement learning
6.3.3 Setup and stimuli
6.3.5 Data analysis
7 ARTICLE j Learning movement kinematics with a targeted sound
7.2 Related Works
7.2.1 Sound-oriented Task
7.2.2 Auditory Feedback in Motion-oriented Task
7.3 Materials and Methods
7.3.1 Experimental Setup
7.3.2 Experimental Procedure
7.3.4 Data Analysis
7.3.5 Angular Velocity Prole Parameters
7.4.1 Exploration Phase
7.4.2 Adaptation Phase
7.4.3 Qualitative Comments of the Subjects
7.5 Discussion and Conclusion
8.1 Main results
8.2 General discussion
8.4 Conclusive words
Appendices and supplementary articles
A ARTICLE j Low-cost motion sensing of table tennis players for real time feedback
A.2 Motion sensing system
A.3.2 Experimental procedure
A.5 Conclusion and perspectives
B Sonication of the coordination of arm movements
B.2 Motion sensing and analysis
B.3 Sonication strategies
C Technical specications of the eyetracker