Wednesday, February 25, 2015

Beyond Random Motor Babbling: How to explore when learning behaviors

Humans can learn or improve skills by practicing, e.g., tennis strokes improve as you spend time whacking balls against a wall.  This process is likely to involve exploration of a problem space, in the case of tennis, trying different the actuator values to develop appropriate responses to a range of different ball trajectories and speeds.

In robot skill learning, an important question is 'how do we do this exploration?'  Many papers use random motor babbling [refs], typically within a safe envelope of values and most commonly with a flat probability distribution within each range [refs].  It is highly unlikely that this is how humans and animals explore.

It has been recognised for some time [ref] that system noise can provide sufficient variability to support exploration leading to the identification of effective solutions.  Pinheiro et al. [1] just showed that such variability is also a sufficient requirement for learning new skills in humans.
Think more about using this to inspire new, biologically more plausible, exploration strategies for robot learning.

[1] João de Paula Pinheiro, Pricila Garcia Marques, Go Tani and Umberto Cesar Corrêa (2015) Diversification of motor skills rely upon an optimal amount of variability of perceptive and motor task demands.  In Adaptive Behavior, (only online so far http://adb.sagepub.com/content/early/2015/02/23/1059712315571369?papetoc ).


Thursday, May 23, 2013

Other health care projects

There are a number of healthcare research projects that are highly relevant to tele-care.  Below are the ones I know about so far:


  •  Sensor Platform for HEalthcare in a Residential Environment (SPHERE), EPSRC

Thursday, October 04, 2012

Tele-assisted living

The idea behind tele-assisted living is that many services can be provided remotely if there is a mobile manipulator connected to the Internet in someone's home.
Tele-operating this platform will remove the requirement for high levels of robot autonomy which we are not likely to see for decades.

Such a platform could also provide a large amount of training data to speed up the development of robot autonomy.  My interest is in the area of robot learning and the delegation of skills from a tele-operator to the robot.

Many research activities are currently ongoing in this area and this blog post is meant as a list of these.

FP7 projects

  • ACCOMPANY: Acceptable robotiCs COMPanions for AgeiNg Years (includes Hertfordshire and Birmingham)
  • AALIANCE2: The European Ambient Assisted Living Innovation Alliance (includes Tunstall Healthcare Ltd., UK)
  • CAPSIL: International Support of a Common Awareness and Knowledge Platform for Studying and Enabling Independent Living (FP7 Support Action, includes Imperial College, London)
  • CONFIDENCE: Ubiquitous Care System to Support Independent Living (no UK partner)
  • DOMEO: Domestic robot for elderly assisteance (no UK partner) 
  • FLORENCE: Multi-Purpose Mobile Robot for Ambient Assisted Living (no UK partners)
  • KSERA: Knowledgeable Service Robots for Ageing (no UK partners)
  • MOBISERV: An Integrated Intelligent Home Environment for the Provision of Health, Nutrition and Mobility Services to the Elderly (includes Bristol)
  • ROBOT-ERA : Implementation and Integration of Advanced Robotic Systems and Intelligent Environments in Real Scenarios for the Ageing Population (includes Plymouth)
  • SCRIPT: Supervised Care and Rehabilitation Involving Personal Tele-Robotics (includes Hertfordshire and Sheffield)
  • SRS: Multi-Role Shadow Robotic System for Independent Living (includes Cardiff and Bedfordshire)

Interest groups

Thursday, April 26, 2012

Horizontal Connections

I have still to find a neural model of the competitive selection mechanism that finds the winning node of a SOM. The process has been parallelised many times for parallel SOM implementations based on FPGAs or CUDA (such as Aquila).  There is a lot of work on horizontal/lateral connections in the cortex.  It would be very interesting to study a SOM implementation using such lateral connections to select SOM winners.

Endogenously Active Elements

Endogenous activity has been raised as an important mechanism for cognition, both in terms of neural self-organisation (Choa and Choi 2012) and for control (Bechtel and Abrahamsen 2010 ).  Such a mechanism would be a very interesting extension to our PLANCS framework for behavior-based robotics (Dahl and Giraud-Carrier 2005).
  • Bechtel, W. and Abrahamsen, A. (2010) Understanding the brain as an endogenously active mechanism. In Proceedings of the 32nd Annual Conference of the Cognitive Science Society, Austin, Texas. (see Bechtel's web site http://mechanism.ucsd.edu/~bill/index.html)
  • Choa, M. W. and Choi, M. Y. (2012) Spontaneous organization of the cortical structure through endogenous neural firing and gap junction transmission, Neural Networks 31:46–52.
  • T. S. Dahl and C. Giraud-Carrier, "Incremental Development of Adaptive Behaviors using Trees of Self-Contained Solutions," Adaptive Behavior, 13(3)243-260, 2005.

Monday, March 19, 2012

Interpolation with SOMs

The traditional use of a SOM is to reduce the dimensionality of the data through vector quantization.
It may be possible, however, to use SOMs to interpolate across a data space based on a small number of data points.  Turns out that not only is it possible, but it has been done by a number of people!

  • Goppert and Rosenstiel (1997) The Continuous Interpolating Self-Organizing Map, Neural Processing Letters, 5:185-192.
  • Yin and Allinson (1999) Interpolating self-organizing map (iSOM), Electronics Letters, 35(19):1649-1650.
  • Kawano, Orii, Shiraishi and Maeda (2010) A Method for Multiple Image Interpolation Employing Self-Organizing Map, Proceedings of the SMC'2010, pages 4035-4040, 10-13 October, Istanbul.
Now, applying this to RL so that we can construct an attractor landscape is a very interesting idea. 

Wednesday, February 15, 2012

Incremental exploration

Learn from demonstration typically means learning from training data that are in the form of a relatively small number of complex sequences of observations and potentially actions. The strength of this learning paradigm is that the data provided is related to the crucial areas of the problem space. In the case of reinforcement learning, this would involve the key reward states and effective paths to these states from relevant starting states. However, due to the restricted part of a problem space that can be covered using this form of learning, it typically leads to brittle behaviours that are not able to compensate for perturbations that place the robot outside the known area. One solution to this problem is to use the training data to learn a policy that generalises across large areas of the problem space, such as the Nonlinear Dynamical Systems presented by Ijspeert et al. [3]. Another approach is to hard code a mechanism for returing to the known area such as the extension to Gaussian Mixture Models presented by Calinon [2]. Abbeel and Ng [1], argued, from their experience in the domain of autonomous helicopter control, that an explicit exploration policy is not required in order to improve performance up to or beyond that of the teacher. Instead, the natural perturbations would provide sufficient exploration.

Bill Smart and Leslie Kaelbling [5] developed the JAQL (Joystick and Q-learning?) algorithm to overcome this problem. The JAQL algorithms has two different learning phases. In the first phase, the robot is driven through the "interesting" parts of the problem space by a hand coded controller or by a human controller using a joystick. In the second phase, the policy learned was in control and responsible for further exploration, running in a more standard reinforcement learning mode. The JAQL algorithm has an explicit exploration policy designed to work with policies learnt from demonstration.

The JAQL exploration policy creates slight deviations from the greedy action by adding a small amount of Gaussian noise [4]. This policy creates actions that are "similar to, but different from", the greedy action.

Our RLSOM algorithm has so far been applied only to learning by demonstration, but should be capable of handling learning from exploration without other modifications that a reasonable exploration policy. This is one of the most exciting direction in which to take our research.

[1] Pieter Abbeel and Andrew Y. Ng, Exploration and apprenticeship learning in reinforcement learning. In the Proceedings of the 22nd International Conference on Machine Learning (ICML'05), pp1-8, August 7-11, Bonn, Germany, 2005.

[2] Sylvain Calinon, Robot Programming by Demonstration: A Probabilistic Approach. EPFL/CRC Press, 2009.

[3] Auke J. Ijspeert, Jun Nakanishi and Stefan Schaal, Movement imitation with nonlinear dynamical systems in humanoid robots. In the Proceedings of the International Conference on Robotics and Automation (ICRA'02), pp1398-1403, May 11 - 15, Washington, DC, 2002.

[4] William D. Smart, Making Reinforcement Learning Work on Real Robots. Ph.D. thesis, Department of Computer Science, Brown University, 2002.

[5] William D. Smart and Leslie Pack Kaelbling, Reinforcement Learning for Robot Control. In Mobile Robots XVI (Proceedings of the SPIE 4573), pp92-103, Douglas W. Gage and Howie M. Choset (eds.), Boston, Massachusetts, 2001.