Reinforcement learning steps robotic prosthetics forward

Researchers from Arizona State University, North Carolina State University and the University of North Carolina have developed an intelligent system for “tuning” powered prosthetic knees, allowing users to walk comfortably with the prosthetic device in minutes rather than hours. The system is the first to rely solely on reinforcement learning to tune a robotic prosthesis.

Jennie Si, a professor in the School of Electrical, Computer and Energy Engineering in ASU’s Ira A. Fulton Schools of Engineering, co-authored a paper released this week in the journal IEEE Transactions on Cybernetics.  

Using the reinforcement learning framework, Si developed the algorithm that essentially “teaches” a prosthetic device to adapt to a user’s normal walking gait using data collected from a suite of sensors in the device and the person’s natural walking pattern.

Normally when someone receives a robotic prosthetic knee, a human practitioner works with them to modify a handful of parameters to specifically accommodate their walking gait. The process can take hours.

However, the new tuning system relies on a computer program that uses reinforcement learning to modify 12 control parameters that address prosthesis dynamics, such as joint stiffness, throughout the entire gait cycle. With this system, a person using a powered prosthetic knee can walk on a level surface in about 10 minutes.

“I recognized the unprecedented challenge of learning to control, in real time, a prosthetic device that is simultaneously affected by the human user. This is a co-adaptation problem that does not have a readily available solution from either classical control designs or the current, state-of-the-art reinforcement learning-controlled robots,” Si said.

She has been working on reinforcement learning from the dynamic system control perspective to account for sensor noise, interference from the environment and the demand of system safety and stability.

“A computer model adapts parameters on the device and compares the patient’s gait to the profile of a normal walking gait in real time,” said Helen Huang, co-author of the work and a professor in the Joint Department of Biomedical Engineering at NC State and UNC. “The model can tell which parameter settings improve performance and which settings impair performance. Using reinforcement learning, the computational model can quickly identify the set of parameters that allows the patient to walk normally. Existing approaches relying on trained clinicians can take half a day.”

While the work is currently done in a controlled, clinical setting, one goal would be to develop a wireless version of the system, which would allow users to continue fine-tuning the powered prosthesis parameters when being used in real-world environments.

“This work was done for scenarios in which a patient is walking on a level surface, but in principle, we could also develop reinforcement learning controllers for situations such as ascending or descending stairs,” Si said.

Huang says researchers hope to make the process even more efficient.

“For example, we think we may be able to improve the process by identifying combinations of parameters that are more or less likely to succeed and training the model to focus first on the most promising parameter settings.”

The researchers note that, while this work is promising, many questions need to be addressed before it is available for widespread use.

“The prosthesis tuning goal in this study is to meet normative knee motion in walking,” Huang explained. “We did not consider other gait performance, such as gait symmetry, or the user’s preference. As another example, we can use our tuning method to fine-tune the device outside of clinics and labs to make the system adaptive over time with the user’s need. However, we need to ensure the safety in real-world use since errors in control might lead to stumbling and falls. Additional testing is needed to show safety.”

If the system does prove to be effective and enter widespread use, it would likely reduce costs for prosthetic knee users by limiting the need to make clinical visits to work with practitioners.

Today three biomedical engineering doctoral students round out the research team — Xiang Gao at ASU and Yue Wen and Andrea Brandt at NC State and UNC.

Si and Huang met while Huang was a graduate student in the Fulton Schools, where she earned her master’s and doctorate degrees. The longtime collaborators plan to continue their high-impact work in this area. 

“We are thrilled to find out that our reinforcement learning control algorithm actually did learn to make the prosthetic device work as part of a human body in such an exciting applications setting,” Si said.

Matt Shipman, research communications lead at North Carolina State University, contributed to this story.