Deep Reinforcement Learning for Creating Advanced Humanoid Robotic Soccer Skills
Luís Paulo Reis, University of Porto, Portugal
Analysis and Control Design Tools for Dynamical Systems with Time-Delay
Wim Michiels, KU Leuven, Belgium
Lessons from Adaptive Control: Towards Real-Time Machine Learning
Anuradha Annaswamy, MIT, United States
Autonomous Driving: The Hidden Enabling Technology for a Sustainable Mobility Model
Sergio M. Savaresi, Politecnico di Milano, Italy
Deep Reinforcement Learning for Creating Advanced Humanoid Robotic Soccer Skills
Brief Bio
Luis Paulo Reis is an Associate Professor with Habilitation at the University of Porto in Portugal and Director of LIACC – Artificial Intelligence and Computer Science Laboratory. He is an IEEE Senior Member and he was president of the Portuguese Society for Robotics and of the Portuguese Association for Artificial Intelligence. He is Co-Director of LIACD - First Degree in Artificial Intelligence and Data Science. During the last 25 years, he has lectured courses, at the University, on Artificial Intelligence, Intelligent Robotics, Multi-Agent Systems, Simulation and Modelling, Games and Interaction, Educational/Serious Games and Computer Programming. He was the principal investigator of more than 20 research projects in those areas. He won more than 60 scientific awards including winning more than 15 RoboCup international competitions (including the last 3 editions of the Simulation 3D League - Humanoid Robots) and best papers at conferences such as ICEIS, Robotica, IEEE ICARSC and ICAART. He supervised 24 PhD and 160 MSc theses to completion and is supervising 12 PhD theses. He evaluated more than 50 projects and proposals for FP6, FP7, Horizon2020, FCT, and ANI. He was a plenary speaker at several international conferences such as ICAART, ICINCO, LARS/SBR, WAF, IcSports, SYROCO, CLAWAR, WCQR, ECIAIR, DATA/DELTA and IC3K. He organized more than 70 international scientific events and belonged to the Program Committee of more than 300 scientific events. He is the author of more than 450 publications in international conferences and journals (indexed at SCOPUS or Web of Knowledge).
Abstract
This talk focuses on Deep Reinforcement Learning (DRL) to create robust humanoid robotic skills and its application in the context of the FC Portugal 3D team, RoboCup simulation 3D league champion in 2022 and 2023. The talk will outline the fundamental principles of DRL, including its distinguishing features, such as learning from delayed rewards, handling the exploration-exploitation trade-off, and operating in complex, highly dimensional, dynamic environments. The talk will outline several methodologies developed in the context of the FC Portugal team to be able to use DRL, in a very efficient way, speeding up its training by more than 100 times, for creating advanced humanoid robotic skills such as kicking, running and dribbling.
Analysis and Control Design Tools for Dynamical Systems with Time-Delay
Wim Michiels
KU Leuven
Belgium
![]()
Brief Bio
Wim Michiels (1974) obtained a MSc degree in Electrical Engineering and a PhD degree in Computer Science from KU Leuven, in 1997 and 2002. He was a postdoctoral fellow of the Research Foundation Flanders (2002-2008) and a research associate at the Eindhoven University of Technology (2007). In 2008 he was appointed as an assistant professor at KU Leuven (associated professor 2012, professor 2017, full professor 2021), where he leads a research team within the Numerical Analysis and Applied Mathematics (NUMA) Section. His research interests include mathematical systems theory, dynamical systems, control and optimization, numerical linear algebra and scientific computing. His work focuses on the analysis and control of systems described by functional differential equations and other infinite-dimensional systems, on control of systems with a network structure, and on large-scale linear algebra problems in the context of control and optimization. He has published in a variety of journals in the area of computational and applied mathematics, control theory, optimization and dynamical systems. He is lead author of the book Stability, Control and Computation of Time-Delay Systems, SIAM, 2014 (2nd edition). He coordinated the H2020 Innovative Training Network UCoCoS, on the analysis and control of complex systems. He has been Associate Editor for the IEEE Transactions on Automatic Control (2014-2019), Systems and Control Letters (2010-2014 and 2023-), Communications in Nonlinear Science and Numerical Simulation (2015-2018), Calcolo (2019-), Frontier in Control Engineering (2020-) and area editor of the Springer book series Advances in Delays and Dynamics (2014-). He has been an IPC chair of the IFAC Workshop on Time-Delay systems (2016, 2021) and the IFAC Workshop on Periodic Control Systems (2016) and has been co-organizer of the ILAS 2016 conference in Leuven. He established and currently leads the IFAC Working Group on Time-Delay Systems (with G. Orosz). He a passionate teacher of six yearly courses at KU Leuven and has vast experiences as lecturer in international PhD training programs (SOCN, CISM, DISC, EECI). He has been a member of the KU Leuven Research Council.
Abstract
Time-delays are important components of many systems from engineering, physics and life science, due to the fact that the transfer of material, energy and information is mostly not instantaneous. They appear for instance as computation and communication lags, they model transport phenomena and hereditary and they arise as feedback delays in control loops. The inclusion of time-delays in the mathematical models gives rise to a description in terms of delay-differential equations.
From a qualitative point of view, the presence of time-delays in dynamical systems may induce complex behavior and this behavior is not always intuitive. Even if the system's equation is scalar, oscillations and chaotic behavior may occur. But on the other hand time-delayed feedback is typically used for stabilizing chaotic systems. Time-delays in control loops are usually associated with degradation of performance and robustness, but there are situation where time-delays are beneficial and even used as controller parameters. Delays may interact with different scales of the system: whereas sometimes very large delays can be tolerated, there are situations where an arbitrarily small delay may destabilize a stable system.
The aim of my talk is to present a control oriented guided tour on time-delay systems. I will first review basic properties of time-delay systems and discuss a flexible modeling framework for interconnected systems, in terms of delay differential-algebraic equations. Next I will address analysis and design methods for structured feedback controllers, relying on non-smooth optimization and illustrated by means of the new software package TDS-CONTROL. Examples from various application domains complete the presentation, with particular emphasis on delay based vibration control methodologies.
Lessons from Adaptive Control: Towards Real-Time Machine Learning
Anuradha Annaswamy
MIT
United States
![]()
Brief Bio
Dr. Anuradha Annaswamy is Founder and Director of the Active-Adaptive Control Laboratory in the Department of Mech. Eng. at MIT. Her research interests span adaptive control theory and its applications to several engineering systems including to aerospace, automotive, propulsion, and energy systems, cyber-enabled energy grids, and urban mobility. She has received best paper awards (Axelby; CSM), Distinguished Member and Distinguished Lecturer awards from the IEEE Control Systems Society (CSS), Best Paper award from IFAC for Annual Reviews in Control (2021-23), and a Presidential Young Investigator award from NSF. She is a Fellow of IEEE and IFAC. She is the recipient of the Distinguished Alumni award from Indian Institute of Science for 2021.Anu Annaswamy is the author of a graduate textbook on adaptive control and several journal and conference publications, and co-editor of two vision documents on smart grids, two editions of the Impact of Control Technology report, and the 2023 CSS report “Control for Societal-scale Challenges: Road Map 2030”. She is also a coauthor of a 2021 National Academy of Sciences, Engineering, and Medicine (NASEM) Committee report on the Future of Electric Power in the United States, and a 2023 NASEM report on the Role of Net-metering in the Evolving Electricity System. She served as the President of CSS in 2020. She is a Faculty Lead in the Electric Power Systems workstream in the MIT Future Energy Systems Center.
Abstract
The fields of adaptive control and machine learning have evolved in parallel over the past few decades, with a significant overlap in goals, problem statements, and tools. Machine learning as a field has focused on computer based systems that improve through experience. Often times the process of learning is encapsulated in the form of a parameterized model such as a neural network, whose weights are trained in order to approximate a function. The field of adaptive control, on the other hand, has focused on the process of controlling engineering systems in order to accomplish regulation and tracking of critical variables of interest. Learning is embedded in this process via online estimation of the underlying parameters. Whether in machine learning or adaptive control, this learning occurs through the use of input-output data. In both cases, the main algorithm used for updating the parameters is based on a gradient descent-like algorithm. Related tools of analysis, convergence, and robustness in both fields have a tremendous amount of similarity. As the scope of problems in both topics increases, the associated complexity and challenges increase as well. In order to address learning and decision-making in real time, it is essential to understand these similarities and connections to develop new methods, tools, and algorithms.
This talk will examine the similarities and interconnections between adaptive control and optimization methods commonly employed in machine learning. Concepts in stability, performance, safety, and learning, common to both fields will be discussed. Building on the similarities in update laws and common concepts, new intersections and opportunities for improved algorithm analysis will be explored. High-order tuners and time-varying learning rates have been employed in adaptive control leading to very interesting results in dynamic systems with delays. We will explore how these methods can be leveraged to lead to provably correct methods for learning in real-time with guaranteed fast convergence. Examples will be drawn from a range of engineering applications.
Autonomous Driving: The Hidden Enabling Technology for a Sustainable Mobility Model
![]()
Brief Bio
Sergio M. Savaresi received the M.Sc. in Electrical Engineering (Politecnico di Milano, 1992), the Ph.D. in Systems and Control Engineering (Politecnico di Milano, 1996), and the M.Sc. in Applied Mathematics (Catholic University, Brescia, 2000). After the Ph.D. he worked as management consultant at McKinsey&Co, Milan Office. He is Full Professor in Automatic Control at Politecnico di Milano since 2006. Since January 2023 he is the Chair of the Department of Electronics, Computer Sciences and Bioengineering (DEIB), Politecnico di Milano. He is author of more than 500 scientific publications. His main interests are in the areas of vehicles control, machine learning, and control applications, with special focus on smart mobility. He has been manager and technical leader of more than 400 research projects in cooperation with leading companies in the automotive industry. He is co-founder of 10 high-tech startup companies. He is the team leader of PoliMOVE, the winner of the the Autonomous Challenge @ CES 2022 (first ever high-speed fully-autonomous head-to-head multi-agent race).
Abstract
In the next 30 years a revolution is expected in the mobility model: the traditional personal mobility model (based on big, fossil-fuel-powered, personal-ownership cars) will be almost entirely replaced by Mobility-As-A-Service, autonomous, electric/H2 cars. This “revolution” aims to make a quantum leap in the overall efficiency of the mobility system, and to contribute to the improvement of the safety and sustainability of vehicles. This revolution will also deeply affect the structure of the entire automotive industry (layers, players, etc.).
Among the main technology megatrends, the autonomous-driving technology has a special/key role: not only is (by far) the most challenging from a technical point of view, but it will play the role of booster/catalyzer of all the other megatrends.
The plenary speech aims to provide a high-level overview of this technology revolution, highlighting the role and the impact of the autonomous-driving technology.
The role of autonomous motorsport within this technology roadmap will be also briefly presented and discussed.