Reinforcement Learning for Extended Intelligence
Shie Mannor, Technion - Israel Institute of Technology, Israel
Neural-fly Enables Rapid Learning for Agile Flight in Strong Winds for Drones
Soon-Jo Chung, California Institute of Technology, United States
Control of Uncertainty or Control with Uncertainty? A New Control Design Paradigm for Stochastic Systems
Panagiotis Tsiotras, Georgia Institute of Technology, United States
Using Delays for Control
Emilia Fridman, Tel Aviv University, Israel
Reinforcement Learning for Extended Intelligence
Shie Mannor
Technion - Israel Institute of Technology
Israel
Brief Bio
Shie Mannor earned a PhD in Electrical Engineering from the Technion at 2002. Shie was then a Fulbright postdoctoral associate at LIDS (MIT) for two years. Shie was an assistant professor and then an associate professor at the Department of Electrical and Computer Engineering at McGill University from July 2004 until August 2010, where he held the Canada Research Chair in Machine Learning from 2005 to 2009. Shie has been a professor of Electrical and Computer Engineering at the Technion since 2008 where where he is the incumbent Edwards Chair in Engineering.
Shie has published over 80 journal papers and over 200 conference papers in leading venues and holds 10 patents. His papers were cited over 20,900 times and his h-index is 72. Shie’s research interests include machine learning and data science with an emphasis on reinforcement learning, planning and control, and analysis and control of large-scale systems.
Shie has been active in the high-tech industry in Israel, Canada and the US where he has consulted to dozens of companies over the years. He co-founded four companies: Kashya (founded at 2000 in Israel and acquired by EMC in 2006) WideSail Technologies (founded at 2007 in Montreal), Jether Energy Research (founded at 2014 in Israel), and Amooka AI (founded in 2018 in Israel and acquired by Ford). Shie was the chief scientist of SAIPS, the Israeli AI R&D subsidiary of the Ford Motor Company 2018-2020. Shie joined Nvidia research in April 2020 where he is a distinguished scientist.
Abstract
In this talk I will start from giving a broad overview of my research, focusing on the essential elements needed for scaling reinforcement learning to real-world problems. I will present a scheme called "extended intelligence" that concerns the design of systems that participate as responsible, aware and robust elements of more complex systems. I will then deep dive into the question of how to create control policies from existing historical data and how to sample trajectories so that future control policies would have less uncertain return. This question has been central in reinforcement learning in the last decade if not more, and involves methods from statistics, optimization, and control theory. We will focus on one the possible remedies to uncertainty in sequential decision problems: using risk measures such as the conditional value-at-risk as the objective to be optimized rather than the ubiquitous expected reward. We consider the complexity and efficiency of evaluating and optimizing risk measures. Our main theme is that considering risk is essential to obtain resilience to model uncertainty and model mismatch. We then turn our attention to online approaches that adapt on-the-fly to the level of uncertainty of a given trajectory, thus achieving robustness without being overly conservative. If time permits, I will shortly discuss a couple of real-world applications my group has been working: one in energy management and one in healthcare.
Neural-fly Enables Rapid Learning for Agile Flight in Strong Winds for Drones
Soon-Jo Chung
California Institute of Technology
United States
Brief Bio
Soon-Jo Chung is Bren Professor of Aerospace and Control and Dynamical Systems in the California Institute of Technology. Prof. Chung is also a Research Scientist of the NASA Jet Propulsion Laboratory. Prof. Chung received the S.M. degree in Aeronautics and Astronautics and the Sc.D. degree in Estimation and Control with a minor in Optics from MIT in 2002 and 2007, respectively. He received the B.S. degree in Aerospace Engineering from KAIST in 1998. From 2009 to 2016, Prof. Chung was an associate professor and an assistant professor at the University of Illinois at Urbana-Champaign. Professor Chung's research focuses on distributed spacecraft systems, space autonomous systems, and aerospace robotics, and in particular, on the theory and application of complex nonlinear dynamics, control, estimation, guidance, and navigation of autonomous space and air vehicles.
He is the recipient of the UIUC Engineering Dean's Award for Excellence in Research, the Arnold Beckman Faculty Fellowship of the U of Illinois Center for Advanced Study, the AFOSR Young Investigator Program (YIP) award, the NSF CAREER award, a 2020 Honorable Mention for the IEEE Robotics and Automation Letters Best Paper Award, three best conference paper awards (2015 AIAA GNC, 2009 AIAA Infotech, 2008 IEEE EIT), and five best student paper or finalist awards. He also received multiple teaching awards including the UIUC List of Teachers Ranked as Excellent and the instructor/advisor for the 1st place national winning team of the AIAA Undergraduate Team Space Design Competition. The work and robots of Prof. Chung’s and his colleagues have received extensive media coverage. The robotic bat, called Bat Bot, was placed in a special exhibit at the Museum of Arts and Crafts in Hamburg along with the work of virtuosos like Albrecht Dürer and Alexander von Humboldt.
Prof. Chung is an Associate Editor of the IEEE Transactions on Automatic Control and the AIAA Journal of Guidance, Control, and Dynamics. He was an Associate Editor of the IEEE Transactions on Robotics, and the Guest Editor of a Special Section on Aerial Swarm Robotics published in the IEEE Transactions on Robotics. He is an Associate Fellow of AIAA.
Abstract
Executing safe and precise flight maneuvers in dynamic high-speed winds is important for the ongoing commoditization of uninhabited aerial vehicles (UAVs). However, because the relationship between various wind conditions and its effect on aircraft maneuverability is not well understood, it is challenging to design effective robot controllers using traditional control design methods. We present Neural-Fly, a learning-based approach that allows rapid online adaptation by incorporating pre-trained representations through deep learning. Neural-Fly builds on a key observation that aerodynamics in different wind conditions share a common representation and that the wind-specific part lies in a low-dimensional space. To that end, Neural-Fly uses a proposed learning algorithm, domain adversarial invariant meta-learning (DAIML), to learn the shared representation, only using 12 min of flight data. With the learned representation as a basis, Neural-Fly then uses a composite adaptation law to update a set of linear coefficients for mixing the basis elements. When evaluated under challenging wind conditions generated with the Caltech Real Weather Wind Tunnel, with wind speeds up to 43.6 km/hour (12.1 m/s), Neural-Fly achieves precise flight control with substantially smaller tracking error than state of-the-art nonlinear and adaptive controllers. In addition to strong empirical performance, the exponential stability of Neural-Fly results in robustness guarantees. Last, our control design extrapolates to unseen wind conditions, is shown to be effective for outdoor flights with only onboard sensors, and can transfer across drones with minimal performance degradation. Science Robotics Paper: https://www.science.org/stoken/author-tokens/ST-463/full
Control of Uncertainty or Control with Uncertainty? A New Control Design Paradigm for Stochastic Systems
Panagiotis Tsiotras
Georgia Institute of Technology
United States
Brief Bio
Dr. Panagiotis Tsiotras is the David and Andrew Lewis Endowed Chair Professor at the School of Aerospace Engineering at Georgia Tech. At Georgia Tech, he is also the Director of the Dynamics and Control Systems Laboratory and an Associate Director for the Institute for Robotics and Intelligent Machines (IRIM). His current research interests are in optimal and nonlinear control and their connections with AI and applications to aerial, space and ground vehicle autonomy. He holds degrees in Mechanical Engineering, Aerospace Engineering, and Mathematics. He is currently the Chief Editor of the Frontiers of Robotics and AI in the area of Space Robotics and an Associate Editor for Dynamical Games and Applications. Previously, he served at the Editorial Boards of the AIAA Journal of Guidance, Control, and Dynamics, the IEEE Transactions of Automatic Control, the IEEE Control Systems Magazine, and the Journal of Dynamical and Control Systems. He is the recipient of the NSF Career Award, the IEEE Excellence Award in Aerospace Control, the Outstanding Aerospace Engineer award from Purdue, and the Sigma Xi Research Excellence Award. He is a Fellow of AIAA, IEEE, and AAS.
Abstract
Uncertainty propagation and mitigation is at the core of all robotic and control systems. The standard approach so far has followed the spirit of controlling a system “with uncertainties,” as opposed to the direct control “of uncertainties.” Recent advances from controllability of the covariance of the distribution of the state trajectories provide us with a new tool to control stochastic systems with strict performance guarantees. In this talk I will review some recent results on covariance control for discrete stochastic systems subject to probabilistic (chance) constraints and will demonstrate the approach on several control and robot motion planning problems under uncertainty. The resulting theory has several connections to the classical Optimal Mass Transport (OMT), it is elegant, and numerically efficient (often resulting in a convex program).
Using Delays for Control
Emilia Fridman
Tel Aviv University
Israel
Brief Bio
Emilia Fridman received the M.Sc. degree from Kuibyshev State University, USSR, in 1981 and the Ph.D. degree from Voronezh State University, USSR, in 1986, all in mathematics. From 1986 to 1992 she was an Assistant and Associate Professor in the Department of Mathematics at Kuibyshev Institute of Railway Engineers, USSR. Since 1993 she has been at Tel Aviv University, where she is currently Professor of Electrical Engineering-Systems. She has held visiting positions at the Weierstrass Institute for Applied Analysis and Stochastics in Berlin (Germany), INRIA in Rocquencourt (France), Ecole Centrale de Lille (France), Valenciennes University (France), Leicester University (UK), Kent University (UK), CINVESTAV (Mexico), Zhejiang University (China), St. Petersburg IPM (Russia), Melbourne University (Australia), Supelec (France), KTH (Sweden).
Her research interests include time-delay systems, networked control systems, distributed parameter systems, robust control, singular perturbations and nonlinear control. She has published two monographs and more than 200 articles in international scientific journals. She serves/served as Associate Editor in Automatica, SIAM Journal on Control and Optimization and IMA Journal of Mathematical Control and Information. In 2014 she was Nominated as a Highly Cited Researcher by Thomson ISI. Since 2018, she has been the incumbent for Chana and Heinrich Manderman Chair on System Control at Tel Aviv University. She is IEEE Fellow since 2019. In 2021 she was recipient of IFAC Delay Systems Life Time Achievement Award and of Kadar Award for outstanding research in Tel Aviv University. She is currently a member of the IFAC Council.
Abstract
In this talk by "using delays" I understand either Time-Delay Approaches to control problems (that originally may be free of delays) or intentional inserting delays to the feedback. I will start with an old Time-Delay approach - to sampled-data control. In application to network-based control with communication constraints, this is the only approach that allows treating transmission delays larger than the sampling intervals. I will continue with "using artificial delays" via simple Lyapunov functionals that lead to feasible LMIs for small delays and to simple sampled-data implementation. Finally I will present a New Time-Delay approach - this time to Averaging. The existing results on averaging (that have been developed for about 60 years starting from the works of Bogoliubov and Mitropolsky) are qualitative: the original system is stable for small enough values of the parameter if the averaged system is stable. Our approach provides the first Quantitative bounds on the small parameter making averaging-based control (including Vibrational Control and Extremum Seeking) reliable.