From Shallow to Deep Kernel Machines
Johan Suykens, ESAT-Stadius & Leuven.AI Institute, KU Leuven, Belgium, Belgium
Available Soon
Bogdan Epureanu, University of Michigan, United States, United States
Observability and Diagnosability of Hybrid Dynamical Systems: Theory and Applications
Maria Domenica Di Benedetto, University of L'Aquila, Italy, Italy
Brief Bio
Johan Suykens is a full Professor with KU Leuven. He is author of the books "Artificial Neural Networks for Modelling and Control of Non-linear Systems" (Springer) and "Least Squares Support Vector Machines" (World Scientific), co-author of the book "Cellular Neural Networks, Multi-Scroll Chaos and Synchronization" (World Scientific) and editor of the books "Nonlinear Modeling: Advanced Black-Box Techniques", "Advances in Learning Theory: Methods, Models and Applications" and "Regularization, Optimization, Kernels, and Support Vector Machines". He has served as associate editor for the IEEE Transactions on Neural Networks and Learning Systems and the IEEE Transactions on Artificial Intelligence and action editor of Neural Networks. He is a recipient of the International Neural Networks Society INNS 2000 Young Investigator Award for significant contributions in the field of neural networks. He has been recently awarded the 2024 IEEE CIS Neural Networks Pioneer Award. He has served as a Director and Organizer of the NATO Advanced Study Institute on Learning Theory and Practice 2002, as a program co-chair for the International Joint Conference on Neural Networks 2004 and the International Symposium on Nonlinear Theory and its Applications 2005, as an organizer of the International Symposium on Synchronization in Complex Networks 2007, as co-organizer of the NIPS 2010 workshop on Tensors, Kernels and Machine Learning, as chair of ROKS 2013 International Workshop on Advances in Regularization, Optimization, Kernel methods and Support vector machines, and as chair of DEEPK 2024 International Workshop on Deep Learning and Kernel Machines. He has been awarded an ERC Advanced Grant 2011 and 2017, has been elevated IEEE Fellow 2015 for developing least squares support vector machines, and is an ELLIS Fellow. He is currently serving as program director for the Master AI program at KU Leuven.
Abstract
Kernel methods have proven effective in many engineering domains, including system identification, control, robotics, signal processing, and decision‑making under uncertainty. Kernel-based modelling has a long tradition in machine learning, with successful applications ranging from function estimation in reproducing kernel Hilbert spaces to support vector machines and Gaussian processes. Within this landscape, least squares support vector machines (LS-SVMs) offer a versatile primal-dual characterization for classification, regression, principal and canonical component analysis, spectral clustering, recurrent modelling, approximate solutions to partial differential equations, and optimal control, among other tasks.
This talk explores a series of extensions that move from shallow to deep kernel machines and establish connections to deep architectures and generative AI. We introduce multi-level feature learning through Deep Kernel Principal Component Analysis, based on restricted kernel machines whose model representations follow from LS-SVMs via conjugate feature duality. These representations naturally relate to Restricted Boltzmann Machines and Deep Boltzmann Machines, and in the shallow case can be aligned with the evidence lower bound of variational autoencoders.
We further show how attention mechanisms in transformers correspond to asymmetric kernel functions involving distinct feature maps for queries and keys. Through a kernel singular value decomposition, this leads to a primal-dual characterization within the LS-SVM framework. The resulting Primal-Attention method provides a regularized loss formulation that enables efficient transformer training and low-rank representations.
Overall, these developments offer a unifying perspective linking kernel methods, deep architectures, and generative modelling, opening new directions for theory, algorithms, and applications.
Brief Bio
Maria Domenica Di Benedetto is Professor Emeritus at the University of L’Aquila (Italy). She received her PhD (Doctorat d'État ès Sciences) from Université Paris-Sud (Orsay, France). She has been Adjunct Professor and McKay Professor at the University of California, Berkeley, and has held visiting positions at MIT, the University of Michigan, Ann Arbor, and the École Centrale de Nantes (France).
She founded the Italian Center of Excellence for Research DEWS and served as its Principal Investigator and Director from 2001 to 2019. She was President of the Italian Association of Researchers in Automatic Control (SIDRA) from 2013 to 2019, and President of the European Embedded Control Institute from 2009 to 2025.
She is an IFAC Fellow and a Life Fellow of the IEEE. She received the IEEE CSS Distinguished Member Award in 2024. She has been Vice-President Member Activities IEEE-CSS, Chair of the IFAC Nichols Medal Selection Committee and member of the IFAC Fellow Selection Committee. She is a Distinguished Lecturer of the IEEE Women in Engineering.
Her research interests include nonlinear and hybrid systems, diagnosability and predictability in cyber-physical systems, with applications to automotive systems, traffic control, smart grids, and biological systems. She is Editor of the IEEE Press Book Series in Control Systems Theory and Applications.
Abstract
Safety-critical embedded control systems, such as those in transportation and industrial plants, are becoming increasingly important as autonomy advances. These systems are naturally modeled as hybrid dynamical systems, characterized by the interaction of continuous and discrete dynamics. In their design, it is crucial to develop methods capable of dealing with such heterogeneous behaviors.
In many practically relevant scenarios, for example in automotive applications or in networked systems subject to intermittent sensing, only partial information about the system’s state is available. This makes state observability under hybrid dynamics, i.e., the possibility to reconstruct the hybrid system’s internal state from available measurements, essential for the implementation of control algorithms. Beyond this practical role, observability is also important as an intrinsic structural property of a system, underpinning the solution of problems such as fault diagnosis or detection of malicious attacks in safety- and security-critical applications.
While observability has been extensively studied in the continuous domain since the 1960s and in the discrete domain since the 1980s, its analysis for hybrid systems poses nontrivial additional challenges. In this talk, we show how hybrid features influence observability and related properties, including diagnosability and predictability, which are closely connected to safety and security. These features give rise to behaviors that cannot be directly inferred from classical results for traditional dynamical systems. To address the resulting complexity, we also discuss an approximate approach based on symbolic models, which provide finite-state abstractions of hybrid systems with guaranteed precision.
Examples drawn from applications highlight the relevance of hybrid modeling and observability analysis in safety-critical contexts.