Contraction-Aware Reinforcement Learning for Nonlinear Control with Statistical Robustness

Abstract

We propose contraction-aware reinforcement learning (CARL) to address the limitations of control contraction metrics (CCMs)—Riemannian metrics under which a closed-loop system is incrementally exponentially stable—in nonlinear path-tracking problems. While CCM-synthesized policies ensure pointwise satisfaction of the contraction conditions, they may not ensure long-term optimality over both transient and steady-state regimes, and their myopic nature makes them susceptible to learning biases when approximate dynamics are used to formulate the CCMs. CARL integrates the two: CCMs provide dynamics-informed feedback for learning a policy with a stability guarantee, while RL provides a framework for minimizing cumulative tracking error under approximate dynamics. Given a pretrained dynamics model, CARL simultaneously learns to generate CCMs and optimizes a policy for rewards defined by those metrics.

Key Contributions

Certification becomes convergence: We recast the problem of certifying stability for highly nonlinear dynamics as the problem of converging to the optimal policy—so that stability follows from policy convergence rather than from an explicit certificate.
Robustness beyond control-affine dynamics: Control-theoretic CCM synthesis presumes a control-affine model, which few highly nonlinear robots actually admit. By learning through a high-fidelity simulator instead, CARL implicitly relies on the true dynamics rather than a malformed control-affine approximation, yielding greater robustness.

Path-Tracking Results

Method Overview — Contraction-Aware Policy Optimization

The core of CARL is a joint learning scheme in which the contraction metric and the policy are optimized together, so that trajectory-level tracking performance is maximized while incremental exponential stability is maintained.

1. Contraction as Dynamics-Informed Reward

A system is contracting under a Riemannian metric \(M(x) \succ 0\) if the differential dynamics satisfy

\[\dot{M} + \text{sym}\!\left( M \frac{\partial f_{cl}}{\partial x} \right) + 2\lambda M \preceq 0,\]

which guarantees that any two closed-loop trajectories converge at rate \(\lambda\). Classical synthesis enforces this pointwise. CARL instead uses the degree of satisfaction of this condition, together with the Riemannian tracking error, to define a reward maximized over an entire trajectory:

\[J(\theta) = \mathbb{E}\left[ \sum_{t} \gamma^t \, r_{\text{CCM}}(x_t, u_t; M_\psi) \right].\]

Long-term optimality is therefore optimized directly, rather than assumed to follow from pointwise conditions.

2. Simultaneous Metric and Policy Learning

CARL maintains a metric network producing \(M_\psi(x)\) and a policy \(\pi_\theta(u \mid x, x_{\text{ref}})\). The metric is trained to yield a valid contraction metric for the pretrained dynamics model, while the policy is optimized with RL against the reward that metric induces. Because both are learned together, the metric adapts to the region of state space the policy actually visits, instead of certifying the entire state space up front.

3. Stability from Convergence

This coupling is what turns certification into convergence: at the optimum of \(J(\theta)\), the policy has maximized contraction-consistent behavior along its own trajectory distribution, so stability is inherited from policy convergence. And because the rollouts come from a simulator rather than a control-affine model, the guarantee is not tied to a representation that highly nonlinear robots rarely satisfy.

Cite this work

Cho, Minjae, Hiroyasu Tsukamoto, and Huy T. Tran. "Contraction-Aware Reinforcement Learning for Nonlinear Control with Statistical Robustness." IEEE Transactions on Robotics (2026), pp. 1-20. doi:10.1109/TRO.2026.3727138

@ARTICLE{11664333,
  author={Cho, Minjae and Tsukamoto, Hiroyasu and Tran, Huy T.},
  journal={IEEE Transactions on Robotics},
  title={Contraction-Aware Reinforcement Learning for Nonlinear Control with Statistical Robustness},
  year={2026},
  volume={},
  number={},
  pages={1-20},
  keywords={Dynamics;Modeling;Optimization;Tracking;Trajectory;Equations;Algorithms;Timing;Learning (artificial intelligence);Measurement;Contraction theory;control contraction metric;reinforcement learning;nonlinear control},
  doi={10.1109/TRO.2026.3727138}}