Sparsity-based Safety Conservatism for Constrained Offline Reinforcement Learning

Abstract

Offline RL is a compelling alternative to online RL when additional experiments are impractical and abundant datasets are available. However, distributional shift — the disparity between the data distribution and the learned policy — poses a risk in offline RL, potentially leading to significant safety breaches. Much prior work addresses extrapolation error (acting outside the data support), while far fewer works offer effective solutions for interpolation error (estimation error within nominally supported but poorly covered regions). Existing remedies typically maximize a potential cost by perturbing the dataset, which introduces a bi-level optimization structure that can be unstable or intractable in high-dimensional tasks. This motivates us to pinpoint areas where hazards may be more prevalent than initially estimated, based on the sparsity of the available data. We present conservative safety metrics grounded in data sparsity that are highly generalizable to any constrained offline RL method and efficacious compared to bi-level cost-upper-bound maximization.

Key Contributions

Sparsity as a safety signal: We show that how densely a region of the dataset is covered is itself a usable proxy for how much the learned cost estimate should be distrusted — no adversarial perturbation of the dataset required.
Single-level, not bi-level: Our conservatism is computed directly from the data geometry, avoiding the instability and computational burden of bi-level cost-maximizing optimization in high-dimensional tasks.
Method-agnostic: The metric is a drop-in addition to existing constrained offline RL algorithms, since it modifies only the cost estimate rather than the underlying optimizer.
Empirical validation: We demonstrate improved constraint satisfaction against bi-level cost-ub-maximization baselines on safety-critical offline benchmarks.

Method Overview — Sparsity-Based Conservatism

Figure 1: Data sparsity as a proxy for the reliability of the learned cost estimate.

1. Measuring Sparsity via Clustering

We partition the offline dataset into \(K\) clusters using K-means, and measure the sparsity of a state-action pair \((s,a)\) by its distance to the assigned cluster centroid relative to the local density of that cluster:

\[\sigma(s,a) \;\propto\; \big\| (s,a) - c_{k(s,a)} \big\|,\]

where \(c_k\) denotes the \(k\)-th centroid. Points that sit far from any well-populated centroid lie in thinly sampled regions, exactly where a learned cost critic has seen the least evidence.

2. Sparsity-Weighted Cost Overestimation

Rather than trusting the learned cost \(Q_C(s,a)\) uniformly, we inflate it in proportion to local sparsity:

\[\widehat{Q}_C(s,a) := Q_C(s,a) + \beta \, \sigma(s,a),\]

so that the policy treats poorly covered regions as more hazardous than the raw estimate suggests. The coefficient \(\beta\) trades off conservatism against performance.

3. Constrained Policy Optimization

The inflated cost is then used in the standard constrained offline objective,

\[\max_{\pi} \; \mathbb{E}\big[Q_R(s,a)\big] \quad \text{s.t.} \quad \mathbb{E}\big[\widehat{Q}_C(s,a)\big] \leq h,\]

which keeps the policy away from regions where an apparent safety margin is really just a lack of data. Because the modification lives entirely inside the cost estimate, any constrained offline RL algorithm can adopt it unchanged.

Cite this work

Cho, Minjae, and Chuangchuang Sun. "Sparsity-based Safety Conservatism for Constrained Offline Reinforcement Learning." arXiv preprint arXiv:2407.13006 (2024).

@article{cho2024sparsity,
  title   = {Sparsity-based Safety Conservatism for Constrained Offline Reinforcement Learning},
  author  = {Cho, Minjae and Sun, Chuangchuang},
  journal = {arXiv preprint arXiv:2407.13006},
  year    = {2024}
}