Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery

Abstract

Meta-reinforcement learning (Meta-RL) enables fast adaptation to new testing tasks, yet it remains challenging to learn performant policies across multiple complex and high-dimensional tasks. We propose HiMeta, a novel architecture with three hierarchical levels for 1) learning task representations, 2) discovering task-agnostic macro-actions in an automated manner, and 3) learning primitive actions. The macro-action guides low-level primitive policy learning to more efficiently transition to goal states, which mitigates the issue that a policy may forget previously learned behavior while learning new, conflicting tasks. The task-agnostic nature of the macro-actions is enabled by removing task-specific components from the state space, making them amenable to re-composition across different tasks and leading to fast adaptation to unseen tasks.

Key Contributions

Automated macro-action discovery: We discover reusable macro-actions without hand-designed subgoals or task-specific supervision, by learning them on a state space stripped of task-specific components.
Tri-level hierarchy: We formulate meta-RL as three coupled levels—task representation, macro-action, and primitive action—so that adaptation happens at the level where it is cheapest.
Stable training scheme: Deeper hierarchies are notoriously unstable; we mitigate this with independently tailored training schemes for each level.
Improved performance: Experiments on the MetaWorld benchmark show improved sample efficiency and success rate over prior state-of-the-art meta-RL methods.

Adaptation to Unseen MetaWorld Tasks

Figure 1: HiMeta adapting to manipulation tasks in the MetaWorld benchmark.

Method Overview — A Tri-Level Hierarchy

1. Task Representation Level

Given trajectories collected from a task \(\mathcal{T}_i\), we infer a latent task embedding \(z_i\) that summarizes what distinguishes this task from the others. Adaptation to a new task then reduces to inferring \(z\) from a handful of transitions, rather than re-learning control from scratch.

2. Macro-Action Level

The middle level operates on a task-agnostic state space \(\bar{s}\), obtained by removing task-specific components (e.g., goal coordinates) from the full state \(s\). On this reduced space we learn macro-actions

\[\mu: \bar{s} \mapsto \bar{s}^{g},\]

each of which proposes a goal state for the low-level policy to reach. Because \(\bar{s}\) carries no task identity, the resulting macro-actions describe generic ways of moving through the environment and can be re-composed across tasks.

3. Primitive Action Level

The low-level policy \(\pi(a \mid s, \bar{s}^{g})\) is rewarded for transitioning toward the macro-action’s proposed goal state. This dense, macro-action-induced signal replaces the sparse task reward that ordinarily drives interference: because the low-level objective is expressed in task-agnostic terms, learning a new task reshapes the high-level composition rather than overwriting the primitive skills.

Stabilizing the Hierarchy

Training three levels jointly is unstable, as each level’s objective is non-stationary from the perspective of the others. We therefore train each level with its own tailored scheme and update cadence, so that a level only adapts once the level below it is sufficiently reliable.

Cite this work

Cho, M., & Sun, C. (2026). "Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery." Scientific Reports. https://doi.org/10.1038/s41598-026-57247-1

@article{cho2026hierarchical,
  title   = {Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery},
  author  = {Cho, Minjae and Sun, Chuangchuang},
  journal = {Scientific Reports},
  year    = {2026},
  doi     = {10.1038/s41598-026-57247-1}
}