Stratifying Reinforcement Learning with Signal Temporal Logic: Connecting Deep RL Geometry to Decision Spaces

New research establishes a correspondence between stratification theory from mathematics and Signal Temporal Logic (STL), providing a fresh framework for analyzing the geometry of spaces generated by deep reinforcement learning agents.

The Core Idea

The paper develops stratification-based semantics for Signal Temporal Logic, where each atomic predicate is interpreted as a membership test in a stratified space. This reveals that most STL formulas induce a stratification of space-time.

Why It Matters

1. Analyzing DRL embeddings — The framework provides tools to analyze the structure of embedding spaces generated by deep RL agents and relate them to the geometry of the decision space

1. Tool reuse — Enables reuse of existing high-dimensional analysis tools from mathematics and motivates new computational techniques

1. Practical grounding — Demonstrated on Minigrid games, showing how STL robustness can serve as reward signals

Key Innovations

• Computational signatures — Proposed efficient signatures based on persistent homology for stratified spaces
• STL as reward — Using the robustness of STL formulas directly as reward functions for RL training
• Geometric interpretation — Connects the abstract logic of temporal specifications to concrete geometric structures in neural network embeddings

Applications

• Robotics — Specifying safety constraints as temporal logic properties
• Autonomous systems — Verifying that agent behavior satisfies temporal specifications
• Game AI — Using temporal logic to shape reward landscapes