Observability¶

RLM Code ships with a pluggable, multi-sink observability architecture that captures every run start, step event, and run completion, then fans them out to one or more telemetry backends in real time. Whether you are debugging locally with JSONL files or shipping distributed traces to a production Jaeger cluster, the system adapts without code changes -- you only toggle environment variables.

Architecture at a Glance¶

flowchart LR
    Runner["RLMRunner"] -->|events| Obs["RLMObservability"]
    Obs --> S1["LocalJSONLSink"]
    Obs --> S2["MLflowSink"]
    Obs --> S3["OpenTelemetrySink"]
    Obs --> S4["LangSmithSink"]
    Obs --> S5["LangFuseSink"]
    Obs --> S6["LogfireSink"]
    Obs --> S7["CompositeSink"]
    S7 --> S7a["Custom Sink A"]
    S7 --> S7b["Custom Sink B"]

The central coordinator, RLMObservability, iterates over its list of sinks and calls each one inside a try/except guard. A failing sink never crashes the run.

The RLMObservabilitySink Protocol¶

Every sink -- built-in or custom -- must satisfy the RLMObservabilitySink structural protocol defined in rlm_code.rlm.observability:

class RLMObservabilitySink(Protocol):
    """Sink contract for RLM observability events."""

    name: str

    def status(self) -> dict[str, Any]:
        """Return sink status for CLI visibility."""
        ...

    def on_run_start(
        self,
        run_id: str,
        *,
        task: str,
        environment: str,
        params: dict[str, Any],
    ) -> None:
        """Hook called at run start."""
        ...

    def on_step(
        self,
        run_id: str,
        *,
        event: dict[str, Any],
        cumulative_reward: float,
    ) -> None:
        """Hook called after each step event."""
        ...

    def on_run_end(
        self,
        run_id: str,
        *,
        result: Any,
        run_path: Path,
    ) -> None:
        """Hook called once at run completion."""
        ...

Available Sinks¶

RLM Code provides 7 sinks out of the box:

Automatic Activation from Environment Variables¶

When RLMObservability.default() is called (which happens automatically at the start of every RLM run), it reads the following environment variables and instantiates sinks accordingly:

Runtime Sink Management¶

The RLMObservability coordinator exposes three methods for runtime sink management:

Adding a Sink¶

from rlm_code.rlm.observability import RLMObservability

obs = RLMObservability.default(workdir=workdir, run_dir=run_dir)

# Add a custom sink at runtime
obs.add_sink(my_custom_sink)

Removing a Sink¶

removed = obs.remove_sink("mlflow")  # Returns True if found and removed

Retrieving a Sink¶

otel_sink = obs.get_sink("opentelemetry")
if otel_sink:
    trace_id = otel_sink.get_trace_id(run_id)

Querying Sink Status¶

for sink_status in obs.status():
    print(f"{sink_status['name']}: enabled={sink_status['enabled']}, "
          f"available={sink_status['available']}, detail={sink_status['detail']}")

Example output:

local-jsonl: enabled=True, available=True, detail=/home/user/.rlm_code/rlm/observability
mlflow: enabled=True, available=True, detail=http://localhost:5000
opentelemetry: enabled=False, available=False, detail=disabled
langsmith: enabled=False, available=False, detail=disabled
langfuse: enabled=False, available=False, detail=disabled
logfire: enabled=False, available=False, detail=disabled

Event Flow¶

Every RLM run follows this lifecycle through the observability system:

sequenceDiagram
    participant R as RLMRunner
    participant O as RLMObservability
    participant S as Sink (any)

    R->>O: on_run_start(run_id, task, env, params)
    O->>S: on_run_start(...)

    loop Each Iteration
        R->>O: on_step(run_id, event, cumulative_reward)
        O->>S: on_step(...)
    end

    R->>O: on_run_end(run_id, result, run_path)
    O->>S: on_run_end(...)

Quick Start¶

Enable MLflow and OpenTelemetry alongside the default local sink:

export DSPY_RLM_MLFLOW_ENABLED=true
export MLFLOW_TRACKING_URI=http://localhost:5000
export DSPY_RLM_OTEL_ENABLED=true
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

rlm-code run --task "Build a DSPy signature" --environment dspy

All three sinks will receive the same events in parallel.

What's Next¶