Structured Logging for Training Jobs

Training jobs produce a lot of output: losses, learning rates, evaluation scores, warnings, timings. Most of it ends up as free-form text in a log file that is easy to read once and hard to use again. Writing one JSON object per line instead costs almost nothing and makes the logs something we can query.

Why plain text falls short

A line such as epoch 3 step 1200 loss 0.412 is clear to a person. To a program it is a string that needs a regular expression, and the expression breaks the day someone adds a field or changes the order. Comparing two runs means writing a parser first.

Structured logs keep the same information in a form that every tool already understands. The cost is a slightly less pretty terminal.

One event per line

The format is called JSON Lines: each line is a complete JSON object. A small helper is all we need.

import json
import sys
import time

def log(event, **fields):
    record = {"ts": time.time(), "event": event, **fields}
    sys.stdout.write(json.dumps(record) + "\n")
    sys.stdout.flush()

log("train_step", run="baseline", epoch=3, step=1200, loss=0.412, lr=3e-4)

{"ts": 1767225600.0, "event": "train_step", "run": "baseline", "epoch": 3, "step": 1200, "loss": 0.412, "lr": 0.0003}

Where the lines go

Writing to standard output keeps the helper simple and lets the job scheduler decide where logs end up. When a job runs on a machine we control, it is still worth writing a copy to a file named after the run, next to the checkpoints. Then the logs travel with the model, and a checkpoint found months later can be matched to the exact run that produced it.

Flushing after every line matters more than it looks. A job that crashes or is killed loses whatever is sitting in the buffer, and the last few lines before a crash are usually the ones we need most.

Choosing fields

Every record should carry enough context to stand alone: a timestamp, an event name, and a run identifier. After that, use the same field names everywhere. If evaluation records call it step, training records should not call it iteration. A short list of agreed names, written down next to the helper, prevents most drift.

Reading the logs back

Because each line is a JSON object, loading a log into a data frame is one call. From there, filtering by event and plotting a metric takes a few more.

import pandas as pd

df = pd.read_json("baseline.jsonl", lines=True)
steps = df[df.event == "train_step"]
steps.groupby("epoch").loss.mean().round(3)

epoch
1    0.871
2    0.553
3    0.418
Name: loss, dtype: float64

Comparing runs

When every run writes the same fields, comparing runs is a concatenation and a group-by on the run column. Command-line tools such as jq work just as well for a quick look without starting Python.

What to leave out

Not everything belongs in the event stream. Large arrays such as gradients or full predictions are better saved as separate files, with the log recording only their path. Secrets and personal data should never be logged at all. And very frequent events, such as every single mini-batch on a fast model, can be sampled or aggregated before writing so the log stays small enough to open.

The habit is simple: log events, not sentences. Future comparisons, dashboards and debugging sessions all start from the same clean record.