Build a lightweight experiment tracking convention

Experiment tracking begins with a complete record, not a particular platform. Each run should answer which code, data, parameters and environment produced which artifact and metric, and what decision followed.

Eight required fields

The local deployment lab creates a JSON-compatible record with run ID, UTC creation time, code revision, data hash, parameters, metrics, artifact hash and decision.

from deployment_cases import experiment_case

result = experiment_case()
assert result["complete"] is True
assert result["record"]["metrics"]["held_out_test_auc"] > 0
print(result["required_fields"])

The recorded held-out AUC is executed on the original synthetic fixture and is approximately 0.785. It is a test description, not a validation metric for choosing another candidate. The record’s decision is recorded_for_review.

Separate run from release

A run may fail, be exploratory or remain unselected. Preserve it. A release points to a reviewed run plus schema, environment, approval and operating policy. Do not overwrite a run when parameters change; create a new ID.

Record metric name, population, split and threshold, not only a number. Attach curves and slice reports as artifacts. Log random seeds and resource settings. Keep secrets and private rows outside public metadata.

Start simple and automate checks

A versioned JSON file and structured artifact folder can support a small project. Add a database or tracking server when concurrency, search and access control require it. Whatever the tool, validate required fields and hash artifacts in CI.

Tracking does not prevent test tuning by itself. Enforce data roles in the evaluation process and mark exploratory metrics clearly.

The Data Science course uses lightweight lineage to support comparison, handover and rollback.

Exercise

Create records for three candidates, including one failed run. Write a script that rejects missing data hash, metric population or artifact hash and generates a comparison table without selecting on test.

Continue learning

This article is part of the Model deployment and MLOps sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Version a model and its training data together.
• Continue with Compare offline metrics with online decision outcomes.

Reference: MLflow tracking concepts.