Build a point-in-time correct feature join

A standard join can attach the latest feature ever recorded, including values created after the prediction. A point-in-time join selects the latest snapshot for the same entity whose feature timestamp is at or before that row’s prediction timestamp.

Three auditable joins

The local lab joins three prediction rows to account spend snapshots.

from data_engineering_cases import point_in_time_case

result = point_in_time_case()
assert result["joined_rows"] == result["labels"] == 3
assert result["future_feature_violations"] == 0
for row in result["rows"]:
    print(row)

Account A-1 at 15 January receives the 10 January value 18. At 22 January it receives the 20 January value 25. Account A-2 at 20 January receives its 5 January value 7. No feature timestamp is in the future.

Define availability, not only event time

A snapshot timestamp may describe the source event while the value became available later. For strict historical reproduction, join on availability time or store both. Corrections and backfills should not rewrite what a model could have known without versioned as-of access.

Sort within entity, enforce uniqueness of snapshot keys and define exact-match behavior. Test missing histories and simultaneous updates. Store the selected feature timestamp with the training row.

Use one definition for offline and online

The online feature path should implement the same window and cutoff. Compare golden entities between offline and serving. Leakage tests should assert every source timestamp and availability timestamp respects prediction time.

The Data Science course treats point-in-time joins as a hard gate for credible model evaluation.

Exercise

Add a feature event that occurred before prediction but was ingested after it. Compare event-time and availability-time joins and document which one production could reproduce.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Handle late-arriving events in a training dataset.
• Continue with Data lineage: trace a model feature back to its source.

Reference: pandas merge_asof API.