Market-basket analysis without confusing popularity with affinity

A product can appear in many baskets simply because it is popular. Market-basket analysis asks whether two products appear together more often than their individual popularity would suggest. Support, confidence and lift answer different parts of that question.

A rule with high confidence is not automatically a strong association, and an association does not establish that recommending one item will cause an additional purchase.

Start with baskets rather than item quantities

The synthetic basket fixture contains six baskets:

Treat each product as present or absent within a basket. Buying three cartons of milk should not create three independent basket observations. The lab's primary key is basket plus product, which prevents duplicated presence rows.

Bread appears in four baskets, milk in four, and both together in three. For the rule bread → milk:

• Support is 3/6 = 50%: the share of all baskets containing both.
• Confidence is 3/4 = 75%: the share of bread baskets also containing milk.
• Lift is (3/4)/(4/6) = 1.125: confidence relative to milk's overall prevalence.

The 75% confidence sounds substantial, but milk already appears in two-thirds of all baskets. Lift reveals the more modest relative association in this sample. IBM's association-rule lift documentation explains this baseline comparison.

Reproduce both a positive and a negative association

from collections import defaultdict
from math import isclose
from build_and_verify import database

db = database()
baskets = defaultdict(set)
for basket, product in db.execute('SELECT basket_id,product FROM baskets'):
    baskets[basket].add(product)
db.close()

def rule(antecedent, consequent):
    n = len(baskets)
    left = sum(antecedent in items for items in baskets.values())
    right = sum(consequent in items for items in baskets.values())
    joint = sum({antecedent, consequent} <= items for items in baskets.values())
    if not n or not left or not right:
        return None
    return {'baskets': n, 'joint': joint, 'support': joint/n,
            'confidence': joint/left, 'lift': (joint/left)/(right/n)}

bread_milk = rule('bread', 'milk')
milk_eggs = rule('milk', 'eggs')
assert bread_milk['joint'] == 3
assert bread_milk['support'] == .5 and bread_milk['confidence'] == .75
assert isclose(bread_milk['lift'], 1.125)
assert milk_eggs['confidence'] == .25
assert isclose(milk_eggs['lift'], .75)
assert rule('absent_product', 'milk') is None
print(bread_milk, milk_eggs)

Milk → eggs has confidence 25%, below eggs' overall presence of one-third. Its lift is 0.75. A low absolute confidence threshold could retain this rule despite the negative association relative to that baseline.

The function returns no result when a required denominator is zero. An undefined rule should not be presented as a measured lift of zero.

Preserve the population that produced the rule

This code derives the basket population from product rows, which is appropriate for these six completed, nonempty baskets. If a business includes empty carts, abandoned baskets or orders containing only filtered-out products, retain a separate basket header table. Otherwise product filtering silently changes the denominator.

Specify whether returns, canceled orders, bundles and promotions belong in the analysis. A forced bundle can create co-occurrence through merchandising rules rather than a customer's independent preference. Availability also matters: an out-of-stock item cannot co-occur even if customers wanted it.

Evaluate a recommendation before rollout

Six baskets demonstrate arithmetic, not reliable commercial evidence. At larger scale, searching thousands of pairs can produce apparently interesting rules by chance. Require adequate counts, examine stability in a later period and avoid selecting solely by the largest lift among tiny groups.

To assess a recommendation, measure incremental contribution and user experience in a suitable experiment. Displaying a recommendation could replace an item the customer already intended to buy, increase returns or add friction. Association strength alone cannot answer those questions.

Exercise: add many milk-only baskets. Recalculate confidence and lift for bread → milk. Explain why confidence can stay fixed while lift falls as the consequent becomes more common.

NeuraPath's Data Analytics with Generative AI course connects descriptive analysis with testable decisions. A useful basket project makes the baseline, sample size and proposed intervention explicit.

Continue learning

This article is part of the Customer and product analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in RFM analysis: validate segments against business actions.
• Continue with Analyze repeat purchases when customer identity changes.