Campaign Response Heterogeneity

Pooled comparison and fixed-effects estimates are different quantities

Pooled comparison (any campaign type active vs none, no controls, household-weeks):

The two tables answer different questions: the first pools every campaign type, the second compares one type against the household's own other weeks, so the second table is not an adjusted version of the first.

Household and week fixed effects (all nine type × tier indicators estimated together in one regression, each read against the same household's own other weeks, with the other campaign types' activity controlled):

Not shown: TypeA × high-spending (β −0.16, raw p 0.8337) and all three mid-spending types (raw p 0.2333 / 0.7591 / 0.7798), none significant. Campaign reach was 94.1% in the high tier against 25.2% in the low tier.

Does campaign response differ by a household's baseline spending tier?

Concretely: when a campaign is running, does a household's weekly spend move, and does that movement depend on whether the household was a light, medium or heavy spender before any campaign began?

Circularity in tier definition circular reasoning

If tiers are defined using data from the campaign period, the analysis contains this chain: campaign runs → spending rises → household classified as high-spending → "high-spending households spent more during campaigns."

That conclusion would follow whether or not campaigns do anything. Tiers were therefore fixed before the first campaign ran and never recomputed.

Source

Dunnhumby "The Complete Journey": household transactions plus campaign delivery records. Reshaped into a balanced household × week panel covering weeks 33–101, the period from the first campaign launch onward. Week 102 is left out; the data end on day 711, so that week holds only 6 days of transactions.

The dataset was used for educational portfolio purposes. This page covers a campaign × tier analysis that grew out of a team project in which hypotheses were divided among team members.

Baseline tiers: fixed on weeks 17–32, before any campaign

Three tiers instead of five. I also split households into five spending quintiles and counted households per campaign type. Two Type C cells fell below the 30-household threshold (16 in the lowest quintile, 25 in the second), so the fixed-effects models were estimated on three tiers only. No quintile model was run. Even with three tiers, Type C × low-spending has 28 households, slightly below the threshold.

142 households (5.7%) had no transactions at all in the tier window and were assigned $0, placing them in the low tier. This creates a possible floor effect, since a household with no recorded activity may behave differently from an active low spender.

① Baseline tiers fixed before any campaign ran

Weeks 17–32 only, with the first campaign starting in week 33, so there is no overlap. This implements the rule from section 02 that tiers must not be defined from campaign-period data.

② Household fixed effects

Each household is compared against itself in campaign weeks versus quiet weeks. This removes permanent differences between households in average spending ("this household simply spends more"), which is one reason the pooled comparison cannot be read as an effect.

③ Week fixed effects

Christmas, Thanksgiving and other calendar effects hit every household at once. Week fixed effects absorb them so seasonality is not mistaken for campaign response.

④ Household-clustered standard errors

A 69-week repeated-observation panel has serial correlation within households; ordinary standard errors ignore it and understate p-values.

F-statistic: unclustered 11.544 (p < 0.0001) · clustered 2.313 (p = 0.0135)

Effect of clustering. The F-statistic is lower with clustering than without it, so unclustered standard errors would overstate significance.

Read at face value, this would suggest heterogeneous campaign effects, with light spenders gaining several times more than heavy ones. Two features of the data prevent that reading.

The comparison is pooled across households. Even inside one tier, the households receiving campaigns may simply be different households from those that did not, so selection into receiving campaigns remains part of the comparison.

The active-week sample is small in the low tier. Only 2,924 of the low tier's 57,546 household-weeks (5.1%) were campaign-active, against 25,448 of 57,408 (44.3%) in the high tier. Household-level reach (25.2% vs 94.1%) points the same way.

Relative size = coefficient ÷ the high tier's mean weekly spend in weeks 17–32 ($72.75).

The two comparisons estimate different quantities. The pooled comparison uses any campaign type; the fixed-effects model compares a household's weeks with a given campaign type active against its own other weeks; all nine type × tier indicators are estimated together in one regression, so the other types' activity is controlled. Under fixed effects the high tier shows negative TypeB and TypeC estimates, and none of the three low-tier estimates is distinguishable from zero (smallest raw p = 0.163, TypeC × low, 28 households). Showing what fixed effects change for one group would need a model with a single any-campaign indicator per tier, which was not estimated.

A within-model R² of 0.0004 is low: campaigns explain little of the weekly spending variance within households, although with 172,500 observations a small signal can still be detected. The F-test for poolability (38.138, p < 0.0001) supports accounting for fixed effects.

TypeB misses the BH threshold by a small margin. It is treated as an exploratory signal rather than reported as a finding, because it does not pass the correction.

These counts are nested, not additive. 295 is every high-tier TypeC recipient; 131 of them received TypeC in more than one separate spell (a spell is a run of consecutive active weeks), and the other 164 in exactly one. The split is applied to week-level observations of the same households, so adding the two would double-count.

Both splits stay significant on raw (unadjusted) p-values; these two tests are not part of the nine-test BH set. The negative association therefore appears in both splits and does not rest on repeat-exposure households alone. Whether the larger repeat coefficient reflects reinforced discount-waiting or repeated targeting of already-declining households cannot be separated here.

Result after multiple-comparison correction

Of nine campaign-type × spending-tier combinations, one shows a statistically significant association after multiple-comparison correction.

These coefficients are measured against the low tier, and week fixed effects absorb the change common to everyone (later period = weeks 67–101 versus the earlier weeks). Low-spending non-recipients rose by 165.5% between weeks 33–42 and 92–101, a different window from the regression's, while high-spending non-recipients changed by −1.0% over those same weeks, so much of the High × later period coefficient can come from low-tier growth rather than a high-tier decline. It is not the same quantity as the Type C estimate, which compares a household's Type C weeks with its own other weeks, so the two numbers are not compared.

The two lead coefficients together differ from zero (joint Wald test p = 0.0003). For 260 of the 295 households, at least one Lead4 week had a Type A or B campaign active; across all 1,162 Lead4 household-weeks it was 82%. So a dip before Type C cannot be separated from anticipation or from the effect of those campaigns. For 9 households the lead windows begin before week 33, outside the panel. Two tests were run, with no multiple-comparison adjustment.

Explanations the data cannot separate

Nothing in the data records how campaign recipients were selected, so the reason a given household received a given campaign at a given time cannot be determined from the data. Explanation 2 therefore stays open.

① A pooled comparison does not measure campaign lift here

On this data a pooled active/quiet comparison shows higher spending in active weeks in every tier, largest for light spenders (+242%, +60%, +23%). The household-level estimates answer a different question (one campaign type × tier against the household's own other weeks, not the pooled any-campaign average) and show no significant gain for any campaign type in the low and mid tiers and a negative Type C association in the high tier. A campaign evaluation built on the pooled comparison could misdirect budget.

② Targeting was strongly skewed toward high spenders, which limits measurement

94.1% reach in the high tier against 25.2% in the low tier means the data say little about whether light spenders respond: 25.2% of low-tier households received any campaign, and only 28 received Type C. Deliberately holding out a random slice of each tier would make the next campaign measurable.

③ TypeC among heavy spenders warrants further investigation

The one surviving signal is negative. If it is a genuine response, Type C may be encouraging customers to wait for discounts. If it is targeting, Type C is being sent to customers whose spending has just dipped (−9.02 in the four weeks before), and the dip is attributed to the campaign. The lead-effect result fits this, but also fits anticipation and Type A/B campaigns. Both readings warrant investigation; neither is established.

Parallel trends are not cleanly satisfied

A pre-campaign test on weeks 17–32, checking whether future recipients and non-recipients were already diverging. It is a regression on 32 aggregate points (average weekly spend per household for each of the two groups in each of 16 weeks), and the coefficient is the future-recipient × week interaction, the difference in weekly slope, in dollars per household per week, per week:

Only the low tier shows no detected difference with a sizeable comparison group (624 households). The high tier, where the Type C result is, has too few comparison households for a non-significant result to be informative.

Programming & data handling

Observational analysis & panel methods

Statistics

Practices