Media and publishing/Personalization tests

Test a personalization rule without a release.

Signals tells your experimentation tool what each reader is doing now and has done this month, so a new rule for what they see is a variant it assigns today and measures on the same behavior.

Start free Talk to us

More personalization tests shipped

Ship more personalization tests by running two rules against each other today.

A personalization rule is a condition over what a reader is doing plus the action your site takes. Signals computes the condition's inputs from live reader behavior and serves them to your experimentation tool, so a new rule is a setting it can assign to half of your readers today. The same definitions run over past visits in your warehouse, so the result is measured on the attribute the rule used.

Test when the newsletter prompt appears

Someone on the team wants the newsletter prompt to appear after the second story a reader opens in one section. The current rule waits for the third. Settling it by debate means one rule ships, the other goes unmeasured, and the next change waits for the next release. Both rules read the same count of stories, so the question is a test rather than a release.

What Signals knows

section_counts_session: How many stories they opened in each section this visit
current_section: The section of the story they are reading now

What your experimentation tool does

Assigns each reader to one of the two rules, reads the same story count for both, shows the prompt at the second story for one group and the third for the other, and compares newsletter sign-ups between them.

article.tsx · 15 lines

from snowplow_signals import Attribute, EntityProperty, Event, StreamAttributeGroup, domain_sessionid

page_view = Event(name="page_view", vendor="com.snowplowanalytics.snowplow", version="1-0-0")
section = EntityProperty(vendor="com.example", name="article", major_version=1, path="section")

section_counts_session = Attribute(
    name="section_counts_session",
    type="dict",
    events=[page_view],
    aggregation="category_count",
    property=section,
)

current_section = Attribute(
    name="current_section",
    type="string",
    events=[page_view],
    aggregation="last",
    property=section,
)

# Keyed on the visit, so the counts start from zero each time a reader arrives.
tests_newsletter_prompt = StreamAttributeGroup(
    name="tests_newsletter_prompt",
    version=1,
    attribute_key=domain_sessionid,
    owner="you@example.com",
    attributes=[section_counts_session, current_section],
)

// In the code that renders the article page
const reader = await signals.getServiceAttributes({
  name: "tests_newsletter_prompt",
  attribute_key: "domain_sessionid",
  identifier: sessionId,
});
// { section_counts_session: { politics: 2, sport: 1 }, current_section: "politics" }

const storiesInSection = reader.section_counts_session[reader.current_section] ?? 0;
const variant = experiments.variant("newsletter_prompt_timing", sessionId);
const threshold = variant === "after_second" ? 2 : 3;

if (storiesInSection >= threshold) {
  newsletter.prompt({ section: reader.current_section });
}

Test a homepage that leads with a reader's habit

A reader has visited six times this month, and most of what they open is business coverage. The homepage they land on is the one the desk built for everyone, so the stories they come back for sit below the fold until they scroll. Leading with their section is the obvious rule to try, and the obvious rule still has to show that it keeps readers. The new homepage goes to half of the returning readers, the current one to the rest, and the test says which half comes back more.

What Signals knows

habit_section_30d: The section they read most in the last 30 days
visits_30d: Separate visits in the last 30 days

What your experimentation tool does

Assigns half of the returning readers to the new rule and leads their homepage with the section they keep reading, shows the rest the current homepage, and compares how many readers from each group come back.

homepage.tsx · 14 lines

from datetime import timedelta
from snowplow_signals import Attribute, AtomicProperty, EntityProperty, Event, StreamAttributeGroup, domain_userid

page_view = Event(name="page_view", vendor="com.snowplowanalytics.snowplow", version="1-0-0")

habit_section_30d = Attribute(
    name="habit_section_30d",
    type="string",
    events=[page_view],
    aggregation="most_frequent",
    property=EntityProperty(vendor="com.example", name="article", major_version=1, path="section"),
    period=timedelta(days=30),
)

visits_30d = Attribute(
    name="visits_30d",
    type="int32",
    events=[page_view],
    aggregation="approx_count_distinct",
    property=AtomicProperty(name="domain_sessionid"),
    period=timedelta(days=30),
)

tests_habit_homepage = StreamAttributeGroup(
    name="tests_habit_homepage",
    version=1,
    attribute_key=domain_userid,
    owner="you@example.com",
    attributes=[habit_section_30d, visits_30d],
)

// In the code that renders the homepage
const reader = await signals.getServiceAttributes({
  name: "tests_habit_homepage",
  attribute_key: "domain_userid",
  identifier: domainUserId,
});
// { habit_section_30d: "business", visits_30d: 6 }

const returning = reader.visits_30d >= 3;
const variant = returning ? experiments.variant("habit_homepage", domainUserId) : "control";

if (variant === "habit_section") {
  homepage.leadWith({ section: reader.habit_section_30d });
}

Hold out a group so the whole program can be measured

Every rule has been tested against the rule before it, and the team can say each change helped. Harder to say is what personalization as a whole is worth against a site that treats every reader the same. That site stopped existing when the first rule shipped. A holdout brings it back for a small share of readers. Their attributes are computed like everyone else's and then ignored, so the only difference between the groups is whether the rules ran.

What Signals knows

habit_section_30d: The section they read most in the last 30 days
visits_30d: Separate visits in the last 30 days

What your experimentation tool does

Assigns a small share of readers to the holdout and shows them the site with every rule switched off, while the same attributes are computed for them as for everyone else, so the gap between the holdout and the rest is what the program is worth.

homepage.tsx · 14 lines

from datetime import timedelta
from snowplow_signals import Attribute, AtomicProperty, EntityProperty, Event, StreamAttributeGroup, domain_userid

page_view = Event(name="page_view", vendor="com.snowplowanalytics.snowplow", version="1-0-0")

habit_section_30d = Attribute(
    name="habit_section_30d",
    type="string",
    events=[page_view],
    aggregation="most_frequent",
    property=EntityProperty(vendor="com.example", name="article", major_version=1, path="section"),
    period=timedelta(days=30),
)

visits_30d = Attribute(
    name="visits_30d",
    type="int32",
    events=[page_view],
    aggregation="approx_count_distinct",
    property=AtomicProperty(name="domain_sessionid"),
    period=timedelta(days=30),
)

# Computed for every reader, holdout included. The holdout is decided after the read, not before it.
tests_holdout = StreamAttributeGroup(
    name="tests_holdout",
    version=1,
    attribute_key=domain_userid,
    owner="you@example.com",
    attributes=[habit_section_30d, visits_30d],
)

// In the code that renders the homepage
const reader = await signals.getServiceAttributes({
  name: "tests_holdout",
  attribute_key: "domain_userid",
  identifier: domainUserId,
});
// { habit_section_30d: "business", visits_30d: 6 }

const variant = experiments.variant("personalization_program", domainUserId);
if (variant === "holdout") return;

if (reader.visits_30d >= 3) {
  homepage.leadWith({ section: reader.habit_section_30d });
}

Measure the test with the definition that served it

The rule under test fires for engaged readers, and the analysis has to count engaged readers in each group. Written twice, once for the page and once for the analysis, the two definitions drift. A lift that showed on the page is then argued over in the review, because nobody is sure the two counts mean the same thing. The definition should be written once and used in both places. Then the attribute the rule read on the page is the attribute the analysis counts in the warehouse.

What Signals knows

long_reads_30d: Articles read to the end in the last 30 days
last_read_days: Days since they last read an article to the end

What your experimentation tool does

Reads each group's result in the warehouse from the same definition the rule read on the page. Your warehouse runs that definition over past visits, so engaged reader means one thing in the test and in the analysis.

article.tsx · 16 lines

from datetime import timedelta
from snowplow_signals import (
    Attribute, Criteria, Criterion, Event, EventProperty, StreamAttributeGroup, domain_userid,
)

article_read = Event(name="article_read", vendor="com.example", version="1-0-0")
finished = Criteria(all=[
    Criterion.gte(EventProperty(vendor="com.example", name="article_read", major_version=1, path="percent_read"), 80),
])

long_reads_30d = Attribute(
    name="long_reads_30d",
    type="int32",
    events=[article_read],
    aggregation="counter",
    criteria=finished,
    period=timedelta(days=30),
    default_value=0,
)

last_read_days = Attribute(
    name="last_read_days",
    type="double",
    events=[article_read],
    aggregation="time_since_last",
    criteria=finished,
    time_unit="d",
)

# One definition. It serves the page live and runs over past visits in your
# warehouse, so the analysis counts the same engaged readers the rule saw.
tests_engaged = StreamAttributeGroup(
    name="tests_engaged",
    version=1,
    attribute_key=domain_userid,
    owner="you@example.com",
    attributes=[long_reads_30d, last_read_days],
)

// In the code that renders the article page
const reader = await signals.getServiceAttributes({
  name: "tests_engaged",
  attribute_key: "domain_userid",
  identifier: domainUserId,
});
// { long_reads_30d: 5, last_read_days: 1.4 }

// The analysis counts engaged readers in the warehouse with the same
// long_reads_30d definition, so it is the same test the page ran.
const engaged = reader.long_reads_30d >= 3;
const variant = experiments.variant("engaged_reader_rail", domainUserId);

if (engaged && variant === "related_rail") {
  rail.show({ mode: "related" });
}

Try it on your own site.

Add the SDK, define one thing you want to know about a visitor, and watch it update while you click around. The free tier runs on the real engine: no card, no sales call.

Start free Talk to an engineer