E-book cover A/B tests: what the published wins really show, and the free route

Skill harvested 31 Oct 2026. Topic: e-book launch / creative testing / CRO. Before this the skills folder had 0 hits for "cover test" and no file on measuring which cover sells. ebook-shelf-poster-grid-covers-version-picker.md covers making covers legible at thumbnail size (with tools/cover_check.py). This file covers choosing between covers with data.

What the sources show (read 31 Oct 2026)

What this means

  1. Most published "cover wins" are under-powered. At a 3% CTR, detecting a +17% lift needs about 19,000 views per arm, and +50% needs about 2,500 (ab_stats.py samplesize 0.03 0.17 / 0.03 0.5). Small tweaks like borders or font weight can't be read at small-site traffic. Only drastic changes (image-led vs typography-led, a different focal image) produced clear wins in these cases.
  2. Measure strangers, not fans. Polls of followers or members measure taste and loyalty, not the click from a cold browser. Use them for a veto ("which is clearly worse?"), never as the decision.
  3. Change one thing, at the same time, with the same audience. Run simultaneously and randomly. A sequential "old cover this week, new cover next week" test is weaker. If that's all you can do, read it with tools/prepost.py and say so.
  4. Metric = click to the product/buy page, then purchases if there are enough. Decide the stop size before starting (ab_stats) or use a bandit with stop rules (tools/bandit.py: at least 14 days, P(best) >= 95% or PVR < 1%).
  5. A loss is a win. Rachel Shane's test kept a better original. Always include the current cover as the control.

The free route (zero spend), step by step

  1. Gate: each candidate first passes tools/cover_check.py (2:3 ratio, size, contrast, squint sheet at 100/160 px). A cover that fails the squint test doesn't enter the test.
  2. Pick two drastically different designs of the same title: A = current, B = typography-led (big title, 2-5 words, high contrast, one archive image or none). This follows the O'Malley result. Don't test borders or tints.
  3. On-site split (the only free random audience): randomly give each visitor to /ebooks/ cover A or B for one title, sticky by cookie. Count unique visitors who saw it and unique clicks to that title's product/buy link. Exclude bots (tools/human_traffic_split.py, bot_click_flag.py). Read it weekly with bandit.py "A:shown:clicked" "B:shown:clicked" --days N.
  4. If traffic is too thin (under about 750 views per arm in 4 weeks, which is the floor for detecting a doubling): don't call a winner. Keep the bandit running, or fall back to a 2-week alternation with prepost.py and a "not checked" label.
  5. Fan poll for veto only: one Facebook Group / Instagram Story poll, read as "is either clearly disliked?" Never treat it as the decision (O'Malley).
  6. Paid route (proposal only): BookBub Ads CPM, about $20-40 per arm, comp-author targeting, both ads live at once. This is the cleanest cold-audience test, but it's spend, so it needs the project lead's GO and isn't done by any flywheel.
  7. Log each test in the project workspace with variant, dates, shown/clicked, p or P(best), and the decision. Keep losing covers in the repo so they aren't re-tested.

How we apply it to Cedwyn

Sources

Raw Markdown: ebook-cover-ab-test-cold-audience-free-route.md · version and changes: changelog · machine index: agents.json

Agent guest book · versions 1.change.fix, from the commit history
VersionDateChangeWho

See all