AI/LLM Experiment Community Standard 0.1

A Draft Standard for AI/LLM Testing: Because We Can Do Better

If you've been in SEO for the last couple of years, you've watched the same thing we have: a flood of tests, case studies, and "here's what works for AI" posts, published fast in a space that's moving faster than anyone can keep up with. 

The excitement is warranted. The stakes are real, the questions matter, and we’re right to be experimenting. 

But in the rush, something has gotten lost. Not every test that's been put forward has the rigor required to actually learn what we think we're learning.

This came into focus for us in two ways at once. Gray Dot tried to run what would have hopefully been a simple, low-fi test, and quickly ran into enough walls that it became clear the challenge wasn't just execution, it was infrastructure. 

Around the same time, The SEO Community Slack (specifically member Josh Squires) started having the same conversation out loud: how do we, collectively, do better? How do we build something that actually compounds: where one test leads to the next, where findings can be challenged and replicated, where the community is learning together rather than shouting into the void?

What you're reading is our first attempt to answer that. Anne Berlin, Tory Gray, Micah Fisher-Kirshner, and Shaun Davidson led this effort, with support from Adrienne Kmetz and The SEO Community, drawing on what we collectively know from years of SEO testing and from the newer, messier frontier of AI. 

It's not a finished product; it's a v1.0, a starting point, and we're saying that explicitly. The space will keep moving, the standard will need to keep up, and we're counting on community feedback to make it better.

What follows is the first piece of that: a working list of the questions and variables we believe every test runner should consider before publishing results. 

This document aims to be a meaningful stake in the ground for what "rigorous enough to learn from" looks like in practice. If you need a quick-start guide, instead, find that here. 

Read them, use them, and tell us what we missed. This only gets better if you're in it with us.

AI/LLM Testing Protocol: Questions Every Test Runner Should Answer

This is a living document. These questions represent our current best thinking on what separates a useful, replicable AI/LLM test from an interesting anecdote. Not every question applies to every test, but if you're skipping one, you should be able to say why.

1. Hypothesis: What Are You Actually Testing?

The most common failure mode in testing isn't bad execution; it's a hypothesis that was never precise enough to be falsifiable in the first place. Before anything else, you need to be able to state clearly what you believe, why, and what result would prove you wrong.

• What is your hypothesis, stated as specifically as possible? ("AI bots prefer structured content" is not a hypothesis. "Adding new FAQ schema to product detail pages increases the rate at which GPT-4o cites specific product specs in response to purchase-intent queries" is.)
• Is your hypothesis causal ("X causes Y") or correlational ("X is associated with Y")? Name which one you're making, and ideally why this method was selected. Each requires different methods and warrant different confidence levels in your conclusions.
• What is your null hypothesis? What result would tell you the thing you're testing doesn't work or doesn't matter?
• What decision or action changes based on the result? If the answer is "nothing yet," this may be better framed as exploratory research than a test.
• What prior tests, claims, or assumptions does this build on or challenge? What gap does it fill?

2. Environment: What Exactly Are You Testing It On?

Findings that don't specify their environment can't be replicated, and therefore can't be trusted. "I tested this on my site" is not detailed enough. The goal is that any stranger reading your results could reproduce your conditions.

The model/platform:

• Which model(s) are you testing? Be specific: brand, product name, and version where available (e.g., Claude vs. ChatGPT, and more specifically - GPT-4o, not just "ChatGPT").

• Are you testing via UI or API? As a paying user or on a free subscription tier? These behave differently and are not interchangeable.
• What is the date of testing? Models often update without announcement. A result from February may not replicate in May on what appears to be the "same" model. Annotating model slug can help, alongside the test dates, especially when attempting to account for compute budget fluctuations.
• What bot type are you testing? And, are you being precise about it? Agent bots, live-fetching bots, and training crawlers have fundamentally different capabilities and limitations. Agents can take screenshots to interpret on-page data; most other bots generally can't, which means rendering, JavaScript execution, and lazy loading matter differently depending on what you're testing. 

• If using the API: what temperature and parameter settings did you use?
• Was memory, personalization, or any other context-carrying feature active?

The site/content environment:

• What type of site are you testing on: by industry, size, and brand/entity awareness? Tactics that work for a well-known enterprise brand don't automatically apply to a small business with low entity recognition, and vice versa. Further, big brands with extensive public history are significantly more likely to have existing LLM parametric knowledge (e.g. insights already in the training materials) vs smaller brands, impacting whether or not live fetching is needed to answer a query - at what frequency, and where that insight can come from (3rd party coverage.) This can and will impact LLM answers. 
• What page type(s) are you testing? (Product detail page, product listing page, blog post, homepage, etc.)
• What format type(s) are involved? (HTML, PDF, Markdown, JPEG, WEBP, iFrame, etc.)
• What code/rendering type? (Code: JSON, APIs, PHP, Java. Rendering type: CSR, SSR, lazy loading.) It’s important to understand what (potentially fetchable) content is in the response HTML vs only in rendered output given the rendering limitations of most AI bots. Further - does that content require user interaction to appear (since non-agent bots can’t access this.)
• What platform/CMS? (WordPress, Webflow, Next.js, Nuxt, etc. These affect hosting, performance, and rendering behavior in ways that can influence results.)
• What language and region? LLMs are generally biased toward English and Western cultural contexts; results may not transfer cleanly across languages or markets.
• What query or page type and user intent? (X vs. Y, best, how to. Informational, navigational, comparisons, purchase, retention.)
• Where on the page is the element you're testing located? (Top of page, body, nav, footer. Placement can affect what gets parsed or prioritized.)
• Are you using new and original content, or dummy content for your tests?

• What is the content length relative to known context window limitations? Can the bot plausibly access all relevant content in a single pass, or is truncation likely?
• What is page speed/performance like? Slow load times affect what gets fetched and processed, particularly for compute-constrained crawlers. Further, issues with Content Layout Shift (CLS) will impact whether or not an agentic bot can successfully take actions like clicking a button (e.g. if the button moved after screenshots were taken.)
• Are you testing on mobile, desktop, or both?
• Are there overlays, popups, or privacy/cookie banners that could interfere with content access?

3. Verification: How Will You Verify What You Think Is Happening, Is Actually Happening?

AI testing has a verification problem: a bot might claim to be one thing and behave as another. Outputs look confident even when they're wrong.

• Do you have access to log files to verify whether your test pages were actually viewed by your target AI user agents? 

• How are you verifying that the user-agent you're attributing crawl activity to is actually who it says it is? Bots can and do crawl under spoofed user-agents. Competitors may fake crawling as a known AI bot, and legitimate AI crawlers may use less obvious identifiers, especially when using 3rd-party services for data scrapping and collection.
  What's your method for validating this? (Log file cross-referencing, reverse DNS lookup/IP validation against published ranges, etc.)
• If you're querying an LLM directly: what are the exact prompts you used, verbatim? Paraphrasing your prompts in the write-up isn’t enough. Small phrasing differences can produce meaningfully different outputs, and readers need the actual text to evaluate or replicate your work.
  Bonus: crafting your prompts with future replication in mind makes them stronger.
• Did you test prompt robustness? I.e., did you run variations of your prompt phrasing to check whether your finding holds across reasonable alternatives, or is it potentially an artifact of one specific wording? (Semantic similarity of the topic matters!)
• Who evaluated the outputs, and were they the same person (or, more ideally, an AI classification system with clearly delineated rules) who wrote the prompts? Evaluator bias is a real risk when the person who designed the test also scores it.

4. Metrics: What Are You Measuring, and How?

Naming a metric isn't the same as having a measurement plan. Ensure the scoring isn’t so vague that it can be unconsciously shaped by what your test hopes to find.

• What is your primary metric? The specific, measurable things you will use to conclude success or failure.
• What does success look like, defined before you see results? What does failure look like?
• What is your baseline or control condition? You cannot measure change without a before-state or a comparable unchanged condition. If you changed schema on page A, for example, what happened on page B where you changed nothing?
• How are you capturing outputs? (API logs, CDN logs, scraping, copy-paste, screenshot, third-party tool.) Note that copy-paste and screenshot methods can introduce formatting artifacts that affect evaluation. Automation of this data collection may mean it’s more replicable by others in a scalable way.
• How are you isolating what you are measuring, while ensuring you aren’t measuring a confluence of factors? How do you know that the change you made isn’t endogenous (not independent) of that result?
• If your measurement involves human judgment (e.g., "did the LLM correctly extract the data?"): what is your scoring rubric? Did more than one person score the outputs? If so, did they agree? Inter-rater reliability matters; if two evaluators score the same output differently, your measurement isn't stable.
• What secondary metrics are you tracking for context that won't be used to draw conclusions?
• Which tools are you using and are they the right fit for what you're measuring? (Visibility benchmarks, share of voice, log file analysis, direct LLM querying, honeypot pages… each answers different questions and has different blind spots.)
• Have you defined and questioned your assumptions about what the results will be? Is there something hidden that could have affected the output?

5. Repeatability: How Many Times Did You Test It?

LLM outputs are non-deterministic. This means, in essence, the same prompt can produce different outputs on consecutive runs. One result is an observation. A pattern across many runs is a potential finding.

• How many times did you run each condition?
  • State n= explicitly.
  • Annotate how many times, over what time period.
  • Annotate impact durability, as possible. e.g. effect holds for 5-10 days after the completion of the test, effect holds til model update, effect holds indefinitely.
  • Annotate when the impact occurs, especially to the extent that it depends on a model update to show results.
  • Acknowledge where n is low and what that means for confidence.

• For direct LLM querying: how many times did you run the same prompt? A single output tells you what the model can produce, but it doesn't tell you what it typically produces. E.g. the ability to extract schema for a specific query test doesn’t inherently mean that the model will always pull schema to answer queries in a predictable and consistent way. “Can” doesn’t mean “will,” especially with variable compute constraints.
• Over what time window did you collect data, and what was the observed crawl frequency? (Especially for crawl/log file tests but really - for all tests.) A bot that visits once in 30 days is a different finding from one that visits consistently and predictably over time.
• What evidence tier does this result sit at? Be honest & label yours.
  • anecdote (n=1, one site, one context)
  • observation (repeated, consistent, one environment)
  • tested finding (controlled conditions, meaningful n)
  • replicated finding (independently reproduced by others)

6. Constants: What Are You Holding Constant?

A variable you didn't control for is an alternative explanation for your result. The goal isn't to control for everything (that's often impossible in real-world AI testing) but to name what you couldn't control so readers can weigh the finding appropriately.

• What did you explicitly hold constant between conditions?
• What could you not control for, and how might that affect the result? Specific categories to address:
  • Model non-determinism (same prompt, different output)
  • Unannounced model updates during the test period
  • Platform-side A/B testing or undisclosed interventions by the AI provider
  • Differences in training data coverage across topics or domains
• How long did the test run, and is it possible the model changed during that period?

7. Interpretation: How Should Others Interpret and Use This Result?

Be explicit about what your result does and doesn't tell you. This protects readers from drawing conclusions you didn't earn, and it protects your credibility.

• What can this result reasonably be applied to, and what should it not be applied to? (By model, by industry, by site type, by page type, etc.)
• What does this test not tell you? What questions remain open that this result can't answer?
• What surprised you, including results that contradicted your hypothesis or went in an unexpected direction? These are often the most useful findings and the most commonly buried.
• What confidence level do you assign to this finding (high, medium, or low) and why?
• Where & how did you establish your HML correlation levels? Ensure you aren’t claiming that low to no correlation claims (~<0.3) aren’t flagged as strong correlation. Using a scatterplot (vs. column or line charts) can help effectively visualize this.
• What follow-on test(s) would most usefully strengthen, challenge, or extend this finding?
• What would a better-resourced or more rigorous version of this test look like?
• Can high-level claims be validated if deeper level segments were not tested?
• Have you checked for heteroskedasticity (a split in the data that hides something deeper in the analysis)? Heteroskedasticity checks are a great way to make sure you're not claiming a correlation on something that isn't - and may provide a deeper gem on what things are really correlated.
• Are you making controversial or exceptional claims? The more extraordinary the claim, the more extraordinary the proof is required.

8. Transparency: Are You Being Transparent About the Full Picture?

Transparency isn't just about methodology. It's about conflicts of interest, access requirements, and the honest acknowledgment of what's missing. A result shared without this context is harder for the community to interpret and leverage with accuracy.

• Do you have any conflict of interest to disclose? This might be a tool, product, or service that benefits from a particular result.
• What access, tools, or budget does replicating this test require? Be specific. A test that requires enterprise log file access or significant API spend means not everyone can replicate on a free account.
• What's explicitly missing from this test that a reader should know before acting on it?
• If you're publishing raw data or outputs alongside the summary: where can readers access them?

9. Replicability: Can Someone Else Run This?

If your test can't be reproduced, it can't be built upon.

• Is this a 0/1 test (establishing whether something is possible at all) or a consistency test (establishing how reliably it happens at scale)?
• Could someone else follow your methodology and expect comparable results? If not, consider what's missing from your documentation.
• Are you interested in others attempting to replicate this? If so, what would be most useful to vary? E.g. model, site type, industry, page type, something else?
• Is there anything about your specific setup that would make replication difficult? Things like proprietary tools, site access, unusual traffic volume.

10. Presentation: How Are You Presenting the Results?

How you show results shapes what people take away from them, often more than the results themselves. Raw data, clear labeling, and honest framing make findings more useful and more credible - and hopefully, more likely to be taken seriously.

• Are you showing raw outputs or data alongside your interpretation, not just your conclusions? Let readers assess the evidence themselves.
• Are you clearly distinguishing between what the data shows and what you conclude from it?
• Do your charts and tables include sample size and date range visibly? (Don’t just bury this in the methodology.)
• Have you used visuals (charts, tables, comparison graphics) to make the key finding legible at a glance? Results that are easy to read are results that get shared, discussed, and built on.

This writeup is a starting point, not a finish line. The questions above won't cover every edge case, and the right answers will keep shifting as the models do. If you need the summary version, find that here. 

But that's exactly why a shared standard matters: not to slow things down, but to make sure the work we're all doing actually builds on itself. 

If you ran a test and these questions would have changed how you set it up or reported it, that's useful information. If there's a variable we missed that burned you, we want to know. The goal is a living resource that gets sharper with every person who uses it and every test that gets run. 

Come find us in The SEO Community, share your feedback, and if you're interested in running tests or proposing new ones - that's exactly the kind of community we're trying to build.