If your team treats a single AI-generated answer as proof that an article “won” a citation, this study should change how you measure results. A new arXiv preprint testing a GPT-5.4 search agent (using Exa results replayed offline) finds that presentation and model randomness shift whether a page is cited far more often than simply moving that page up one slot in the search results.
How the experiment worked
Researchers Sriram Selvam and Anneswa Ghosh analyzed saved transcripts from 130 queries answered by a GPT-5.4 search agent. From those transcripts they selected pages that appeared together in the same five search results and independently supported the same factual point, yielding 113 page pairs for the main tests (a later blinded human check confirmed 103 true matches).
For each pair the team replayed the conversation four ways: swapping which page appeared above the other, and showing each page either as plain paragraphs or as a structured rewrite with headings, lists or a table. Only the agent’s final answer was regenerated in each replay. The rewrites were created by AI (mostly Grok 4.3, with GPT-5.4 as fallback for one pair) and checked to ensure the same facts were present.
Key findings
Looking at the original saved transcripts, a page that appeared first among the five Exa results was cited 85.1% of the time versus 42.8% for a page in fifth position — a raw gap of 42.3 percentage points. The authors note this raw gap mixes position with the underlying quality of pages, since search rankings typically surface more relevant content higher.
When researchers replayed conversations and moved the same page above its matched partner, citation likelihood rose by 7.9 percentage points, but this increase was not statistically significant after correcting for multiple comparisons. In a separate set of 56 pairs where only order was switched, the estimated effect was essentially zero (0.0 points; 95% confidence interval -5.4 to +5.4).
By contrast, structured rewrites (headings and lists) produced a measurable lift in how much credit a page received. Rewritten pages averaged 0.50 more inline citation markers per answer than the same pages shown as plain paragraphs (95% confidence interval 0.20 to 0.84). The binary chance of being cited at all rose by an estimated 4.5 percentage points (95% interval -1.4 to +10.4), although the paper cautions that the experiment was powered to reliably detect effects of roughly 8.5 points or larger.
Model randomness mattered. Re-running the same inputs produced different citation outcomes in 15% of reruns, and the authors estimate about 45% of single-run variation is due to model stochasticity. When the researchers replayed the experiments on Grok 4.3 they saw similar directionality for formatting effects, though fewer than half of Grok’s first replies used the correct citation format.
Why this matters for marketers, publishers and measurement teams
There are two immediate implications. First, correlations between being cited by an AI agent and features like position or JSON-LD do not prove causation. The study references an industry finding that AI-cited pages were more likely to include JSON-LD, but unless you experimentally vary the element you cannot claim a causal effect.
Second, single-answer attribution is noisy. Formatting, the specific model and run-to-run randomness can all flip whether a page is named. That makes a lone AI response a weak signal for declaring SEO or content wins.
Practically: design citation experiments that repeat the same queries multiple times, test across different models and retrieval providers, and control for the retrieval step if you want results that map to live SEO changes. Because this study applied rewrites only to text already retrieved by the search provider, its results do not prove that reformatting a live page will necessarily boost citations in deployed systems.
What to watch next
Look for peer review of this preprint, independent replications across other models and experiments that include crawling and ranking steps. Those follow-ups will show whether structured formatting consistently helps in production AI search or whether the effects are model- and setup-specific. For now, treat modest citation lifts with caution and require replication before translating them into content or measurement decisions.
