Method: Does GEO Work for LLMs? What C-SEO Bench Found

Claims ledger

What the explainer may say about the paper (46)

Each statement was checked by an editor against the paper itself. The quote is the paper’s own words.

1. The paper presents C-SEO Bench, described as the first benchmark to evaluate conversational SEO methods across multiple tasks, domains and numbers of competing actors.

   “We present C-SEO Bench, the first benchmark designed to evaluate C-SEO methods across multiple tasks, domains, and number of actors.”

   Method · Abstract

2. The benchmark covers two search tasks — question answering and product recommendation — with three domains each.

   “We consider two search tasks, question answering and product recommendation, with three domains each.”

   Method · Abstract

3. The headline finding is that most current C-SEO methods are largely ineffective and often hurt document ranking, the opposite of what is expected.

   “Our experiments reveal that most current C-SEO methods are not only largely ineffective but also frequently have a negative impact on document ranking, which is opposite to what is expected.”

   Finding · Abstract

4. Traditional SEO strategies, which improve the ranking of the source in the LLM context, were significantly more effective than C-SEO methods.

   “Instead, traditional SEO strategies, those aiming to improve the ranking of the source in the LLM context, are significantly more effective.”

   Finding · Abstract

5. As more actors adopt C-SEO, the overall gains decrease, which the authors describe as a congested, zero-sum problem.

   “We also observe that as we increase the number of C-SEO adopters, the overall gains decrease, depicting a congested and zero-sum nature of the problem.”

   Finding · Abstract

6. The authors conclude C-SEO will complement rather than replace traditional SEO.

   “Hence, we believe C-SEO will not replace SEO, but will complement it.”

   Finding · Section 1, p. 2

7. Effectiveness is measured as improvement in citation ranking: an effective method should make the LLM cite the modified document earlier than the original.

   “We propose to measure the effectiveness of C-SEO methods by the improvement in the citation ranking, i.e., an effective method should lead the LLM to cite the modified document earlier than it cited the original document.”

   Method · Section 1, p. 2

8. Rank improvement (gain) is defined as the baseline citation rank minus the citation rank after C-SEO optimisation at adoption rate alpha.

   “We define the rank improvement (i.e., gain) for the document i for query q as: ∆q,d = rank0 d − rankαd”

   Method · Section 4.2, p. 6

9. Statistical significance of rank improvements was assessed with a right-tailed Wilcoxon signed-rank test with Holm–Bonferroni correction for multiple comparisons.

   “We access the statistical significance of the rank improvement using the right-tailed Wilcoxon signed-rank test. This test allows us to evaluate whether the rank after applying a C-SEO method is statistically significantly smaller than the original rank. We adjust the p-values using the HolmBonferroni correction [Holm, 1979]”

   Method · Section 4.2, p. 6

10. The authors introduce an AUC metric: the area under the curve of average gains across adoption rates from 0% to 100%.

    “We propose to report the Area Under the Curve (AUC) of the overall gains computed in the range of adoption rate from 0% to 100%”

    Method · Section 4.2, p. 6

11. The benchmark contains more than 1.9k queries and 16k documents across six domains.

    “In total, our C-SEO benchmark contains more than 1.9k queries and 16k documents.”

    Number · Section 4.1, p. 6

12. C-SEO Bench covers 2 tasks and 6 domains, uses real data, and contains 16.3k documents with many adopters.

    “C-SEO Bench (Ours) 2 6 ✓ 16.3k Many”

    Number · Table 1, p. 3 · wording edited by the editor

13. Per-domain dataset sizes are: Retail 500 queries / 5000 docs, Video Games 436 / 4360, Books 249 / 2245, Web 300 / 1500, News 294 / 2375, Debate 142 / 880.

    “Retail 500 5000 Video Games 436 4360 Books 249 2245 Web 300 1500 News 294 2375 Debate 142 880”

    Number · Table 2, p. 5

14. The datasets used include Amazon Shopping Queries, Steam Games from FronkonGames, Natural Questions and Multi News, all licensed for research use.

    “We used the following datasets with the corresponding licenses. In all cases, they allow the use of their data for research purposes, so we agree to their terms of use. • Amazon Shopping Queries [Reddy et al., 2022]. License: Apache 2.0 • Steam Games from FronkonGames. License: CC-by-4.0. • Natural Questions [Kwiatkowski et al., 2019]. License: CC-by-3.0. • Multi News [Fabbri et al., 2019].”

    Method · Appendix B, p. 13 · wording edited by the editor

15. In the product recommendation task the system retrieves 10 unsorted documents and the LLM must recommend the best five products using only the retrieved content.

    “The user submits a query, and the system retrieves 10 documents (product descriptions), which are presented in an unsorted order. The LLM must then recommend the best five products based on the provided descriptions, using only the content of the retrieved documents.”

    Method · Section 4, p. 5

16. Ten C-SEO methods were benchmarked in total: eight from Aggarwal et al. 2024 plus two new ones proposed by the authors.

    “Overall, we benchmark a total of ten methods.”

    Method · Section 5, p. 7

17. The two new methods are Content Improvement (combining all eight transformations) and LLM Guidance, a prompt-based method inspired by the llms.txt standard that prepends a markdown summary to the document.

    “1. Content Improvement: The combination of all the eight transformations above into a single holistic text improvement method. An example is shown in Figure 3. 2. LLM Guidance: Inspired by the LLMs.txt standard4 , this novel prompt-based method generates a markdown summary that is concatenated at the beginning of the original document to guide the LLM about its content.”

    Method · Section 5, p. 7

18. All C-SEO transformations were produced with gpt-4o-mini, and the conversational search engine was run with gpt-4o-mini, Claude 3.5 Haiku, o3 and o4-mini.

    “We use gpt-4o-mini-2024-07-18 to run all the C-SEO methods and run the Conversational Search Engine with gpt-4o-mini-2024-07-18, claude-3-5-haiku-20241022, o3-2025-04-16 and o4-mini-2025-04-16, covering two state-of-the-art chat and reasoning models.”

    Method · Section 6.1, p. 8

19. In the unilateral-adoption experiments a document was selected at random to receive the C-SEO treatment, so that SEO position would not confound the C-SEO evaluation.

    “In the experiments with unilateral adoption (i.e., only one player improving its document), we randomly select a document to apply a C-SEO method. In this way, the ranking from SEO does not impact the evaluation of C-SEO.”

    Method · Section 6.1, p. 8

20. In the multi-actor experiments the maximum number of actors was ten, so 10% adoption means one actor and 100% means ten.

    “In the experiments with increasing number of actors, we use use a maximum of ten. So 10% means one actor, and 100% means ten.”

    Method · Section 6.1, p. 8

21. After applying LLM guidance, 61.0% of retail product ranks stayed the same.

    “For example, 61.0% of the retail product ranks are the same after applying the LLM guidance.”

    Number · Section 6.2, p. 8

22. LLM guidance on retail data produced a positive rank boost in 26.2% of cases and a negative boost in 12.8%, so the average effect was near zero with high variance.

    “guidance on the retail data has a positive boost in 26.2% of the cases and a negative boost in 12.8% of the cases. As a result, the overall average effect is close to zero, but with high variance.”

    Number · Section 6.2, p. 9

23. The authors report only three statistically significant ranking improvements out of 54 cases.

    “Out of 54 cases, we uncover only three where the ranking improvements are statistically significant.”

    Number · Section 6.2, p. 9 · wording edited by the editor

24. Only LLM guidance and content improvement gave significant gains, and only in retail (content improvement) and retail plus video games (LLM guidance); no method worked for question answering or for Haiku 3.5.

    “Among all evaluated methods, only LLM guidance and content improvement yield statistically significant gains. But these gains are only significant on the retail domain for content improvement, and only on the retail and video games domains for LLM guidance. We found no method effective for the question answering task, and no methods effective for the Haiku 3.5 model (results in appendix).”

    Finding · Section 6.2, p. 9

25. Left-tailed tests showed many C-SEO methods significantly reduce rankings: the Statistics method lowered rankings in 19 of 24 settings, product recommendation on Haiku 3.5 showed 26 of 30 significant negative cases, and question answering on o4-mini 19 of 30.

    “for example, the Statistics method decreases rankings in 19 out of 24 evaluated settings. In product recommendation tasks on Haiku 3.5, 26 out of 30 cases show significant negative effects, and in question answering on o4-mini, 19 out of 30 cases do so.”

    Number · Section 6.2, p. 9

26. On GPT-4o-mini, LLM Guidance improved rank by 0.36 on Retail, 0.24 on Games, 0.14 on Books, 0.10 on Web, 0.00 on News and 0.15 on Debate.

    “LLM Guid. 0.36 ±1.47 0.24 ±1.05 0.14 ±1.07 0.10 ±0.99 0.00 ±1.10 0.15 ±1.65”

    Number · Table 3, p. 8 · wording edited by the editor

27. Best SEO on GPT-4o-mini gave rank improvements of 2.77 (Retail), 1.89 (Games), 1.60 (Books), 0.87 (Web), 0.70 (News) and 1.54 (Debate).

    “Best SEO 2.77 ±2.31 1.89 ±2.32 1.60 ±2.04 0.87 ±1.35 0.70 ±1.64 1.54 ±2.07”

    Number · Table 3, p. 8

28. On Claude 3.5 Haiku, the Statistics method lowered rank in every domain: -0.82 on Retail, -0.10 on Games, -0.60 on Books, -0.58 on Web, -0.12 on News and -0.85 on Debate.

    “Statistics -0.82 ±1.47 -0.10 ±1.01 -0.60 ±1.34 -0.58 ±1.18 -0.12 ±0.91 -0.85 ±1.66”

    Number · Table 4, p. 8 · wording edited by the editor

29. On the o3 model none of the document optimisation methods achieved significant gains while Best SEO was always statistically significant.

    “None of the LLM Document Optimization methods achieve significant gains. Best SEO is always statistically significant.”

    Finding · Table 9 caption, p. 15

30. On the o4-mini model none of the document optimisation methods achieved significant gains while Best SEO was always statistically significant.

    “None of the LLM Document Optimization methods achieve significant gains. Best SEO is always statistically significant.”

    Finding · Table 11 caption, p. 16

31. The authors explain the contrast with the optimistic GEO results of Aggarwal et al. 2024 as a difference in metrics: that work reported word count, which does not measure LLM preference the way citation ranking does.

    “The differences are due to the metrics used. They report their main results as word count, i.e., the ratio of words used in the LLM response to talk about the target document. However, this metric does not measure the LLM preference, contrary to the citation ranking that we use. A higher word count does not necessarily correspond to a better citation ranking.”

    Finding · Section 6.2, p. 9

32. The authors state that the results of both papers on LLM preferences do not contradict each other.

    “Therefore, the results of both papers on LLM preferences do not contradict each other.”

    Finding · Section 6.2, p. 9

33. Placing the target document first in the LLM context gave far greater citation ranking gains than any C-SEO method, indicating in-context position dominates.

    “making the target document the first one in the LLM context window leads to far greater citation ranking gains in the LLM response than any C-SEO method. This indicates that the position of a document in the input context has a dominant effect on how the LLM ranks or selects content.”

    Finding · Section 6.3, p. 9

34. Placing the document first in the GPT-4o-mini context improved its rank by 2.77 on Retail, 1.89 on Games, 1.60 on Books, 0.87 on Web, 0.70 on News and 1.54 on Debate.

    “1 2.77 ±2.31 1.89 ±2.32 1.60 ±2.04 0.87 ±1.35 0.70 ±1.64 1.54 ±2.07”

    Number · Table 5, p. 14 · wording edited by the editor

35. In the multi-actor simulation, LLM Guidance reached AUC 1.88 on retail and 1.58 on games, while Content Improvement reached 1.05 and 0.74.

    “LLM Guid. [Retail] (AUC = 1.88) LLM Guid. [Games] (AUC = 1.58) Content Impr. [Retail] (AUC = 1.05) Content Impr. [Games] (AUC = 0.74)”

    Number · Figure 5, p. 10

36. Early adopters of C-SEO see significant gains, but gains fall steadily as more players adopt the same method, revealing congestion and zero-sum dynamics.

    “We observe that while early adopters experience significant gains, these gains decrease steadily as more players adopt the same method. This pattern reveals two key characteristics of C-SEO: it is both a congested and a zero-sum game.”

    Finding · Section 6.4, pp. 9-10

37. The authors warn that using current C-SEO techniques carries potential risks for content creators.

    “At the same time, our negative results show that the use of current C-SEO techniques carries potential risks for content creators.”

    Finding · Section 7, p. 10

38. The conclusions apply only to white-hat C-SEO methods; black-hat (adversarial) methods are outside the paper's scope.

    “Nevertheless, our conclusions apply exclusively to white-hat C-SEO methods, leaving black-hat C-SEO methods outside our scope.”

    Limitation · Section 7 Limitations, p. 10

39. The paper does not examine the interplay between traditional SEO and C-SEO methods.

    “Moreover, we do not examine the potential interplay between traditional SEO and C-SEO methods.”

    Limitation · Section 7 Limitations, p. 10

40. The experiments were conducted only with commercial proprietary models.

    “Our experiments are conducted with commercial proprietary models.”

    Limitation · Section 7 Limitations, p. 10

41. The study does not cover setups where documents do not fit in the LLM's context window.

    “We also do not examine setups where documents do not fit the context of the LLM, leaving experiments on how summaries of documents can affect citation order for future work.”

    Limitation · Section 7 Limitations, p. 10

42. The dataset is restricted to English-language content, which may limit generalisability to other linguistic or cultural contexts.

    “Lastly, our dataset is restricted to English-language content, which may reduce the generalizability of our findings to other linguistic or cultural contexts.”

    Limitation · Section 7 Limitations, p. 10

43. For the books dataset, queries were generated with GPT-4o by first asking for the most popular book genres, then requesting short queries of at most three words.

    “For the books dataset, we generate queries using GPT-4o by first asking for the most popular book genres, and then requesting short queries (maximum of three words) that correspond to searches”

    Method · Section 4.1, p. 5 · wording edited by the editor

44. The web dataset sampled 300 questions from Natural Questions, expanded them with GPT-4o-mini, and retrieved five pages per question via the Brave Search API.

    “we sample 300 questions from the development set of Kwiatkowski et al. [2019]. We generate three variations of every question using query expansion with GPT-4o-mini (prompt in Section F). Then, we use each expanded query to search the web using the Brave Search API and retrieve the most relevant pages, collecting five pages per original question.”

    Method · Section 4.1, p. 6

45. The authors model the interaction among content providers as a non-cooperative game in which each player controls one document and aims to improve its citation rank.

    “We model the interaction among content providers as a noncooperative game. Each player i controls a document di and aims to maximize their utility ui , defined in terms of their citation rank.”

    Method · Section 3, p. 5 · wording edited by the editor

46. Code and data are publicly released on GitHub and Hugging Face.

    “Our code and data are available at https://github.com/parameterlab/c-seo-bench and https://huggingface.co/datasets/parameterlab/c-seo-bench.”

    Method · Abstract

Every site

What each site returned