Cookie Consent by Free Privacy Policy Generator

AI Content Evaluation App with Google Guidelines, advertools, OpenAI API & Dash

Go to the Best Of the SEO Community.

Elias
Elias
Nov 14, 2024, 5:36 AM
 
Evaluating content with AI on a large scale - I have an app for that
 
Recipe:
Google content guidelines + advertools + OpenAI API + Dash
  • ???? Content guidelines: Set of questions to ask about a web page’s content.
  • ???? advertools: crawls the given pages, and extracts the title and body text.
  • ???? OpenAI API: Evaluates the content by answering each of the questions True/False.
  • ???? Dash: A no-code GUI for using all this.
 
The summary is the average score for each question for a bunch of URLs.
 
Todo: Getting a “True” for a question is not necessarily a good thing, and it depends on the phrasing of the question. Answers need to be normalized, True = good, False = bad.
 
If you have traffic data about the pages, even better. You can do the evaluation weighted by traffic.
 
You can play with the app if you want: <https://adver.tools/llm-crawl-eval/>
 
Would love any suggestions, or if you get any issues.
Thanks!
Content Evaluation with ChatGPT
Use an LLM (ChatGPT) to check which of Google's helpful content guidelines are covered by the article.
Forwarded thread from another channel:
Valentin Pletzer
Valentin Pletzer
Nov 14, 2024, 6:09 AM
Love it. One thing I believe that could be improved: Most of the questions result in “True” in the positive case but some like “Does the content have any spelling or stylistic issues?” “False” is the good result. This makes it a bit hard to check on a glance.
Elias
Elias
Nov 14, 2024, 7:21 AM
Thanks @pletzer!
You’re absolutely right. The answers should be normalized 1=good and 0=bad for example, and then the percentages would make sense, and would be clear.
I’ll focus on this next.
Ray Grieselhuber
Ray Grieselhuber
Nov 14, 2024, 7:50 AM
Advertools is one of the best open source projects on the internet!
Bill Scully
Bill Scully
Nov 14, 2024, 5:17 PM
Stuck at Loading for me.
Elias
Elias
Nov 15, 2024, 12:58 AM
Sorry for that, might be too many people.
Please try again and try refreshing the browser. I just checked, it's working.
Mika Lepistö
Mika Lepistö
Nov 15, 2024, 1:35 AM
@elias please post to <#C05H79LAGUF|> when adding your own stuff.
Elias
Elias
Nov 15, 2024, 2:55 AM
@mika OK will do :)
Mark Barrus
Mark Barrus
Nov 15, 2024, 7:05 PM
Excellent tool. Please let me know when you have normalized, "answers 1=good and 0=bad"
Elias
Elias
Nov 15, 2024, 7:57 PM
Happy you liked it @markbarrus
Will post an update when done.
Thanks!
Mark Barrus
Mark Barrus
Nov 15, 2024, 7:59 PM
Appreciate you. It might also be helpful to have the sliding scale from 1 to 10, or some such.
Kristine Schachinger
Kristine Schachinger
Nov 27, 2024, 12:23 AM
Does it give the same answer if you run the same data say ten times?
Elias
Elias
Nov 27, 2024, 10:27 AM
@schachin Great question.
 
I just tested three URLs ten times each. Attached is the summary and below are some observations/info:
 
• 3 articles x 10 times = 30 times each question was asked
• URLs with a count of 30 (whether True or False) mean that it was perfectly consistent
• In 11 out of 21 questions we have a count of 30
• In the rest it’s mostly around 20:10
• Almost half the time it was 100% consistent, the other half it’s as good as a toss of a 60:40 coin
• Temperature is set to zero, and I used a seed in the prompts to ensure reproducibility
Happy to run this again, using different parameters, URLs, number of tests, or another approach.

Channels

  • agency (3)
  • ai-for-seo (11)
  • ai-llms (24)
  • analytics (13)
  • biz-of-seo (4)
  • career (4)
  • content (15)
  • cro (4)
  • data-viz-reporting (4)
  • datascience (2)
  • ecommerce (3)
  • events (1)
  • freelance (2)
  • general-seo (21)
  • in-house (2)
  • local-seo (12)
  • local-seo,biz-of-seo (1)
  • lounge (1)
  • reading-list-resources (3)
  • search-console (11)
  • shameless-plug (1)
  • technical-seo (37)
  • tools (1)


    • Join The Community

    The SEO Community is a free organization, with private Slack group and events, for those interested in search and AI optimization to learn, discuss, and meet others in the industry.

    Powered by MODX

    Resources

    The Community Story SEO Articles SEO Jobs SEO Conferences Top SEO Discussions Newsletter Free Tools Learn Claude Code

    The SEO Community © 2026

    Shop Merch Code of Conduct Partnerships Sitemap Update cookies preferences Terms of Service Privacy Policy