Cookie Consent by Free Privacy Policy Generator

The Best Of

Go to the Best Of the SEO Community.

Noah
Noah
Jan 19, 2025, 10:05 AM
Forwarded from another channel:
Forwarded thread from another channel:
Elias
Elias
Nov 14, 2024, 5:36 AM
Evaluating content with AI on a large scale - I have an app for that
Recipe:
Google content guidelines + advertools + OpenAI API + Dash
???? Content guidelines: Set of questions to ask about a web page’s content.
???? advertools: crawls the given pages, and extracts the title and body text.
???? OpenAI API: Evaluates the content by answering each of the questions True/False.
???? Dash: A no-code GUI for using all this.
The summary is the average score for each question for a bunch of URLs.
Todo: Getting a “True” for a question is not necessarily a good thing, and it depends on the phrasing of the question. Answers need to be normalized, True = good, False = bad.
If you have traffic data about the pages, even better. You can do the evaluation weighted by traffic.
You can play with the app if you want:
Would love any suggestions, or if you get any issues.
Thanks!
Use an LLM (ChatGPT) to check which of Google's helpful content guidelines are covered by the article.
Valentin Pletzer
Valentin Pletzer
Nov 14, 2024, 6:09 AM
Love it. One thing I believe that could be improved: Most of the questions result in “True” in the positive case but some like “Does the content have any spelling or stylistic issues?” “False” is the good result. This makes it a bit hard to check on a glance.
Elias
Elias
Nov 14, 2024, 7:21 AM
Thanks @Valentin Pletzer!
You’re absolutely right. The answers should be normalized 1=good and 0=bad for example, and then the percentages would make sense, and would be clear.
I’ll focus on this next.
Ray Grieselhuber
Ray Grieselhuber
Nov 14, 2024, 7:50 AM
Advertools is one of the best open source projects on the internet!
Bill Scully
Bill Scully
Nov 14, 2024, 5:17 PM
Stuck at Loading for me.
Elias
Elias
Nov 15, 2024, 12:58 AM
Sorry for that, might be too many people.
Please try again and try refreshing the browser. I just checked, it's working.
Mika Lepistö
Mika Lepistö
Nov 15, 2024, 1:35 AM
@Elias please post to <#C05H79LAGUF|> when adding your own stuff.
Elias
Elias
Nov 15, 2024, 2:55 AM
@Mika Lepistö OK will do :)
Mark Barrus
Mark Barrus
Nov 15, 2024, 7:05 PM
Excellent tool. Please let me know when you have normalized, "answers 1=good and 0=bad"
Elias
Elias
Nov 15, 2024, 7:57 PM
Happy you liked it @Mark Barrus
Will post an update when done.
Thanks!
Mark Barrus
Mark Barrus
Nov 15, 2024, 7:59 PM
Appreciate you. It might also be helpful to have the sliding scale from 1 to 10, or some such.
Kristine Schachinger
Kristine Schachinger
Nov 27, 2024, 12:23 AM
Does it give the same answer if you run the same data say ten times?
Elias
Elias
Nov 27, 2024, 10:27 AM
@Kristine Schachinger Great question.
I just tested three URLs ten times each. Attached is the summary and below are some observations/info:
• 3 articles x 10 times = 30 times each question was asked
• URLs with a count of 30 (whether True or False) mean that it was perfectly consistent
• In 11 out of 21 questions we have a count of 30
• In the rest it’s mostly around 20:10
• Almost half the time it was 100% consistent, the other half it’s as good as a toss of a 60:40 coin
• Temperature is set to zero, and I used a seed in the prompts to ensure reproducibility
Happy to run this again, using different parameters, URLs, number of tests, or another approach.

Our Values

What we believe in

Building friendships

Kindness

Giving

Elevating others

Creating Signal

Discussing ideas respectfully

What has no home here

Diminishing others

Gatekeeping

Taking without giving back

Spamming others

Arguing

Selling links and guest posts


Sign up for our Newsletter

Join our mailing list for updates

By signing up, you agree to our Privacy Policy and Terms of Service. We may send you occasional newsletters and promotional emails about our products and services. You can opt-out at any time.

Apply now to join our amazing community.

Powered by MODXModx Logo
the blazing fast + secure open source CMS.