Evaluating content with AI on a large scale - I have an app for that
Recipe:
Google content guidelines + advertools + OpenAI API + Dash
Content guidelines: Set of questions to ask about a web page’s content.
advertools: crawls the given pages, and extracts the title and body text.
OpenAI API: Evaluates the content by answering each of the questions True/False.
Dash: A no-code GUI for using all this.
The summary is the average score for each question for a bunch of URLs.
Todo: Getting a “True” for a question is not necessarily a good thing, and it depends on the phrasing of the question. Answers need to be normalized, True = good, False = bad.
If you have traffic data about the pages, even better. You can do the evaluation weighted by traffic.
You can play with the app if you want: <https://adver.tools/llm-crawl-eval/>
Would love any suggestions, or if you get any issues.
Thanks!
