For the last couple of days, I’ve been playing with creating a pipeline to generate YouTube titles based on trending videos.
I use a popular creator’s newsletter content to get the Good and Bad titles along with his feedback on every title. The pipeline in itself is easy to setup. Every week I get an email which is stored in my backend > vectorised.
The hard part is creating the evals. I started with creating a set of 25 prompts, but it proved to be ineffective. I then expanded to 100 prompts and the quality did improve, but it’s not hardened according to me. And I could expand to say 500 prompts, but not sure I’m going to see a comparable increase in the output quality. And rating the 500 eval prompts is a big task in itself.