←
Mike Taylor37,085 posts
Mike Taylor avatar

Mike Taylor

@hammer_mt

🧪 Head of Evals @Every 📕 2x O'Reilly author: Prompt Engineering / DSPy 🎓 400k students on AI Udemy course 🤖 AI engineer since 2020 📈 Built 50 person agency

◷ Joined February 2009▧ 4,134 Photos and videos
Mike Taylor avatar
Mike Taylor@hammer_mt·Oct 11

How to build good evals without getting lost or in trouble. You should see evals as saving past work and replaying it with new models or prompts/skills to see if it improved. This helps you do 3 things: 1. see where it fails so you can tell your boss/users 2. optimize it until it improves 3. make it work on a cheaper / open source model Measuring "if it improved" is 90% of the work and 10% is infra. For infra just tell your agent to store tasks in Cloudflare R2 (a cheap/free way to store files in the cloud) in Harbor task format. This means saving the instructions for the task and how the folder looked before you started the task (does the agent have all the files you had access to, in the state they were in?). When you run a task your agent should pull down the files, run a docker container (a sandbox on your computer so your agent can't peek at the answer), and send the instructions. Ideally you copy across your auth credentials so you can use your codex/claude sub (otherwise it has to use expensive API tokens). That's probably all the sophistication you need but one more cool thing you can do is mock any tools your agent might use i.e. if your agent usually reads slack channels or searches google or searches your documents you just need a way to fake it and send back something without touching the real services (otherwise it'll get something new every time you run it). Ok onto the good stuff: measuring performance. When you have the output of what the different models or prompts/skills did on your task, you need a way to grade how good or bad that output is or you can't improve it. There are three types of evals: 1. Human 2. Model 3. Code Humans are the most slow and expensive but usually* the most accurate if you're working with domain experts. Do everything you can to bring the AI output into where they already live and work. For an editor put the AI edits in Google docs as comments. Engineers should be reviewing actual PRs. Marketers should be reviewing copy in notion or wherever they normally do it. This decreases cognitive load which is important because their time is valuable and data labeling sucks. Be grateful for anything you can get from them and use their time extremely wisely. Jump on a call with them and get them to talk out loud about what they like or don't like. AI can always turn those transcripts into structured data afterwards. The good news is when you've got some feedback out of them you can train an AI judge by rewriting it's prompt until it agrees 90% with human judgements. Once you have that you can run a/b tests and optimize the prompt/skill for your original task and have the model grade it until you find something that performs better/cheaper. Simple yes/no questions work best, assign 1 or 0 for each specific thing that could go right or wrong, so it's impossible to misinterpret the grade. For example did this article have a specific AI tell (one judgement for each type of tell), did the powerpoint use our brand colors, or is it following our coding conventions? Models are cheaper and faster at grading than humans, so you can run many more tests and automatically optimize your agents when you have an AI judge you can trust. If you get here you're in the top 5% of AI users today and life will continue to improve for you. You'll have proof of what works or doesn't and actually be able to see the frontier and when it shifts. Look at the data when something rests badly and you'll learn something new about your domain. This is when you'll realize that the human is not the ceiling. Quite often the human domain experts get things wrong and explore methods for superhuman results. For more sophistication you want to invest in building something closer to an RL environment with deterministic code tests that run faster and cheaper at scale, more on that another time. Good luck building evals - this is one of the things AI still does badly and there's a great career to be had!

Dan Shipper avatar
⟳ Mike Taylor retweeted
Dan Shipper@danshipper·1h

i know it's a bummer to get negative feedback, but this is a bit unfair. @every we always re-test new release candidates down to the wire. the video was filmed on monday and the vibe check was written end to end yesterday. i don't see any comms from the @openai team that there were new changes to test over the last 3-4 days. we absolutely would've started from scratch if there were!

Dan Shipper avatar
⟳ Mike Taylor retweeted
Dan Shipper@danshipper·1h

i know it's a bummer to get negative feedback, but this is a bit unfair. @every we always re-test new release candidates down to the wire. the video was filmed on monday and the vibe check was written end to end yesterday. i don't see any comms from the @openai team that there were new changes to test over the last 3-4 days. we absolutely would've started from scratch if there were!

Theo - t3.gg avatar
⟳ Mike Taylor retweeted
Theo - t3.gg@theo·9h

For those curious, Claude's dominance in T3 Code did not take a hit from the OpenAI release today

Post attachment
Nate Berkopec avatar
⟳ Mike Taylor retweeted
Nate Berkopec@nateberkopec·Sep 30

remember how people used to spend years learning vim or emacs? haha take a big fat L buddy, I used Sublime Text until December and now I just talk to Claude through this gimp mask

Post attachment
Mike Taylor avatar
Mike Taylor@hammer_mt·11h

Cosigned

All this pangram controversy (ie. the latest Caine Prize) could be avoided by allowing writers to use whatever tool (AI included) they want and judging stories by their own merit This graph will only continue to grow and the literary world will tear itself apart

Post attachment
Aki|AIに任せて寝る avatar
⟳ Mike Taylor retweeted
Aki|AIに任せて寝る@Aki_LIG·13h

AIに「自分らしい文章」を書かせる方法が、ついに見つかった 結論 AIに良い文章を書かせる秘訣は、上手な指示を書くことではない。実際に文章を書かせて、比較し、指示を改善し続けることだった。 引用元の記事の著者は、6年間、自分らしい文章をAIに書かせようとしてきた。そして、従来とは違う方法で大きな改善に成功した。 1. 従来の方法は、なぜダメだったのか? 過去の文章をAIに読ませて「私の文体を分析して、同じように書いて」と頼む。 一見よさそうだが、AIは特徴を誇張してしまう。 例えば「短い文章が多い」「ユーモアを使う」と分析すると、そればかりを繰り返して不自然になる。 実験では、文体を指示しない方が良い結果になることさえあった。 2. 成功した方法は何か? やったことはシンプル。 お手本を集める:自分が実際に書いた文章を12〜25本ほど用意する。 内容と文体を分ける:元の文章から、主張・事実・エピソードだけを取り出す。 AIに書かせて比較する:同じ内容を異なる指示で書かせ、どちらが本人の文章に近いかを別のAIに判定させる。 指示を改善する:良くなった変更だけ残し、悪くなったら元に戻す。これを繰り返す。 重要なのは、AIに「もっと自然に」と頼むのではなく、何が不自然だったのかを具体的に見つけ、次の指示に反映すること。 3. どれくらい改善したのか? 著者の比較実験では、Claude Opus 5.5の成績が次のように変化した。 文体の指示なし 16% 最適化した指示あり 93% 著者独自の比較評価での成績。文章品質の絶対点ではない。 さらに驚くことに、改善の大部分は最初の3回程度の試行で得られた。 実際の執筆も、以前は丸1日かかっていたところ、約1時間で進められるようになったという。ただし、本人による修正は引き続き必要だった。 4. 最も大切な学び この記事から持ち帰るべきことは3つ。 ① AIの出力を良くしたければ、指示より先に評価方法を考える。 良くなったかどうかを判断できなければ、改善はできない。 ② AIの失敗を次の改善材料にする。 「気に入らない」で終わらせず、何が違うのかを記録して修正する。 ③ AIに考えを丸投げしない。 著者が強調しているのは、何を伝えるかは人間が決めるということ。AIが得意なのは、その考えを適切な文章にすることだ。 この記事の本当の価値 これは文章作成だけの話ではない。 お手本を用意する → AIに作らせる → 比較して評価する → 改善する → 再び試す。 この仕組みを作れば、文章だけでなく、コード、デザイン、動画制作などにも応用できる可能性がある。 ただし、良し悪しをきちんと判定できる評価方法が必要だ。 AIを賢く使う鍵は、最初から完璧な指示を書くことではなく、AIが自分の失敗から改善できる仕組みを作ること。 これがこの記事の一番大きな学びだと思う。

I wrote about how I automated my writing with my automated writing skill using my automated writing skill. Enjoy! every.to/also-true-for-human…

Cody Plofker avatar
⟳ Mike Taylor retweeted
Cody Plofker@codyplof·20h

Here are my top Runneth use cases so far: 1. Ad uploading - It handles all my naming conventions, utms, etc. Does it way faster and better than I could. Saves hours and hours per week. 2. Image Gen - It's really good at image generation. Much better than I was able to get in claude. 3. Landing Pages - As you know I went deep into building design systems and landing pages in Claude. I was early, but it's so much easier now. Runneth can design landers and website pages super well. I've been super impressed here. 4. Editing. Although early, Runneth can run on Opus and edit videos super well. 5. Trybe Management - Runneth is trained on all of the Trybe feedback I have given. It automatically analyzes for safe zones, compliance, and general performance and drafts feedback.

What are your favorite use cases /w Runneth atm?

Dan Shipper avatar
⟳ Mike Taylor retweeted
Dan Shipper@danshipper·15h

if your team isn't generating videos like this instead of decks to explain new product direction...ngmi

I’m leading a sprint to reboot @every’s Checks product, and I put together a Christopher Alexander inspired project pattern language based on my interviews with the team. Then I made this explainer video to summarize the findings. What a time to be alive

4:40
Dennis Hong avatar
⟳ Mike Taylor retweeted
Dennis Hong@dennisihong·19h

"Do not own stuff that gets worse as intelligence gets better."

Post attachment
Becky Isjwara avatar
⟳ Mike Taylor retweeted
Becky Isjwara@beckyisj·Nov 17

so i'm a level 8 for AI-nativeness, but @hammer_mt has recently introduced 2 new levels that I need to aspire towards. thankfully @every is giving me concrete steps on how to level up

Post attachment