←
Dscout6,142 posts
Dscout avatar

Dscout

@dscout

Dscout is an AI-powered human signal platform that helps product and model builders collect and act on hi-fidelity feedback, fast.

◷ Joined December 2011▧ 492 Photos and videos
Dscout avatar
Dscout@dscout·Oct 21

We need humans, or human proxies, in reinforcement learning and evaluation loops now more than ever. Agents keep topping benchmarks, then stumbling in the real world. No surprise -- for one, there’s a level of scientific rigor needed to ensure benchmarks don’t test models with tasks they’ve seen in training data. For another, tasks and rubrics included in benchmarks are usually much simpler than what real users hold as good. Finally, tasks and answer keys are often wrong (about 6.5% of MMLU questions have errors, Terminal Bench was recently found to contain errors). How many legal documents can be reviewed without back-and-forth with a client? How many tax returns can be prepared without input from specialists, departments, and regulatory websites? To handle real-world workflows, agents need to learn the real-world messiness during training. That often means integrating humans into training and evaluation environments. But people don’t scale, so teams turn to LLMs to play the user. New research from Meta shows why that’s a problem. The MIMESIS paper measured how often the same GPT-5.5 agent succeeds: With real users: 63.6% success. With frontier-model playing real users: 82.4–84.4%. Too tidy, helpful, and nice. With pretrained Qwen3.5-4B/9B playing real users: Success drops by 16.0 and 14.9 points. Too obstructive. The authors trained simulators on how humans actually interact, closing the gap. Every trained simulator landed within 10.3 points of real users; every off-the-shelf model missed by at least 14.9. The best, MIMESIS-4B and 9B, came within 2.4 points. The key takeaway: trained simulators captured how people actually behave, their directness, emotions, and context, while exposing agents to more diverse interaction paths. The team then added a coaching loop, using the simulator’s private reasoning and next reaction as feedback for the agent. Agents trained this way outperformed those trained against GPT-5.5 across all nine unseen user simulators. Kudos to Hoang Phan and the MIMESIS team at @Meta. This is rigorous, practical work for the industry Our read: The next leap in agents comes from understanding how people really behave. They’re vague, impatient, and change their minds mid-sentence. That’s where we are spending time at Dscout, and we’re happy to help anyone experimenting here. What does your “user” look like when you test your agents? #AIAgents #HumanAI #UserResearch

Dscout avatar
Dscout@dscout·Oct 10

Perplexity thread:

New research: We post-trained a Computer model to learn from its own errors using hint-guided self-distillation. In a live A/B test, a later trained checkpoint reduced tool-call failures by 21.2% relative to an earlier checkpoint.

Post attachment
Dscout avatar
Dscout@dscout·Oct 10

A big shoutout to the researchers/builders/founders that joined our #SFTechWeek dinner to discuss how the next wave of RL & post-training will be driven by real users, in real context, in the loop. @siyan_zhao's work on on-policy self-distillation showed how agents can learn from their own failed trajectories. Now that idea is showing up in production. @perplexity_ai shared a great example last month: use real user sessions to see where an agent went wrong, turn that into better feedback, and train the system not to repeat the same mistake. In the same vein, where the agent was in fact successful, use those trajectories as demonstration data that gets imitated. As agents take on longer, messier, real-world work, the real-user feedback loop helps hone in on "Did the model get the answer right?", "Where did the agent get stuck?", "What did the user correct?" and "Which steps worked? Which failed?" That's exactly what we're helping unlock at @dscout. Thanks all -- to more such riffs.

Post attachmentPost attachmentPost attachmentPost attachment
Dscout avatar
Dscout@dscout·26 Jun 2024

Looking forward to making new friends at #config2024!

Figma@figma26 Jun 2024

Thanks to our sponsors for building #Config2024 with us. Stop by the Maker Space to see their booths starting tomorrow

0:10
Dscout avatar
Dscout@dscout·13 Sep 2023

Validating makes it seem like you're already correct. It also signals a one-and-done approach to testing. Was the design validated or not? Instead, try using words will help you (and stakeholders) shift into a mindset of curiosity and open-mindedness. ⤵️ #UXResearch #UX

Post attachment
Dscout avatar
Dscout@dscout·6 Sep 2023

If you want to take your #usability testing up a notch, spend: ➖ less time on manual tasks ➕ more time gathering and discussing insights Our newest feature will help you do all of that and more. Learn more about the "more" here: bit.ly/44GRUk7 #UX

1:23
Dscout avatar
Dscout@dscout·23 Aug 2023

Feeling analysis paralysis around the glut of tools, solutions, and vendors available for research? Use this list to narrow on the offering that best serves your organization's ongoing needs: bit.ly/44op48f #UX #UserResearch

Dscout avatar
Dscout@dscout·16 Aug 2023

Hey there👋 We're not frequent Tweeters (Xers?). But if you need: • The latest from our People Nerds community➡️join us on LinkedIn • Support➡️email help@dscout.com • Anything else➡️email info@dscout.com Don't be a stranger.

Dscout avatar
Dscout@dscout·10 Aug 2023

This checklist has it all: know what to do before, during, and after a usability testing session—including a template on hand to make everything easier 🗒️ bit.ly/3KBGk2B #UXResearch

Dscout avatar
Dscout@dscout·7 Aug 2023

Once you've delivered your insights, don't just let them stagnate. Using ideation and co-creation can jumpstart them into meaningful action: bit.ly/4793jLX #UXR