←
Anurag Jha14,789 posts
Anurag Jha avatar

Anurag Jha

@anuragdotdev

19 | Code • Build • Ship | DM for work/Collab

◷ Joined November 2023▧ 1,923 Photos and videos
Anurag Jha avatar
Anurag Jha@anuragdotdev·Oct 17

Next topic: Validation and Transformation. Another article in the backend series: medium.com/@anuragdotdev/val… I wrote about what happens when data enters your backend and how to ensure your application handles it correctly. Input validation, data transformation, Zod, Express middleware, the 400 vs 500 problem, mass assignment, database constraints, testing, and more. I didn't want to write another article that just shows a Zod schema and calls it validation. I wanted to understand the whole thing properly. From untrusted client input to validated data reaching the service and database. It took me a while to put this together, but I wanted it to be something I'd actually want to read while learning backend development.

Post attachment
Anurag Jha avatar
Anurag Jha@anuragdotdev·45m

Hydrogen was better than neon in building site, tbh this is straight up slop

First test of Gemini 4 Neon does NOT look promising

0:14
Anurag Jha avatar
Anurag Jha@anuragdotdev·47m

Gemini 4 Neon is LIVE. Google just dropped a new Flash model built on Hydrogen, and the docs claim it comes close to Carbon at a fraction of the cost. Some interesting specs and capabilities: > 1M context window > Up to 1M output tokens with long decode > Reasoning up to XHigh > Computer use > Code execution > Function calling > Google Search & Maps grounding > Structured outputs > Video, audio, image and PDF input Pricing: > $3 input > $0.15 cached input > $15 output per 1M tokens The model is live on Gemini API and available to Pro and Ultra users in AI Studio. Running my tests on Neon right now. Will share the results as soon as they're done.

Post attachment

Breaking 🚨: Gemini Neon is released, showing on cloud Official announcement soon

Anurag Jha avatar
⟳ Anurag Jha retweeted
Anurag Jha@anuragdotdev·Nov 17

Google is naming models after elements from the periodic table. > Argon > Hydrogen > Carbon > Krypton OpenAI is going with space-inspired names. > Astra > Sol > Terra > Luna And Anthropic is basically naming its models after literature. > Mythos > Fable > Opus > Sonnet > Haiku Not gonna lie, all three have their own vibe Model naming is getting interesting 💀

Anurag Jha avatar
Anurag Jha@anuragdotdev·1h

Those 27 countries are : Australia, Austria, Belgium, Canada, Cyprus, Estonia, Finland, France, Germany, Greece, Hong Kong, India, Ireland, Italy, Lithuania, Luxembourg, Malaysia, Mexico, the Netherlands, Norway, the Philippines, Portugal, Singapore, Slovenia, Spain, Sweden, and the United Kingdom

Send money abroad, right from X. International wires (beta) are now live. Send to bank accounts in 27 countries from the Money tab.

0:20
Anurag Jha avatar
⟳ Anurag Jha retweeted
Anurag Jha@anuragdotdev·Nov 17

Evals are probably one of the most overlooked skills in AI engineering. > 1. Start by understanding failures, not chasing metrics Take 100+ real production traces and inspect them manually. Write down what went wrong in plain language, then group similar failures into categories. Count how often each category appears. Your three most common failure types should define your first evaluation metrics. > 2. Create a versioned golden dataset Maintain a dataset containing inputs, expected outputs or grading rubrics, and useful metadata such as intent, difficulty, and user type. Include rare but important scenarios, not just common and easy examples. Version it alongside your prompts so every evaluation result can be reproduced. > 3. Generate synthetic data with proper coverage Don't simply ask an LLM to generate random test questions. Define dimensions like persona, intent, complexity, and edge cases. Combine them into structured test scenarios, then convert those scenarios into realistic user inputs. This gives you better coverage and fewer repetitive examples. > 4. Start with deterministic checks Validate JSON schemas, required fields, regex patterns, length limits, code execution, and SQL queries against a test database. These checks are fast, cheap, and produce consistent results. Use LLM-based grading only for things traditional code cannot reliably evaluate. > 5. Evaluate retrieval and generation independently For RAG systems, measure retrieval using recall@k, MRR, or nDCG against labeled relevant chunks. Evaluate generation separately using faithfulness: whether every claim in the response is actually supported by the retrieved context. > 6. Evaluate the entire agent trajectory Record every agent step, including tool names, arguments, responses, and token usage. Measure tool selection accuracy, argument correctness, unnecessary steps, and error recovery. > 7. Understand pass@k and pass^k pass@k measures whether a task succeeds at least once across k attempts. pass^k measures whether it succeeds on every attempt. At 80% per-run success, the probability of succeeding in all 5 runs is only around 33%. > 8. Use simulated users for multi-turn evaluations Give an LLM a specific persona, hidden objective, and set of constraints. Let it interact with your system until the objective is achieved or a turn limit is reached. Evaluate task completion, number of turns required, and policy violations. > 9. Treat your LLM judge like a classifier Manually label at least 100 examples as pass or fail. Split them into training, development, and test sets. Improve your judge prompt using the development set, then measure true positive rate and true negative rate on unseen test data. > 10. Reduce bias in pairwise LLM judging LLM judges can favour the first answer, the second answer, or simply the longer response. Compare outputs twice with their positions reversed. Only trust verdicts that remain consistent. > 11. Don't ignore statistical significance Getting 80% accuracy on 100 examples gives you a 95% confidence interval of roughly ±8 percentage points. Moving from 80% to 83% doesn't automatically mean your system improved. > 12. Integrate evaluations into CI/CD Run your evaluation suite whenever a PR changes prompts, models, tools, or retrieval configurations. Define separate performance thresholds for different failure categories. Block deployments when critical thresholds are violated. > 13. Monitor evaluations in production Sample a portion of real user traffic and evaluate it asynchronously using automated judges. Monitor changes in individual failure categories and set alerts for unexpected regressions. > 14. Track cost and latency alongside quality Measure tokens consumed per request, cost per successfully completed task, and p50/p95 latency. If a new implementation maintains the same quality while reducing inference costs by 40%, that's a meaningful engineering improvement. > 15. Build a continuous feedback loop Every meaningful production failure should become a new labeled test case. Every new model release should run against your existing evaluation suite before you consider switching. Prompts will change. Models will get replaced. Frameworks will come and go. But a strong evaluation system keeps getting more valuable over time.

Anurag Jha avatar
Anurag Jha@anuragdotdev·2h

Hey @grok Build a premium, modern, responsive portfolio for me using HTML, CSS, and JavaScript. Use everything you know about my skills, projects, experience, and achievements. Take inspiration from Linear and Vercel. Make it minimal, aesthetic, and interactive. Deliver the complete, copy-paste-ready code in a single HTML file.

Anurag Jha avatar
Anurag Jha@anuragdotdev·2h

They built the hype for Neon and released Genie. Google just announced Genie 4. > Generate interactive worlds using text prompts > Modify environments in real time > Native audio and dynamic soundscapes > Explore longer, connected worlds > Gemini-powered agent for world creation > Share and remix worlds with the community We're moving from generating videos to generating worlds you can actually explore and interact with. Interesting direction for world models congrats to @OfficialLoganK and Team

Genie 4 is now available in Project Genie. It brings native audio, real-time editing, and a new agent to help you create interactive, virtual worlds.

1:21
Anurag Jha avatar
Anurag Jha@anuragdotdev·3h

Let's see if Gemini hydrogen is actually AGI or not........ You guys can play GTA 6 I am not interested anymore

Post attachment
Anurag Jha avatar
⟳ Anurag Jha retweeted
Anurag Jha@anuragdotdev·5h

This lecture is all you need to understand about transformers: >Background on NLP and tasks >Tokenization >Embeddings >Word to EVC >RNN >LSTM >Attention mechanism >Transformer architecture >End-to-end example Now, don't give yourself an excuse for not getting a perfect resource to start

Post attachment