←
hatchet316 posts
hatchet avatar

hatchet

@hatchet_dev

the orchestration engine for everything async. billions of tasks run, from the fortune 100 to linear, greptile, enter + cursor.

◷ Joined February 2024▧ 105 Photos and videos
hatchet avatar
hatchet@hatchet_dev·Nov 9

introducing hatchet v2 durable execution, durable queues, durable streams, sandbox integrations, serverless support, and operators also we raised $5.7M

despite coding agents, modern apps are fragile. without proper architecture, you can spit out as much code as you’d like, but you’re just adding levels to a house of cards. introducing Hatchet v2 & our $5.7M seed round. our bet is that great orchestration is the difference between a reliable app and a broken one. Hatchet v2 is the universal orchestration layer: durable execution, durable queues and durable streams, sandbox integrations, serverless support, and operators. this is the architecture used by high-growth companies like @greptile, @linear, @getenter_ai, and many others, who collectively process over 5 billion tasks/month on Hatchet. we’ve spent over two years obsessing over making our engine performant and scalable. this release extends the existing @hatchet_dev engine to support any sandbox provider, serverless environment, edge function, or container.

1:34
Megan O'Leary avatar
⟳ hatchet retweeted
Megan O'Leary@meganeoleary·21h

ok so i have toast, how do i make it super help @PostgreSQL

Post attachment

modern data layers are bifurcating into either really fast, local NVMes or really slow, massively scalable object storage. or both. it's pretty obvious when NVMe is useful (incredibly fast disk i/o). and it's also obvious when S3 is useful. but systems that combine the two are the most interesting, because you're taking a big bet on data access patterns. for example, the reason this works so well for us is that if our users are accessing a payload that was written in the past 24 hours, you are fairly likely to be latency sensitive (obviously not true for everyone, but enough of our users). so you'd want single-digit millisecond queue times. but if you're accessing data that was written over 24 hours ago (for example, resuming a workflow after 24 hours of inactivity), you are not latency sensitive to a ~100ms queue time. every data stack which uses NVMe + S3 needs a similar argument for their data access, or it doesn't work. so while disaggregated storage + compute OLTP databases are really cool, I've never worked on a stack where I've wanted to use them in production. once you're sure of the access pattern, matt's two blog posts on how this actually gets built are definitely worth a read!

hatchet avatar
hatchet@hatchet_dev·23h

new blog post: how we built index files on S3 to offload data from our NVMe disks on Hatchet Cloud. everything is S3 now. Postgres, Git, Kafka, and even POSIX filesystems are being reinvented with S3 as a foundational layer. earlier this year, we hopped on the bandwagon and decided to offload a large part of the Hatchet payload data to S3 to save us money on disk. we affectionally called the resulting table Supertoast. but our initial implementation didn't go far enough. and thus, "Supertoast, Redux". hatchet.run/blog/supertoast-…

hatchet avatar
hatchet@hatchet_dev·Nov 16

durable tasks meet durable filesystems. that's twice the durability. hatchet 🤝 e2b

E2B@e2bNov 16

Two new integrations landed in the E2B Cookbook this week: Hatchet and Anchor Browser. 1. Hatchet Use @hatchet_dev with E2B to run background jobs and agents reliably, with every step executing in its own isolated sandbox. Long-running agent jobs rarely fail cleanly. A model call times out, a worker restarts, a step crashes halfway through. Hatchet handles scheduling, retries, and recovery, so a single failure is retried instead of bringing down the entire run. Hatchet decides when each step runs and how to recover when it fails. E2B gives each step a secure, isolated machine for agent-generated code. Steps run in parallel without interfering with one another, and a crash inside a sandbox stays inside it. The result is agent pipelines that are both reliable and safe. Check out the cookbook example- a Hatchet durable task runs Python in an E2B sandbox, kills it, and reconnects to find its files intact: github.com/e2b-dev/e2b-cookb… 2. Anchor Browser Use Anchor Browser with E2B to give your agents a cloud browser that can navigate websites, sign in, and complete tasks across web apps, while the sandbox stays lightweight. Much of the most valuable work on the web sits behind a login, and many sites block bots outright. @AnchorBrowser gives agents safe, managed access, with built-in authentication, stealth, and proxies, so they can reach the apps they need. Paired with E2B, your agent has both halves of a computer. The sandbox is where it installs packages, runs scripts and calls models. Anchor is where it browses the web like a person, signed in to real apps. In a single run, an agent can pull data from a web app and process it in code. Because the browser runs as a managed Anchor session instead of the sandbox, sandboxes start faster and you can watch every session live. Get started with the cookbook example: github.com/e2b-dev/e2b-cookb…

Post attachment
hatchet avatar
hatchet@hatchet_dev·Nov 16

big news from our friends at @tryrevyl (revyl runs on hatchet)

hayden@hxydenNov 16

Cloud simulators should be cheap enough that anyone can use them! That's why Revyl gives every developer 1,200 free cloud simulator minutes a month. What would you build with a free fleet of simulators?

0:17
Jean-Baptiste Souvestre avatar
⟳ hatchet retweeted
Jean-Baptiste Souvestre@jbsouvestre·Nov 16

this is not easy to convey in a short video but everything showcased here is replayable and survives a worker crash, thanks to @hatchet_dev sandboxes and durable streams every agent turn and SQL call is a durable task; kill the worker mid-race and it resumes from the event log without re-calling Claude or re-running the query

we built a Postgres query optimizer with @hatchet_dev sandboxes and durable streams. I spend a lot of time optimizing queries. even the best coding agents are actually quite bad at this. they'll add an index or modify the query structure and more often than not won't pick the best approach (i.e. every index comes with write overhead, which often isn't considered). this is partially because it's very non-obvious how to squeeze performance out of a query without the right environment. I've been on teams that have cloned (anonymized) production datasets just because query issues are often impossible to reproduce locally. so how do you get around this problem? while query writing is meh at best, coding agents are really good at two things: (1) parallelization and (2) iterating towards a verifiable goal with the right environment. and thus our Postgres query optimizer was born. it runs a full postgres instance in a sandbox and the agents race to build the most performant query given a specific problem. nice work @jbsouvestre!

0:16
hatchet avatar
hatchet@hatchet_dev·Nov 16

🪓 + 🐘 = 📈

we built a Postgres query optimizer with @hatchet_dev sandboxes and durable streams. I spend a lot of time optimizing queries. even the best coding agents are actually quite bad at this. they'll add an index or modify the query structure and more often than not won't pick the best approach (i.e. every index comes with write overhead, which often isn't considered). this is partially because it's very non-obvious how to squeeze performance out of a query without the right environment. I've been on teams that have cloned (anonymized) production datasets just because query issues are often impossible to reproduce locally. so how do you get around this problem? while query writing is meh at best, coding agents are really good at two things: (1) parallelization and (2) iterating towards a verifiable goal with the right environment. and thus our Postgres query optimizer was born. it runs a full postgres instance in a sandbox and the agents race to build the most performant query given a specific problem. nice work @jbsouvestre!

0:16
hatchet avatar
hatchet@hatchet_dev·Nov 13

claude is down. microsoft is down. it's basically a snow day, so pour yourself a nice cuppa and read our latest blog post ⬇️

so I pissed off temporal's ceo a few months ago by making the argument that old-school durable execution is much less important than it used to be, or more specifically, is very architecturally lacking. this is really a conversation we've been having internally for the past 6 months, because we've been trying to reconcile how to model the two different aspects of durable state that agents need: filesystem state and session state. these are distinct and if you're building an agentic application I think you need to understand the difference (which I talk about in the article). it turns out that the biggest innovations in filesystem durability are coming out of the sandbox providers like @archildata @e2b @modal @freestyle_dev , to name a few. it's easy to have sandbox-provider-fatigue if you view them as compute resellers, but if you cut through the noise, some of them are doing super interesting work on the filesystem durability layer (and the isolation layer of course). I go into more detail in the article, but there's a correct way to use these providers in conjunction with durable execution. that's why we released the @hatchet_dev sandbox package earlier this week and why we're integrating with really great sandbox providers. full article: hatchet.run/blog/agent-state…

Megan O'Leary avatar
⟳ hatchet retweeted
Megan O'Leary@meganeoleary·Nov 12

i'm using ray.hatchet.run to stage future sunset photoshoots of my cat. stellar use case shoutout @hatchet_dev and @CloudflareDev

0:51

my CTO gave me 350k of infra credits and I spent them all rendering mushroom clouds I made a ray tracing app in Rust that lets you prompt SDFs to render realistic images (using @hatchet_dev durable streams and @cloudflare workers). Try it out at ray.hatchet.run before we run out of money

2:02
hatchet avatar
hatchet@hatchet_dev·Nov 12

we support our engineers' special interests: rust, ray-tracing, and infra spendmaxxing

my CTO gave me 350k of infra credits and I spent them all rendering mushroom clouds I made a ray tracing app in Rust that lets you prompt SDFs to render realistic images (using @hatchet_dev durable streams and @cloudflare workers). Try it out at ray.hatchet.run before we run out of money

2:02
Megan O'Leary avatar
⟳ hatchet retweeted
Megan O'Leary@meganeoleary·Nov 10

i built a live news network for my cat (rory) @hackernews -> @hatchet_dev durable stream -> browser reporting cat

0:33

we built durable streams on postgres. through our analysis, we explore how heraclitus (greek philosopher) was wrong about two things: 1. durable streams 2. that bathing in cow dung and lying in the sun was an appropriate remedy for fluid retention hatchet.run/blog/durable-str…

hatchet avatar
hatchet@hatchet_dev·Nov 10

hatchet 🤝🤝 ubicloud

@hatchet_dev v2 runs on PostgreSQL! Some numbers: * Peak at 237K TPS * Writes peak at 377L rows/s * Stores 33TB of "hot" data in @UbicloudHQ Everything runs faster compared to AWS Postgres thanks to local NVMe. Also saves $500K in annual costs. Excited about Hatchet's v2 & seed round! Congrats to @abelanger5 and the team on the launch.

Ben avatar
⟳ hatchet retweeted
Ben@bensenescu·Nov 9

the engineering behind @hatchet_dev is wild

despite coding agents, modern apps are fragile. without proper architecture, you can spit out as much code as you’d like, but you’re just adding levels to a house of cards. introducing Hatchet v2 & our $5.7M seed round. our bet is that great orchestration is the difference between a reliable app and a broken one. Hatchet v2 is the universal orchestration layer: durable execution, durable queues and durable streams, sandbox integrations, serverless support, and operators. this is the architecture used by high-growth companies like @greptile, @linear, @getenter_ai, and many others, who collectively process over 5 billion tasks/month on Hatchet. we’ve spent over two years obsessing over making our engine performant and scalable. this release extends the existing @hatchet_dev engine to support any sandbox provider, serverless environment, edge function, or container.

1:34
Landseer Enga avatar
⟳ hatchet retweeted
Landseer Enga@LandseerEnga·Nov 9

We use hatchet for everything and I cannot recommend it enough. Amazing product and amazing team. Hatchet 2.0 🚀

despite coding agents, modern apps are fragile. without proper architecture, you can spit out as much code as you’d like, but you’re just adding levels to a house of cards. introducing Hatchet v2 & our $5.7M seed round. our bet is that great orchestration is the difference between a reliable app and a broken one. Hatchet v2 is the universal orchestration layer: durable execution, durable queues and durable streams, sandbox integrations, serverless support, and operators. this is the architecture used by high-growth companies like @greptile, @linear, @getenter_ai, and many others, who collectively process over 5 billion tasks/month on Hatchet. we’ve spent over two years obsessing over making our engine performant and scalable. this release extends the existing @hatchet_dev engine to support any sandbox provider, serverless environment, edge function, or container.

1:34
Sean Callahan avatar
⟳ hatchet retweeted
Sean Callahan@ScalaHanSolo·Nov 9

Over the last few months, I've been migrating much of Linear's workflow related work to Hatchet, and I could not be happier with the whole package.

despite coding agents, modern apps are fragile. without proper architecture, you can spit out as much code as you’d like, but you’re just adding levels to a house of cards. introducing Hatchet v2 & our $5.7M seed round. our bet is that great orchestration is the difference between a reliable app and a broken one. Hatchet v2 is the universal orchestration layer: durable execution, durable queues and durable streams, sandbox integrations, serverless support, and operators. this is the architecture used by high-growth companies like @greptile, @linear, @getenter_ai, and many others, who collectively process over 5 billion tasks/month on Hatchet. we’ve spent over two years obsessing over making our engine performant and scalable. this release extends the existing @hatchet_dev engine to support any sandbox provider, serverless environment, edge function, or container.

1:34