
Our very own Vivek Garg spoke recently about building the data platform for an agent-native world, and the downsampling barriers to retaining agent event history which is increasingly important to overcome for regulatory needs. The talk covers an approach to building data foundations that can accept more data without losing control, including ingestion at 1.5M events/sec at peak into a single Apache Iceberg table with new data queryable in about 30 seconds, and these three takeaways: 1️⃣Write files scales with workers. Committing to the table does not, because every commit has to advance the same table state. 2️⃣Put one committer in front of the table. Workers keep writing Parquet in parallel, and the committer batches their files into one commit. 3️⃣Remove the commit path conflicts doesn’t make commits free. Each commit adds metadata, so pick an interval that meets your freshness target and keep snapshots and manifests trimmed. You can watch the full talk here: youtube.com/watch?v=dLRiNA0O… A big thank you to @streamnativeio for hosting!













