←
Gordon Murray13,766 posts
Gordon Murray avatar

Gordon Murray

@gortron

Creator of Firn: vector + full-text search on object storage. Platform & data infra engineer. AWS, Terraform, Flink. AWS Solutions Architect.

◷ Joined March 2007▧ 627 Photos and videos
Gordon Murray avatar
Gordon Murray@gortron·Jun 12

Firn has a Python package now! Just `pip install firn` Vector and full-text search on object storage, straight from Python. The engine embeds in-process, so there's no server: point it at a local folder or an S3 bucket. Imagine how happy your SysOps team will be with no cluster to manage. Or your CFO, with no managed solution to approve

Ovais Tariq avatar
⟳ Gordon Murray retweeted
Ovais Tariq@ovaistariq·10h

"S3-compatible" should be measurable. So we wrote an open-source test suite and ran it against Tigris, R2 and GCS. Out of 94 tests, Tigris passed 83, R2 53 and GCS 48.

Gordon Murray avatar
Gordon Murray@gortron·Nov 15

I ran into this with a Paperclip experiment I ran: a concurrent-scan test could miss incorrect output. Independent review caught it tough along with claims the evidence didn’t support. The tests themselves needed checking gordonmurray.ie/data/2026/10…

AWS Emulator is a cool project, but notice the language in the article: Agent reviewers are called gates, agent-written tests are called guardrails. Degrading the meaning of the terms to the equivalent of "make no mistakes".

Gordon Murray avatar
Gordon Murray@gortron·Nov 15

Clear read-only checks are useful. I built Out of State for a related question: which AWS resources have no match in the Terraform states I supply? It distinguishes missing matches from incomplete scans. It doesn’t check configuration drift. github.com/gordonmurray/outo…

Terraform needs a dedicated drift detection command. And there's an interesting debate happening around this. Today, teams can use: terraform plan -refresh-only -detailed-exitcode But should infrastructure drift detection have its own command? Imagine having: terraform drift A dedicated command that: → Checks infrastructure for out-of-band changes. → Is guaranteed to be read-only. → Returns clear drift-specific exit codes. → Can safely run in scheduled CI/CD checks. There's an open feature request for this in HashiCorp's Terraform repository. A Terraform maintainer has raised an interesting counterpoint: The existing plan functionality already detects drift. Do we really need another command? Personally, I like the idea of explicit read-only operations in infrastructure automation. Less ambiguity. Fewer opportunities for accidental changes. But I want to hear from people running Terraform at scale. Would you prefer: A) A dedicated terraform drift command B) Existing terraform plan + refresh-only + JSON parsing Which approach would you trust more in production?

Fahad Shah avatar
⟳ Gordon Murray retweeted
Fahad Shah@FAHADSH97656634·Nov 11

Object-storage-native is the future of search infra! @turbopuffer, @trychroma, @pinecone, @lancedb, @milvusio, @apurva1618, @jeffreyhuber, @Sirupsen

Elasticsearch is dead, long live Elasticsearch! Object-storage-native is the future of search infrastructure, and that's why it has become the foundation of modern search projects like turbopuffer, LanceDB, Chroma, Milvus, Pinecone, and OpenData Vector. By object-storage-native, we mean object storage as the primary storage layer, not merely as a backup or cold-storage destination. But object storage doesn't have to be part of the direct execution path for every operation. In this case, the hot path is accelerated by: - Memory and local SSD/NVMe caches - Metadata caching - Asynchronous prefetching - Intelligent indexing and data locality That's how you get both: - Cloud-scale economics - High performance At RisingWave, we've been investing in native search capabilities for quite some time, including full-text and vector search over streaming data. We're building these capabilities for AI agents that require high-concurrency, low-latency queries, all powered by an object-storage-native architecture. Stay tuned for more exciting updates! The future is: - S3-native - Cache-native - Disaggregated - Intelligent data locality and movement Object storage is the durable storage layer. Caching is the performance layer. That's the architectural shift happening across the data infra space.

Post attachment
SeaweedFS avatar
⟳ Gordon Murray retweeted
SeaweedFS@SeaweedFS·Nov 9

🚀 SeaweedFS supports Amazon S3 Vectors API! • AWS CLI & boto3 • Vector similarity + metadata filters • Lance storage + LanceDB interoperability S3 objects, tables, and vectors in one storage system! github.com/seaweedfs/seaweed… cc @lancedb @changhiskhan @eddyxu @xuanwo

Buswe avatar
⟳ Gordon Murray retweeted
Buswe@buswe_com·Nov 8

Pairs nicely with import blocks: feed that list into terraform plan -generate-config-out and the HCL writes itself.

Gordon Murray avatar
Gordon Murray@gortron·Nov 8

For anyone wanting to try the primary-key and snapshot behaviour, my Flink/Paimon/MinIO demo includes SQL examples for updating an existing key and reading an earlier snapshot. A small local setup to explore the mechanics. github.com/gordonmurray/apac…

mosespawar.blog/blog/the-pro… The problem of continuously updated lakehouse tables. Originally published: 4 November 2026 a) Lakehouse formats like Iceberg work well for analytical workloads where data is mostly appended and queries read large amounts of historical data. But there are so many usecases such as customer profiles and inventory and order status which receive millions of updates and deletes every day and at this rate Iceberg has to rewrite many data files or accumulate many delete files which is super expensive. Apache Paimon is a table format which tries to solve this problem. b) Lets first look at LSM trees or Log Structured Merge Trees as they are the foundation of Paimon. Instead of modifying records in place an LSM tree buffers new records in memory and writes them into new sorted files so multiple versions of the same record can exist in different files. A reader can return the latest version or read previous available versions. A background compaction periodically merges the files and removes obsolete versions. c) Paimon brings this LSM architecture to lakehouse. A table is divided into partitions and buckets and each record within a partition is assigned to a bucket by hashing its primary key. All versions of same record hence land in same bucket. Each bucket is its own LSM tree with files organized into levels and this allows frequent updates without rewriting the entire dataset. d) Paimon provides Iceberg-like snapshots and manifests. Streaming engines like Flink continuously writes updates and commits a new snapshot at each checkpoint and analytical engines can query a consistent snapshot while writes continue. Paimon also produces a changelog so downstream streaming jobs can process incremental updates rather than repeatedly scanning the entire table. e) Paimon supports three table modes that help choose between write throughput and read performance. Merge on Read is the default and keeps multiple versions in files and merges them during reads. Merge on Write uses deletion vectors to mark obsolete records so reads can skip them without merging. Copy on Write runs a full compaction on every commit so reads see fully merged files. f) There are a few important tradeoffs. Paimon makes frequent updates cheaper but it moves that cost into background compaction which runs continuously and needs tuning to keep reads fast and data fresh. We also need to be careful about large bucket skew because a few oversized buckets can slow down both compaction and reads. Paimon also has a smaller ecosystem of query engines than Iceberg. #DataEngineering #Lakehouse

Gordon Murray avatar
Gordon Murray@gortron·Nov 8

Asking for proof of work is important. In my Paperclip experiments, independent review caught a test that could miss incorrect output and claims the evidence didn’t support. The agents corrected both before sign-off. gordonmurray.ie/data/2026/10…

dax@thdxrNov 6

so far this is working great. i tried this in the past and it wasn't as great. some details - we have long running sessions per project - opus is told to do all work in deepseek subagents - be strict about scope when creating a task - ask for proof of work instead of a summary - do a one step review of the diff to make sure it's not totally off opus is amazing at this role

Gordon Murray avatar
Gordon Murray@gortron·Nov 6

What can it do? Obvious question IMHO and not in the FAQ

Asimov 1 is an open-source humanoid robot you can build, train, and customize. - 1.2 m tall, 35 kg - 25 powered joints - Open hardware with off-the-shelf parts - Open-source SDK and simulation models Available as a DIY kit. Order yours at menlo.ai/order

Post attachmentPost attachmentPost attachmentPost attachment
Gordon Murray avatar
Gordon Murray@gortron·Nov 3

Refreshed my data engineering skills for Claude Code with November 2026 releases, compatibility guidance, and migration notes. Also added a dedicated Apache Flink Agents skill covering streaming AI agents, durable execution, and recovery. github.com/gordonmurray/data…