Firn has a Python package now!
Just `pip install firn`
Vector and full-text search on object storage, straight from Python. The engine embeds in-process, so there's no server: point it at a local folder or an S3 bucket.
Imagine how happy your SysOps team will be with no cluster to manage. Or your CFO, with no managed solution to approve
"S3-compatible" should be measurable. So we wrote an open-source test suite and ran it against Tigris, R2 and GCS. Out of 94 tests, Tigris passed 83, R2 53 and GCS 48.
Every company needs a context graph. We built Omnigraph to be the foundation for it.
Open-source MIT and it lives in your own bucket.
github.com/ModernRelay/omnig…
I ran into this with a Paperclip experiment I ran: a concurrent-scan test could miss incorrect output.
Independent review caught it tough along with claims the evidence didn’t support.
The tests themselves needed checking
gordonmurray.ie/data/2026/10…
AWS Emulator is a cool project, but notice the language in the article: Agent reviewers are called gates, agent-written tests are called guardrails.
Degrading the meaning of the terms to the equivalent of "make no mistakes".
Clear read-only checks are useful. I built Out of State for a related question: which AWS resources have no match in the Terraform states I supply?
It distinguishes missing matches from incomplete scans. It doesn’t check configuration drift.
github.com/gordonmurray/outo…
Terraform needs a dedicated drift detection command.
And there's an interesting debate happening around this.
Today, teams can use:
terraform plan -refresh-only -detailed-exitcode
But should infrastructure drift detection have its own command?
Imagine having:
terraform drift
A dedicated command that:
→ Checks infrastructure for out-of-band changes.
→ Is guaranteed to be read-only.
→ Returns clear drift-specific exit codes.
→ Can safely run in scheduled CI/CD checks.
There's an open feature request for this in HashiCorp's Terraform repository.
A Terraform maintainer has raised an interesting counterpoint:
The existing plan functionality already detects drift. Do we really need another command?
Personally, I like the idea of explicit read-only operations in infrastructure automation.
Less ambiguity. Fewer opportunities for accidental changes.
But I want to hear from people running Terraform at scale.
Would you prefer:
A) A dedicated terraform drift command
B) Existing terraform plan + refresh-only + JSON parsing
Which approach would you trust more in production?
Elasticsearch is dead, long live Elasticsearch!
Object-storage-native is the future of search infrastructure, and that's why it has become the foundation of modern search projects like turbopuffer, LanceDB, Chroma, Milvus, Pinecone, and OpenData Vector.
By object-storage-native, we mean object storage as the primary storage layer, not merely as a backup or cold-storage destination.
But object storage doesn't have to be part of the direct execution path for every operation.
In this case, the hot path is accelerated by:
- Memory and local SSD/NVMe caches
- Metadata caching
- Asynchronous prefetching
- Intelligent indexing and data locality
That's how you get both:
- Cloud-scale economics
- High performance
At RisingWave, we've been investing in native search capabilities for quite some time, including full-text and vector search over streaming data.
We're building these capabilities for AI agents that require high-concurrency, low-latency queries, all powered by an object-storage-native architecture.
Stay tuned for more exciting updates!
The future is:
- S3-native
- Cache-native
- Disaggregated
- Intelligent data locality and movement
Object storage is the durable storage layer.
Caching is the performance layer.
That's the architectural shift happening across the data infra space.
I made a thing this weekend: outofstate.dev
It lists the AWS resources that aren't in your Terraform state yet. RDS, load balancers, S3, Lambda and NAT gateways so far.
An agent could do this too, but this way you don't burn a pile of tokens on it. And it isn't Rust, its Go!
For anyone wanting to try the primary-key and snapshot behaviour, my Flink/Paimon/MinIO demo includes SQL examples for updating an existing key and reading an earlier snapshot.
A small local setup to explore the mechanics.
github.com/gordonmurray/apac…
mosespawar.blog/blog/the-pro…
The problem of continuously updated lakehouse tables.
Originally published: 4 November 2026
a) Lakehouse formats like Iceberg work well for analytical workloads where data is mostly appended and queries read large amounts of historical data. But there are so many usecases such as customer profiles and inventory and order status which receive millions of updates and deletes every day and at this rate Iceberg has to rewrite many data files or accumulate many delete files which is super expensive. Apache Paimon is a table format which tries to solve this problem.
b) Lets first look at LSM trees or Log Structured Merge Trees as they are the foundation of Paimon. Instead of modifying records in place an LSM tree buffers new records in memory and writes them into new sorted files so multiple versions of the same record can exist in different files. A reader can return the latest version or read previous available versions.
A background compaction periodically merges the files and removes obsolete versions.
c) Paimon brings this LSM architecture to lakehouse. A table is divided into partitions and buckets and each record within a partition is assigned to a bucket by hashing its primary key. All versions of same record hence land in same bucket.
Each bucket is its own LSM tree with files organized into levels and this allows frequent updates without rewriting the entire dataset.
d) Paimon provides Iceberg-like snapshots and manifests. Streaming engines like Flink continuously writes updates and commits a new snapshot at each checkpoint and analytical engines can query a consistent snapshot while writes continue. Paimon also produces a changelog so downstream streaming jobs can process incremental updates rather than repeatedly scanning the entire table.
e) Paimon supports three table modes that help choose between write throughput and read performance. Merge on Read is the default and keeps multiple versions in files and merges them during reads. Merge on Write uses deletion vectors to mark obsolete records so reads can skip them without merging. Copy on Write runs a full compaction on every commit so reads see fully merged files.
f) There are a few important tradeoffs. Paimon makes frequent updates cheaper but it moves that cost into background compaction which runs continuously and needs tuning to keep reads fast and data fresh. We also need to be careful about large bucket skew because a few oversized buckets can slow down both compaction and reads. Paimon also has a smaller ecosystem of query engines than Iceberg.
#DataEngineering#Lakehouse
Asking for proof of work is important. In my Paperclip experiments, independent review caught a test that could miss incorrect output and claims the evidence didn’t support. The agents corrected both before sign-off.
gordonmurray.ie/data/2026/10…
so far this is working great. i tried this in the past and it wasn't as great. some details
- we have long running sessions per project
- opus is told to do all work in deepseek subagents
- be strict about scope when creating a task
- ask for proof of work instead of a summary
- do a one step review of the diff to make sure it's not totally off
opus is amazing at this role
Asimov 1 is an open-source humanoid robot you can build, train, and customize.
- 1.2 m tall, 35 kg
- 25 powered joints
- Open hardware with off-the-shelf parts
- Open-source SDK and simulation models
Available as a DIY kit.
Order yours at menlo.ai/order
Refreshed my data engineering skills for Claude Code with November 2026 releases, compatibility guidance, and migration notes.
Also added a dedicated Apache Flink Agents skill covering streaming AI agents, durable execution, and recovery.
github.com/gordonmurray/data…