←
Al Brown204 posts
Al Brown avatar

Al Brown

@sdairs

Product @ ClickHouse Thinking about agentic experience & identity

◷ Joined April 2015▧ 33 Photos and videos
Al Brown avatar
Al Brown@sdairs·5h

5/5 Per-path metadata locks let unrelated tables commit concurrently. Whole-part removal can use batched deletes without temporary file copies. The common theme is reducing object-storage requests and the time spent coordinating them.

Al Brown avatar
Al Brown@sdairs·5h

4/5 The S3 metadata path makes fewer listing requests by listing key prefixes in parallel and collecting files across directories. One PR test cut metadata load from 116 to 37 seconds for about 36K directories and 500K objects.

Al Brown avatar
Al Brown@sdairs·5h

3/5 MergeTree ALTER operations and mutations check that the disk supports hard links, so this setting enables those paths. Changed columns still need new files, with the saving coming from sharing the unchanged ones.

Al Brown avatar
Al Brown@sdairs·5h

2/5 The metadata uses explicit file lists to track those references. The setting is off by default because servers that don't understand the new representation can't read a disk containing hard links. Turning it off later doesn't convert existing metadata back.

Al Brown avatar
Al Brown@sdairs·5h

1/5 ClickHouse 26.10 (@ClickhouseDB) can share unchanged column files between parts on plain_rewritable object-storage disks. With enable_hard_links = 1, a mutated part can reference existing blobs instead of copying them.

0:40
Al Brown avatar
Al Brown@sdairs·Nov 17

5/5 Text filters need String columns, math needs numeric operands, and numeric stats use Float64. Some schemaless operations have limits, including unpack_json without an explicit fields list. This isn't complete VictoriaLogs compatibility.

Al Brown avatar
Al Brown@sdairs·Nov 17

4/5 The parser translates the pipes into an ordinary SELECT for ClickHouse's analyzer and execution engine, while the source table keeps its storage layout. /dialect clickhouse switches the interactive client back.

Al Brown avatar
Al Brown@sdairs·Nov 17

3/5 Choose the table with logsql_table and map _time and _msg with logsql_time_column and logsql_message_column. Enable enable_logsql_dialect = 1, then switch with /dialect logsql in an interactive client or SET dialect in a query file.

Al Brown avatar
Al Brown@sdairs·Nov 17

2/5 _time:5m error | stats by (service) count() as errors | sort by (errors desc) | limit 10 finds recent error messages, counts them by service and returns ten results in descending count order.

Al Brown avatar
Al Brown@sdairs·Nov 17

1/5 ClickHouse 26.10 (@ClickhouseDB) adds compact log queries through the experimental LogsQL dialect. VictoriaLogs-style filters and pipes can find messages, calculate statistics, sort and limit results over an existing ClickHouse log table.

1:05
Al Brown avatar
Al Brown@sdairs·Nov 17

carbon is excellent...when it works. google finally managed to release a model I want to use, but fumbled serving it so hard that its just impractical

It's essentially malpractice to ignore Gemini 4 Carbon at this point, especially if you're doing ML engineering stuff. Carbon does things Fable/Astra cannot do.

Al Brown avatar
Al Brown@sdairs·Nov 17

mostly agree, but I'd say the harness is the UI. My harness can generate a UI if I need/want one, I don't see why I'd need that pushed into the product. I can imagine future products being almost entirely headless.

UI is dead. The new UI is MCP. Nobody wants to spend time in your product

Al Brown avatar
Al Brown@sdairs·Nov 16

6/6 The completion-rate figures on the slide are for workload-level memory spilling, a separate feature from the scheduling policies discussed here.

Al Brown avatar
Al Brown@sdairs·Nov 16

5/6 These policies order queries inside one workload, while the workload hierarchy shares resources between workloads. The default stays 'fifo'; choose 'las' explicitly. CPU accounting also needs CPU slot preemption.

Al Brown avatar
Al Brown@sdairs·Nov 16

4/6 With fair scheduling, a long query's weight can fall after a CPU-time, IO or wall-clock threshold. weight_lowering_cpu_seconds = 10 with weight_lowering_factor = 0.1 reduces its weight after ten CPU-seconds, which need not take ten wall-clock seconds.