←
Apache Hudi1,091 posts
Apache Hudi avatar

Apache Hudi

@apachehudi

Official twitter handle of Apache Hudi, an open data lakehouse platform. s.apache.org/hudi-faq

◷ Joined January 2019▧ 311 Photos and videos
Apache Hudi avatar
Apache Hudi@apachehudi·17 Dec 2024

Hudi 1.0 is the most powerful release to date for data lakehouses. Read the blog for details: Secondary Indexing, Expression Indexes, Partial Updates, Non-blocking Concurrency Control, New LSM timeline, +more: hudi.apache.org/blog/2024/12… #datalakehouse #opentableformat

Apache Hudi avatar
Apache Hudi@apachehudi·Nov 12

For capital-markets workloads — real-time reconciliation, audit-grade history, late-arriving CDC, reporting that can't miss a beat — Hudi's primitives fit the operational profile trading platforms demand. Links ↓ #ApacheHudi #FinTech

Apache Hudi avatar
Apache Hudi@apachehudi·Nov 12

At OSS Data Summit 2023, Balaji Varadarajan and Pritam Dey walked through how Robinhood's lean data team scaled this into a multi-petabyte lakehouse — exponential data growth without proportional team growth.

Apache Hudi avatar
Apache Hudi@apachehudi·Nov 12

Why CoW for the trading platform: read performance for analytics + regulatory queries. Update-write cost was the price they paid for query speed downstream. Incremental changelog ingestion — never re-read full tables.

Apache Hudi avatar
Apache Hudi@apachehudi·Nov 12

Robinhood took data lake freshness from 24 hours to 5-15 minutes by switching to Apache Hudi. Kafka → Hudi DeltaStreamer → S3 (CoW). Exactly-once via Hudi commits ↓

Post attachment
Apache Hudi avatar
Apache Hudi@apachehudi·Nov 11

Why this fits regulated FS: Hudi open table format keeps data-asset storage optimized and interoperable across services. Glue Catalog makes assets discoverable across domains. Event-driven refresh keeps products current. Federated governance via Lake Formation.

Apache Hudi avatar
Apache Hudi@apachehudi·Nov 11

AWS Glue as workflow + catalog backbone. Hudi as the open table format for data assets on S3. Event-driven pipelines so data products refresh on change, not fixed schedules. Each domain owns its data flow.

Apache Hudi avatar
Apache Hudi@apachehudi·Nov 11

Stifel Financial Corp. modernized their data platform on AWS with Apache Hudi, Glue, and an event-driven, domain-first architecture. Data-mesh-meets-Hudi for regulated wealth management ↓

Post attachment
Apache Hudi avatar
Apache Hudi@apachehudi·Nov 5

INSERT_OVERWRITE / DELETE_PARTITION — atomic batch ops. Old file groups marked REPLACE_COMMIT, new ones written. Lazy cleanup. Each maps to a specific action type on the timeline. Links ↓ #ApacheHudi #DataEngineering

Apache Hudi avatar
Apache Hudi@apachehudi·Nov 5

BULK_INSERT — sort-based, scales to hundreds of TB initial loads. Skips in-memory heuristics. Best-effort file sizing — pair with clustering to consolidate. DELETE — soft (keep key, null fields) or hard (remove all evidence).

Apache Hudi avatar
Apache Hudi@apachehudi·Nov 5

UPSERT — default. Index-tags records as insert/update. File-sizing on. Never duplicates. Best for CDC. INSERT — same heuristics, skips index lookup. Faster. Best for log de-dup.

Apache Hudi avatar
Apache Hudi@apachehudi·Nov 5

Pick the right Hudi write operation. Each one trades index lookup against file sizing against initial-load speed.

Post attachment