←
Tom Nicholas501 posts
Tom Nicholas avatar

Tom Nicholas

@TEGNicholasCode

OSS for science, #python, @pangeo_data, @xarray_dev team. Ex-fusion, oceanography, now @_cworthy 🦋 bsky.app/profile/tegnicholas…

◷ Joined February 2020▧ 25 Photos and videos
Tom Nicholas avatar
Tom Nicholas@TEGNicholasCode·19 Dec 2024

At AGU I talked to NASA people about how agencies could better support open-source tools they rely on. I argued that our recent collaboration between Xarray and NASA ESDIS on xarray.DataTree was a good model to copy - read about how it happened here! xarray.dev/blog/datatree

Xarray avatar
⟳ Tom Nicholas retweeted
Xarray@xarray_dev·9 Jan 2025

Our friend's over at @zarr_dev made a big release today! Xarray v2025.01.1 was also released today with full support for Zarr-Python 3 🚀

🎉 Zarr-Python 3 is here! 🎉 - Full support for Zarr v3 spec - Chunk-sharding for more efficient data storage - Major performance boosts with async I/O & parallel compression 💻 pip install --upgrade zarr Blog post: zarr.dev/blog/zarr-python-3-…

Tom Nicholas avatar
Tom Nicholas@TEGNicholasCode·4 Dec 2024

Completely agree - "in theory" we have the simple scalability of the cloud, but in practice it's often a headache, for no good reason, which prevents adoption by most users (including many scientists)

New Post: Cloud Computing is Broken matthewrocklin.com/cloud-is-… Investor asks: "What's next for Data/Cloud Infrastructure?" My answer: "Boring stuff. People struggle with basics." Cloud feels like MP3 players before iPod. In theory everything is good. In practice adoption is low

Ian Schuler avatar
⟳ Tom Nicholas retweeted
Ian Schuler@ianschuler·18 Nov 2024

That said, it isn't 100% clear that NASA's best move is to immediately convert 10000+ data sets into cutting edge ARCO formats. Kerchunk and Virtual Zarr offer benefits of ARCO while keeping data in the native formats.

Joe Hamman avatar
⟳ Tom Nicholas retweeted
Joe Hamman@_jhamman·24 Oct 2024

We've talked a lot about #Icechunk's performance this week 🚀. But the Zarr-Python 3 results are also very encouraging! We're a few weeks away from the 3.0 launch but what this chart shows is that the new AsyncIO + multi-threading functionality in Zarr is going to be really good.

ALSO this release is the first to be compatible with the much anticipated v3 implementation of zarr-python! (still on its beta branch right now) This brings big performance benefits when reading @zarr_dev on S3 via async and (b) compatibility with @EarthmoverHQ 's Icechunk.

Post attachment
Tom Nicholas avatar
Tom Nicholas@TEGNicholasCode·24 Oct 2024

All these integrations represent literally years-worth of effort, all coming out at once 🤯 And that's not even mentioning all the other changes you see in a typical xarray release!

Tom Nicholas avatar
Tom Nicholas@TEGNicholasCode·24 Oct 2024

ALSO this release is the first to be compatible with the much anticipated v3 implementation of zarr-python! (still on its beta branch right now) This brings big performance benefits when reading @zarr_dev on S3 via async and (b) compatibility with @EarthmoverHQ 's Icechunk.

Post attachment
Tom Nicholas avatar
Tom Nicholas@TEGNicholasCode·24 Oct 2024

This was a huge effort involving many people. It started in 2021 when I prototyped DataTree on a long Amtrak journey, and was only pushed over the finish line with generous help from people at @NASAEarthData GESDIS. If there's a feature you want to see in xarray then hit us up!

Tom Nicholas avatar
Tom Nicholas@TEGNicholasCode·24 Oct 2024

xarray.DataTree is a huge feature addition - since the the dawn of time xarray top-level classes were `DataArray` and `Dataset` (and `Variable`), now we have `DataTree` too! Read about it here github.com/pydata/xarray/dis…