Comparing performance between access multi-file patterns

[danielfromearth]

Perhaps we can create a blog post, white paper, or some other artifact, explaining the results of our performance assessment.

Access methods to compare:

• earthaccess.open_virtual_dataset()
• earthaccess.open_virtual_mfdataset()
• pydap
• earthaccess.download()
• xarray.open_mfdataset()

Tentatively, we want to access as Zarr v3, and our "moon shot" would be Icechunk.

(This came up in a discussion during the earthaccess hackday on 15 April 2025, with @battistowx, @betolink, @Mikejmnez, @rwegener2)

[danielfromearth]

A new notebook (in a discussion-987/comparing-access-performance branch) shows initial test results in the second markdown cell, titled "Test Report (so far)".

[betolink]

This has been on the list forever, I'm happy we are moving along!

[Mikejmnez]

This is great. I will look onto replicating these access workflows with

xr.open_dataset(opendap_dap4url, engine='pydap')

and

xr.open_mfdataset(list_opendap_dap4urls, engine='pydap', **open_params)

and will share a notebook later today.

[danielfromearth]

Opened draft PR #989 for this effort.

[danielfromearth]

Quick chart showing the results from the initial tests: