Tutorial

Intake-thredds provides an interface that combines functionality from siphon and intake to retrieve data from THREDDS data servers. This tutorial provides an introduction to the API and features of intake-thredds. Let’s begin by importing intake.

import intake

Loading a catalog

You can load data from a THREDDS catalog by providing the URL to a valid THREDDS catalog:

cat_url = 'https://psl.noaa.gov/thredds/catalog/Datasets/noaa.ersst/catalog.xml'
catalog = intake.open_thredds_cat(cat_url, name='noaa-ersst-catalog')
catalog

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

noaa-ersst-catalog:
  args:
    name: noaa-ersst-catalog
    url: https://psl.noaa.gov/thredds/catalog/Datasets/noaa.ersst/catalog.xml
  description: ''
  driver: intake_thredds.cat.ThreddsCatalog
  metadata:
    dataType: Grid
    inherited: true
    serviceName: all

Using the catalog

Once you’ve loaded a catalog, you can display its contents by iterating over its entries:

list(catalog)

['err.mnmean.v3.nc',
 'sst.mnmean.v3.nc',
 'sst.mnmean.v4.nc',
 'sst.mon.1971-2000.ltm.v4.nc',
 'sst.mon.19712000.ltm.v3.nc',
 'sst.mon.1981-2010.ltm.v3.nc',
 'sst.mon.1981-2010.ltm.v4.nc',
 'sst.mon.v4.ltm.1971-2000.nc',
 'sst.mon.v4.ltm.1981-2010.nc',
 'sst.mon.v4.ltm.nc']

Once you’ve identified a dataset of interest, you can access it as follows:

source = catalog['err.mnmean.v3.nc']
print(source)

sources:
  err.mnmean.v3.nc:
    args:
      chunks: {}
      urlpath: https://psl.noaa.gov/thredds/dodsC/Datasets/noaa.ersst/err.mnmean.v3.nc
    description: THREDDS data
    driver: intake_xarray.opendap.OpenDapSource
    metadata:
      catalog_dir: null

Loading a dataset

To load a dataset of interest, you can use the to_dask() method which is available on a source object:

%%time
ds = source().to_dask()
ds

CPU times: user 454 ms, sys: 16.5 ms, total: 470 ms
Wall time: 2.71 s

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 128MB
Dimensions:    (lat: 89, lon: 180, time: 1994, nbnds: 2)
Coordinates:
  * lat        (lat) float32 356B 88.0 86.0 84.0 82.0 ... -84.0 -86.0 -88.0
  * lon        (lon) float32 720B 0.0 2.0 4.0 6.0 ... 352.0 354.0 356.0 358.0
  * time       (time) datetime64[ns] 16kB 1854-01-01 1854-02-01 ... 2020-02-01
Dimensions without coordinates: nbnds
Data variables:
    time_bnds  (time, nbnds) float64 32kB dask.array<chunksize=(1994, 2), meta=np.ndarray>
    err        (time, lat, lon) float32 128MB dask.array<chunksize=(1994, 89, 180), meta=np.ndarray>
Attributes: (12/13)
    title:                           NOAA Extended Reconstructed SST V3
    Conventions:                     CF-1.0
    history:                         Thu Jul  1 14:04:15 2010: ncatted -O -a ...
    comments:                        The extended reconstructed sea surface t...
    platform:                        Model
    source:                          NOAA/NESDIS/National Climatic Data Center
    ...                              ...
    citation:                        Smith, T.M., R.W. Reynolds, Thomas C. Pe...
    dataset_title:                   Extended Reconstructed Sea Surface Tempe...
    source_doc:                      https://www.ncdc.noaa.gov/data-access/ma...
    _NCProperties:                   version=2,netcdf=4.6.3,hdf5=1.10.5
    References:                      https://www.psl.noaa.gov/data/gridded/da...
    DODS_EXTRA.Unlimited_Dimension:  time

xarray.Dataset

• Dimensions:
  • lat: 89
  • lon: 180
  • time: 1994
  • nbnds: 2
• Coordinates: (3)
  • lat
    (lat)
    float32
    88.0 86.0 84.0 ... -86.0 -88.0

    units :

    degrees_north

    long_name :

    Latitude

    actual_range :

    [88.0, -88.0]

    standard_name :

    latitude

    axis :

    Y

    coordinate_defines :

    center

    array([ 88.,  86.,  84.,  82.,  80.,  78.,  76.,  74.,  72.,  70.,  68.,  66.,
            64.,  62.,  60.,  58.,  56.,  54.,  52.,  50.,  48.,  46.,  44.,  42.,
            40.,  38.,  36.,  34.,  32.,  30.,  28.,  26.,  24.,  22.,  20.,  18.,
            16.,  14.,  12.,  10.,   8.,   6.,   4.,   2.,   0.,  -2.,  -4.,  -6.,
            -8., -10., -12., -14., -16., -18., -20., -22., -24., -26., -28., -30.,
           -32., -34., -36., -38., -40., -42., -44., -46., -48., -50., -52., -54.,
           -56., -58., -60., -62., -64., -66., -68., -70., -72., -74., -76., -78.,
           -80., -82., -84., -86., -88.], dtype=float32)

  • lon
    (lon)
    float32
    0.0 2.0 4.0 ... 354.0 356.0 358.0

    units :

    degrees_east

    long_name :

    Longitude

    actual_range :

    [0.0, 358.0]

    standard_name :

    longitude

    axis :

    X

    coordinate_defines :

    center

    array([  0.,   2.,   4.,   6.,   8.,  10.,  12.,  14.,  16.,  18.,  20.,  22.,
            24.,  26.,  28.,  30.,  32.,  34.,  36.,  38.,  40.,  42.,  44.,  46.,
            48.,  50.,  52.,  54.,  56.,  58.,  60.,  62.,  64.,  66.,  68.,  70.,
            72.,  74.,  76.,  78.,  80.,  82.,  84.,  86.,  88.,  90.,  92.,  94.,
            96.,  98., 100., 102., 104., 106., 108., 110., 112., 114., 116., 118.,
           120., 122., 124., 126., 128., 130., 132., 134., 136., 138., 140., 142.,
           144., 146., 148., 150., 152., 154., 156., 158., 160., 162., 164., 166.,
           168., 170., 172., 174., 176., 178., 180., 182., 184., 186., 188., 190.,
           192., 194., 196., 198., 200., 202., 204., 206., 208., 210., 212., 214.,
           216., 218., 220., 222., 224., 226., 228., 230., 232., 234., 236., 238.,
           240., 242., 244., 246., 248., 250., 252., 254., 256., 258., 260., 262.,
           264., 266., 268., 270., 272., 274., 276., 278., 280., 282., 284., 286.,
           288., 290., 292., 294., 296., 298., 300., 302., 304., 306., 308., 310.,
           312., 314., 316., 318., 320., 322., 324., 326., 328., 330., 332., 334.,
           336., 338., 340., 342., 344., 346., 348., 350., 352., 354., 356., 358.],
          dtype=float32)

  • time
    (time)
    datetime64[ns]
    1854-01-01 ... 2020-02-01

    long_name :

    Time

    delta_t :

    0000-01-00 00:00:00

    avg_period :

    0000-01-00 00:00:00

    prev_avg_period :

    0000-00-07 00:00:00

    standard_name :

    time

    axis :

    T

    actual_range :

    [19723.0, 80384.0]

    _ChunkSizes :

    1

    array(['1854-01-01T00:00:00.000000000', '1854-02-01T00:00:00.000000000',
           '1854-03-01T00:00:00.000000000', ..., '2019-12-01T00:00:00.000000000',
           '2020-01-01T00:00:00.000000000', '2020-02-01T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (2)
  • time_bnds
    (time, nbnds)
    float64
    dask.array<chunksize=(1994, 2), meta=np.ndarray>

    long_name :

    Time Boundaries

    _ChunkSizes :

    [1, 2]

    Array Chunk Bytes 31.16 kiB 31.16 kiB Shape (1994, 2) (1994, 2) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • err
    (time, lat, lon)
    float32
    dask.array<chunksize=(1994, 89, 180), meta=np.ndarray>

    long_name :

    Monthly Mean Error Estimate of Sea Surface Temperature

    units :

    -

    precision :

    2

    least_significant_digit :

    1

    var_desc :

    Normalized Error Variance

    dataset :

    NOAA Extended Reconstructed SST V3

    level_desc :

    Surface

    statistic :

    Mean

    parent_stat :

    Mean

    actual_range :

    [0.04, 1.0]

    valid_range :

    [0.0, 1.0]

    _ChunkSizes :

    [1, 89, 180]

    Array Chunk Bytes 121.86 MiB 121.86 MiB Shape (1994, 89, 180) (1994, 89, 180) Dask graph 1 chunks in 2 graph layers Data type float32 numpy.ndarray

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Index([ 88.0,  86.0,  84.0,  82.0,  80.0,  78.0,  76.0,  74.0,  72.0,  70.0,
            68.0,  66.0,  64.0,  62.0,  60.0,  58.0,  56.0,  54.0,  52.0,  50.0,
            48.0,  46.0,  44.0,  42.0,  40.0,  38.0,  36.0,  34.0,  32.0,  30.0,
            28.0,  26.0,  24.0,  22.0,  20.0,  18.0,  16.0,  14.0,  12.0,  10.0,
             8.0,   6.0,   4.0,   2.0,   0.0,  -2.0,  -4.0,  -6.0,  -8.0, -10.0,
           -12.0, -14.0, -16.0, -18.0, -20.0, -22.0, -24.0, -26.0, -28.0, -30.0,
           -32.0, -34.0, -36.0, -38.0, -40.0, -42.0, -44.0, -46.0, -48.0, -50.0,
           -52.0, -54.0, -56.0, -58.0, -60.0, -62.0, -64.0, -66.0, -68.0, -70.0,
           -72.0, -74.0, -76.0, -78.0, -80.0, -82.0, -84.0, -86.0, -88.0],
          dtype='float32', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([  0.0,   2.0,   4.0,   6.0,   8.0,  10.0,  12.0,  14.0,  16.0,  18.0,
           ...
           340.0, 342.0, 344.0, 346.0, 348.0, 350.0, 352.0, 354.0, 356.0, 358.0],
          dtype='float32', name='lon', length=180))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1854-01-01', '1854-02-01', '1854-03-01', '1854-04-01',
                   '1854-05-01', '1854-06-01', '1854-07-01', '1854-08-01',
                   '1854-09-01', '1854-10-01',
                   ...
                   '2019-05-01', '2019-06-01', '2019-07-01', '2019-08-01',
                   '2019-09-01', '2019-10-01', '2019-11-01', '2019-12-01',
                   '2020-01-01', '2020-02-01'],
                  dtype='datetime64[ns]', name='time', length=1994, freq=None))

• Attributes: (13)

  title :

  NOAA Extended Reconstructed SST V3

  Conventions :

  CF-1.0

  history :

  Thu Jul 1 14:04:15 2010: ncatted -O -a _FillValue,err,o,s,32767 err.mnmean.v3b.nc created 09/2007 by CAS

  comments :

  The extended reconstructed sea surface temperature (ERSST) was constructed using the most recently available Comprehensive Ocean-Atmosphere Data Set (COADS) SST data and improved statistical methods that allow stable reconstruction using sparse data.Currently, ERSST version 2 ( ERSST.v2 ) and version 3 ( ERSST.v3 ) are available from NCDC. ERSST.v3 is an improved extended reconstruction over version 2. Most of the improvements are justified by testing with simulated data. The major differences are caused by the improved low-frequency ( LF ) tuning of ERSST.v3 which reduces the SST anomaly damping before 1930 using the optimized parameters. Beginning in 1985, ERSST.v3 is also improved by explicitly including bias-adjusted satellite infrared data from AVHRR.

  platform :

  Model

  source :

  NOAA/NESDIS/National Climatic Data Center

  institution :

  NOAA/NESDIS/National Climatic Data Center

  citation :

  Smith, T.M., R.W. Reynolds, Thomas C. Peterson, and Jay Lawrimore 2007: Improvements to NOAA's Historical Merged Land-Ocean Surface Temperature lysis (1880-2006). In press. Journal of Climate (ERSSTV3). Smith, T.M., and R.W. Reynolds, 2003: Extended Reconstruction of Global Sea Surface Temperatures Based on COADS Data (1854-1997). Journal of Climate, 16, 1495-1510. ERSSTV1 Smith, T.M., and R.W. Reynolds, 2004: Improved Extended Reconstruction of SST (1854-1997). Journal of Climate, 17, 2466-2477.ERSSTv2

  dataset_title :

  Extended Reconstructed Sea Surface Temperature (ERSST)

  source_doc :

  https://www.ncdc.noaa.gov/data-access/marineocean-data/extended-reconstructed-sea-surface-temperature-ersst-v3b

  _NCProperties :

  version=2,netcdf=4.6.3,hdf5=1.10.5

  References :

  https://www.psl.noaa.gov/data/gridded/data.noaa.ersst.html

  DODS_EXTRA.Unlimited_Dimension :

  time

The to_dask() reads only metadata needed to construct an xarray.Dataset. The actual data are streamed over the network when computation routines are invoked on the dataset. By default, intake-thredds uses chunks={} to load the dataset with dask using a single chunk for all arrays. You can use a different chunking scheme by prividing a custom value of chunks before calling .to_dask():

%%time
# Use a custom chunking scheme
ds = source(chunks={'time': 100, 'lon': 90}).to_dask()
ds

CPU times: user 245 ms, sys: 5.74 ms, total: 250 ms
Wall time: 2.48 s

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 128MB
Dimensions:    (lat: 89, lon: 180, time: 1994, nbnds: 2)
Coordinates:
  * lat        (lat) float32 356B 88.0 86.0 84.0 82.0 ... -84.0 -86.0 -88.0
  * lon        (lon) float32 720B 0.0 2.0 4.0 6.0 ... 352.0 354.0 356.0 358.0
  * time       (time) datetime64[ns] 16kB 1854-01-01 1854-02-01 ... 2020-02-01
Dimensions without coordinates: nbnds
Data variables:
    time_bnds  (time, nbnds) float64 32kB dask.array<chunksize=(100, 2), meta=np.ndarray>
    err        (time, lat, lon) float32 128MB dask.array<chunksize=(100, 89, 90), meta=np.ndarray>
Attributes: (12/13)
    title:                           NOAA Extended Reconstructed SST V3
    Conventions:                     CF-1.0
    history:                         Thu Jul  1 14:04:15 2010: ncatted -O -a ...
    comments:                        The extended reconstructed sea surface t...
    platform:                        Model
    source:                          NOAA/NESDIS/National Climatic Data Center
    ...                              ...
    citation:                        Smith, T.M., R.W. Reynolds, Thomas C. Pe...
    dataset_title:                   Extended Reconstructed Sea Surface Tempe...
    source_doc:                      https://www.ncdc.noaa.gov/data-access/ma...
    _NCProperties:                   version=2,netcdf=4.6.3,hdf5=1.10.5
    References:                      https://www.psl.noaa.gov/data/gridded/da...
    DODS_EXTRA.Unlimited_Dimension:  time

xarray.Dataset

• Dimensions:
  • lat: 89
  • lon: 180
  • time: 1994
  • nbnds: 2
• Coordinates: (3)
  • lat
    (lat)
    float32
    88.0 86.0 84.0 ... -86.0 -88.0

    units :

    degrees_north

    long_name :

    Latitude

    actual_range :

    [88.0, -88.0]

    standard_name :

    latitude

    axis :

    Y

    coordinate_defines :

    center

    array([ 88.,  86.,  84.,  82.,  80.,  78.,  76.,  74.,  72.,  70.,  68.,  66.,
            64.,  62.,  60.,  58.,  56.,  54.,  52.,  50.,  48.,  46.,  44.,  42.,
            40.,  38.,  36.,  34.,  32.,  30.,  28.,  26.,  24.,  22.,  20.,  18.,
            16.,  14.,  12.,  10.,   8.,   6.,   4.,   2.,   0.,  -2.,  -4.,  -6.,
            -8., -10., -12., -14., -16., -18., -20., -22., -24., -26., -28., -30.,
           -32., -34., -36., -38., -40., -42., -44., -46., -48., -50., -52., -54.,
           -56., -58., -60., -62., -64., -66., -68., -70., -72., -74., -76., -78.,
           -80., -82., -84., -86., -88.], dtype=float32)

  • lon
    (lon)
    float32
    0.0 2.0 4.0 ... 354.0 356.0 358.0

    units :

    degrees_east

    long_name :

    Longitude

    actual_range :

    [0.0, 358.0]

    standard_name :

    longitude

    axis :

    X

    coordinate_defines :

    center

    array([  0.,   2.,   4.,   6.,   8.,  10.,  12.,  14.,  16.,  18.,  20.,  22.,
            24.,  26.,  28.,  30.,  32.,  34.,  36.,  38.,  40.,  42.,  44.,  46.,
            48.,  50.,  52.,  54.,  56.,  58.,  60.,  62.,  64.,  66.,  68.,  70.,
            72.,  74.,  76.,  78.,  80.,  82.,  84.,  86.,  88.,  90.,  92.,  94.,
            96.,  98., 100., 102., 104., 106., 108., 110., 112., 114., 116., 118.,
           120., 122., 124., 126., 128., 130., 132., 134., 136., 138., 140., 142.,
           144., 146., 148., 150., 152., 154., 156., 158., 160., 162., 164., 166.,
           168., 170., 172., 174., 176., 178., 180., 182., 184., 186., 188., 190.,
           192., 194., 196., 198., 200., 202., 204., 206., 208., 210., 212., 214.,
           216., 218., 220., 222., 224., 226., 228., 230., 232., 234., 236., 238.,
           240., 242., 244., 246., 248., 250., 252., 254., 256., 258., 260., 262.,
           264., 266., 268., 270., 272., 274., 276., 278., 280., 282., 284., 286.,
           288., 290., 292., 294., 296., 298., 300., 302., 304., 306., 308., 310.,
           312., 314., 316., 318., 320., 322., 324., 326., 328., 330., 332., 334.,
           336., 338., 340., 342., 344., 346., 348., 350., 352., 354., 356., 358.],
          dtype=float32)

  • time
    (time)
    datetime64[ns]
    1854-01-01 ... 2020-02-01

    long_name :

    Time

    delta_t :

    0000-01-00 00:00:00

    avg_period :

    0000-01-00 00:00:00

    prev_avg_period :

    0000-00-07 00:00:00

    standard_name :

    time

    axis :

    T

    actual_range :

    [19723.0, 80384.0]

    _ChunkSizes :

    1

    array(['1854-01-01T00:00:00.000000000', '1854-02-01T00:00:00.000000000',
           '1854-03-01T00:00:00.000000000', ..., '2019-12-01T00:00:00.000000000',
           '2020-01-01T00:00:00.000000000', '2020-02-01T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (2)
  • time_bnds
    (time, nbnds)
    float64
    dask.array<chunksize=(100, 2), meta=np.ndarray>

    long_name :

    Time Boundaries

    _ChunkSizes :

    [1, 2]

    Array Chunk Bytes 31.16 kiB 1.56 kiB Shape (1994, 2) (100, 2) Dask graph 20 chunks in 2 graph layers Data type float64 numpy.ndarray

  • err
    (time, lat, lon)
    float32
    dask.array<chunksize=(100, 89, 90), meta=np.ndarray>

    long_name :

    Monthly Mean Error Estimate of Sea Surface Temperature

    units :

    -

    precision :

    2

    least_significant_digit :

    1

    var_desc :

    Normalized Error Variance

    dataset :

    NOAA Extended Reconstructed SST V3

    level_desc :

    Surface

    statistic :

    Mean

    parent_stat :

    Mean

    actual_range :

    [0.04, 1.0]

    valid_range :

    [0.0, 1.0]

    _ChunkSizes :

    [1, 89, 180]

    Array Chunk Bytes 121.86 MiB 3.06 MiB Shape (1994, 89, 180) (100, 89, 90) Dask graph 40 chunks in 2 graph layers Data type float32 numpy.ndarray

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Index([ 88.0,  86.0,  84.0,  82.0,  80.0,  78.0,  76.0,  74.0,  72.0,  70.0,
            68.0,  66.0,  64.0,  62.0,  60.0,  58.0,  56.0,  54.0,  52.0,  50.0,
            48.0,  46.0,  44.0,  42.0,  40.0,  38.0,  36.0,  34.0,  32.0,  30.0,
            28.0,  26.0,  24.0,  22.0,  20.0,  18.0,  16.0,  14.0,  12.0,  10.0,
             8.0,   6.0,   4.0,   2.0,   0.0,  -2.0,  -4.0,  -6.0,  -8.0, -10.0,
           -12.0, -14.0, -16.0, -18.0, -20.0, -22.0, -24.0, -26.0, -28.0, -30.0,
           -32.0, -34.0, -36.0, -38.0, -40.0, -42.0, -44.0, -46.0, -48.0, -50.0,
           -52.0, -54.0, -56.0, -58.0, -60.0, -62.0, -64.0, -66.0, -68.0, -70.0,
           -72.0, -74.0, -76.0, -78.0, -80.0, -82.0, -84.0, -86.0, -88.0],
          dtype='float32', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([  0.0,   2.0,   4.0,   6.0,   8.0,  10.0,  12.0,  14.0,  16.0,  18.0,
           ...
           340.0, 342.0, 344.0, 346.0, 348.0, 350.0, 352.0, 354.0, 356.0, 358.0],
          dtype='float32', name='lon', length=180))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1854-01-01', '1854-02-01', '1854-03-01', '1854-04-01',
                   '1854-05-01', '1854-06-01', '1854-07-01', '1854-08-01',
                   '1854-09-01', '1854-10-01',
                   ...
                   '2019-05-01', '2019-06-01', '2019-07-01', '2019-08-01',
                   '2019-09-01', '2019-10-01', '2019-11-01', '2019-12-01',
                   '2020-01-01', '2020-02-01'],
                  dtype='datetime64[ns]', name='time', length=1994, freq=None))

• Attributes: (13)

  title :

  NOAA Extended Reconstructed SST V3

  Conventions :

  CF-1.0

  history :

  Thu Jul 1 14:04:15 2010: ncatted -O -a _FillValue,err,o,s,32767 err.mnmean.v3b.nc created 09/2007 by CAS

  comments :

  The extended reconstructed sea surface temperature (ERSST) was constructed using the most recently available Comprehensive Ocean-Atmosphere Data Set (COADS) SST data and improved statistical methods that allow stable reconstruction using sparse data.Currently, ERSST version 2 ( ERSST.v2 ) and version 3 ( ERSST.v3 ) are available from NCDC. ERSST.v3 is an improved extended reconstruction over version 2. Most of the improvements are justified by testing with simulated data. The major differences are caused by the improved low-frequency ( LF ) tuning of ERSST.v3 which reduces the SST anomaly damping before 1930 using the optimized parameters. Beginning in 1985, ERSST.v3 is also improved by explicitly including bias-adjusted satellite infrared data from AVHRR.

  platform :

  Model

  source :

  NOAA/NESDIS/National Climatic Data Center

  institution :

  NOAA/NESDIS/National Climatic Data Center

  citation :

  Smith, T.M., R.W. Reynolds, Thomas C. Peterson, and Jay Lawrimore 2007: Improvements to NOAA's Historical Merged Land-Ocean Surface Temperature lysis (1880-2006). In press. Journal of Climate (ERSSTV3). Smith, T.M., and R.W. Reynolds, 2003: Extended Reconstruction of Global Sea Surface Temperatures Based on COADS Data (1854-1997). Journal of Climate, 16, 1495-1510. ERSSTV1 Smith, T.M., and R.W. Reynolds, 2004: Improved Extended Reconstruction of SST (1854-1997). Journal of Climate, 17, 2466-2477.ERSSTv2

  dataset_title :

  Extended Reconstructed Sea Surface Temperature (ERSST)

  source_doc :

  https://www.ncdc.noaa.gov/data-access/marineocean-data/extended-reconstructed-sea-surface-temperature-ersst-v3b

  _NCProperties :

  version=2,netcdf=4.6.3,hdf5=1.10.5

  References :

  https://www.psl.noaa.gov/data/gridded/data.noaa.ersst.html

  DODS_EXTRA.Unlimited_Dimension :

  time

Working with nested catalogs

In some scenarious, a THREDDS catalog can reference another THREDDS catalog. This results into a nested structure consisting of a parent catalog and children catalogs:

cat_url = 'https://psl.noaa.gov/thredds/catalog.xml'
catalog = intake.open_thredds_cat(cat_url)
list(catalog)

['Datasets', 'Projects', 'Aggregations']

print(list(catalog['Datasets']))

['20thC_ReanV2', '20thC_ReanV2c', '20thC_ReanV3', 'ATOMIC', 'COBE', 'COBE2', 'CarbonTracker', 'E3SM', 'E3SM_LE', 'ICOADS3.0.0_2015-2021', 'LIM', 'NARR', 'S2S', 'SERDP_regimeshifts', 'Timeseries', 'cmap', 'cmems', 'coads', 'cpc_blended_olr-1deg', 'cpc_blended_olr-2.5deg', 'cpc_global_precip', 'cpc_global_temp', 'cpc_us_hour_precip', 'cpc_us_precip', 'cpcsoil', 'cru', 'dai_pdsi', 'ghcncams', 'ghcngridded', 'gistemp', 'godas', 'gpcc', 'gpcp', 'icoads', 'icoads2.5', 'interp_OLR', 'jmatemp', 'kaplan_sst', 'livneh', 'marinehw', 'mlost', 'mlostv3b', 'ncep', 'ncep.marine', 'ncep.pac.ocean', 'ncep.reanalysis', 'ncep.reanalysis.dailyavgs', 'ncep.reanalysis.derived', 'ncep.reanalysis2', 'ncep.reanalysis2.dailyavgs', 'ncep.reanalysis2.derived', 'noaa.ersst', 'noaa.ersst.v3', 'noaa.ersst.v4', 'noaa.ersst.v5', 'noaa.oisst.v2', 'noaa.oisst.v2.derived', 'noaa.oisst.v2.highres', 'noaa_hrc', 'noaaglobaltemp', 'noaamergedtemp', 'nodc.woa94', 'nodc.woa98', 'olrcdr', 'prec', 'precl', 'rte-rrtmgp', 'snowcover', 'udel.airt.precip', 'uninterp_OLR']

print(list(catalog['Datasets']['ncep.reanalysis.dailyavgs']))

['other_gauss', 'pressure', 'surface', 'surface_gauss', 'tropopause']

To load data from such a nested catalog, intake-thredds provides a special source object THREDDSMergedSource accessible via the .open_thredds_merged() function. The inputs for this function consists of:

• url: top level URL of the THREDDS catalog

• path: a list of paths for child catalogs to descend down. The paths can include glob characters (*, ?). These glob characters are used for matching.

source = intake.open_thredds_merged(
    cat_url, path=['Datasets', 'ncep.reanalysis.dailyavgs', 'surface', 'air*sig995*194*.nc']
)
print(source)

sources:
  thredds_merged:
    args:
      path:
      - Datasets
      - ncep.reanalysis.dailyavgs
      - surface
      - air*sig995*194*.nc
      url: https://psl.noaa.gov/thredds/catalog.xml
    description: ''
    driver: intake_thredds.source.THREDDSMergedSource
    metadata: {}

To load the data into an xarray Dataset, you can invoke the .to_dask() method. Internally, THREDDSMergedSource does the following:

• descend down the given paths and collect all available datasets.

• load each dataset in a dataset.

• combine all loaded datasets into a single dataset.

%%time
ds = source.to_dask()
ds

Dataset(s):   0%|                                        | 0/2 [00:00<?, ?it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  50%|████████████████                | 1/2 [00:02<00:02,  2.43s/it]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s): 100%|████████████████████████████████| 2/2 [00:04<00:00,  2.44s/it]

Dataset(s): 100%|████████████████████████████████| 2/2 [00:04<00:00,  2.44s/it]

CPU times: user 904 ms, sys: 34.9 ms, total: 939 ms
Wall time: 6.49 s

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 31MB
Dimensions:    (lon: 144, time: 731, lat: 73, nbnds: 2)
Coordinates:
  * lon        (lon) float32 576B 0.0 2.5 5.0 7.5 ... 350.0 352.5 355.0 357.5
  * time       (time) datetime64[ns] 6kB 1948-01-01 1948-01-02 ... 1949-12-31
  * lat        (lat) float32 292B 90.0 87.5 85.0 82.5 ... -85.0 -87.5 -90.0
Dimensions without coordinates: nbnds
Data variables:
    air        (time, lat, lon) float32 31MB dask.array<chunksize=(366, 73, 144), meta=np.ndarray>
    time_bnds  (time, nbnds) float64 12kB dask.array<chunksize=(366, 2), meta=np.ndarray>
Attributes:
    Conventions:                     COARDS
    title:                           mean daily NMC reanalysis (1948)
    description:                     Data is from NMC initialized reanalysis\...
    platform:                        Model
    history:                         created 99/05/11 by Hoop (netCDF2.3)\nCo...
    dataset_title:                   NCEP-NCAR Reanalysis 1
    References:                      http://www.psl.noaa.gov/data/gridded/dat...
    _NCProperties:                   version=2,netcdf=4.6.3,hdf5=1.10.5
    DODS_EXTRA.Unlimited_Dimension:  time

xarray.Dataset

• Dimensions:
  • lon: 144
  • time: 731
  • lat: 73
  • nbnds: 2
• Coordinates: (3)
  • lon
    (lon)
    float32
    0.0 2.5 5.0 ... 352.5 355.0 357.5

    units :

    degrees_east

    long_name :

    Longitude

    actual_range :

    [0.0, 357.5]

    standard_name :

    longitude

    axis :

    X

    array([  0. ,   2.5,   5. ,   7.5,  10. ,  12.5,  15. ,  17.5,  20. ,  22.5,
            25. ,  27.5,  30. ,  32.5,  35. ,  37.5,  40. ,  42.5,  45. ,  47.5,
            50. ,  52.5,  55. ,  57.5,  60. ,  62.5,  65. ,  67.5,  70. ,  72.5,
            75. ,  77.5,  80. ,  82.5,  85. ,  87.5,  90. ,  92.5,  95. ,  97.5,
           100. , 102.5, 105. , 107.5, 110. , 112.5, 115. , 117.5, 120. , 122.5,
           125. , 127.5, 130. , 132.5, 135. , 137.5, 140. , 142.5, 145. , 147.5,
           150. , 152.5, 155. , 157.5, 160. , 162.5, 165. , 167.5, 170. , 172.5,
           175. , 177.5, 180. , 182.5, 185. , 187.5, 190. , 192.5, 195. , 197.5,
           200. , 202.5, 205. , 207.5, 210. , 212.5, 215. , 217.5, 220. , 222.5,
           225. , 227.5, 230. , 232.5, 235. , 237.5, 240. , 242.5, 245. , 247.5,
           250. , 252.5, 255. , 257.5, 260. , 262.5, 265. , 267.5, 270. , 272.5,
           275. , 277.5, 280. , 282.5, 285. , 287.5, 290. , 292.5, 295. , 297.5,
           300. , 302.5, 305. , 307.5, 310. , 312.5, 315. , 317.5, 320. , 322.5,
           325. , 327.5, 330. , 332.5, 335. , 337.5, 340. , 342.5, 345. , 347.5,
           350. , 352.5, 355. , 357.5], dtype=float32)

  • time
    (time)
    datetime64[ns]
    1948-01-01 ... 1949-12-31

    long_name :

    Time

    delta_t :

    0000-00-01 00:00:00

    avg_period :

    0000-00-01 00:00:00

    standard_name :

    time

    axis :

    T

    actual_range :

    [1297320.0, 1306080.0]

    coordinate_defines :

    start

    _ChunkSizes :

    1

    array(['1948-01-01T00:00:00.000000000', '1948-01-02T00:00:00.000000000',
           '1948-01-03T00:00:00.000000000', ..., '1949-12-29T00:00:00.000000000',
           '1949-12-30T00:00:00.000000000', '1949-12-31T00:00:00.000000000'],
          dtype='datetime64[ns]')

  • lat
    (lat)
    float32
    90.0 87.5 85.0 ... -87.5 -90.0

    units :

    degrees_north

    actual_range :

    [90.0, -90.0]

    long_name :

    Latitude

    standard_name :

    latitude

    axis :

    Y

    array([ 90. ,  87.5,  85. ,  82.5,  80. ,  77.5,  75. ,  72.5,  70. ,  67.5,
            65. ,  62.5,  60. ,  57.5,  55. ,  52.5,  50. ,  47.5,  45. ,  42.5,
            40. ,  37.5,  35. ,  32.5,  30. ,  27.5,  25. ,  22.5,  20. ,  17.5,
            15. ,  12.5,  10. ,   7.5,   5. ,   2.5,   0. ,  -2.5,  -5. ,  -7.5,
           -10. , -12.5, -15. , -17.5, -20. , -22.5, -25. , -27.5, -30. , -32.5,
           -35. , -37.5, -40. , -42.5, -45. , -47.5, -50. , -52.5, -55. , -57.5,
           -60. , -62.5, -65. , -67.5, -70. , -72.5, -75. , -77.5, -80. , -82.5,
           -85. , -87.5, -90. ], dtype=float32)

• Data variables: (2)
  • air
    (time, lat, lon)
    float32
    dask.array<chunksize=(366, 73, 144), meta=np.ndarray>

    long_name :

    mean Daily Air temperature at sigma level 995

    units :

    degK

    precision :

    2

    least_significant_digit :

    1

    GRIB_id :

    11

    GRIB_name :

    TMP

    var_desc :

    Air temperature

    dataset :

    NCEP Reanalysis Daily Averages

    statistic :

    Mean

    parent_stat :

    Individual Obs

    actual_range :

    [188.29001, 315.3]

    valid_range :

    [185.16, 331.16]

    level_desc :

    0.995 sigma

    _ChunkSizes :

    [1, 73, 144]

    Array Chunk Bytes 29.31 MiB 14.68 MiB Shape (731, 73, 144) (366, 73, 144) Dask graph 2 chunks in 5 graph layers Data type float32 numpy.ndarray

  • time_bnds
    (time, nbnds)
    float64
    dask.array<chunksize=(366, 2), meta=np.ndarray>

    _ChunkSizes :

    [1, 2]

    Array Chunk Bytes 11.42 kiB 5.72 kiB Shape (731, 2) (366, 2) Dask graph 2 chunks in 5 graph layers Data type float64 numpy.ndarray

• Indexes: (3)
  • lon
    PandasIndex

    PandasIndex(Index([  0.0,   2.5,   5.0,   7.5,  10.0,  12.5,  15.0,  17.5,  20.0,  22.5,
           ...
           335.0, 337.5, 340.0, 342.5, 345.0, 347.5, 350.0, 352.5, 355.0, 357.5],
          dtype='float32', name='lon', length=144))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1948-01-01', '1948-01-02', '1948-01-03', '1948-01-04',
                   '1948-01-05', '1948-01-06', '1948-01-07', '1948-01-08',
                   '1948-01-09', '1948-01-10',
                   ...
                   '1949-12-22', '1949-12-23', '1949-12-24', '1949-12-25',
                   '1949-12-26', '1949-12-27', '1949-12-28', '1949-12-29',
                   '1949-12-30', '1949-12-31'],
                  dtype='datetime64[ns]', name='time', length=731, freq=None))

  • lat
    PandasIndex

    PandasIndex(Index([ 90.0,  87.5,  85.0,  82.5,  80.0,  77.5,  75.0,  72.5,  70.0,  67.5,
            65.0,  62.5,  60.0,  57.5,  55.0,  52.5,  50.0,  47.5,  45.0,  42.5,
            40.0,  37.5,  35.0,  32.5,  30.0,  27.5,  25.0,  22.5,  20.0,  17.5,
            15.0,  12.5,  10.0,   7.5,   5.0,   2.5,   0.0,  -2.5,  -5.0,  -7.5,
           -10.0, -12.5, -15.0, -17.5, -20.0, -22.5, -25.0, -27.5, -30.0, -32.5,
           -35.0, -37.5, -40.0, -42.5, -45.0, -47.5, -50.0, -52.5, -55.0, -57.5,
           -60.0, -62.5, -65.0, -67.5, -70.0, -72.5, -75.0, -77.5, -80.0, -82.5,
           -85.0, -87.5, -90.0],
          dtype='float32', name='lat'))

• Attributes: (9)

  Conventions :

  COARDS

  title :

  mean daily NMC reanalysis (1948)

  description :

  Data is from NMC initialized reanalysis (4x/day). These are the 0.9950 sigma level values.

  platform :

  Model

  history :

  created 99/05/11 by Hoop (netCDF2.3) Converted to chunked, deflated non-packed NetCDF4 2014/09

  dataset_title :

  NCEP-NCAR Reanalysis 1

  References :

  http://www.psl.noaa.gov/data/gridded/data.ncep.reanalysis.html

  _NCProperties :

  version=2,netcdf=4.6.3,hdf5=1.10.5

  DODS_EXTRA.Unlimited_Dimension :

  time

Caching

Under the hood intake-thredds uses the driver='opendap' from intake-xarray by default. You can also choose driver='netcdf', which in combination with fsspec caches files by appending simplecache:: to the url, see fsspec docs.

import os

import fsspec

# specify caching location, where to store files to with their original names
fsspec.config.conf['simplecache'] = {'cache_storage': 'my_caching_folder', 'same_names': True}

cat_url = 'https://psl.noaa.gov/thredds/catalog.xml'
source = intake.open_thredds_merged(
    f'simplecache::{cat_url}',
    path=['Datasets', 'ncep.reanalysis.dailyavgs', 'surface', 'air.sig995.194*.nc'],
    driver='netcdf',  # specify netcdf driver to open HTTPServer
)
print(source)

sources:
  thredds_merged:
    args:
      driver: netcdf
      path:
      - Datasets
      - ncep.reanalysis.dailyavgs
      - surface
      - air.sig995.194*.nc
      url: simplecache::https://psl.noaa.gov/thredds/catalog.xml
    description: ''
    driver: intake_thredds.source.THREDDSMergedSource
    metadata:
      fsspec_pre_url: 'simplecache::'

%time ds = source.to_dask()

Dataset(s):   0%|                                        | 0/2 [00:00<?, ?it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  50%|████████████████                | 1/2 [00:00<00:00,  1.60it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s): 100%|████████████████████████████████| 2/2 [00:00<00:00,  2.66it/s]

Dataset(s): 100%|████████████████████████████████| 2/2 [00:00<00:00,  2.42it/s]

CPU times: user 568 ms, sys: 58.7 ms, total: 626 ms
Wall time: 2.39 s

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

assert os.path.exists('my_caching_folder/air.sig995.1949.nc')

# after caching very fast
%time ds = source.to_dask()

CPU times: user 6 µs, sys: 0 ns, total: 6 µs
Wall time: 8.82 µs

Multi-file concat_kwargs with cfgrib engine

Another example demonstrating how to use caching consists of reading in ensemble members in parallel for GEFS. The example below reads in 21 ensemble members for a single timestep. It also demonstrates usage of xarray_kwargs which are passed on to xarray for opening the files, which in this cases uses the cfgrib engine:

cat_url = 'https://www.ncei.noaa.gov/thredds/catalog/model-gefs-003/202008/20200831/catalog.xml'
source = intake.open_thredds_merged(
    f'simplecache::{cat_url}',
    path=["NCEP gens-a Grid 3 Member-Forecast *-372 for 2020-08-31 00:00*"],
    driver="netcdf",
    concat_kwargs={"dim": "number"},
    xarray_kwargs=dict(
        engine="cfgrib",
        backend_kwargs=dict(
            filter_by_keys={"typeOfLevel": "heightAboveGround", "cfVarName": "t2m"}
        ),
    ),
)
source.to_dask()

Dataset(s):   0%|                                       | 0/21 [00:00<?, ?it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):   5%|█▍                             | 1/21 [00:00<00:18,  1.08it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  10%|██▉                            | 2/21 [00:01<00:14,  1.30it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  14%|████▍                          | 3/21 [00:02<00:12,  1.45it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  19%|█████▉                         | 4/21 [00:02<00:11,  1.53it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  24%|███████▍                       | 5/21 [00:03<00:09,  1.60it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  29%|████████▊                      | 6/21 [00:03<00:09,  1.62it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  33%|██████████▎                    | 7/21 [00:04<00:08,  1.63it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  38%|███████████▊                   | 8/21 [00:05<00:07,  1.65it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  43%|█████████████▎                 | 9/21 [00:05<00:07,  1.68it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  48%|██████████████▎               | 10/21 [00:06<00:06,  1.68it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  52%|███████████████▋              | 11/21 [00:07<00:07,  1.43it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  57%|█████████████████▏            | 12/21 [00:07<00:06,  1.50it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  62%|██████████████████▌           | 13/21 [00:08<00:05,  1.54it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  67%|████████████████████          | 14/21 [00:09<00:04,  1.54it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  71%|█████████████████████▍        | 15/21 [00:09<00:03,  1.60it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  76%|██████████████████████▊       | 16/21 [00:10<00:03,  1.62it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  81%|████████████████████████▎     | 17/21 [00:10<00:02,  1.64it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  86%|█████████████████████████▋    | 18/21 [00:11<00:01,  1.65it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  90%|███████████████████████████▏  | 19/21 [00:12<00:01,  1.60it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s):  95%|████████████████████████████▌ | 20/21 [00:12<00:00,  1.62it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

Dataset(s): 100%|██████████████████████████████| 21/21 [00:13<00:00,  1.65it/s]

Dataset(s): 100%|██████████████████████████████| 21/21 [00:13<00:00,  1.58it/s]

/home/docs/checkouts/readthedocs.org/user_builds/intake-thredds/conda/latest/lib/python3.11/site-packages/intake_xarray/base.py:21: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
  'dims': dict(self._ds.dims),

<xarray.Dataset> Size: 5MB
Dimensions:            (number: 21, latitude: 181, longitude: 360)
Coordinates:
  * number             (number) int64 168B 20 19 18 17 16 15 14 ... 5 4 3 2 1 0
    time               datetime64[ns] 8B 2020-08-31
    step               timedelta64[ns] 8B 15 days 12:00:00
    heightAboveGround  float64 8B 2.0
  * latitude           (latitude) float64 1kB 90.0 89.0 88.0 ... -89.0 -90.0
  * longitude          (longitude) float64 3kB 0.0 1.0 2.0 ... 357.0 358.0 359.0
    valid_time         datetime64[ns] 8B 2020-09-15T12:00:00
Data variables:
    t2m                (number, latitude, longitude) float32 5MB dask.array<chunksize=(1, 181, 360), meta=np.ndarray>
Attributes:
    GRIB_edition:            2
    GRIB_centre:             kwbc
    GRIB_centreDescription:  US National Weather Service - NCEP
    GRIB_subCentre:          2
    Conventions:             CF-1.7
    institution:             US National Weather Service - NCEP
    history:                 2024-04-01T18:19 GRIB to CDM+CF via cfgrib-0.9.1...

xarray.Dataset

• Dimensions:
  • number: 21
  • latitude: 181
  • longitude: 360
• Coordinates: (7)
  • number
    (number)
    int64
    20 19 18 17 16 15 ... 5 4 3 2 1 0

    long_name :

    ensemble member numerical id

    units :

    1

    standard_name :

    realization

    array([20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10,  9,  8,  7,  6,  5,  4,  3,
            2,  1,  0])

  • time
    ()
    datetime64[ns]
    2020-08-31

    long_name :

    initial time of forecast

    standard_name :

    forecast_reference_time

    array('2020-08-31T00:00:00.000000000', dtype='datetime64[ns]')

  • step
    ()
    timedelta64[ns]
    15 days 12:00:00

    long_name :

    time since forecast_reference_time

    standard_name :

    forecast_period

    array(1339200000000000, dtype='timedelta64[ns]')

  • heightAboveGround
    ()
    float64
    2.0

    long_name :

    height above the surface

    units :

    m

    positive :

    up

    standard_name :

    height

    array(2.)

  • latitude
    (latitude)
    float64
    90.0 89.0 88.0 ... -89.0 -90.0

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    stored_direction :

    decreasing

    array([ 90.,  89.,  88.,  87.,  86.,  85.,  84.,  83.,  82.,  81.,  80.,  79.,
            78.,  77.,  76.,  75.,  74.,  73.,  72.,  71.,  70.,  69.,  68.,  67.,
            66.,  65.,  64.,  63.,  62.,  61.,  60.,  59.,  58.,  57.,  56.,  55.,
            54.,  53.,  52.,  51.,  50.,  49.,  48.,  47.,  46.,  45.,  44.,  43.,
            42.,  41.,  40.,  39.,  38.,  37.,  36.,  35.,  34.,  33.,  32.,  31.,
            30.,  29.,  28.,  27.,  26.,  25.,  24.,  23.,  22.,  21.,  20.,  19.,
            18.,  17.,  16.,  15.,  14.,  13.,  12.,  11.,  10.,   9.,   8.,   7.,
             6.,   5.,   4.,   3.,   2.,   1.,   0.,  -1.,  -2.,  -3.,  -4.,  -5.,
            -6.,  -7.,  -8.,  -9., -10., -11., -12., -13., -14., -15., -16., -17.,
           -18., -19., -20., -21., -22., -23., -24., -25., -26., -27., -28., -29.,
           -30., -31., -32., -33., -34., -35., -36., -37., -38., -39., -40., -41.,
           -42., -43., -44., -45., -46., -47., -48., -49., -50., -51., -52., -53.,
           -54., -55., -56., -57., -58., -59., -60., -61., -62., -63., -64., -65.,
           -66., -67., -68., -69., -70., -71., -72., -73., -74., -75., -76., -77.,
           -78., -79., -80., -81., -82., -83., -84., -85., -86., -87., -88., -89.,
           -90.])

  • longitude
    (longitude)
    float64
    0.0 1.0 2.0 ... 357.0 358.0 359.0

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    array([  0.,   1.,   2., ..., 357., 358., 359.])

  • valid_time
    ()
    datetime64[ns]
    2020-09-15T12:00:00

    standard_name :

    time

    long_name :

    time

    array('2020-09-15T12:00:00.000000000', dtype='datetime64[ns]')

• Data variables: (1)
  • t2m
    (number, latitude, longitude)
    float32
    dask.array<chunksize=(1, 181, 360), meta=np.ndarray>

    GRIB_paramId :

    167

    GRIB_dataType :

    pf

    GRIB_numberOfPoints :

    65160

    GRIB_typeOfLevel :

    heightAboveGround

    GRIB_stepUnits :

    1

    GRIB_stepType :

    instant

    GRIB_gridType :

    regular_ll

    GRIB_NV :

    0

    GRIB_Nx :

    360

    GRIB_Ny :

    181

    GRIB_cfName :

    air_temperature

    GRIB_cfVarName :

    t2m

    GRIB_gridDefinitionDescription :

    Latitude/longitude. Also called equidistant cylindrical, or Plate Carree

    GRIB_iDirectionIncrementInDegrees :

    1.0

    GRIB_iScansNegatively :

    0

    GRIB_jDirectionIncrementInDegrees :

    1.0

    GRIB_jPointsAreConsecutive :

    0

    GRIB_jScansPositively :

    0

    GRIB_latitudeOfFirstGridPointInDegrees :

    90.0

    GRIB_latitudeOfLastGridPointInDegrees :

    -90.0

    GRIB_longitudeOfFirstGridPointInDegrees :

    0.0

    GRIB_longitudeOfLastGridPointInDegrees :

    359.0

    GRIB_missingValue :

    3.4028234663852886e+38

    GRIB_name :

    2 metre temperature

    GRIB_shortName :

    2t

    GRIB_totalNumber :

    20

    GRIB_units :

    K

    long_name :

    2 metre temperature

    units :

    K

    standard_name :

    air_temperature

    Array Chunk Bytes 5.22 MiB 254.53 kiB Shape (21, 181, 360) (1, 181, 360) Dask graph 21 chunks in 64 graph layers Data type float32 numpy.ndarray

• Indexes: (3)
  • number
    PandasIndex

    PandasIndex(Index([20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1,
           0],
          dtype='int64', name='number'))

  • latitude
    PandasIndex

    PandasIndex(Index([ 90.0,  89.0,  88.0,  87.0,  86.0,  85.0,  84.0,  83.0,  82.0,  81.0,
           ...
           -81.0, -82.0, -83.0, -84.0, -85.0, -86.0, -87.0, -88.0, -89.0, -90.0],
          dtype='float64', name='latitude', length=181))

  • longitude
    PandasIndex

    PandasIndex(Index([  0.0,   1.0,   2.0,   3.0,   4.0,   5.0,   6.0,   7.0,   8.0,   9.0,
           ...
           350.0, 351.0, 352.0, 353.0, 354.0, 355.0, 356.0, 357.0, 358.0, 359.0],
          dtype='float64', name='longitude', length=360))

• Attributes: (7)

  GRIB_edition :

  2

  GRIB_centre :

  kwbc

  GRIB_centreDescription :

  US National Weather Service - NCEP

  GRIB_subCentre :

  2

  Conventions :

  CF-1.7

  institution :

  US National Weather Service - NCEP

  history :

  2024-04-01T18:19 GRIB to CDM+CF via cfgrib-0.9.10.4/ecCodes-2.34.1 with {"source": "my_caching_folder/gens-a_3_20200831_0000_372_20.grb2", "filter_by_keys": {"typeOfLevel": "heightAboveGround", "cfVarName": "t2m"}, "encode_cf": ["parameter", "time", "geography", "vertical"]}