1-day rain plots

Plots daily rainfall over 5 days of the East Coast Low AUS2200 simulation.

from dask.distributed import Client
from glob import glob
import xarray as xr
import pandas

flist=sorted(glob('/g/data/v45/cc6171/experiments/ECL-2016_smallshift/netcdf/u-cs142/2016060*T*Z/aus2200/d0198/RA3/um/umnsa_spec_*.nc'))

client=Client()

/g/data/hh5/public/apps/cms_conda/envs/analysis3-24.04/lib/python3.10/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 37389 instead
  warnings.warn(

Modifying workers

/g/data/hh5/public/apps/dask-optimiser/schedplugin.py:160: UserWarning: Attempted illegal binding: []
  warn(f"Attempted illegal binding: {(sorted(list(self.slots_available - taken_slots))[:self.worker.state.nthreads])}")
/g/data/hh5/public/apps/dask-optimiser/schedplugin.py:160: UserWarning: Attempted illegal binding: []
  warn(f"Attempted illegal binding: {(sorted(list(self.slots_available - taken_slots))[:self.worker.state.nthreads])}")
/g/data/hh5/public/apps/dask-optimiser/schedplugin.py:160: UserWarning: Attempted illegal binding: []
  warn(f"Attempted illegal binding: {(sorted(list(self.slots_available - taken_slots))[:self.worker.state.nthreads])}")
/g/data/hh5/public/apps/dask-optimiser/schedplugin.py:160: UserWarning: Attempted illegal binding: []
  warn(f"Attempted illegal binding: {(sorted(list(self.slots_available - taken_slots))[:self.worker.state.nthreads])}")

client

Client

Client-967d2a8d-216a-11ef-b858-000007affe80

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: /node/gadi-cpu-spr-0218.gadi.nci.org.au/13627/proxy/37389/status

Launch dashboard in JupyterLab

Cluster Info

LocalCluster

b6cb1c90

Dashboard: /node/gadi-cpu-spr-0218.gadi.nci.org.au/13627/proxy/37389/status	Workers: 4
Total threads: 4	Total memory: 0 B
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-121cf7f7-38f5-4b1f-9e25-ba7c2f564c34

Comm: tcp://127.0.0.1:42913	Workers: 4
Dashboard: /node/gadi-cpu-spr-0218.gadi.nci.org.au/13627/proxy/37389/status	Total threads: 4
Started: Just now	Total memory: 0 B

Workers

Worker: 0

Comm: tcp://127.0.0.1:41847	Total threads: 1
Dashboard: /node/gadi-cpu-spr-0218.gadi.nci.org.au/13627/proxy/33621/status	Memory: 0 B
Nanny: tcp://127.0.0.1:40577
Local directory: /jobfs/117041814.gadi-pbs/dask-scratch-space/worker-51zs6anx

Worker: 1

Comm: tcp://127.0.0.1:36723	Total threads: 1
Dashboard: /node/gadi-cpu-spr-0218.gadi.nci.org.au/13627/proxy/33947/status	Memory: 0 B
Nanny: tcp://127.0.0.1:37665
Local directory: /jobfs/117041814.gadi-pbs/dask-scratch-space/worker-i6bhti7j

Worker: 2

Comm: tcp://127.0.0.1:45801	Total threads: 1
Dashboard: /node/gadi-cpu-spr-0218.gadi.nci.org.au/13627/proxy/33645/status	Memory: 0 B
Nanny: tcp://127.0.0.1:35097
Local directory: /jobfs/117041814.gadi-pbs/dask-scratch-space/worker-aabzzg64

Worker: 3

Comm: tcp://127.0.0.1:39321	Total threads: 1
Dashboard: /node/gadi-cpu-spr-0218.gadi.nci.org.au/13627/proxy/46389/status	Memory: 0 B
Nanny: tcp://127.0.0.1:35941
Local directory: /jobfs/117041814.gadi-pbs/dask-scratch-space/worker-ursgrl9a

ds = xr.open_mfdataset(flist,parallel=True,preprocess=lambda x: x['fld_s04i201'] + x['fld_s04i202'])

ds

<xarray.DataArray (time_0: 720, lat: 2120, lon: 2600)> Size: 16GB
dask.array<concatenate, shape=(720, 2120, 2600), dtype=float32, chunksize=(2, 1060, 1300), chunktype=numpy.ndarray>
Coordinates:
  * lat                       (lat) float64 17kB -48.79 -48.77 ... -6.852 -6.832
  * lon                       (lon) float64 21kB 114.3 114.3 ... 165.7 165.7
    theta_level_height        float64 8B 5.0
    model_theta_level_number  int32 4B 1
    sigma_theta               float64 8B 0.9994
    height                    float64 8B 10.0
    height_0                  float64 8B 1.5
  * time_0                    (time_0) datetime64[ns] 6kB 2016-06-03T00:05:00...

xarray.DataArray

• time_0: 720
• lat: 2120
• lon: 2600

• dask.array<chunksize=(2, 1060, 1300), meta=np.ndarray>

  Array Chunk Bytes 14.78 GiB 10.51 MiB Shape (720, 2120, 2600) (2, 1060, 1300) Dask graph 1440 chunks in 601 graph layers Data type float32 numpy.ndarray

• Coordinates: (8)
  • lat
    (lat)
    float64
    -48.79 -48.77 ... -6.852 -6.832

    axis :

    Y

    bounds :

    lat_bnds

    units :

    degrees_north

    standard_name :

    latitude

    array([-48.787998, -48.7682  , -48.748398, ...,  -6.871399,  -6.851597,
            -6.831799])

  • lon
    (lon)
    float64
    114.3 114.3 114.3 ... 165.7 165.7

    axis :

    X

    bounds :

    lon_bnds

    units :

    degrees_east

    standard_name :

    longitude

    array([114.260002, 114.2798  , 114.299599, ..., 165.680603, 165.700409,
           165.7202  ])

  • theta_level_height
    ()
    float64
    5.0

    bounds :

    theta_level_height_bnds

    units :

    m

    long_name :

    level_height

    positive :

    up

    array(5.)

  • model_theta_level_number
    ()
    int32
    1

    units :

    1

    standard_name :

    model_level_number

    positive :

    up

    array(1, dtype=int32)

  • sigma_theta
    ()
    float64
    0.9994

    bounds :

    sigma_theta_bnds

    units :

    1

    long_name :

    sigma

    array(0.99942381)

  • height
    ()
    float64
    10.0

    units :

    m

    standard_name :

    height

    positive :

    up

    array(10.)

  • height_0
    ()
    float64
    1.5

    units :

    m

    standard_name :

    height

    positive :

    up

    array(1.5)

  • time_0
    (time_0)
    datetime64[ns]
    2016-06-03T00:05:00 ... 2016-06-...

    axis :

    T

    bounds :

    time_0_bnds

    standard_name :

    time

    array(['2016-06-03T00:05:00.000000000', '2016-06-03T00:15:00.000000000',
           '2016-06-03T00:24:59.999999744', ..., '2016-06-07T23:35:00.000000256',
           '2016-06-07T23:45:00.000000000', '2016-06-07T23:55:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Index([ -48.78799819946289,   -48.7681999206543,  -48.74839782714844,
           -48.728599548339844, -48.708797454833984,  -48.68899917602539,
            -48.66919708251953,  -48.64939880371094,  -48.62959671020508,
           -48.609798431396484,
           ...
            -7.009998321533203,  -6.990200042724609,   -6.97039794921875,
            -6.950599670410156,  -6.930797576904297,  -6.910999298095703,
            -6.891197204589844,   -6.87139892578125,  -6.851596832275391,
            -6.831798553466797],
          dtype='float64', name='lat', length=2120))

  • lon
    PandasIndex

    PandasIndex(Index([114.26000213623047, 114.27980041503906, 114.29959869384766,
           114.31940460205078, 114.33920288085938, 114.35900115966797,
           114.37879943847656, 114.39860534667969, 114.41840362548828,
           114.43820190429688,
           ...
           165.54200744628906, 165.56179809570312, 165.58160400390625,
           165.60140991210938, 165.62120056152344, 165.64100646972656,
           165.66079711914062, 165.68060302734375, 165.70040893554688,
           165.72019958496094],
          dtype='float64', name='lon', length=2600))

  • time_0
    PandasIndex

    PandasIndex(DatetimeIndex([          '2016-06-03 00:05:00',
                             '2016-06-03 00:15:00',
                   '2016-06-03 00:24:59.999999744',
                   '2016-06-03 00:35:00.000000256',
                             '2016-06-03 00:45:00',
                   '2016-06-03 00:54:59.999999744',
                   '2016-06-03 01:05:00.000000256',
                             '2016-06-03 01:15:00',
                             '2016-06-03 01:25:00',
                             '2016-06-03 01:35:00',
                   ...
                             '2016-06-07 22:25:00',
                             '2016-06-07 22:35:00',
                             '2016-06-07 22:45:00',
                   '2016-06-07 22:54:59.999999744',
                   '2016-06-07 23:05:00.000000256',
                             '2016-06-07 23:15:00',
                   '2016-06-07 23:24:59.999999744',
                   '2016-06-07 23:35:00.000000256',
                             '2016-06-07 23:45:00',
                             '2016-06-07 23:55:00'],
                  dtype='datetime64[ns]', name='time_0', length=720, freq=None))

• Attributes: (0)

da_fin=ds.resample(time_0='1D',offset='9h').sum()

(ds.sel(time_0=slice(pandas.Timestamp('2016-06-03T09:00:00'),pandas.Timestamp('2016-06-04T09:00:00'))).sum(dim='time_0') - da_fin[1]).plot()

<matplotlib.collections.QuadMesh at 0x14f0440f26e0>

da_fin[0].plot()

<matplotlib.collections.QuadMesh at 0x14f04382d2a0>

da_fin[1].plot()

<matplotlib.collections.QuadMesh at 0x14f00818b760>

da_fin[2].plot()

<matplotlib.collections.QuadMesh at 0x14f0a0847b50>

da_fin[3].plot()

<matplotlib.collections.QuadMesh at 0x14f0490af880>

da_fin[4].plot()

<matplotlib.collections.QuadMesh at 0x14f0440855a0>

da_fin

<xarray.DataArray (time_0: 6, lat: 2120, lon: 2600)> Size: 132MB
dask.array<transpose, shape=(6, 2120, 2600), dtype=float32, chunksize=(1, 1060, 1300), chunktype=numpy.ndarray>
Coordinates:
  * lat                       (lat) float64 17kB -48.79 -48.77 ... -6.852 -6.832
  * lon                       (lon) float64 21kB 114.3 114.3 ... 165.7 165.7
    theta_level_height        float64 8B 5.0
    model_theta_level_number  int32 4B 1
    sigma_theta               float64 8B 0.9994
    height                    float64 8B 10.0
    height_0                  float64 8B 1.5
  * time_0                    (time_0) datetime64[ns] 48B 2016-06-02T09:00:00...

xarray.DataArray

• time_0: 6
• lat: 2120
• lon: 2600

• dask.array<chunksize=(1, 1060, 1300), meta=np.ndarray>

  Array Chunk Bytes 126.16 MiB 5.26 MiB Shape (6, 2120, 2600) (1, 1060, 1300) Dask graph 24 chunks in 635 graph layers Data type float32 numpy.ndarray

• Coordinates: (8)
  • lat
    (lat)
    float64
    -48.79 -48.77 ... -6.852 -6.832

    axis :

    Y

    bounds :

    lat_bnds

    units :

    degrees_north

    standard_name :

    latitude

    array([-48.787998, -48.7682  , -48.748398, ...,  -6.871399,  -6.851597,
            -6.831799])

  • lon
    (lon)
    float64
    114.3 114.3 114.3 ... 165.7 165.7

    axis :

    X

    bounds :

    lon_bnds

    units :

    degrees_east

    standard_name :

    longitude

    array([114.260002, 114.2798  , 114.299599, ..., 165.680603, 165.700409,
           165.7202  ])

  • theta_level_height
    ()
    float64
    5.0

    bounds :

    theta_level_height_bnds

    units :

    m

    long_name :

    level_height

    positive :

    up

    array(5.)

  • model_theta_level_number
    ()
    int32
    1

    units :

    1

    standard_name :

    model_level_number

    positive :

    up

    array(1, dtype=int32)

  • sigma_theta
    ()
    float64
    0.9994

    bounds :

    sigma_theta_bnds

    units :

    1

    long_name :

    sigma

    array(0.99942381)

  • height
    ()
    float64
    10.0

    units :

    m

    standard_name :

    height

    positive :

    up

    array(10.)

  • height_0
    ()
    float64
    1.5

    units :

    m

    standard_name :

    height

    positive :

    up

    array(1.5)

  • time_0
    (time_0)
    datetime64[ns]
    2016-06-02T09:00:00 ... 2016-06-...

    array(['2016-06-02T09:00:00.000000000', '2016-06-03T09:00:00.000000000',
           '2016-06-04T09:00:00.000000000', '2016-06-05T09:00:00.000000000',
           '2016-06-06T09:00:00.000000000', '2016-06-07T09:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Index([ -48.78799819946289,   -48.7681999206543,  -48.74839782714844,
           -48.728599548339844, -48.708797454833984,  -48.68899917602539,
            -48.66919708251953,  -48.64939880371094,  -48.62959671020508,
           -48.609798431396484,
           ...
            -7.009998321533203,  -6.990200042724609,   -6.97039794921875,
            -6.950599670410156,  -6.930797576904297,  -6.910999298095703,
            -6.891197204589844,   -6.87139892578125,  -6.851596832275391,
            -6.831798553466797],
          dtype='float64', name='lat', length=2120))

  • lon
    PandasIndex

    PandasIndex(Index([114.26000213623047, 114.27980041503906, 114.29959869384766,
           114.31940460205078, 114.33920288085938, 114.35900115966797,
           114.37879943847656, 114.39860534667969, 114.41840362548828,
           114.43820190429688,
           ...
           165.54200744628906, 165.56179809570312, 165.58160400390625,
           165.60140991210938, 165.62120056152344, 165.64100646972656,
           165.66079711914062, 165.68060302734375, 165.70040893554688,
           165.72019958496094],
          dtype='float64', name='lon', length=2600))

  • time_0
    PandasIndex

    PandasIndex(DatetimeIndex(['2016-06-02 09:00:00', '2016-06-03 09:00:00',
                   '2016-06-04 09:00:00', '2016-06-05 09:00:00',
                   '2016-06-06 09:00:00', '2016-06-07 09:00:00'],
                  dtype='datetime64[ns]', name='time_0', freq='D'))

• Attributes: (0)