Temperature Animation

Create an animation of the surface temperature across the entire AUS2200 domain for a complete simulation in parallel using Xarray.

import xarray as xr
from glob import glob
from dask.distributed import Client
import matplotlib.pyplot as plt
import pandas as pd

file_list = sorted(glob('/g/data/gx60/dr4292/experiments/ECL-shifted-evolvingSST/netcdf/u-cs142-20160603T0000-continuous-140E-evolvingSST/*/aus2200/d0198/RA3/um/umnsa_spec*'))

client=Client()
client

2024-06-11 12:23:33,952 - distributed.preloading - INFO - Creating preload: /g/data/hh5/public/apps/dask-optimiser/schedplugin.py
2024-06-11 12:23:33,984 - distributed.utils - INFO - Reload module schedplugin from .py file
2024-06-11 12:23:34,048 - distributed.preloading - INFO - Import preload module: /g/data/hh5/public/apps/dask-optimiser/schedplugin.py

Modifying workers

Client

Client-9a08e09b-2799-11ef-a20f-00000813fe80

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: /node/gadi-cpu-spr-0098.gadi.nci.org.au/52617/proxy/8787/status

Launch dashboard in JupyterLab

Cluster Info

LocalCluster

95fafa1a

Dashboard: /node/gadi-cpu-spr-0098.gadi.nci.org.au/52617/proxy/8787/status	Workers: 4
Total threads: 4	Total memory: 0 B
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-0f745ce2-8ff3-47d4-82bb-54d95d456290

Comm: tcp://127.0.0.1:35419	Workers: 4
Dashboard: /node/gadi-cpu-spr-0098.gadi.nci.org.au/52617/proxy/8787/status	Total threads: 4
Started: Just now	Total memory: 0 B

Workers

Worker: 0

Comm: tcp://127.0.0.1:39799	Total threads: 1
Dashboard: /node/gadi-cpu-spr-0098.gadi.nci.org.au/52617/proxy/35585/status	Memory: 0 B
Nanny: tcp://127.0.0.1:45567
Local directory: /jobfs/117857481.gadi-pbs/dask-scratch-space/worker-svqrc4hr

Worker: 1

Comm: tcp://127.0.0.1:40109	Total threads: 1
Dashboard: /node/gadi-cpu-spr-0098.gadi.nci.org.au/52617/proxy/41377/status	Memory: 0 B
Nanny: tcp://127.0.0.1:32889
Local directory: /jobfs/117857481.gadi-pbs/dask-scratch-space/worker-1vri8v2j

Worker: 2

Comm: tcp://127.0.0.1:43757	Total threads: 1
Dashboard: /node/gadi-cpu-spr-0098.gadi.nci.org.au/52617/proxy/33443/status	Memory: 0 B
Nanny: tcp://127.0.0.1:34421
Local directory: /jobfs/117857481.gadi-pbs/dask-scratch-space/worker-8kub71ir

Worker: 3

Comm: tcp://127.0.0.1:46723	Total threads: 1
Dashboard: /node/gadi-cpu-spr-0098.gadi.nci.org.au/52617/proxy/41219/status	Memory: 0 B
Nanny: tcp://127.0.0.1:41795
Local directory: /jobfs/117857481.gadi-pbs/dask-scratch-space/worker-bp1yuo9k

mf_ds = xr.open_mfdataset(file_list,preprocess=lambda x: x['fld_s00i024'],parallel=True,chunks={'lat':-1,'lon':-1})

mf_ds

<xarray.Dataset> Size: 16GB
Dimensions:                   (time: 720, lat: 2120, lon: 2600)
Coordinates:
  * time                      (time) datetime64[ns] 6kB 2016-06-03T00:10:00 ....
  * lat                       (lat) float64 17kB -48.79 -48.77 ... -6.852 -6.832
  * lon                       (lon) float64 21kB 114.3 114.3 ... 165.7 165.7
    theta_level_height        float64 8B 5.0
    model_theta_level_number  int32 4B 1
    sigma_theta               float64 8B 0.9994
    height                    float64 8B 10.0
    height_0                  float64 8B 1.5
Data variables:
    fld_s00i024               (time, lat, lon) float32 16GB dask.array<chunksize=(1, 2120, 2600), meta=np.ndarray>

xarray.Dataset

• Dimensions:
  • time: 720
  • lat: 2120
  • lon: 2600
• Coordinates: (8)
  • time
    (time)
    datetime64[ns]
    2016-06-03T00:10:00 ... 2016-06-08

    axis :

    T

    standard_name :

    time

    array(['2016-06-03T00:10:00.000000000', '2016-06-03T00:19:59.999999744',
           '2016-06-03T00:30:00.000000000', ..., '2016-06-07T23:40:00.000000256',
           '2016-06-07T23:50:00.000000000', '2016-06-08T00:00:00.000000000'],
          dtype='datetime64[ns]')

  • lat
    (lat)
    float64
    -48.79 -48.77 ... -6.852 -6.832

    axis :

    Y

    bounds :

    lat_bnds

    units :

    degrees_north

    standard_name :

    latitude

    array([-48.787998, -48.7682  , -48.748398, ...,  -6.871399,  -6.851597,
            -6.831799])

  • lon
    (lon)
    float64
    114.3 114.3 114.3 ... 165.7 165.7

    axis :

    X

    bounds :

    lon_bnds

    units :

    degrees_east

    standard_name :

    longitude

    array([114.260002, 114.2798  , 114.299599, ..., 165.680603, 165.700409,
           165.7202  ])

  • theta_level_height
    ()
    float64
    5.0

    bounds :

    theta_level_height_bnds

    units :

    m

    long_name :

    level_height

    positive :

    up

    array(5.)

  • model_theta_level_number
    ()
    int32
    1

    units :

    1

    standard_name :

    model_level_number

    positive :

    up

    array(1, dtype=int32)

  • sigma_theta
    ()
    float64
    0.9994

    bounds :

    sigma_theta_bnds

    units :

    1

    long_name :

    sigma

    array(0.99942381)

  • height
    ()
    float64
    10.0

    units :

    m

    standard_name :

    height

    positive :

    up

    array(10.)

  • height_0
    ()
    float64
    1.5

    units :

    m

    standard_name :

    height

    positive :

    up

    array(1.5)

• Data variables: (1)
  • fld_s00i024
    (time, lat, lon)
    float32
    dask.array<chunksize=(1, 2120, 2600), meta=np.ndarray>

    standard_name :

    surface_temperature

    long_name :

    SURFACE TEMPERATURE AFTER TIMESTEP

    units :

    K

    um_stash_source :

    m01s00i024

    grid_mapping :

    latitude_longitude

    Array Chunk Bytes 14.78 GiB 21.03 MiB Shape (720, 2120, 2600) (1, 2120, 2600) Dask graph 720 chunks in 241 graph layers Data type float32 numpy.ndarray

• Indexes: (3)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex([          '2016-06-03 00:10:00',
                   '2016-06-03 00:19:59.999999744',
                             '2016-06-03 00:30:00',
                             '2016-06-03 00:40:00',
                             '2016-06-03 00:50:00',
                             '2016-06-03 01:00:00',
                   '2016-06-03 01:10:00.000000256',
                             '2016-06-03 01:20:00',
                             '2016-06-03 01:30:00',
                             '2016-06-03 01:40:00',
                   ...
                             '2016-06-07 22:30:00',
                             '2016-06-07 22:40:00',
                   '2016-06-07 22:49:59.999999744',
                             '2016-06-07 23:00:00',
                             '2016-06-07 23:10:00',
                             '2016-06-07 23:20:00',
                             '2016-06-07 23:30:00',
                   '2016-06-07 23:40:00.000000256',
                             '2016-06-07 23:50:00',
                             '2016-06-08 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=720, freq=None))

  • lat
    PandasIndex

    PandasIndex(Index([ -48.78799819946289,   -48.7681999206543,  -48.74839782714844,
           -48.728599548339844, -48.708797454833984,  -48.68899917602539,
            -48.66919708251953,  -48.64939880371094,  -48.62959671020508,
           -48.609798431396484,
           ...
            -7.009998321533203,  -6.990200042724609,   -6.97039794921875,
            -6.950599670410156,  -6.930797576904297,  -6.910999298095703,
            -6.891197204589844,   -6.87139892578125,  -6.851596832275391,
            -6.831798553466797],
          dtype='float64', name='lat', length=2120))

  • lon
    PandasIndex

    PandasIndex(Index([114.26000213623047, 114.27980041503906, 114.29959869384766,
           114.31940460205078, 114.33920288085938, 114.35900115966797,
           114.37879943847656, 114.39860534667969, 114.41840362548828,
           114.43820190429688,
           ...
           165.54200744628906, 165.56179809570312, 165.58160400390625,
           165.60140991210938, 165.62120056152344, 165.64100646972656,
           165.66079711914062, 165.68060302734375, 165.70040893554688,
           165.72019958496094],
          dtype='float64', name='lon', length=2600))

• Attributes: (0)

def make_frame(f,ts,i,lat,lon):
    fig = plt.figure()
    ax=fig.add_subplot(111)
    pc = ax.pcolormesh(lon,lat,f - 273.15,shading='nearest',vmin=-5,vmax=45)
    fig.colorbar(pc,label="Surface Temperature (°C)")
    t = pd.to_datetime(ts)
    plt.title(str(ts))
    plt.savefig(f'/home/563/dr4292/animation/frame{i:04}.png')
    return i

out = xr.apply_ufunc(make_frame,
    mf_ds['fld_s00i024'],
    mf_ds['fld_s00i024'].time,
    xr.DataArray([i for i in range(720)],{'time':mf_ds.time}),
    mf_ds['fld_s00i024'].lat,
    mf_ds['fld_s00i024'].lon,
    input_core_dims=[['lat','lon'],[],[],['lat',],['lon',]],
    exclude_dims=set(("lat","lon")),
    vectorize=True,
    dask="parallelized"
)
#for i,f in enumerate(mf_ds['fld_s00i024']):
#    f.plot(vmin=260, vmax=350)
#    plt.savefig(f'/home/563/dr4292/animation/frame{i:04}.png')
#    plt.close()

out.compute()

!ffmpeg -framerate 10 -pattern_type glob -i '/home/563/dr4292/animation/frame*.png' -c:v libx264 -r 30 -pix_fmt yuv420p output.mp4