Running the seasonal forecast

This page provides documentation on how to run the NorCPM seasonal forecast. In general there are four steps to this process:

  1. Update the input data for assimilation

  2. Run forward the analysis to the present day, using new observational data

  3. Run the prediction

  4. Post-processing

The forecast should always be provided on the evening of the 14th of the month, as our collaborators need it on the 15th.

The Met Office provides monthly updates to EN4.2.2. observational temperature and salinity profiles, and these updates happen on the 9th-17th of each month. If this update can be included into the assimilation then this is good, as it gives us an additional month of temperature and salinity data to assimilate. You can check when the last update was on the Met Office site (see bottom of page, or look at file links). I would recommend running the analysis on the 12th/13th of the month. It takes ~3 hours to run. I would then recommend running the prediction on the 12/13th, this takes ~2 hours. The post processing takes another ~2 hours. Therefore, if there are no issues the whole workflow takes around 1 working day, but doing this on the 12/13th allows for potential issues (e.g. machine instability) to be addressed. It is important to keep an eye out for Betzy downtime announcements in the week preceding the 15th to ensure there is no significant disruption.

1 Update the data for assimilation

The first step in running the seasonal forecast is to update the observational data that is assimilated into the analysis. The steps below show how it is done for the forecast initialized in December 2024 on Betzy.

Update the temperature and salinity profile data

cd /cluster/projects/nn9039k/inputdata/obs/TEM/EN422

Download the temperature and salinity profiles from the UK Met Office (https://www.metoffice.gov.uk/hadobs/en4/download-en4-2-2.html).

./retrieveEN4.sh

Update NOAA OISST ver.2.1 data for the two previous months

If the daily OISST data server (https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/) is available:

cd /cluster/projects/nn9039k/inputdata/obs/SST/NOAA/hires

Download daily OISST high resolution data from NOAA.

./download_oisst_hires.sh 2024 10 2024 11

Load the shared conda environment on Betzy.

source /cluster/projects/nn9039k/people/pgchiu/conda/py3env.sh

Finally, compute monthly mean.

./interpolate_oisst_hires.sh 2024 10 2024 11

Note

Alternatively, if the daily OISST data is not available: Download the yearly OISST high resolution data from the server (https://psl.noaa.gov/thredds/catalog/Datasets/noaa.oisst.v2.highres/catalog.html).

cd /cluster/projects/nn9039k/inputdata/obs/SST/NOAA/hires_v2

Download the yearly OISST data.

./download_oisst_hires_v2.sh 2024

After the yearly OISST high resolution data is downloaded, the daily data needs to be extracted. Load the shared conda environment.

source /cluster/projects/nn9039k/people/pgchiu/conda/py3env.sh

Run a python script to extract daily OISST from one yearly NetCFD file.

python extract_daily_oisst.py 202410 202411

Finally, compute monthly mean.

./interpolate_oisst_hires_v2.sh 2024 10 2024 11

Now the SST, temperature and salinity profiles are updated.

When it comes to running the analysis, it is important to note that some of the temperature and salinity profiles are initially uploaded by the Met Office as ‘preliminary’, and then are replaced later by ‘final’ files.

Similarly, daily OISST high resolution data are initially uploaded by NOAA as ‘preliminary’ (e.g., oisst-avhrr-v02r01.20241031_preliminary.nc), and then are replaced later by final’ files (e.g., oisst-avhrr-v02r01.20241031.nc).

We always want to run the analysis with the best possible data, which means that each month, we will re-run the part of the analysis that used ‘preliminary’ data last month. This means that it is important to keep track of what the latest observational files are each month.

Note that the data servers are sometimes unavailable due to maintenance downtime or capacity problems. You need to try again later.

2 Install NorCPM forecast system

Create a personal folder in nn9039k on Betzy.

mkdir -p /cluster/projects/nn9039k/people/$USER
cd /cluster/projects/nn9039k/people/$USER

Download the code with git clone.

git clone ssh://git@github.com/NorESMhub/NorCPM.git norcpm-cf

3 Create workflow script

We recommend creating a workflow Shell Script under norcpm-cf/setup/noresm1 so that you don’t need to change arguments or edit the setup files whenever you run simulations.

cd /cluster/projects/nn9039k/people/$USER/norcpm-cf/setup/noresm1

Copy the code below into the workflow script.

workflow_seasonal_forecast.sh:

#!/bin/sh -e
: ${INIDATE_ANALYSIS:=2024-08-15} #<--change the date!!! e.g., 2024-08-15 for Dec forecast
: ${INIDATE_FORECAST:=2024-11-15} #<--change the date!! e.g., 2024-11-15 for Dec forecast
: ${CHMOD_DATE:=20241115} #<--change the date!! e.g., 20241115 for Dec forecast
: ${SETTING_FILE_ANALYSIS:=norcpm-cf-system1_assim_19811115_continue20240815.sh}
: ${SETTING_FILE_FORECAST:=norcpm-cf-system1_hindcast_20230415.sh}
: ${ACCOUNT:=nn9873k} # nn9873k: Climate Futures; nn9039k: BCPU

export INIDATE_ANALYSIS INIDATE_FORECAST CHMOD_DATE SETTING_FILE_ANALYSIS SETTING_FILE_FORECAST ACCOUNT

### Analysis experiment ###

if [[ $1 && $1 == "create_analysis" ]]
then
      echo create analysis experiment
      ./create_ensemble.sh $SETTING_FILE_ANALYSIS REF_DATES=$INIDATE_ANALYSIS
fi

if [[ $1 && $1 == "run_analysis_stage1" ]] # propagate NorCPM for 3 months (skip first assimilation)
then
      echo submit analysis experiment stage 1
      ./submit_ensemble.sh $SETTING_FILE_ANALYSIS ACCOUNT=$ACCOUNT RESTART=2 SKIP_ASSIM_START=1 SKIP_ASSIM_FIRST=1
fi

if [[ $1 && $1 == "run_analysis_stage2" ]] # propagate NorCPM for another month, only assimilating SST
then
      echo submit analysis experiment stage 2
      ./submit_ensemble.sh $SETTING_FILE_ANALYSIS WALLTIME='01:00:00' ACCOUNT=$ACCOUNT RESTART=0 SKIP_ASSIM_START=0 SKIP_ASSIM_FIRST=0 OBSLIST=SST PRODUCERLIST=NOAA REF_PERIODLIST=1982-2016 COMBINE_ASSIM=1
fi

if [[ $1 && $1 == "backup_analysis" ]]
then
      echo backup analysis
      rsync -uav /cluster/work/users/$USER/archive/norcpm-cf-system1_assim/norcpm-cf-system1_assim_19811115 /nird/datalake/NS9873K/norcpm/raw/norcpm-cf-system1/norcpm-cf-system1_assim/
fi


### Prediction ###

if [[ $1 && $1 == "setup_forecast" ]]
then
      echo create forecast
      ./create_ensemble.sh $SETTING_FILE_FORECAST START_DATE=$INIDATE_FORECAST REF_DATES=$INIDATE_FORECAST
fi

if [[ $1 && $1 == "run_forecast" ]]
then
      echo run forecast
      ./submit_ensemble.sh $SETTING_FILE_FORECAST START_DATE=$INIDATE_FORECAST REF_DATES=$INIDATE_FORECAST ACCOUNT=$ACCOUNT
fi

if [[ $1 && $1 == "merge_forecast" ]]
then
      echo merge forecast
      sbatch ../../tools/mergediag/mergediag.betzy.sh /cluster/work/users/$USER/archive/norcpm-cf-system1_hindcast/norcpm-cf-system1_hindcast_`echo $INIDATE_FORECAST | sed 's/-//g'`
fi

if [[ $1 && $1 == "backup_forecast" ]]
then
      echo backup forecast
      mkdir -p /nird/datalake/NS9873K/norcpm/raw/norcpm-cf-system1/norcpm-cf-system1_hindcast/norcpm-cf-system1_hindcast_`echo $INIDATE_FORECAST | sed 's/-//g'`
      rsync --info=progress2 -uav /cluster/work/users/$USER/archive/norcpm-cf-system1_hindcast/norcpm-cf-system1_hindcast_`echo $INIDATE_FORECAST | sed 's/-//g'`/norcpm-cf-system1_hindcast_`echo $INIDATE_FORECAST | sed 's/-//g'`_mem01-60 /nird/datalake/NS9873K/norcpm/raw/norcpm-cf-system1/norcpm-cf-system1_hindcast/norcpm-cf-system1_hindcast_`echo $INIDATE_FORECAST | sed 's/-//g'`/
      chmod -R go+rx /nird/datalake/NS9873K/norcpm/raw/norcpm-cf-system1/norcpm-cf-system1_hindcast/norcpm-cf-system1_hindcast_$CHMOD_DATE
fi

Note

Note that INIDATE_ANALYSIS, INIDATE_FORECAST, CHMOD_DATE need to be updated in the script before you start a new experiment in the next month.

4 Create analysis experiment (if not exist)

Create the analysis experiment if it does not exist.

cd /cluster/projects/nn9039k/people/$USER/norcpm-cf/setup/noresm1

We recommend starting tmux before starting the setup because it may take much time to set up the experiment.

tmux
./workflow_seasonal_forecast.sh create_analysis

5 Run the analysis

Run the analysis for the step1. This propagates NorCPM for 3 months and skip first assimilation.

cd /cluster/projects/nn9039k/people/$USER/norcpm-cf/setup/noresm1
./workflow_seasonal_forecast.sh run_analysis_stage1

Run the analysis for the step2. This propagates NorCPM for another month, but only assimilates SST.

cd /cluster/projects/nn9039k/people/$USER/norcpm-cf/setup/noresm1
./workflow_seasonal_forecast.sh run_analysis_stage2

Monitor the job status of your job.

squeue -u $USER

6 Backup output from the analysis

Back up the output from the analysis experiment, after the analysis experiment is completed.

cd /cluster/projects/nn9039k/people/$USER/norcpm-cf/setup/noresm1

We recommend using tmux because it may take much time to back up the output.

tmux
./workflow_seasonal_forecast.sh backup_analysis

7 Create prediction

Create the prediction.

cd /cluster/projects/nn9039k/people/$USER/norcpm-cf/setup/noresm1

We recommend starting tmux before starting the setup because it may take much time to set up the experiment.

tmux
./workflow_seasonal_forecast.sh setup_forecast

8 Run the prediction

Run the prediction.

cd /cluster/projects/nn9039k/people/$USER/norcpm-cf/setup/noresm1
./workflow_seasonal_forecast.sh run_forecast

Monitor the job status of your job.

squeue -u $USER

9 Merge output from the prediction

Merge the output from the prediction.

cd /cluster/projects/nn9039k/people/$USER/norcpm-cf/setup/noresm1
./workflow_seasonal_forecast.sh merge_forecast

Monitor the job status of your job.

squeue -u $USER

10 Backup output from the prediction

Back up the output from the prediction, after the prediction is completed.

cd /cluster/projects/nn9039k/people/$USER/norcpm-cf/setup/noresm1

We recommend using tmux because it may take much time to back up the output.

tmux
./workflow_seasonal_forecast.sh backup_forecast