2.1. Land DA Workflow (Hera/Orion/Hercules/Gaea-C6)
This chapter provides instructions for building and running the Unified Forecast System (UFS) Land DA System using a Jan. 19-20, 2025 00z sample LND warmstart case using ERA5 and IMS data and the 3D-Var algorithm with the UFS Noah-MP land component and data atmosphere (DATM) component.
This case corresponds to the January 2025 Gulf Coast Blizzard, which brought unprecedented snowfall to the entire Gulf Coast. Leading up to the event, the polar vortex stretched far south and met with unusually warm Gulf waters. In response, the National Weather Service (NWS) issued a series of winter storm warnings, extreme cold warnings, and even blizzard warnings — the first ever in some areas. New Orleans, LA received a record 8 inches of snow, and the surrounding coastal areas likewise saw record-breaking snowfall and cold temperatures.
Attention
These steps are designed for use on Level 1 systems (e.g., Hera, Orion) and may require significant changes on other systems. It is recommended that users on other systems run the containerized version of Land DA. Users may reference Chapter 2.2: Containerized Land DA Workflow for instructions.
2.1.1. Create a Working Directory
Users can either create a new directory for their Land DA work (e.g., landda) or choose an existing directory, depending on preference. Then, users should navigate to this directory. For example, to create a new directory and navigate to it, run:
mkdir /path/to/landda
cd /path/to/landda
where /path/to/landda is the path to the directory where the user plans to run Land DA experiments (e.g., /Users/Joe.Schmoe/landda). In the experiment configuration file, this directory is referred to as ${exp_basedir}.
Optionally, users can save this directory path in an environment variable (e.g., $LANDDAROOT) to avoid typing out full path names later.
export LANDDAROOT=`pwd`
In this documentation, $LANDDAROOT is used, but users are welcome to choose another name for this variable if they prefer.
2.1.2. Get Code
Clone the Land DA workflow repository. To clone the develop branch, run:
git clone -b develop --recursive https://github.com/ufs-community/land-DA_workflow.git
To clone the most recent release, run the same command with release/public-v2.0.0 in place of develop:
git clone -b release/public-v2.0.0 --recursive https://github.com/ufs-community/land-DA_workflow.git
Attention
When working with a release branch, be sure to follow instructions in the release branch documentation. View the v2.0.0 release documentation here.
2.1.3. Build the Land DA System
Navigate to the
sorcdirectory.cd $LANDDAROOT/land-DA_workflow/sorcRun the build script
app_build.sh:./app_build.shUsers may need to press the
Enterkey to advance the build once the list of currently loaded modules appears. If the code successfully compiles, the console output should include:[100%] Completed 'ufs_model.fd' [100%] Built target ufs_model.fd ... exit 0
Additionally, the
execdirectory will contain the following executables:apply_incr.execalcfIMS.exetile2tile_converter.exeufs_model
2.1.4. Configure an Experiment
Several sample experiment configurations come with the Land DA System. Although this chapter outlines how to run the config.LND.era5.3dvar.ims.warmstart.yaml case, the following cases are available for use in the land-DA_workflow/parm/config_samples directory:
config.ATML.3dvar.ghcn.coldstart.yamlconfig.LND.era5.letkf.ghcn.coldstart.yamlconfig.ATML.3dvar.ghcn.warmstart.yamlconfig.LND.gswp3.3dvar.ghcn.coldstart.yamlconfig.LND.era5.3dvar.ims.warmstart.yamlconfig.LND.gswp3.letkf.ghcn.warmstart.yaml
The sample configuration files are named based on their features:
Atmospheric forcing data (
gswp3orera5) — if anyDA algorithm (
letkfor3dvar)
Users are encouraged to explore and modify the options available!
2.1.4.1. Load the Workflow Environment
To load the workflow environment, run:
cd $LANDDAROOT/land-DA_workflow
module use modulefiles
module load wflow_<platform>
conda activate land_da
where <platform> is hera, orion, or hercules.
This activates the land_da conda environment, and the user typically sees (land_da) in front of the Terminal prompt at this point.
2.1.4.2. Modify the Workflow Configuration YAML
Copy the experiment settings into config.yaml:
cd $LANDDAROOT/land-DA_workflow/parm
cp config_samples/config.LND.era5.3dvar.ims.warmstart.yaml config.yaml
Users will need to configure the account variable in config.yaml and choose an EXP_CASE_NAME if a different name for the experiment is desired:
account:A valid account name. Most NOAA RDHPCS systems require a valid account name; other systems may not (in which case, any value will do).
EXP_CASE_NAME:This variable can be changed to any name the user wants (but note that whitespace and some punctuation characters are not allowed). However, the best names will indicate useful information about the experiment. This documentation useslnd_era5_warmstart_00to indicate that it is an ERA5-LND warmstart case.
Note
To determine an appropriate account field for Level 1 systems that use the Slurm job scheduler, run saccount_params. On other systems, running groups will return a list of projects that the user has permissions for. Not all listed projects/groups have an HPC allocation, but those that do are potentially valid account names.
Users may configure other elements of an experiment in config.yaml if desired. For example, users may wish to alter DATE_FIRST_CYCLE, DATE_LAST_CYCLE, and/or DATE_CYCLE_FREQ_HR to indicate a different start cycle, end cycle, and increment. Users may also wish to change the DA algorithm from 3dvar to letkf via the JEDI_ALGORITHM variable. Users who wish to run a more complex experiment may change the values in config.yaml using information from Section 3.1: Workflow Configuration Parameters.
2.1.4.2.1. Data
Table 2.1 shows the locations of pre-staged data on NOAA RDHPCS (e.g., Hera, Orion). These data locations are already linked to the Land DA System during the build but are provided here for informational purposes.
Platform |
Data Location |
|---|---|
Hera |
/scratch2/NAGAPE/epic/UFS_Land-DA_v2.1/inputs |
Hercules & Orion |
/work/noaa/epic/UFS_Land-DA_v2.1/inputs |
Gaea-C6 |
/gpfs/f6/bil-fire8/world-shared/UFS_Land-DA_v2.1/inputs |
Users who have difficulty accessing the data on Hera, Orion, Hercules, or Gaea-C6 may download it according to the instructions in Section 2.2.2. Its subdirectories are soft-linked to the land-DA_workflow/fix directory by the build script (sorc/app_build.sh); when downloading new data, it should be placed in or linked to the fix directory.
2.1.4.3. Set Up the Workflow
Generate the experiment directory by running:
./setup_wflow_env.py -p=<platform>
where <platform> is hera, orion, hercules, or gaeac6.
If the command runs without issue, this script will print override messages, experiment details, and “0 errors found” messages to the console, similar to the following excerpts:
Python Log Level= str: INFO, attr: 20
INFO::/path/to/setup_wflow_env.py::L34:: Current directory (PARMdir): /work/noaa/epic/username/hercules/landda/land-DA_workflow/parm
INFO::/path/to/setup_wflow_env.py::L36:: Home directory (HOMEdir): /work/noaa/epic/username/hercules/landda/land-DA_workflow
INFO::/path/to/setup_wflow_env.py::L38:: Experimental base directory (exp_basedir): /work/noaa/epic/username/hercules/landda
hercules
INFO::/path/to/setup_wflow_env.py::L168:: Experimental case directory /work/noaa/epic/username/hercules/landda/exp_case/lnd_era5_warmstart_00 has been created.
INFO::/path/to/setup_wflow_env.py::L175:: Rocoto YAML template: /work/noaa/epic/username/hercules/landda/land-DA_workflow/parm/templates/template.land_analysis.yaml
**************************************************
Overriding ACCOUNT = epic
Overriding APP = LND
Overriding ATMOS_FORC = era5
...
Overriding partition_default = hercules
Overriding queue_default = batch
Overriding res_p1 = 97
**************************************************
LND_LAYOUT_X: 1
DATM_STREAM_FN_LAST_DATE:
JEDI_PY_VER: python3.10
...
model_ver: v2.1.0
OUTPUT_FH: 1 -1
DT_RUNSEQ: 3600
KEEPDATA: YES
INFO::/path/to/uwtools/config/validator.py::L76::0 schema-validation errors found in Rocoto config
INFO::/path/to/uwtools/rocoto.py::L66::0 Rocoto XML validation errors found
The setup script (./setup_wflow_env.py) will create an experiment directory, located by default at ../../exp_case/${EXP_CASE_NAME}/. It will populate this directory with the experiment configuration file (land_analysis.yaml), the workflow XML file (land_analysis.xml), and the workflow launch script (launch_rocoto_wflow.sh), as well as several directories described in Table 2.2 below.
File/Directory Name |
Description |
|---|---|
|
Symlink to the |
|
Combines information from the user’s |
|
Workflow XML file used by the Rocoto workflow manager to determine which tasks (or “jobs”) to submit to the batch system and when to submit them (e.g., when task dependencies are satisfied) |
|
Workflow launch script |
|
Symlink to the directory containing log files for the Rocoto workflow ( |
|
Symlink to the |
For a deeper understanding of the setup_wflow_env.py script, see Figure 3.1.
2.1.5. Run the Experiment
To run the experiment, users can automate job submission via crontab or submit tasks manually via rocotorun.
2.1.5.1. Workflow Overview
Each Land DA experiment includes multiple tasks that must be run in order to satisfy the dependencies of later tasks. These tasks are housed in the J-job scripts contained in the jobs directory.
J-job Task |
Description |
Application/Configuration |
|---|---|---|
PREP_DATA |
Prepares the observation / DATM forcing data files |
LND/ATML |
FCST_IC |
Generates initial conditions (IC) files for the ATML configuration only |
ATML |
JCB |
Generates JEDI configuration YAML file |
LND/ATML |
PRE_ANAL |
Transfers the snow depth data from the restart files to the surface data files |
LND |
ANALYSIS |
Runs JEDI and adds the increment to the surface data files |
LND/ATML |
POST_ANAL |
Transfers the JEDI snow depth result from the surface data files to the restart files |
LND/ATML |
FORECAST |
Runs the forecast model |
LND/ATML |
PLOT_STATS |
Plots the results of the ANALYSIS and FORECAST tasks |
LND/ATML |
2.1.5.2. Automated Run
To automate task submission, users must be on a system where cron is available. On Orion, cron is only available on the orion-login-1 node, and likewise on Hercules, it is only available on hercules-login-1, so users will need to work on those nodes when running cron jobs on Orion/Hercules.
To submit jobs automatically via crontab, users should navigate to the experiment directory and launch the workflow with the add argument:
cd ../../exp_case/<EXP_CASE_NAME>
./launch_rocoto_wflow.sh add
where <EXP_CASE_NAME> is replaced with the actual name of the experiment directory (e.g., lnd_era5_warmstart_00/).
To check the status of the experiment, see Section 2.1.5.4 on tracking experiment progress.
Note
If users run into issues with the launch script, they can run conda deactivate before running the launch script.
2.1.5.3. Manual Submission
To run the experiment manually, issue a rocotorun command from the experiment directory:
cd ../../exp_case/<EXP_CASE_NAME>
rocotorun -w land_analysis.xml -d land_analysis.db
where <EXP_CASE_NAME> is replaced with the actual name of the experiment directory (e.g., lnd_era5_warmstart_00/).
Users will need to issue the rocotorun command multiple times. The tasks must be run in order, and rocotorun initiates the next task once its dependencies have completed successfully.
Details on checking experiment status are provided in the next section.
2.1.5.4. Track Progress
To check on the job status, users on a system with a Slurm job scheduler may run:
squeue -u $USER
To view the experiment status, run:
rocotostat -w land_analysis.xml -d land_analysis.db
If rocotorun was successful, the rocotostat command will print a status report to the console. For example:
CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION
=======================================================================================================
202501190000 jcb 5428846 SUCCEEDED 0 1 3.0
202501190000 prep_data 5428847 SUCCEEDED 0 1 30.0
202501190000 pre_anal 5428848 SUCCEEDED 0 1 8.0
202501190000 analysis 5428985 SUCCEEDED 0 1 72.0
202501190000 post_anal 5429034 SUCCEEDED 0 1 3.0
202501190000 forecast 5429128 QUEUED - 0 0.0
202501190000 plot_stats - - - - -
=======================================================================================================
202501200000 jcb 5428849 SUCCEEDED 0 1 11.0
202501200000 prep_data 5428850 SUCCEEDED 0 1 30.0
202501200000 pre_anal 5428851 SUCCEEDED 0 1 3.0
202501200000 analysis 5428986 SUCCEEDED 0 1 71.0
202501200000 post_anal 5429035 SUCCEEDED 0 1 3.0
202501200000 forecast druby://130.18.14.151:46755 SUBMITTING - 0 0.0
202501200000 plot_stats - - - - -
Note that the status table printed by rocotostat only updates after each rocotorun command (whether issued manually or via cron automation). For each task, a log file is generated. These files are stored in $LANDDAROOT/ptmp/test_*/com/output/logs.
The experiment has successfully completed when all tasks say SUCCEEDED under STATE. Other potential statuses are: QUEUED, SUBMITTING, RUNNING, and DEAD. Users may view the log files to determine why a task may have failed.
2.1.5.5. Check Experiment Output
As the experiment progresses, it will generate a number of directories to hold intermediate and output files. The structure of those files and directories appears below:
$LANDDAROOT (<exp_basedir>): Base directory
├── land-DA_workflow (<HOMElandda>): Home directory of the land DA workflow
│ ├── jobs
│ ├── modulefiles
│ ├── parm
│ ├── scripts
│ ├── sorc
│ └── ush
├── exp_case
│ └── $EXP_CASE_NAME
│ ├── com_dir --> symlinked to ptmp/test_*/com/landda/v2.1.0
│ ├── land_analysis.yaml
│ ├── land_analysis.xml
│ ├── launch_rocoto_wflow.sh
│ ├── log_dir --> symlinked to ptmp/test_*/com/output/logs
│ └── tmp_dir --> symlinked to ptmp/test_*/com/tmp
└── ptmp (<PTMP>)
└── test_* (<envir>)
└── com (<COMROOT>)
│ ├── landda (<NET>)
│ │ └── vX.Y.Z (<model_ver>)
│ │ └── landda.YYYYMMDD (<RUN>.<PDY>): Directory containing the output files
│ │ ├── datm
│ │ ├── hofx
│ │ ├── obs
│ │ └── plot
│ └── output
│ └── logs (<LOGDIR>): Directory containing the log files for the Rocoto workflow
└── tmp (<DATAROOT>)
├── [task_name].${PDY}${cyc}.<jobid> (<DATA>): Working directory for a specific task and cycle
└── DATA_SHARE
├── INPUT_DATM
├── hofx: Directory containing the soft links to the results of the analysis task for plotting
├── hofx_omb
└── RESTART: Directory containing the soft links to the restart files for the next cycles
Each variable in parentheses and angle brackets (e.g., (<VAR>)) is the name for the directory defined in the file land_analysis.yaml (derived from template.land_analysis.yaml or config.yaml) or in the NCO Implementation Standards. For example, the <envir> variable is set to “test” (i.e., envir: "test") in template.land_analysis.yaml. In the future, this directory structure will be further modified to meet the NCO Implementation Standards.
Check for the output files for each cycle in the experiment directory:
ls -l $LANDDAROOT/ptmp/test_*/com/landda/<model_ver>/landda.YYYYMMDD
where YYYYMMDD is the cycle date, and <model_ver> is the model version (currently v2.1.0 in the develop branch). The experiment should generate several restart files.
2.1.5.5.1. Plotting Results
Additionally, in the plot subdirectory, users will find a variety of plots for each cycle, including scatter plots (hofx_omb_<obs_type>_YYYYMMDD_scatter.png) and histograms (hofx_omb_<obs_type>_YYYYMMDD_histogram.png).
The scatter plot is named OBS-BKG (i.e., Observation Minus Background [OMB]), and it depicts a map of snow depth results. Blue points indicate locations where the observed values are less than the background values, and red points indicate locations where the observed values are greater than the background values. The title lists the mean and standard deviation of the absolute value of the OMB values.
The histogram plots OMB values on the x-axis and frequency density values on the y-axis. The title of the histogram lists the mean and standard deviation of the real value of the OMB values.
|
|
Note
There are many options for viewing plots, and instructions for this are highly machine dependent. Users should view the data transfer documentation for their system to secure copy files from a remote system (such as RDHPCS) to their local system.
Another option is to download Xming (for Windows) or XQuartz (for Mac), use the -X option when connecting to a remote system via SSH, and run:
module load imagemagick
display file_name.png
where file_name.png is the name of the file to display/view. Depending on the system, users may need to install imagemagick and/or adjust other settings (e.g., for X11 forwarding). Users should contact their machine administrator with any questions.