How to bacy

This is a small guide on how to use bacy to run assimilation cycles in practice.

Prerequisites

Before running you need: * a path to a directory containing a bacy with initial conditions for your experiment * a icon gridfile, a radiation gridfile and extpar-Data. You can create these using the icongridgen or via the web interface at DWD (user icon-web, pw: icon@dwd). * a working bacy, cloned eihter from git@gitlab.dkrz.de:dace/dace_bacy.git or from another bacy using bacy/bacy_clone

Running bacy

To run cycles you need to go to to the cycle module: bacy/modules/cycle/. There you can find the scripts prep_cycle and cycle.

Running prep_cycle

To initialize bacy run prep_cycle in the cycle directory. The script describes itself like this:

SCRIPT for preparing a data assimilation cycle

USAGE: ./prep_cycle [-h] model mode frombacypath startdate

   - '-h':    help (optional)
   - 'model':        ICON|COSMO|ICON-LAM
   - 'mode':         ASS|MAIN|SUPP|URDA|CLIMA
   - 'frombacypath': path from which bacy experiment to start the cycle
   - 'startdate':    format yyyymmddhhmmss|yyyymmddhh
   - 'enddate':      format yyyymmddhhmmss|yyyymmddhh
                     (only for mode MAIN|SUPP|URDA)"

DESCRIPTION: serves as preparation of script 'cycle';
   links all module relevant input data for a given start date (as specified by
   'startdate') from an existing bacy experiment (as specified by 'frombacypath')
   into its own cycling directory tree (also containing 'data', 'feedback' and
   'veri' subdirectories, but only with initial time-stamp subfolders);

So you can call it like:

./prep_cycle ICON-LAM ASS /hpc/path/to/initial/conditons/bacy/ 2023112812

This will prepare a cycle with the ICON large area model in assimilation mode with initial condition from /hpc/path/to/initial/conditons/bacy/ starting at the 28.11.2023 - 12 UTC.

Running cycle

After prearation you can run the cycle by calling cycle with similar arguments. The script describes itself like this:

SCRIPT for running a data assimilation cycle

USAGE: ./cycle [-h] [-d] \
         [--cycle[=][ ]value] [--task[=][ ]value] [--subtask[=][ ]value] \
         model mode startdate enddate

  options:
  - '-h': help (shows usage)
  - '-d': dry run (does not submit any long-time jobs)
  - '--cycle'  : from which cycle   to resume work (requires a number)
  - '--task'   : from which task    to resume work (requires a number)
  - '--subtask': from which subtask to resume work (requires a number)
  arguments:
  - 'model':     ICON|COSMO|ICON-LAM
  - 'mode':      ASS|MAIN|VERI|VERIENS|SUPP|URDA|CLIMA
  - 'startdate': start of bacy experiment (format yyyymmddhhmmss|yyyymmddhh)
  - 'enddate'  : end   of bacy experiment (format yyyymmddhhmmss|yyyymmddhh)

  Creating the file 'STOP_CYCLE' in the 'cycle' directory will interrupt
  all running cycles. To continue the experiment delete the file and restart
  the cycle(s) or run the 'RESTART_CYCLE_*' script(s)'.

DESCRIPTION: Depending on mode, the scripts runs specific cycles:
   ASS:     a data assimilation cycle by calling the modules 'core',
            'more', 'sst', 'sma', 'snow', and 'mec'(COSMO) in the correct order
   MAIN:    long forecasts (building upon an ASS cycle)
   VERI:    mec for deterministic verification against observations
            (building upon a MAIN cycle)
   VERIENS: mec for ensemble verification against observations
            (building upon a MAIN cycle)
   SUPP:    supplemental assimilation of a previous cycle
            (building upon an ASS cycle)
   URDA:    an ultra rapid data assimilation cycle
   CLIMA:   climate data assimilation cycle with ICON-SML

So you can call it like:

./cycle ICON-LAM ASS 2023112812 2023112912

This will start a cycle prepared above going from the 28.11.2023 - 12 UTC until the 29.11.2023 - 12 UTC.

Using nohup

For longer experiments you wankt to be able to leave the command shell and let the cycles take care of its own. To do so, you can start the process using nohup:

nohup ./cycle ICON-LAM ASS 2023112812 2023112912 &> cycle.log &

This will start the process in the background, writing its output to the file cycle.log.

Pro Tip: You can read the output using less. While in less press SHIFT + F to read it like a feed. To exit this mode press CTRL + C and then q to exit less.

Interrupting and Restarting the cycle

To interrupt a cycle you can create a file named STOP_CYCLE (eg. touch STOP_CYCLE) in the cycle directory. After completion of the current step, the cycle will break.

To restart a interrupted cycle at any of the past times, delete STOP_CYCLE and use the flags --cycle and --task of the cycle script. For example

./cycle --cycle=5 --task=7 ICON-LAM ASS 2023112812 2023112912

restarts the former cycle at 28.11.2023 - 16 UTC, at the 7th task, which is the core module (assimilation). This of course also works with nohup.

You can also change the enddate when resarting a cycle. Just make sure you also change the name of a logfile bacy/data/cycle_icon-lam_ass.log_20230928120000_20231129120000 so the last number matches your new endtime.

Adapting bacy

Bacy is modular and quite flexible. The challenge is to know where to find the settings you want to change. A few hints: * "global" settings can be changed in modules/common/bacy_conf.sh This includes parameters like the ensemble size (BA_ENSIZE_ASS), on-off switches for snow, surface and sst analysis (BA_RUN_SNW, BA_RUN_SMA, BA_RUN_SST) and several settings regarding the icon grid (serach for GRID) * each module has - namelists, which can be found in the modules/module-name/const/ folder. Only the namelists without the .default ending affect the run. Inside there probably all paramters of the said module can be set. - the file module_name_conf.sh containing high-level configurations of the module - the scripts prep_module-name, module-name, save_module-name which make use of prep_module-name_fcns.sh, module-name_fcns.sh, save_module-name_fcns.sh. Inside said functions the real work of the module (file logistics, putting the namelists together and filling them with values, starting and monitoring the jobs and saving the output) is done. You can insert ksh prompts inside the functions to get your code executed at the desired time. Be aware that many functions differentiate between cycle and model modes, so make sure your code is actually executed where you put it. - also see here

---

Last update: December 7, 2023
Created: December 7, 2023