Skip to content

ICON NWP

Disclaimer

The following guidelines explain how to configure and compile ICON on the clusters of IMGW (primarily jet). They should be generic enough to help anyone getting ICON up and running, but they are targeted to users of the limited-area-model and large-eddy-simulation versions of ICON.

For more specific guidance on working with the climate modelling version of ICON, please check elsewhere (Gitlab projects of the climate modelling group, or their wiki).

What is ICON

ICON can be downloaded and used under a personal non-commercial research license. An account at is required. Instructions are available on the public ICON page. Useful information can be found in the ICON Tutorial.

How to set up ICON on jet

  1. Load the necessary modules. (base) [serafin@jet01 icon-2.6.6]$ module list Currently Loaded Modulefiles: 1) intel-oneapi-compilers/2022.2.1-zkofgc5 2) intel-oneapi-mpi/2021.7.1-intel-2021.7.1-pt3unoz 3) hdf5/1.12.2-intel-2021.7.1-w5sw2dq 4) netcdf-c/4.7.4-intel-2021.7.1-lnfs5zz 5) netcdf-fortran/4.5.3-intel-2021.7.1-27ldrnt 6) openblas/0.3.18-gcc-8.5.0-zv6qss4 7) netlib-lapack/3.9.1-gcc-8.5.0-ipqdnxj 8) eccodes/2.25.0-intel-2021.7.1-zqawf62

  2. Unpack the tarball of the source code.

    Bash
    1
    tar xvf icon-2.6.6.tar.xz
    The unpacked source code should look like this:
    Bash
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
    (mypy36) [serafin@jet01 test]$ ls icon-2.6.6
total 684K
drwxr-xr-x  5 serafin users 4,0K 20 mar 17.05 adify
drwxr-xr-x 14 serafin users 4,0K 20 mar 17.05 config
drwxr-xr-x  5 serafin users 4,0K 20 mar 17.05 data
drwxr-xr-x 10 serafin users 4,0K 20 mar 17.05 doc
drwxr-xr-x  3 serafin users 4,0K 20 mar 17.05 etc
drwxr-xr-x 18 serafin users 4,0K 20 mar 17.05 externals
drwxr-xr-x  2 serafin users 4,0K 20 mar 17.05 grids
drwxr-xr-x  3 serafin users 4,0K 20 mar 17.05 license
drwxr-xr-x  2 serafin users 4,0K 20 mar 17.05 m4
drwxr-xr-x 18 serafin users  16K 20 mar 17.05 run
drwxr-xr-x  2 serafin users 4,0K 20 mar 17.05 run_art
drwxr-xr-x  3 serafin users 4,0K 20 mar 17.05 schedulers
drwxr-xr-x 20 serafin users 4,0K 20 mar 17.05 scripts
drwxr-xr-x 38 serafin users 4,0K 20 mar 17.05 src
drwxr-xr-x  2 serafin users 4,0K 20 mar 17.05 support
drwxr-xr-x  8 serafin users 4,0K 20 mar 17.05 utils
drwxr-xr-x  2 serafin users 4,0K 20 mar 17.05 vertical_coord_tables
-rw-r--r--  1 serafin users  13K  9 feb 14.48 claw.mk.in
-rw-r--r--  1 serafin users 2,6K  7 feb 11.50 config.h.in
-rwxr-xr-x  1 serafin users 371K  9 feb 14.48 configure
-rw-r--r--  1 serafin users 113K  9 feb 14.48 configure.ac
-rw-r--r--  1 serafin users  641  7 feb 11.50 COPYING
-rw-r--r--  1 serafin users   47  7 feb 11.50 depgen.c.config.in
-rw-r--r--  1 serafin users 1,7K  9 feb 14.48 depgen.f90.config.in
-rw-r--r--  1 serafin users  389  9 feb 14.48 deplist.config.in
-rw-r--r--  1 serafin users  27K  9 feb 14.48 icon.mk.in
-rw-r--r--  1 serafin users 2,9K  9 feb 14.48 inlib.mk.in
-rw-r--r--  1 serafin users 3,8K  9 feb 14.48 Makefile.in
-rwxr-xr-x  1 serafin users 8,8K  9 feb 14.48 make_runscripts
-rw-r--r--  1 serafin users 2,4K  9 feb 14.48 README.art
-rw-r--r--  1 serafin users  73K  9 feb 14.48 README.md
-rw-r--r--  1 serafin users  11K  9 feb 14.48 RELEASE_NOTES.md

  3. Generate a configuration script for the machine where the code has to run. In practice, go to config and create a directory univie in there. In the univie directory, place the following contents in a file called jet.ifort.sh, then make the file executable (chmod u+x jet.ifort.sh):

    Bash
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
    #!/bin/bash
set -eu

module purge
module load intel-oneapi-compilers/2022.2.1-zkofgc5 \
intel-oneapi-mpi/2021.7.1-intel-2021.7.1-pt3unoz \
hdf5/1.12.2-intel-2021.7.1-w5sw2dq \
netcdf-c/4.7.4-intel-2021.7.1-lnfs5zz \
netcdf-fortran/4.5.3-intel-2021.7.1-27ldrnt \
openblas/0.3.18-gcc-8.5.0-zv6qss4 \
netlib-lapack/3.9.1-gcc-8.5.0-ipqdnxj \
eccodes/2.25.0-intel-2021.7.1-zqawf62
#libxml2/2.9.12-intel-20.0.2-anavgpx \

./configure \
CC='/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.5rhel8/intel-oneapi-compilers-2022.2.1-zkofgc52emmuxy3gj4hgwdxtpqx2zvcl/compiler/latest/linux/bin/icx' \
FC='/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/intel-2021.7.1/intel-oneapi-mpi-2021.7.1-pt3unozt2bsezjd65dkutk2vqakctvsa/mpi/2021.7.1/bin/mpiifort' \
FCFLAGS="-I/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/intel-2021.7.1/intel-oneapi-mpi-2021.7.1-pt3unozt2bsezjd65dkutk2vqakctvsa/mpi/2021.7.1/include \
        -I/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/intel-2021.7.1/netcdf-fortran-4.5.3-27ldrntppocl6dfnfb7nknp2yrk362kz/include \
        -O3 -g" \
CFLAGS="-I/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/intel-2021.7.1/netcdf-c-4.7.4-lnfs5zzfvfj7eoprpng73synyohurwub/include" \
CPPFLAGS="-I/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/intel-2021.7.1/eccodes-2.25.0-zqawf62kodmeksghxvk3k7pqlekpvigy/include \
        -I/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/intel-2021.7.1/libxml2-2.10.1-rq3keeuc35vmgpvqk5z2rcimlfanni6x/include/libxml2" \
LDFLAGS="-L/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/intel-2021.7.1/intel-oneapi-mpi-2021.7.1-pt3unozt2bsezjd65dkutk2vqakctvsa/mpi/2021.7.1/lib \
        -L/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/intel-2021.7.1/netcdf-c-4.7.4-lnfs5zzfvfj7eoprpng73synyohurwub/lib \
        -L/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/intel-2021.7.1/netcdf-fortran-4.5.3-27ldrntppocl6dfnfb7nknp2yrk362kz/lib \
        -L/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.5.0/openblas-0.3.18-zv6qss4vqhaqt4a3x4zeh7sbi6buuai3/lib \
        -L/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.5.0/netlib-lapack-3.9.1-ipqdnxjet2tlzossnlxlz33v7f5x6ckf/lib64 \
        -L/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/intel-2021.7.1/libxml2-2.10.1-rq3keeuc35vmgpvqk5z2rcimlfanni6x/lib \
        -L/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/intel-2021.7.1/eccodes-2.25.0-zqawf62kodmeksghxvk3k7pqlekpvigy/lib64" \
LIBS='-leccodes -lxml2 -llapack -lblas -lnetcdff -lnetcdf -lmpi' \
FI_provider=tcp \
--enable-mpi --enable-grib2 --without-external-cdi
    This configure script does two things: i) loads a selection of environment modules; ii) runs the ICON configure scripts passing all the required compiler flags.

  4. Prepare for an out-of-source build. Create a build directory in the source code root and move into it. In the build directory, link the configure file and then run jet.ifort.sh (the configuration script above)

    Bash
    1
2
3
4
    (mypy36) [serafin@jet01 icon-2.6.6]$ mkdir jet
(mypy36) [serafin@jet01 icon-2.6.6]$ cd jet/
(mypy36) [serafin@jet01 jet]$ ln -s ../configure .
(mypy36) [serafin@jet01 jet]$ ../config/univie/jet.ifort.sh
    A side note: without specifying FI_provider=tcp in the configure script, it is still possible to run it as follows:
    Bash
    1
2
3
4
    (mypy36) [serafin@jet01 icon-2.6.6]$ mkdir jet
(mypy36) [serafin@jet01 icon-2.6.6]$ cd jet/
(mypy36) [serafin@jet01 jet]$ ln -s ../configure .
(mypy36) [serafin@jet01 jet]$ srun --mem=20G ../config/univie/jet.ifort.sh
    Why the srun --mem=20G? Because this way the configuration script runs on the compute nodes of jet (so the MPI environment, in particular mpirun, works correctly) and has enough memory to run simple test programmes (actually, it could work with a lot less than 20G).

  5. Compile the code by running the following in the build directory (-j8 is optional, it's only about running multiple compile jobs in parallel):

    Bash
    1
    (mypy36) [serafin@jet01 jet]$ make -j8

  6. Copy in the build directory all the files needed to generate the ICON run scripts. The easiest way to do so is to use the following script (after editing the paths ICON_SRC_DIR and ICON_BUILD_DIR):

    Bash
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
    #!/bin/bash
set -eu

ICON_SRC_DIR=/jetfs/home/serafin/ICON/icon-2.6.6
ICON_BUILD_DIR=/jetfs/home/serafin/ICON/icon-2.6.6/jet

cd $ICON_BUILD_DIR
rsync -uavz $ICON_SRC_DIR/run . --exclude='*in' --exclude='.*'
rsync -uavz $ICON_SRC_DIR/externals . --exclude='.git' \
  --exclude='*.f90' --exclude='*.F90' --exclude='*.c' --exclude='*.h' \
  --exclude='*.Po' --exclude='tests' --exclude='rrtmgp*.nc' --exclude='*.mod' \
  --exclude='*.o'
rsync -uavz $ICON_SRC_DIR/make_runscripts .
ln -sf $ICON_SRC_DIR/data
ln -sf $ICON_SRC_DIR/vertical_coord_tables
    You may call it copy_runscripts.sh, then give it execution permissions and run it:
    Bash
    1
    (mypy36) [serafin@jet01 jet]$ ./copy_runscripts.sh

    How to run ICON on jet

Last update: December 7, 2023
Created: March 20, 2023