Skip to content

European Center for Medium-Range Weather Forecast

ECMWF

website / service status / confluence / support / accounting / jupyterhub

If you need access, talk to your supervisor to create an account for you. You will get a username and a password as well as an OTP device (hardware or smartphone). Accounts are handled via www.ecmwf.int

Available Services @ IMGW:

Connecting to ECMWF Services

Teleport supported versions

Please note that ECMWF does currently only support teleport version up to 13. ecmwf information

from home

Well you need to download the appropriate package from teleport selecting major version 13 and what ever sub version is available, e.g. 13.4.26 (9.10.2024). Choose your OS (Linux, Mac, Windows) and download the package. For Linux you can also try to install this with the package manager:

Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# adjust to recent version
curl https://goteleport.com/static/install.sh | bash -s 13.4.26

# launch an ssh-agent
eval $(ssh-agent)
# login using the ECMWF teleport server
# this opens a browser and you need to login to ECMWF with OTP.
tsh login --proxy=jump.ecmwf.int
If browser window does not open automatically, open it by clicking on the link:
 http://127.0.0.1:38615/f5df50f4-35bf-4f88-a2dc-2a271df6e1d5

# finaly you should get a confirmation
> Profile URL:        https://jump.ecmwf.int:443
  Logged in as:       [MAIL ADDRESS]
  Cluster:            jump.ecmwf.int
  Roles:
  Logins:             [ECMWF USERNAME]
  Kubernetes:         disabled
  Valid until:        2024-10-09 23:18:33 +0200 CEST [valid for 11h58m0s]
  Extensions:         permit-X11-forwarding, permit-agent-forwarding, permit-port-forwarding, permit-pty

look at the SSH config below and you should be fine to connect.

from IMGW

A ECMWF user can connect to the ECS/ATOS using teleport, first load the teleport module and start the ssh-agent:

Using teleport
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
module load teleport
** INFO: Default jumphost now: jump.ecmwf.int
** INFO: Module loaded. SSH Agent required for login, run 'ssh-agentstart',
**       or 'ssh-agentreconnect' ro reconnect to an existing agent.
**       run 'ssh-agent -k' to kill the agent.
         Login run: 'python3 -m teleport.login' and your ECMWF credentials.
         e.g. 'ssh -J <id>@jump.ecmwf.int <id>@ecs-login'
         Check certificates, run: 'tsh status'

# Activate the ssh-agent (required to store the key/certificate)
ssh-agentstart
# or
ssh-agentreconnect
# Check if it is running
ssh-add -l

once you have a running ssh-agent, run a browserless login via python

Connecting to ECMWF
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Login to the default teleport jump host (shell.ecmwf.int) Reading
python3 -m teleport.login
tsh status
# run ssh agent again
ssh-add -l
# now there should be two keys!!!
# Login to ECaccess in Bologna
ssh -J [user]@jump.ecmwf.int [user]@ecs-login
# Login to HPC ATOS
ssh -J [user]@jump.ecmwf.int [user]@hpc-login
# delete current certificates
tsh logout

Configuration

Environment variables configuration:

  • ECMWF_USERNAME - The ECMWF Username
  • ECMWF_PASSWORD - The ECMWF Password
  • TSH_EXEC - The Teleport binary tsh path
  • TSH_PROXY - The ECMWF Teleport proxy

You can set these variables in your ~/.bashrc file to avoid typing these at every login. Please do not save your ECMWF_PASSWORD like this!

It is highly advised to add this to your .ssh/config, although ECMWF has added some information on that too:

.ssh/config
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
Host jump.ecmwf.int a?-* a??-* hpc-* hpc2020-* ecs-*
  User [ECMWF USERNAME]
  IdentityFile ~/.tsh/keys/jump.ecmwf.int/[MAIL ADDRESS]
  CertificateFile ~/.tsh/keys/jump.ecmwf.int/[MAIL ADDRESS]/jump.ecmwf.int-cert.pub
  HostKeyAlgorithms +ssh-rsa*,rsa-sha2-512
  PubkeyAcceptedKeyTypes +ssh-rsa*
  ServerAliveInterval 60
  TCPKeepAlive yes

Host a?-* a??-* hpc-* hpc2020-* ecs-*
  ProxyJump jump.ecmwf.int

SSH-agent

It is required to have an SSH-agent running in order to connect to the ECMWF servers. The teleport module includes a startagent function to allow to reconnect to an existing ssh-agent. Do not start too many agents!

start ssh-agent
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# load the module
module load teleport
# start a new agent or reconnect
ssh-agentstart
# or reconnect
ssh-agentreconnect
# unsure about agents?
userservices sshtools -h
# kill all agents
userservices sshtools -k

ECMWF Access Server (ECS)

There is an issue with ssh-keys

ECS fix ssh-key issue
1
2
3
4
5
6
# connect to ECS following the teleport login procedure above
ssh -J [user]@jump.ecmwf.int [user]@ecs-login
# Generate a new SSH key on ECS, no passphrase.
ssh-keygen -t ed25519
# Add the public key to your own authorized_keys on ECS/HPC
cat .ssh/id_ed25519.pub >> .ssh/authorized_keys

This will solve some ecaccess issues.

Sometimes there are also different issues with the connection. You can search the issues to check if you have a familiar problem.

Storage at ECMWF

There is the information for the new ATOS system: ecmwf wiki.

A short summart

File System Suitable for ... Technology Features Quota
HOME permanent files, e.g. profile, utilities, sources, libraries, etc. NFS It is backed up. 10GB
PERM permanent files. no backup. NFS DO NOT USE IN PARALLEL APPLICATIONS. DO NOT USE FOR JOB STANDARD OUTPUT/ERROR 500 GB
HPCPERM permanent files. no backup Lustre 100 GB for users without HPC access. 1 TB for users with HPC access
SCRATCH all temporary (large) files. Main storage for your jobs and experiments input and output files. Lustre Automatic deletion after 30 days of last access. no backup. 50 TB for users with HPC access. 2 TB for users without HPC access
SCRATCHDIR Big temporary data for an individual session or job, not as fast as TMPDIR but higher capacity. Files accessible from all cluster. Lustre Deleted at the end of session or jobCreated per session/ job as a subdirectory in SCRATCH part of SCRATCH quota
TMPDIR Fast temporary data for an individual session or job, small files only. Local to every node. SSD Deleted at the end of session or job. Created per session/ job space and limits are shared with LOCALSSD

Typically all users from Austria a grouped together into the at group and can share data with each other.

Connecting via ECaccess

using a local installation of ecaccess tools can be used to submit jobs and monitor jobs from a remote location. Documentation @ECMWF

ECAccess module
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# load the ecaccess module
module load ecaccess-webtoolkit/6.3.1
# all available tools
ecaccess                       ecaccess-file-delete
ecaccess-association-delete    ecaccess-file-dir
ecaccess-association-get       ecaccess-file-get
ecaccess-association-list      ecaccess-file-mdelete
ecaccess-association-protocol  ecaccess-file-mget
ecaccess-association-put       ecaccess-file-mkdir
ecaccess-certificate-create    ecaccess-file-modtime
ecaccess-certificate-list      ecaccess-file-move
ecaccess-cosinfo               ecaccess-file-mput
ecaccess-ectrans-delete        ecaccess-file-put
ecaccess-ectrans-list          ecaccess-file-rmdir
ecaccess-ectrans-request       ecaccess-file-size
ecaccess-ectrans-restart       ecaccess-gateway-connected
ecaccess-event-clear           ecaccess-gateway-list
ecaccess-event-create          ecaccess-gateway-name
ecaccess-event-delete          ecaccess-job-delete
ecaccess-event-grant           ecaccess-job-get
ecaccess-event-list            ecaccess-job-list
ecaccess-event-send            ecaccess-job-restart
ecaccess-file-chmod            ecaccess-job-submit
ecaccess-file-copy             ecaccess-queue-list
ECAccess certificate
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# First get a valid certificate to get access
$ ecaccess-certificate-create
Please enter your user-id: [ECMWF short username]
Your passcode: [OTP Code]
# Check if the certifcate is fine
$ ecaccess-certificate-list
chmod                168h     Oct 15 14:26         change file mode
deleteFile           168h     Oct 15 14:26         delete file
deleteJob            168h     Oct 15 14:26         delete a job
getFileList          168h     Oct 15 14:26         get file list
getFileSize          168h     Oct 15 14:26         get file size
getJobList           168h     Oct 15 14:26         job list
getJobResult         168h     Oct 15 14:26         job result
getTempFile          168h     Oct 15 14:26         create temporary file
getTransferList      168h     Oct 15 14:26         get transfer list
mkdir                168h     Oct 15 14:26         make directory
moveFile             168h     Oct 15 14:26         move file
readFile             168h     Oct 15 14:26         read file
rmdir                168h     Oct 15 14:26         remove directory
spoolTransfer        168h     Oct 15 14:26         ectrans request
submitJob            168h     Oct 15 14:26         job submission
writeFile            168h     Oct 15 14:26         write file

if you have troubles or for some other reason, if you remove this file ~/.eccert.crt then your current certificate is gone.

ECAccess gateway
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# check what server you are connected to
$ ecaccess-gateway-name
boaccess.ecmwf.int
# connected ?
$ ecaccess-gateway-connected
yes

# the associations are defined with path and username, password to access the server. See below.
# now it is time to check associations on that server
$ ecaccess-association-list
aurora            aurora.img.univie.ac.at active     scratch
# NAME             GATEWAY                STATUS     COMMENT

# check on a different ecaccess server
$ ecaccess-association-list -gateway ecaccess.img.univie.ac.at
jet               jet01.img.univie.ac.at  active     scratch
# NAME             GATEWAY                STATUS     COMMENT
ECAccess ectrans
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# transfer some files using these associations
#
# ecaccess-ectrans-request -lifeTime 1h -overwrite -onFailure [ASSOCIATION NAME] [SOURCE FILE/DIR]
# lifeTime: how long it will retry to do so
# overwrite: overwrites any files existing
# onFailure: reports back to you.
# there are more options available: ecaccess-ectrans-request --help
$ ecaccess-ectrans-request -lifeTime 1h -overwrite -onFailure aurora [SOURCE FILE/DIR]
# this is an async process. It does not happen right away.
$ ecaccess-ectrans-list
176674485  INIT       aurora            boaccess.ecmwf.int   Oct 09 09:42
# check again
$ ecaccess-ectrans-list
176674485  COPY       aurora            boaccess.ecmwf.int   Oct 09 09:42

if you encounter a STOP or ERROR, then you can also check the gateway (boaccess, imgaccess) to have a look at the message (file transfers).

How to create ecaccess associations

There are two ways to create these associations:

  1. via the web interface:
  2. via the ecaccess-webtoolkit

After creating new associations it takes a while before they actually work (about 10min).

web interface

Depending on where you want to transfer files to, go to:

Steps:

  1. Login with ECMWF username and OTP
  2. Go to ECtrans setup
  3. Click add association (at bottom)
  4. Fill in the association
    • name
    • hostname (login.img.univie.ac.at or jet01 or jet02)
    • directory (/srvfs/scratch/[USERNAME] or something else)
    • comment (giving you a hint where it drops the file sto)
    • login (this is your imgw server username)
    • password (this is your imgw server password)
  5. Click on Create

Later you can also change the password for your associations.

toolkit

You need to have access to an installation of ecaccess-webtoolkit.

Create an association on one gateway server
1
2
3
4
5
6
7
8
9
# load module to give access to the commands
$ module load ecaccess-webtoolkit/6.3.1
# create a certificate
$ ecaccess-certificate-create
Please enter your user-id: [ECMWF short username]
Your passcode: [OTP Code]
# create the template
$ ecaccess-association-get -template [ASSOCIATION NAME] new-association
# now you need to edit that file: new-association

This file serves as a template only the first part is important.

Change:

  • active='yes'
  • comment='Ssomething that explains where is will send the data to'
  • directory='/srvfs/scratch/[USERNAME]' or another directory.
  • hostName='login.img.univie.ac.at' or jet01... or jet02...
  • login='[USERNAME]'
  • protocol='genericSftp'

save the file and then you can add this to the correct gateway. Remember that JET is only available from the gateway (ecaccess.img.univie.ac.at), which is available only from inside the VPN@UNIVIE under ecaccess.wolke.img.univie.ac.at.

new-association
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
##############################################################
# Main Parameters
##############################################################
$name='[ASSOCIATION NAME]';
$active='no';
$comment='';
$grantedUserList='';
$directory='';
$hostName='';
$login='';
$protocol='';

finally you can add your newly created association to the gateway:

Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# add to IMGW ecaccess server (password: your imgw server password)
$ ecaccess-association-put -password -gateway ecaccess.img.univie.ac.at new-association
New password:
# add to boaccess (password: your imgw server password)
$ ecaccess-association-put -password new-association
New password:
# test the association
$ ecaccess-association-list
aurora            login.img.univie.ac.at active     scratch
$ ecaccess-association-list -gateway ecaccess.img.univie.ac.at
jet               jet01.img.univie.ac.at active     scratch
# send a file to both

using ssh-keys

There is another way to overcome the need to continuously changing the password in the association. It is possible to add a ssh-key to the ectrans association.

Steps:

  1. Create a compatible ssh-key
  2. Add ssh public key (e.g. ecmwf.pub) to IPA
  3. Modify the association
Create an ectrans ssh-key
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# generate a ssh-key using the PEM format
ssh-keygen -t rsa -m PEM -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa): ecmwf
Enter passphrase for "test" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in test
Your public key has been saved in test.pub
The key fingerprint is:
SHA256:2FIvXhZASKo/b565cUiWQsImKV63YZhhnx0ySb3Rak8 user@notebook
The key's randomart image is:
+---[RSA 4096]----+
|   o.=+++        |
| o. =o*o.o       |
|+ ++o* .= .      |
|oo.+o oO E .     |
| .. ..B S +      |
|   . + + =       |
|    o o o        |
|     o.=         |
|     o*.         |
+----[SHA256]-----+
#

Add the public key to the IPA. It might take up to 10 min, before the new key is registered by the system. You can check on aurora by running: sss_ssh_authorizedkeys $USER

Now you can modify the association by adding your generated private key:

Modified association
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
...
sftp.port = "22"                                                                                 
sftp.prefix = ""
sftp.privateKey = "
-----BEGIN RSA PRIVATE KEY-----
eJaBR2f80p2qlgapAku1z+PsnY2gjdL7y6iqxnrR19L8/CnM+A2OdU+lSnBv1PS7
VU2/nY4Al6xSJTJOrZ+k9dkyWjbixF1FCpVeNOxqJdqjtcFw/2nX8Mtp+5BOrCxg
rTkoW31foJQL+FNf/VelOPO1xf+YSfKIUmZ7OU3LHrzDm07p0pd/Aclj7Qqf89mp
pjqDXe7/00OuRuda6gu2Sbd4Oro+5ha9jkvfDQpV8Xj5QsLbjnxjp5+J9yUDoujq
...
vlN5kEeFbyB22H5QCkCF4RWVOUfudCTcPNC2DMeR7gtFwlWmxzizZuaVi5v48vP3
8/zt0udPyCyPP2B0NOyJzrDejcvfVQ76SmLGgArjQN3jJDF7p7UausliO2R1SD/p
jJNEf9KEDEeO3COLZrT0tcfTmAEd7OVSURdZKJTXQCPai2LTevTBYJxXHgFFly4Z
-----END RSA PRIVATE KEY-----
"
sftp.sessionTimeOut = "60000"
sftp.suffix = ".tmp"
sftp.usetmp = "yes"
sftp.wmoLikeFormat = "no"
###### END-OF-PROPERTIES ######
';

to either the association file or via the web interface. Then you can remove the password, but leave the login=[USERNAME].

More information on these details can be found here

ECaccess Gateway

The department is running a member state ecaccess gateway service. The purpose of an individual access server is to bridge ECMWF's network with IMGW's network. Hence, protecting these networks. For example, you can access the JET cluster from the department ecaccess server, but not from boaccess server, but from boaccess you can accesss aurora.

There are two gateways:

Please use your ECMWF credentials to login. Documentation @ECMWF

Last update: January 8, 2025
Created: January 26, 2023