SSH
Note | |
---|---|
This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team. |
Overview
SSH is a widely used network protocol and tool to establish a secure communication channel to a remote machine. Its most common use is to get shell access (be able to type command-line commands) on a remote server.
When connecting to a remote machine, the standard way to authenticate is via the remote account's login and password. However, this authentication schema is disabled on Grid'5000 for security reasons (as it is often subject to brute-force attacks). Instead, SSH is used with a key-based authentication mechanism to gain access to the platform, and ssh keys are the recommended authentication method even inside Grid'5000.
It can also be used to transfer files (using SCP, SFTP or RSYNC), connect securely to network services on the remote network (using SSH tunnels or a SSH-based VPN), and even execute graphical applications remotely (using X forwarding).
This documentation is designed to teach you the basics as recommended for use in the Grid'5000 context.
Generating keys for use with Grid'5000
If you do not already have a SSH public/private key pair, the following gives the basic commands to create one, as required to access Grid'5000. See also in the next sections more details about the SSH keys.
Linux and Mac users
Most Linux distribution and Mac OS variants come with the ssh
and ssh-keygen
command line utilities. Generating a key-pair is simply a matter of running the ssh-keygen
and supplying a passphrase. To access Grid'5000 using ssh, you must use a key-pair that whose private key is passphrase protected.
If you follow the defaults, you now have a public key stored in the ~/.ssh/id_rsa.pub
file. You can use it when requesting an account or to update your profile.
Windows users
- All Windows versions, with PuTTYgen
The basic steps for Windows users are well documented in the page Inria maintains for gforge users. You'll only need to remember that the key you generate don't need to be uploaded to InriaGforge, but to the Account management form or the Account request form.
- Since Windows 10, there is new SSH client usage possibilities
- A native OpenSSH (client and server) could be installed. See OpenSSH installation and first use Microsoft documentation
- Windows Subsystem for Linux (WSL) is available to use a complete Linux distribution and thus a complete OpenSSH client too. See about WSL Microsoft documentation
So ssh-keygen command is available, and could be used as in Linux and Mac OS.
SSH configuration for Grid'5000, and first basic connections
After having looked at the explanations above, you might be wondering how servers can get a copy of your public key if you can not connect to them to put it there. In Grid'5000, the answer depends on the type of server.
Access machine configuration
You have to upload your previously generated public key (/home/login/.ssh/id_rsa.pub
) using the Account Management Interface or the Account request form.
Access machines update the repository of trusted keys they use regularly from the user database.
The following command should therefore connect you to Grid'5000, assuming the ssh
command is available and has access to the private key which corresponds to the public key you uploaded in the Account management form.
Whenever you may have more than one SSH key-pair or gave a specific file name to the key you use for Grid'5000, you can use the -i
option of SSH:
In any case, you must be particularly careful about access rights given to the key files and the directory storing them (see .ssh
access rights): only you should be able to read and write the key files (access mode should be 0600) and the directory (access mode should be 0700).
Site frontend machines
Looking at the general configuration for your account on the access machine, you'll find some additional files:
The id_rsa
and id_rsa.pub
files are a passphrase-less pair of keys that where automatically configured for you when your account was created. The public part of that key was added to the repository of trusted keys for all access machines, as you can see when running
The authorized keys file authorizes you to connect to site frontends using either the internal key (generated for you during the account creation) or your external key that you use to connect from the outside to the access machine (the key that is set in your account management interface). As a reminder, it is strongly recommended to protect the external key with a passphrase, but not the internal key (e.g. to avoid having to provide it in scripts). The internal key is less secure. It is strongly advised to not use it outside of Grid'5000.
Reserved machines
When you have reserved nodes in a OAR job, 3 different cases are to be considered when it comes to connect to them via SSH:
- You did not reserve all nodes CPU cores or use the
deploy
job type - oar generates a keypair for your job, that will be used transparently whenever you use
oarsh
. oarsh is a wrapper around the OpenSSHssh
command, which allows to connect to a node in a job (oarsh offers most of the features and options of OpenSSH ssh). - You reserve nodes entirely with OAR (all CPU cores)
- you can ssh to nodes just like to any machine (you can still use
oarsh
). - You used the OAR
deploy
job type - before deploying an environment, you cannot connect to nodes. Once an environment is deployed:
- if the
-k
option ofkadeploy3
is used, the authorized_keys file of your home directory is copied to the ssh configuration of the root account of the deployed nodes, thus you can connect as root. - if you deployed an environment that setup Grid'50000 NFS (e.g. a -nfs or -big environment variant of the Grid'5000 supported environments), you should be able to ssh normally to the deployed nodes (Note that other users may not, as this may be restricted automatically).
Use SSH to connect to another site
The figure below shows how you just connected from your local machine to access
, and then to the site frontend in nancy
. Site frontends (named f
site
.
site
.grid5000.fr
or simply site
.grid5000.fr
) are the machines you will use to interact with Grid'5000 tools such as OAR and Kadeploy.
Those machines are virtual machines, and must not be used for CPU or I/O intensive tasks (nodes must be reserved and used instead).
More advanced SSH usage, handling files
Basic files copy
Grid'5000 users have home directories in every Grid'5000 sites. But each site home directory is accessible in the access machines in a directory named after the site name.
To copy myfile
from my workstation to my nancy home:
To copy myremotefile
from my lyon home to my workstation:
To copy mydir
from my workstation to my nancy home:
rsync files copy/synchronisation
Copying mydir
to my rennes home with rsync through SSH:
tar/ssh files copy
Copying mydir
to my rennes home with tar through SSH:
Mounting remote filesystem (sshfs)
To mount a remote filesystem using sshfs:
To unmount the remote filesystem :
TCP port forwarding
A SSH connection allows to tunnelize some other TCP connections. This is achieved using the "-L", "-R" or "-D" options. This provides some alternative to using Grid'5000 VPN which connects your machine directly in the Grid'5000 network. SSH tunnels are lighter, easier to setup but are limited to TCP connections.
Forwarding a local port
You can forward a local TCP port to a host inside Grid'5000 (or more generally behind a firewall) using the "-L" option. Typically, this allows to connect to an TCP server running inside Grid'5000 from your workstation connected to your lab's network.
In the illustration below, the user uses the following command:
where:
ssh_gateway
is the ssh target host, which can directly connect to remote_server in TCP, e.g. ssh_gateway = access.grid5000.frremote_server
is the remote machine which actually runs the service you want to connect to, e.g. an apache server or jupyter notebook.2222
is an example of TCP port opened on the remote_server. For instance in case of a web server it could be 80 or 443.1111
is an example of TCP port opened locally, that your local software client will connect to (can be the same port number as the remote port, as long as it is allowed: must be > 1024 unless you run ssh as root)
Note | |
---|---|
A common practice is to add the -v option to the ssh command, so that ssh be more verbose and displays explicitly the port forwarding activity. |
Forwarding a remote port
You can forward a remote TCP port from a target machine in Grid'5000 back to a host located in the local network your workstation is connected to, using the "-R" option. This is typically useful if you have a service provided on your intranet, which you would like to reach from Grid'5000. See also below the "-g" option which may be useful to add.
In the illustration below, the user uses the following command:
where:
ssh_gateway
is the ssh target host, on which TCP port 1111 will be opened, listen for a connection to forward back. For instance, ssh_gateway = access.grid5000.frlocal_server
is the local server hostname in your local network, which actually runs the service you want to connect to2222
is an example of TCP port, which the service running on the local server is listening to. For instance in case of a web server it could be 80 or 443.1111
is an example of TCP port the ssh target host will listen to and forward back (can be the same port number as the local server port, as long as it is allowed: must be > 1024 unless you connect as root on the remote server, which is mostly not the case !)
Tunnelizing for others
"-L" and "-R" options open TCP sockets listening only on localhost for the forwarding, which means only the ssh target host can use the forwarding. Adding the "-g" parameter, you allow connections from other hosts than localhost to use your SSH tunnels. Be warned that anybody within your network may access the tunneled host this way, which may be a security issue.
SOCKS proxy
The OpenSSH ssh client embeds a SOCKS5 proxy, which can be activate using the "-D" option. Then it can be configured in the application (e.g. web browser), so that any connection from the application is made originated from the ssh remote machine, hence typically from within Grid'5000.
In the illustration below, the user uses the following command:
where:
ssh_gateway
is the ssh target host, e.g. ssh_gateway = access.grid5000.fr7777
is the TCP port the SOCKS5 proxy is listening to on the local machine.
In the web browser, the configuration to use is thus: SOCKS5 -> 'localhost:'7777
Note that many tools can make use of such a SOCKS5 proxy.
- It can generally be configured in the network settings of the operating system or browser (but it's often not the most convenient).
- A very popular and handy add-on for managing proxies in web browsers is FoxyProxy Standard, which makes switching from proxy to proxy very easy, or even automatic (based on patterns of target URLs):
- Finally, some other tools allow to proxify command line applications transparently:
tsocks
orredsocks
, which are available as Debian packages for instance.
Note | |
---|---|
If using SSH tunneling does not suit your needs, for instance because your client application does not support the SOCKS5 protocol, or because you do not manage to get things working as you like, mind looking at Grid'5000 VPN, which provides another way to reach services in Grid'5000 from the outside. |
Easing SSH connections from the outside to Grid'5000
Setting up aliases or defaults
You may provide a SSH configuration file that defines default parameters for your SSH connections inside Grid'5000. This file is ~/.ssh/config
in your Grid'5000 home directories (one per site). In this configuration file, you may for instance specify:
- alias for long hostnames
- username to be used to connect some machines if this is not the one used locally (no more need for either
-l username
orusername@
statement). - X11 forwarding enabling
- TCP port forwarding
- ProxyCommand or ProxyJump (see below)
- and more (see
man ssh_config
)
Using SSH ProxyCommand feature to ease the access to hosts inside Grid'5000
You can setup your local (on your laptop or workstation) ~/.ssh/config
so that you can directly connect any host inside Grid'5000 (behind Grid'5000 firewalls) from your workstation using the OpenSSH ProxyCommand option.
- First, setup an alias for the global access machine
# Alias for the gateway (not really needed, but convenient)
Host g5k
User g5k_login
Hostname access.grid5000.fr
ForwardAgent no
Your are now able to connect to your Grid'5000 site without being bothered with options such as to specify your Grid'5000 login each time:
- Second, setup the glob alias for the access to any machine inside Grid'5000
(hosts inside Grid'5000, except the access machines, are not reachable from the outside, Internet)
# Direct connection to hosts within Grid'5000 which are not reachable directly
Host *.g5k
User g5k_login
ProxyCommand ssh g5k -W "$(basename %h .g5k):%p"
ForwardAgent no
The .g5k suffix is just an (arbitrary) suffix that you will now specify in your ssh command lines, in order to access hosts inside Grid'5000.
- Accessing the frontend of my site (e.g. grenoble):
- Accessing the frontend of another site:
- Accessing a node belonging to one of my job
- Copying files (any command such as scp, rsync, sshfs, can use the .g5k suffix)
- Global picture
Using SSH with ProxyJump to access hosts inside Grid'5000
The ProxyJump feature was introduced in OpenSSH version 7.3, and allows a ssh connection jumping over several hops (instead of sshing to the several hops one after another). A ProxyJump is defined using the "-J" option in the ssh command line.
For instance for Grid'5000, one can run to connect to Nancy's frontend:
Like OpenSSH "ProxyCommand", "ProxyJump" can also be setup in the .ssh/config
file.
Host fnancy
User g5k_login
ProxyJump access.grid5000.fr
and then directly connect with:
However, using "ProxyJump" is far less versatile than using "ProxyCommand", as it does not allow using the substitution variables "%h" and "%p" and use a subcommand (e.g. call "basename"). Thus, a general/wildcard suffix can't be used like proposed with the ProxyCommand configuration, allowing to access to any hosts inside Grid'5000. Using the "ProxyCommand" feature is then by far more powerful.
One case where ProxyJump may prove useful, is to copy file back from Grid'5000 to a machine behind a firewall, like in an intranet lab. For instance, assuming my_station is behind my_lab_bastion, I could use the ProxyJump feature as follows:
workstation :
|
scp -o "ProxyJump=my_lab_login @my_lab_bastion " a_file my_lab_login @my_workstation :some_place/ |
Same can be used with rsync as well.
Easing SSH connections inside Grid'5000
Look at the Advanced OAR tutorial, for more information about the use of the ssh
or it's OAR wrapper oarsh
to connect to machines reserved in jobs.
SSH keys handling
SSH key types
SSH support 3 types of keys:
- rsa1: RSA algorithm, SSH protocol version 1 (default key name: identity), DEPRECATED
- rsa: RSA algorithm, SSH protocol version 2 (default key name: id_rsa)
- dsa: DSA algorithm, SSH protocol version 2 (default key name: id_dsa), DEPRECATED since OpenSSH 7
You are advised to use a rsa key.
About the public keys format supported in Grid'5000
The only format of the public_keys supported in Grid'5000 is the openSSH format. You MUST provide and use ssh public keys in this format.
- Examples
- SSH2 like public key (NOT SUPPORTED) :
---- BEGIN SSH2 PUBLIC KEY ---- Comment: rsa-key-20090623 AAAAB3NzaC1yc2EAAAABJQAAAIEA1YO87ubDgjQmCEdyX98UZ1RaBNAEXNGUNX2t D/lEw7MPShJKpVYpcj4JhrOqTc0QXIcLqefkucDaoAIlEAp7e5aShWhWFtYR5Mwn qAF1hrMBMF0xJIqgZjUWUPxvvFVeQXkObUWQkRyj5AjlG9+qQDLOoD9GgBOqfLDV edGCLoM= ---- END SSH2 PUBLIC KEY ----
- OpenSSH like public key :
ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEA1YO87ubDgjQmCEdyX98UZ1RaBNAEXNGUNX2tD/lEw7MPShJKpVYpcj4JhrOqTc0QXIcLqefkucDaoAIlEAp7e5aShWhWFtYR5MwnqAF1hrMBMF0xJIqgZjUWUPxvvFVeQXkObUWQkRyj5AjlG9+qQDLOoD9GgBOqfLDVedGCLoM=
To convert ssh public keys from SSH2 to OpenSSH, see this tutorial.
SSH keys vs. password, passphrase
An SSH key-pair is a set of 2 files. A rsa key-pair for instance is in your ~/.ssh
directory:
id_rsa
: the private key, to be kept on a safe storage, no permissions on the file for anybody except you (i.e. mode 600).id_rsa.pub
: the public key, to be dispatched on the systems that you want to connect using SSH (within the authorized_keys files)
SSH connections using SSH keys do not use the unix account password.
Passphrase: the general case
In order to protect your private key from being maliciously used to access your systems (if one steals your private key file for instance), you must provide a passphrase. Empty passphrase should only be used for system accounts keys that need automatic (none-interactive) login on remote systems (no passphrase eases scripting) or for SSH host key generation (normal users should not have to care about this).
For a regular user's usage, you actually won't have to enter your passphrase each time you connect to a system using SSH if you use a ssh-agent. This ssh-agent stores the key so that you only need to enter your passphrase once.
The Grid'5000 case
It is common usage on HPC cluster to use SSH keys without passphrase, so that connection to nodes can be scripted, and never interrupted by any authentication interactive process. Furthermore, using a ssh-agent is often not a good solution, because it doesn't work well with shell scripts, non-interactive sessions, screen sessions ... because you are always asked to enter your passphrase once.
In the Grid'5000 context, the advice then is to use:
- a SSH key with passphrase to access the access machines (this is usually your SSH key used in other context than Grid'5000 as well).
- another dedicated SSH key without passphrase, stored on your Grid'5000 home directories for internal connections. This is the Grid'5000 internal ssh key pair. See below.
About the Grid'5000 internal ssh key pair
When your Grid5000 account is created, an internal public/private ssh key pair is automatically generated for you. This ssh key pair allows you to:
- connect from frontend to frontend between grid5000 sites without entering your password.
- connect to the reserved nodes from one of the frontends.
This key pair is 'internal key' as it is only used inside the Grid5000 network. Therefore, this key is not copied on access-north or access-south and thus cannot be used to access grid5000 from outside. For that purpose, you have to generate your own ssh key. More information here.
The internal key is decomposed in a private and a public key respectively stored in ~/.ssh/id_rsa
and ~/.ssh/id_rsa.pub
. The public key is also appended to the ~/.ssh/authorized_keys
file), making it an authorized key for connections.
Warning | |
---|---|
You cannot remove the internal ssh key or modify it as it is periodically rewritten. If you need to regenerate a new key, you can contact us and we will generate a new one for you. |
SSH Key generation
Please use the command ssh-keygen -t rsa
to generate your SSH RSA key, and give a non-empty passphrase. The public key can then be used to encrypt messages that only the private key will be able to decrypt.
$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/bob/.ssh/id_rsa): Enter passphrase (empty for no passphrase): ******************** Enter same passphrase again: ******************** Your identification has been saved in /home/bob/.ssh/id_rsa. Your public key has been saved in /home/bob/.ssh/id_rsa.pub. The key fingerprint is: 00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:ff bob@leponge
SSH Key usage
Once you generated your key, you need to dispatch its public part on every server you may want to access.
This is automatically done in Grid'5000.
However, here is some general information.
To dispatch your id_rsa.pub
public key on the server named myserver (login mylogin):
A command named ssh-copy-id may be provided within the SSH package installed on your system. In such a case, you may simply run:
NB: this requires your key(s) to be stored in a SSH agent, see below.
Or you can use the following command:
Using the SSH agent
- Storing keys and passphrases
To tell the ssh-agent to store your key AND remember the passphrase so that you do not have to enter it everytime you connect via SSH to a remote machine, just do a:
$ ssh-add Enter passphrase for /home/bob/.ssh/id_rsa: ******************** Identity added: /home/bob/.ssh/id_rsa (/home/bob/.ssh/id_rsa)
You may look at the keys the agent actually stores with the following command:
$ ssh-add -l 2048 SHA256:Bx7zKapa1Gz04+LtvxTsQCkvojKjB2iPmx2OLxfkXjl /home/bob/.ssh/id_rsa (RSA)
- forwarding the agent (useful if chaining SSH connections)
If not activated by default, you may need to run the SSH command with the -A
parameter in order your key storage agent to be forwarded on connected hosts, so that you can chain other SSH command from this host without password or passphrase afterward. (try a ssh-add -l
on the remote host to check if your key is forwarded). You may also add a "ForwardAgent yes" statement in your ~/.ssh/config
file.
Warning | |
---|---|
Be warned that forwarding the agent (and then the data it stores) on a untrusted remote server is unsafe. |
Tips
How to avoid SSH host key checking?
With the StrictHostKeyChecking
option, SSH host key checking can be turned off. This option can be set in the ~/.ssh/config
file:
StrictHostKeyChecking no
Or it can be passed on the command line:
How not to get tons of SSH errors about Man-in-the-middle attacks while deploying images ?
If you get the following error when you try to connect a machine using ssh
:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is 6e:44:89:6d:ac:fc:d8:84:fd:2b:fb:22:e5:ba:5c:88. Please contact your system administrator.
This is because SSH get worried by the fact that the machine answering to the connection is not the same from run to run. This is actually really logical if you just redeployed the image so it is not same system that is answering...
Technically, the file /etc/ssh/ssh_host_dsa_key.pub
is likely to be different in your own deployed image and in the default image. SSH will thus freak out since such replacement usually denote that someone is intercepting the communication and pretend to be the server to get informations from you.
If you don't want to care about this issues, there are several solutions:
- Add
StrictHostKeyChecking=no
to your.ssh/config
file to explain SSH to ignore about those errors. - Pass this option (
StrictHostKeyChecking=no
) on the command line to ssh (using -o) - Make sure that you have the same
host_dsa_key
in your own images than in defaults one. They can usually be found in the pre/post install scripts of your site.
Outside of Grid'5000 scope, the correct solution is to fix your ~/.ssh/known_hosts
, either by hand or using the command ssh-keygen
-R hostname
.
bash-completion
The bash-completion package eases the ssh command usage by providing completion for hostnames and more.
On Debian:
Escape characters
- ~.
Disconnect you, even if your remote command hangs.
- ~C
Open command line. Allows the addition of port forwardings using the -L and -R options.
See `man ssh' for more details
SSH clients software
Un*x clients
- OpenSSH
OpenSSH is available in every good linux distro, and every *BSD, and Mac OS X.
Mac OS X tools
- SSHKeychain is a Cocoa application that acts as a frontend to
ssh-agent
. It uses the Mac OS X Keychain to store the key (so you don't have to remember another password), and it can manage a set of tunnels. - Cyberduck is a Cocoa FTP and SFTP client.
Windows clients
- OpenSSH on Windows 10+
OpenSSH is natively available since Windows 10 April 2018 Update.
- OpenSSH/Cygwin
OpenSSH is available with Cygwin. You may then find the same features in your SSH client even if you run Windows. Furthermore, Cygwin also embeds many other GNU Un*x like tools, and even a FREE X server for windows. USE CYGWIN ;). URL: http://www.cygwin.com/
- Putty
Free windowish SSH client: http://www.chiark.greenend.org.uk/~sgtatham/putty/ About proxy commands with putty: https://stackoverflow.com/questions/28926612/putty-configuration-equivalent-to-openssh-proxycommand
More
Depending on your client, you will be able to configure ssh to use more fancy options.
For more OpenSSH ssh client options, see man ssh
and man ssh_config
, or look at the official openSSH website.
For more informations about the OpenSSH/SSH Protocols, see OpenSSH/SSH Protocols page on wikibooks.org.