Reconfigurable Firewall: Difference between revisions
(→Example usage: Add EnOSlib usage) |
|||
(31 intermediate revisions by 5 users not shown) | |||
Line 1: | Line 1: | ||
In IPv4, all communications from inside Grid'5000 to the outside | {{Portal|User}} | ||
{{Portal|Tutorial}} | |||
{{Portal|Network}} | |||
{{Status|In_production}} | |||
{{Pages|KaVLAN}} | |||
{{TutorialHeader}} | |||
= Quick overview = | |||
Using IPv6 and Grid'5000 reconfigurable firewall, it is possible to connect directly from the Internet to Grid'5000 nodes. You need to reserve some nodes (one or more), [[IPv6#First_step:_obtaining_an_IPv6_address|activate IPv6]] on these nodes, then issue some requests to the firewall API to open the firewall for these nodes. You can jump to the [[#Example usage]] section if you want to skip the detailed explanation and directly see how to do that. | |||
= Background = | |||
In IPv4, all communications from inside Grid'5000 to the outside Internet are NATed to the single public IP address of Grid'5000. It is possible to initiate a network connection from inside Grid'5000 to the outside Internet and the connection tracking of the Grid'5000 firewalls will allow communications in most situations (i.e. for protocols that are supported by standard connection tracking mechanisms). | |||
But two things are not possible, in IPv4: | But two things are not possible, in IPv4: | ||
* Initiating a direct network connection from the outside | * Initiating a direct network connection from the outside Internet to the inside of Grid'5000 (by direct connection we mean not through VPN or SSH tunnel). It is impossible to allow such connections since Grid'5000 nodes do not have public IPv4 addresses. | ||
* having connections initiated inside Grid'5000 to the outside | * having connections initiated inside Grid'5000 to the outside Internet, on protocols which are not supported by standard connection tracking mechanisms ("standard connection tracking mechanisms" is deliberately vague, it depends on the specific firewall version which may change). | ||
In IPv6, all Grid'5000 | In IPv6, all Grid'5000 nodes have a public, globally routable IPv6 addresses, and there is a Reconfigurable Firewall service which allows users to request firewall openings for specific IPv6 addresses, allowing connections initialed from the outside Internet to target specific IPv6 addresses inside Grid'5000. This page discusses the usage of this Reconfigurable Firewall service. | ||
= | = Features = | ||
The Reconfigurable Firewall (aka Dynamic Firewall) service is called g5kfw. It provides a REST API allowing: | The Reconfigurable Firewall (aka Dynamic Firewall) service is called ''g5kfw''. It provides a REST API allowing: | ||
* To request firewall openings associated with an OAR job. The request is checked to ensure that the requested openings only targets IPv6 addresses of nodes which belong exclusively to the OAR job (It is not possible to request an opening to a shared resource) | * To request firewall openings associated with an OAR job. The request is checked to ensure that the requested openings only targets IPv6 addresses of nodes which belong exclusively to the OAR job of user performing the request. (It is not possible to request an opening to a shared resource, ie. you need to reserve a full host with all its cpu cores to be allowed to open the firewall). | ||
* There can be several openings associated with an OAR job. | * There can be several openings associated with an OAR job. | ||
* At the end of the OAR job, all openings are removed. | * At the end of the OAR job, all openings are removed. | ||
* An opening is: | * An opening is: | ||
** A list of destination hostnames (which resolve to IPv6 addresses) or IPv6 addresses. These are the addresses of | ** A list of destination hostnames (which resolve to IPv6 addresses) or IPv6 addresses. These are the IPv6 addresses of Grid'5000 nodes for which a connection from the Internet will be allowed, | ||
** The kind of protocols | ** The kind of protocols authorized. There are two kinds of protocols: | ||
*** "tcp+udp": will open only for TCP / UDP flows. In this case, the list of TCP/UDP ports or port ranges that are to be allowed are also needed. "tcp+udp" is the default. | *** "tcp+udp": will open only for TCP / UDP flows. In this case, the list of TCP/UDP ports or port ranges that are to be allowed are also needed. "tcp+udp" is the default. | ||
*** "all": means open any protocol, meaning any IPv6 packet are allowed. In this case, there is no concept of ports. | *** "all": means open any protocol, meaning any IPv6 packet are allowed. In this case, there is no concept of ports. | ||
** Optional source hostnames (which resolve to IPv6 addresses) or IPv6 addresses (/128) or networks, to restrict the opening to specific sources outside Grid'5000. If none given, there is no restriction. | ** Optional source hostnames (which resolve to IPv6 addresses) or IPv6 addresses (/128) or networks, to restrict the opening to specific sources outside Grid'5000. If none given, there is no restriction. | ||
* Additionally, for any destination for which an opening is requested, ICMPv6 is also automatically | * Additionally, for any destination for which an opening is requested, ICMPv6 is also automatically authorized. | ||
= Known limitations = | = Known limitations = | ||
Due to the way Grid'5000 | Due to the way Grid'5000 Linux-based firewalls behave (interaction between nftables rules and connection tracking mechanisms), when firewall openings are removed (either because the user explicitly asks for removal or because the OAR job ends), existing established connections will '''not''' be closed. | ||
= API details = | = API details = | ||
The reconfigurable Firewall API resource URLs are of the form [https://api.grid5000.fr/stable/sites/<site>/firewall/<jobid> https://api.grid5000.fr/stable/sites/<site>/firewall/<jobid>] where ''<site>'' ''<jobid>'' are the Grid'5000 site and the OAR job number for which one requests openings. | |||
The reconfigurable Firewall API resource URLs are of the form [https://api.grid5000.fr/ | |||
There are three operations than can be performed on these resource URLs: | There are three operations than can be performed on these resource URLs: | ||
* POST: adds openings to the current openings for the OAR job. The openings are given in the POST body as a | * POST: adds openings to the current openings for the OAR job. The openings are given in the POST body as a JSON payload, with the format: | ||
<syntaxhighlight lang="json"> | <syntaxhighlight lang="json"> | ||
[ | [ | ||
{ | { | ||
"addr": "space separated list of hostnames or ipv6 host (/128) addresses" or array of, | "addr": "space separated list of hostnames or ipv6 host (/128) addresses" or array of, | ||
"proto": (optional) protocols to open, can be "tcp+udp" (the default) or "all" | "proto": (optional) protocols to open, can be "tcp+udp" (the default) or "all" | ||
"port": (mandatory if proto=="tcp+udp", forbidden if proto=="all") integer, or "space separated list of ports or port ranges" or array of. A port range is in the form "port1..port2", | |||
"src_addr": (optional) same as addr, for source addresses. Also possible to pass IPv6 network addresses (/x with x < 128) | "src_addr": (optional) same as addr, for source addresses. Also possible to pass IPv6 network addresses (/x with x < 128) | ||
}, | }, | ||
Line 45: | Line 57: | ||
* GET: returns all openings for an OAR job. | * GET: returns all openings for an OAR job. | ||
Following verification are performed: | |||
* check that user making the request has permission on the job | * check that user making the request has permission on the job | ||
* for an opening | * for an opening request, check that all requested destination addresses belong to the job | ||
* for an opening | * for an opening request, check that the job has started | ||
If the request is accepted, a success code is returned, which can be 200 if all the firewalls could be immediately configured, or 202 if one or more of the firewalls could not yet be configured. If one of the firewall could not be immediately configured (maybe there is a network issue, or the firewall is rebooting...) the service will retry to apply the opening every X seconds (X currently being set to 60 seconds) | If the request is accepted, a success HTTP code is returned, which can be 200 if all the firewalls could be immediately configured, or 202 if one or more of the firewalls could not yet be configured. If one of the firewall could not be immediately configured (maybe there is a network issue, or the firewall is rebooting...) the service will retry to apply the opening every X seconds (X currently being set to 60 seconds) | ||
If the request is refused or fails for any reason, an error code is issued (4xx or 5xx), and the response body contains a message with an explanation of the error. | If the request is refused or fails for any reason, an error code is issued (4xx or 5xx), and the response body contains a message with an explanation of the error. | ||
= | = Using the reconfigurable firewall from EnOSlib = | ||
[https://discovery.gitlabpages.inria.fr EnOSlib] has support for the reconfigurable firewall API, which makes it easy to use. | |||
Example code is provided here: [https://discovery.gitlabpages.inria.fr/enoslib/tutorials/grid5000.html#reconfigurable-firewall-open-ports-to-the-external-world EnOSlib tutorial to open ports with the reconfigurable firewall] | |||
= Example usage tutorials = | |||
{{Note|text=You can test this functionality from anywhere outside Grid'5000, like a campus, your home, or any server / VM, provided that you have IPv6 connectivity to the Internet.}} | |||
== Opening inbound SSH from the Internet == | |||
Connect to a Grid'5000 site and submit a job: | |||
<syntaxhighlight lang="shell"> | <syntaxhighlight lang="shell"> | ||
USER@fSITE:~$ oarsub | USER@fSITE:~$ oarsub -l nodes=1,walltime=1:00:00 -I | ||
USER@NODE:~$ | USER@NODE:~$ echo $OAR_JOB_ID | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Line 71: | Line 91: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
The request for opening | The request for firewall opening can be issued from anywhere, but it's more convenient to issue it from a frontend as it will avoid having to authenticate. | ||
The opening request must include the IPv6 address of the node, which is in the form <code>CLUSTERNAME-NODENUMBER-ipv6.SITE.grid5000.fr</code>, so you just need to add <code class="replace">-ipv6</code> inside the node name <code class="replace">NODE</code> (for example, if <code>NODE=sagittaire-12.lyon.grid5000.fr</code>, then <code>IPV6NODE=sagittaire-12</code><code class="replace">-ipv6</code><code>.lyon.grid5000.fr</code>). | |||
Then we send the request to the dynamic firewall service: | |||
<syntaxhighlight lang="shell"> | |||
USER@flyon:~$ curl -i https://api.grid5000.fr/stable/sites/SITE/firewall/OAR_JOB_ID -d '[{"addr": "IPV6NODE", "port": 22}]' | |||
</syntaxhighlight> | |||
If everything is ok, the answer code should be 200 or 202, and the response body should be similar to the request (except that all DNS names are converted to IPv6, and the response format is more "rigid"). | |||
For more complex requests, it's possible to have the JSON in a file: | |||
<syntaxhighlight lang="shell"> | <syntaxhighlight lang="shell"> | ||
USER@fSITE:~$ cat <<EOF > ~/example-firewall-opening.json | USER@fSITE:~$ cat <<EOF > ~/example-firewall-opening.json | ||
Line 81: | Line 112: | ||
] | ] | ||
EOF | EOF | ||
USER@flyon:~$ curl -i https://api.grid5000.fr/stable/sites/SITE/firewall/OAR_JOB_ID -d @/home/USER/example-firewall-opening.json | |||
USER@flyon:~$ curl - | |||
</syntaxhighlight> | </syntaxhighlight> | ||
Then you can try sending pings to the node: | Then you can try sending pings to the node: | ||
<syntaxhighlight lang="shell"> | <syntaxhighlight lang="shell"> | ||
LOCALWORKSTATION:~$ | LOCALWORKSTATION:~$ ping6 -c 5 IPV6NODE | ||
</syntaxhighlight> | </syntaxhighlight> | ||
If | If you have an SSH key on your workstation which allows you to connect to Grid'5000 resources (it depends on your SSH setup), you should also be able to connect directly to the node: | ||
<syntaxhighlight lang="shell"> | <syntaxhighlight lang="shell"> | ||
LOCALWORKSTATION:~$ ssh USER@IPV6NODE | LOCALWORKSTATION:~$ ssh USER@IPV6NODE | ||
Line 102: | Line 129: | ||
Then, as soon as the OARJOB is finished, the firewall opening will be deleted. | Then, as soon as the OARJOB is finished, the firewall opening will be deleted. | ||
== Restricting access to specific source addresses == | |||
For security reasons, you might not want to allow the whole Internet to access your nodes. It's possible to restrict firewall rules to specific source IP addresses. Assuming the IPv6 network of your home or lab is <code class="file">2001:db8:1842:1234::/64</code>: | |||
<syntaxhighlight lang="shell"> | |||
USER@flyon:~$ curl -i https://api.grid5000.fr/stable/sites/SITE/firewall/OAR_JOB_ID -d '[{"addr": "IPV6NODE", "src_addr": "2001:db8:1842:1234::/64", "port": 22}]' | |||
</syntaxhighlight> | |||
Now SSH to the node should work from your home/lab, but not from anywhere else. | |||
== Allowing any network protocol == | |||
You may want to experiment with protocols other than UDP and TCP, such as GRE, SCTP, IPv6-in-IPv6 tunnels... In this case, you should specifically allow "all" protocols: | |||
<syntaxhighlight lang="shell"> | |||
USER@flyon:~$ curl -i https://api.grid5000.fr/stable/sites/SITE/firewall/OAR_JOB_ID -d '[{"addr": "IPV6NODE", "proto": "all"}]' | |||
</syntaxhighlight> | |||
Note that, as a side-effect, it also allows TCP and UDP traffic on all destination ports. Use with caution or restrict the source address. | |||
= Testing end to end IPv6 connectivity = | |||
To be able to use IPv6 from anywhere to a node in grid5000, it is mandatory to have end to end IPv6 between the two. If you need to check that you have IPv6 connectivity on the path from your external host to Grid5000, you can check by pinging <code>ping6.grid5000.fr</code>: | |||
<syntaxhighlight lang="shell"> | |||
yourexternalhost$ ping -6 ping6.grid5000.fr | |||
</syntaxhighlight> | |||
If this ping fails, the issue is most likely on your side (do you have IPv6 at your home or on your campus?). |
Latest revision as of 19:56, 10 July 2024
![]() |
Note |
---|---|
This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team. |
Quick overview
Using IPv6 and Grid'5000 reconfigurable firewall, it is possible to connect directly from the Internet to Grid'5000 nodes. You need to reserve some nodes (one or more), activate IPv6 on these nodes, then issue some requests to the firewall API to open the firewall for these nodes. You can jump to the #Example usage section if you want to skip the detailed explanation and directly see how to do that.
Background
In IPv4, all communications from inside Grid'5000 to the outside Internet are NATed to the single public IP address of Grid'5000. It is possible to initiate a network connection from inside Grid'5000 to the outside Internet and the connection tracking of the Grid'5000 firewalls will allow communications in most situations (i.e. for protocols that are supported by standard connection tracking mechanisms).
But two things are not possible, in IPv4:
- Initiating a direct network connection from the outside Internet to the inside of Grid'5000 (by direct connection we mean not through VPN or SSH tunnel). It is impossible to allow such connections since Grid'5000 nodes do not have public IPv4 addresses.
- having connections initiated inside Grid'5000 to the outside Internet, on protocols which are not supported by standard connection tracking mechanisms ("standard connection tracking mechanisms" is deliberately vague, it depends on the specific firewall version which may change).
In IPv6, all Grid'5000 nodes have a public, globally routable IPv6 addresses, and there is a Reconfigurable Firewall service which allows users to request firewall openings for specific IPv6 addresses, allowing connections initialed from the outside Internet to target specific IPv6 addresses inside Grid'5000. This page discusses the usage of this Reconfigurable Firewall service.
Features
The Reconfigurable Firewall (aka Dynamic Firewall) service is called g5kfw. It provides a REST API allowing:
- To request firewall openings associated with an OAR job. The request is checked to ensure that the requested openings only targets IPv6 addresses of nodes which belong exclusively to the OAR job of user performing the request. (It is not possible to request an opening to a shared resource, ie. you need to reserve a full host with all its cpu cores to be allowed to open the firewall).
- There can be several openings associated with an OAR job.
- At the end of the OAR job, all openings are removed.
- An opening is:
- A list of destination hostnames (which resolve to IPv6 addresses) or IPv6 addresses. These are the IPv6 addresses of Grid'5000 nodes for which a connection from the Internet will be allowed,
- The kind of protocols authorized. There are two kinds of protocols:
- "tcp+udp": will open only for TCP / UDP flows. In this case, the list of TCP/UDP ports or port ranges that are to be allowed are also needed. "tcp+udp" is the default.
- "all": means open any protocol, meaning any IPv6 packet are allowed. In this case, there is no concept of ports.
- Optional source hostnames (which resolve to IPv6 addresses) or IPv6 addresses (/128) or networks, to restrict the opening to specific sources outside Grid'5000. If none given, there is no restriction.
- Additionally, for any destination for which an opening is requested, ICMPv6 is also automatically authorized.
Known limitations
Due to the way Grid'5000 Linux-based firewalls behave (interaction between nftables rules and connection tracking mechanisms), when firewall openings are removed (either because the user explicitly asks for removal or because the OAR job ends), existing established connections will not be closed.
API details
The reconfigurable Firewall API resource URLs are of the form https://api.grid5000.fr/stable/sites/<site>/firewall/<jobid> where <site> <jobid> are the Grid'5000 site and the OAR job number for which one requests openings.
There are three operations than can be performed on these resource URLs:
- POST: adds openings to the current openings for the OAR job. The openings are given in the POST body as a JSON payload, with the format:
[
{
"addr": "space separated list of hostnames or ipv6 host (/128) addresses" or array of,
"proto": (optional) protocols to open, can be "tcp+udp" (the default) or "all"
"port": (mandatory if proto=="tcp+udp", forbidden if proto=="all") integer, or "space separated list of ports or port ranges" or array of. A port range is in the form "port1..port2",
"src_addr": (optional) same as addr, for source addresses. Also possible to pass IPv6 network addresses (/x with x < 128)
},
]
- DELETE: delete all openings for the OAR job.
- GET: returns all openings for an OAR job.
Following verification are performed:
- check that user making the request has permission on the job
- for an opening request, check that all requested destination addresses belong to the job
- for an opening request, check that the job has started
If the request is accepted, a success HTTP code is returned, which can be 200 if all the firewalls could be immediately configured, or 202 if one or more of the firewalls could not yet be configured. If one of the firewall could not be immediately configured (maybe there is a network issue, or the firewall is rebooting...) the service will retry to apply the opening every X seconds (X currently being set to 60 seconds)
If the request is refused or fails for any reason, an error code is issued (4xx or 5xx), and the response body contains a message with an explanation of the error.
Using the reconfigurable firewall from EnOSlib
EnOSlib has support for the reconfigurable firewall API, which makes it easy to use.
Example code is provided here: EnOSlib tutorial to open ports with the reconfigurable firewall
Example usage tutorials
![]() |
Note |
---|---|
You can test this functionality from anywhere outside Grid'5000, like a campus, your home, or any server / VM, provided that you have IPv6 connectivity to the Internet. |
Opening inbound SSH from the Internet
Connect to a Grid'5000 site and submit a job:
USER@fSITE:~$ oarsub -l nodes=1,walltime=1:00:00 -I
USER@NODE:~$ echo $OAR_JOB_ID
From now on, take note of the SITE
, OAR_JOB_ID
and NODE
.
Activate IPv6 on the node:
USER@NODE:~$ sudo-g5k dhclient -6 br0
The request for firewall opening can be issued from anywhere, but it's more convenient to issue it from a frontend as it will avoid having to authenticate.
The opening request must include the IPv6 address of the node, which is in the form CLUSTERNAME-NODENUMBER-ipv6.SITE.grid5000.fr
, so you just need to add -ipv6
inside the node name NODE
(for example, if NODE=sagittaire-12.lyon.grid5000.fr
, then IPV6NODE=sagittaire-12
-ipv6
.lyon.grid5000.fr
).
Then we send the request to the dynamic firewall service:
USER@flyon:~$ curl -i https://api.grid5000.fr/stable/sites/SITE/firewall/OAR_JOB_ID -d '[{"addr": "IPV6NODE", "port": 22}]'
If everything is ok, the answer code should be 200 or 202, and the response body should be similar to the request (except that all DNS names are converted to IPv6, and the response format is more "rigid").
For more complex requests, it's possible to have the JSON in a file:
USER@fSITE:~$ cat <<EOF > ~/example-firewall-opening.json
[
{
"addr": "IPV6NODE",
"port": 22
}
]
EOF
USER@flyon:~$ curl -i https://api.grid5000.fr/stable/sites/SITE/firewall/OAR_JOB_ID -d @/home/USER/example-firewall-opening.json
Then you can try sending pings to the node:
LOCALWORKSTATION:~$ ping6 -c 5 IPV6NODE
If you have an SSH key on your workstation which allows you to connect to Grid'5000 resources (it depends on your SSH setup), you should also be able to connect directly to the node:
LOCALWORKSTATION:~$ ssh USER@IPV6NODE
USER@NODE:~$ ...
Then, as soon as the OARJOB is finished, the firewall opening will be deleted.
Restricting access to specific source addresses
For security reasons, you might not want to allow the whole Internet to access your nodes. It's possible to restrict firewall rules to specific source IP addresses. Assuming the IPv6 network of your home or lab is 2001:db8:1842:1234::/64
:
USER@flyon:~$ curl -i https://api.grid5000.fr/stable/sites/SITE/firewall/OAR_JOB_ID -d '[{"addr": "IPV6NODE", "src_addr": "2001:db8:1842:1234::/64", "port": 22}]'
Now SSH to the node should work from your home/lab, but not from anywhere else.
Allowing any network protocol
You may want to experiment with protocols other than UDP and TCP, such as GRE, SCTP, IPv6-in-IPv6 tunnels... In this case, you should specifically allow "all" protocols:
USER@flyon:~$ curl -i https://api.grid5000.fr/stable/sites/SITE/firewall/OAR_JOB_ID -d '[{"addr": "IPV6NODE", "proto": "all"}]'
Note that, as a side-effect, it also allows TCP and UDP traffic on all destination ports. Use with caution or restrict the source address.
Testing end to end IPv6 connectivity
To be able to use IPv6 from anywhere to a node in grid5000, it is mandatory to have end to end IPv6 between the two. If you need to check that you have IPv6 connectivity on the path from your external host to Grid5000, you can check by pinging ping6.grid5000.fr
:
yourexternalhost$ ping -6 ping6.grid5000.fr
If this ping fails, the issue is most likely on your side (do you have IPv6 at your home or on your campus?).