Subnet reservation

Problem statement

Users deploying VMs on Grid'5000 need to attribute IP addresses to them. Usually, what is done is that users take IPs from "free to use" IPs as described in Grid5000:Network. When several users use this technique, they may both provide the same IPs to their machines, thus creating a conflict.

To overcome this issue, OAR can reserve subnets, enabling scheduling and sharing among users.

Warning

Using the subnet reservation in OAR, you will NOT get subnet isolation. For this purpose, you must use KaVLAN. The subnets obtained here just correspond to an IP range you can use. Gateways, broadcast and netmask are the same for all subnets and correspond to the /14 of each site. Therefore, the netmask will always be 255.252.0.0, and the broadcast and gateway will always correspond to the values given here

Available IP ranges

Each site of Grid'5000 is allocated a /14 ip block giving thus a total of 262142 different IPs. Therefore, the maximum range of IPs has to be in this /14 subnet.

The /14 address block available on each site is divided in four /16 blocks as described here. The first three /16 are used for IP reservation through OAR.

Concerning the smallest subnet size, it is a /22, which is equivalent to 1022 hosts per subnet. We do not allow smaller subnets to avoid a scheduling overhead in OAR. Therefore, there are 192 reservable /22 subnets.

Requesting subnets

Using the Grid'5000 API

For an understandable use of the Grid'5000 API, see the API page and the practicals

Authentication

This example using the Grid'5000 API uses the Restfully, a wrapper of REST requests.

If you operate from outside Grid'5000, you need to use your login and password to be able to connect to the API.

  echo "grid5000:
    username: your-grid5000-login
    password: your-grid5000-password" >> ~/.restclient && chmod 600 ~/.restclient

If you operate from within Grid'5000, the authentication is currently automatically handled for you (since you already had to connect to the frontend via SSH).

Code example

The following code will reserve one /22 subnet and one node.

#!/usr/bin/env ruby
require 'rubygems'
require 'restfully' # gem install restfully
require 'yaml'

config = YAML.load_file(File.expand_path("~/.restclient"))['grid5000']
config[:base_uri] = "https://api.grid5000.fr/sid/sites/rennes/jobs.json"

Restfully::Session.new(config) do |jobs, session|
  jobs.submit({:resources => "slash_22=1+nodes=1,walltime=00:30:00", :command => "sleep 500"})
end

From the frontend of a site

Subnet reservation through OAR is similar to normal resource reservation.

To reserve 4 /22 subnets and 2 nodes, just type:

frontend:

oarsub -l slash_22=4+nodes=2 -I

You can of-course have more complex request. To obtain 4 /22 on different /19 subnets, you can type:

frontend:

oarsub -l slash_19=4/slash_22=1+nodes=2/core=1 -I

Working with your subnets

Once you have subnets reserved, you can work with them using the g5k-subnet gem.

Using the Grid'5000 API

Authentication

See Authentication

Getting the g5k-subnet gem

The gemfile is located on a server in Grid'5000 that is not accessible from outside. The simplest way to install the gem is described in the following.

Then, clone it on your local machine:

localhost:

git clone https://gitlab.inria.fr/grid5000/g5k-subnets

Finally, install it:

localhost:

rake build

localhost:

gem install g5k-subnets

Code example

See the API documentation to learn to use the API. The following code will get the subnets of the job 123456789 of Rennes site. Using the Subnet class, we compress the subnets returned by the API (only /21 are returned in the API). You can see different printing options in the cfg definition.

#!/usr/bin/env ruby

require 'rubygems'
require 'restfully' # gem install restfully
require 'yaml'
require 'g5k-subnets'

config = YAML.load_file(File.expand_path("~/.restclient"))['grid5000']
config[:base_uri] = "https://api.grid5000.fr/sid/sites/rennes/jobs/123456789.json"

subnets = nil
Restfully::Session.new(config) do |job, session|
  subnets = job['resources_by_type']['subnets']
end

s = Grid5000::Subnets.new(subnets, {:broadcast => false, :netmask => false, :prefix => true, :ips => false, :summarize => true, :gateway => false})
s.print

If the job requested 3 /21 subnets, running this sample code may print:

#get_g5k_subnets_api.rb
10.8.0.0/20
10.8.32.0/21

From inside Grid'5000

The simplest way to get the list of your allocated subnets is to use the g5k-subnets script provided on the head node of the submission. OAR provides a property file ($OAR_RESOURCE_PROPERTIES_FILE) where the resources are described. By default, the script uses this file if no other is specified with the -f option.

# g5k-subnets
10.8.0.0
10.8.8.0

The -p option prints the CIDR format

# g5k-subnets -p
10.8.0.0/21
10.8.8.0/21

Different other printing options are available (-b to display broadcast address, -n to see the netmask, and -a is equivalent to -bnp):

# g5k-subnets -a
10.8.0.0/21	10.11.255.255	255.255.252.0	10.11.255.254
10.8.8.0/21	10.11.255.255	255.255.252.0	10.11.255.254

You can also summarize the subnets into a larger one if they are contiguous:

# g5k-subnets -sp
10.8.0.0/20

However, when not contiguous, the networks are not merged.

# g5k-subnets -p
10.8.0.0/21
10.8.16.0/21
# g5k-subnets -ps
10.8.0.0/21
10.8.16.0/21

Another option available is to get the subnets of any job (running or terminated) using the -j option. This can only be used from a computer where oarstat is usable. Because it uses oarstat that will query the OAR database, it is slower than just reading a property file

# g5k-subnets -j 123456 -p
10.8.0.0/21
10.8.16.0/21

Subnets and properties

Warning

If you need to use properties on nodes, you can not use the usual "-p" as demonstrated in the following example: $ oarsub -I -l slash_22=1+nodes=2 -p "cluster = 'parapluie'" # Set default walltime to 3600. There are not enough resources for your request OAR_JOB_ID=-5 # Error: oarsub failed, please verify your request syntax. The "-p" property is expressed on all resources asked, and subnets don't have a "cluster" property set to "parapluie".

To reach this objective, you must put the property just in from on the type of resource:

# oarsub -I -l slash_22=1+{"cluster='parapluie'"}nodes=2 
[ADMISSION RULE] Set default walltime to 3600.
[ADMISSION RULE] Modify resource description with type constraints
Generate a job key...
OAR_JOB_ID=377169
Interactive mode : waiting...
Starting...

Advanced usage

Using Grid'5000 gateways

Gateways addresses of each site are given here.

You can also get the gateways addresses using the subnet class:

require 'g5k-subnets'
s = Grid5000::Subnets.new(['10.8.0.0/21'], {:gateway => true})
puts s.gateway # will print 10.11.255.254

You can get the gateway with the command too:

# g5k-subnets  -gN
10.159.255.254

Obtaining a unique mac address

If you deploy VMs and have reserved subnets, you can obtain unique mac addresses corresponding to the IP.

	Note
	The DHCP servers respects the mac - ip association as given by g5k-subnets, and will allocate IPs in accordance with the mac addresses.

You can display a corresponding mac address with g5k-subnets:

# g5k-subnets -im
10.158.16.1     00:16:3E:9E:10:01
....

g5k-subnet also provide 2 additional binaries : g5k-ip2mac and g5k-mac2ip:

# g5k-ip2mac 10.158.16.1
00:16:3E:9E:10:01