Subnet reservation

Problem statement

Users deploying VMs on Grid'5000 need to attribute IP addresses to them. Usually, what is done is that users take IPs from "free to use" IPs as described in Virtual Network Interlink. When several users use this technique, they may both provide the same IPs to their machines, thus creating a conflict.

To overcome this issue, OAR can reserve subnets, enabling scheduling and sharing among users.

Warning

Using the subnet reservation in OAR, you will NOT get subnet isolation. For this purpose, you must use KaVLAN. The subnets obtained here just correspond to an IP range you can use. Gateways, broadcast and netmask are the same for all subnets and correspond to the /14 of each site. Therefore, the netmask will always be 255.252.0.0, and the broadcast and gateway will always correspond to the values given here

Available IP ranges

Each site of Grid'5000 is allocated a /14 ip block giving thus a total of 262142 different IPs. Therefore, the maximum range of IPs has to be in this /14 subnet.

The /14 address block available on each site is divided in four /16 blocks as described here. The first three /16 are used for IP reservation through OAR.

Concerning the smallest subnet size, it is a /22, which is equivalent to 1022 hosts per subnet. We do not allow smaller subnets to avoid a scheduling overhead in OAR. Therefore, there are 192 reservable /22 subnets.

Requesting subnets

Using the Grid'5000 API

For an understandable use of the Grid'5000 API, see the API page and the practicals

Authentication

This example using the Grid'5000 API uses the Restfully, a wrapper of REST requests.

If you operate from outside Grid'5000, you need to use your login and password to be able to connect to the API.

  echo "grid5000:
    username: your-grid5000-login
    password: your-grid5000-password" >> ~/.restclient && chmod 600 ~/.restclient

If you operate from within Grid'5000, the authentication is currently automatically handled for you (since you already had to connect to the frontend via SSH).

Code example

The following code will reserve one /22 subnet and one node.

#!/usr/bin/env ruby
require 'rubygems'
require 'restfully' # gem install restfully
require 'yaml'

config = YAML.load_file(File.expand_path("~/.restclient"))['grid5000']
config[:base_uri] = "https://api.grid5000.fr/sid/sites/rennes/jobs.json"

Restfully::Session.new(config) do |jobs, session|
  jobs.submit({:resources => "slash_22=1+nodes=1,walltime=00:30:00", :command => "sleep 500"})
end

From the frontend of a site

Subnet reservation through OAR is similar to normal resource reservation.

To reserve 4 /22 subnets and 2 nodes, just type:

frontend:

oarsub -l slash_22=4+nodes=2 -I

You can of-course have more complex request. To obtain 4 /22 on different /19 subnets, you can type:

frontend:

oarsub -l slash_19=4/slash_22=1+nodes=2/core=1 -I

Working with your subnets

Once you have subents reserved, you can work with them using the g5k-subnet gem.

Using the Grid'5000 API

	Warning
	The 2.0 API does not take subnets into account (as as of 03/02/2011 sid API does not either). The following code is just here to be ready when the new API server implementation will be deployed.

Authentication

See Authentication

Getting the g5k-subnet gem

The gemfile is located on a server in Grid'5000 that is not accessible from outside. The simplest way to install the gem is described in the following.

First, get the gem file inside grid'5000:

frontend:

gem fetch g5k-subnets --source http://gems.grid5000.fr

Then, copy it on your local machine:

localhost:

scp frontend:g5k-subnets*.gem .

Finally, install it:

localhost:

gem install g5k-subnets*.gem

Code example

See the tutorial to learn to use the API. The following code will get the subnets of a job. Using the Subnet class, we compress the subnets returned by the API (only /21 are returned in the API). You can see different printing options in the cfg definition.

#!/usr/bin/env ruby

require 'rubygems'
require 'restfully' # gem install restfully
require 'yaml'
require 'g5k-subnets'

config = YAML.load_file(File.expand_path("~/.restclient"))['grid5000']
config[:base_uri] = "https://api.grid5000.fr/sid/sites/rennes/jobs/123456789.json"

subnets = nil
Restfully::Session.new(config) do |job, session|
  subnets = job['resources_by_type']['subnets']
end

s = Grid5000::Subnets.new(subnets, {:broadcast => false, :netmask => false, :prefix => true, :ips'=> false, :summarize => true, :gateway => false})
s.print

If the job requested 3 /21 subnets, running this sample code may print:

#get_g5k_subnets_api.rb
10.8.0.0/20
10.8.32.0/21

From inside Grid'5000

The simplest way to get the list of your allocated subnets is to use the g5k-subnets script provided on the head node of the submission. OAR provides a property file ($OAR_RESOURCE_PROPERTIES_FILE) where the resources are described. By default, the script uses this file if no other is specified with the -f option.

# g5k-subnets
10.8.0.0
10.8.8.0

The -p option prints the CIDR format

# g5k-subnets -p
10.8.0.0/21
10.8.8.0/21

Different other printing options are available (-b to display broadcast address, -n to see the netmask, and -a is equivalent to -bnp):

# g5k-subnets -a
10.8.0.0/21	10.11.255.255	255.255.252.0	10.11.255.254
10.8.8.0/21	10.11.255.255	255.255.252.0	10.11.255.254

You can also summarize the subnets into a larger one if they are contiguous:

# g5k-subnets -sp
10.8.0.0/20

However, when not contiguous, the networks are not merged.

# g5k-subnets -p
10.8.0.0/21
10.8.16.0/21
# g5k-subnets -ps
10.8.0.0/21
10.8.16.0/21

Another option available is to get the subnets of any job (running or terminated) using the -j option. This can only be used from a computer where oarstat is usable. Because it uses oarstat that will query the OAR database, it is slower than just reading a property file

# g5k-subnets -j 123456 -p
10.8.0.0/21
10.8.16.0/21

Subnets and properties

Warning

If you need to use properties on nodes, you can not use the usual "-p" as demonstrated in the following example: # oarsub -I -l slash_22=1+nodes=2 -p "cluster = 'parapluie'" [ADMISSION RULE] Set default walltime to 3600. [ADMISSION RULE] Modify resource description with type constraints Generate a job key... There are not enough resources for your request OAR_JOB_ID=-5 Oarsub failed: please verify your request syntax or ask for support to your admin. The "-p" property is expressed on all resources asked, and subnets don't have a "cluster" property set to "parapluie".

To reach this objective, you must put the property just in from on the type of resource:

# oarsub -I -l slash_22=1+{"cluster='parapluie'"}nodes=2 
[ADMISSION RULE] Set default walltime to 3600.
[ADMISSION RULE] Modify resource description with type constraints
Generate a job key...
OAR_JOB_ID=377169
Interactive mode : waiting...
Starting...

Advanced usage

Using Grid'5000 gateways

Gateways addresses of each site are given here.

You can also get the gateways addresses using the subnet class:

require 'g5k-subnets'
s = Grid5000::Subnets.new(['10.8.0.0/21'], {:gateway => true})
puts s.gateway # will print 10.11.255.254

You can get the gateway with the command too:

# g5k-subnets  -gN
10.159.255.254

Obtaining a unique mac address

If you deploy VMs and have reserved subnets, you can obtain unique mac addresses corresponding to the IP.

You can display a corresponding mac address with g5k-subnets:

# g5k-subnets -im
10.158.16.1     00:16:3E:9E:10:01
....

g5k-subnet also provide 2 additional binaries : g5k-ip2mac and g5k-mac2ip:

# g5k-ip2mac 10.158.16.1
00:16:3E:9E:10:01