Subnet reservation
Problem statement
Users deploying VMs on Grid'5000 need to attribute IP addresses to them. Usually, what is done is that users take IPs from "free to use" IPs as described in Grid5000:Network. When several users use this technique, they may both provide the same IPs to their machines, thus creating a conflict.
To overcome this issue, OAR can reserve subnets, enabling scheduling and sharing among users.
|
Warning |
|---|---|
Using the subnet reservation in OAR, you will NOT get subnet isolation. For this purpose, you must use KaVLAN. The subnets obtained here just correspond to an IP range you can use. | |
Available IP ranges
Each site of Grid'5000 is allocated a /14 ip block giving thus a total of 262142 different IPs. Therefore, the maximum range of IPs has to be in this /14 subnet.
The /14 address block available on each site is divided in four /16 blocks as described here. The first three /16 are used for IP reservation through OAR.
Concerning the smallest subnet size, it is a /22, which is equivalent to 1022 hosts per subnet. We do not allow smaller subnets to avoid a scheduling overhead in OAR. Therefore, there are 192 reservable /22 subnets.
Requesting subnets
Using the Grid'5000 API
For an understandable use of the Grid'5000 API, see the API page and the practicals
Authentication
This example using the Grid'5000 API uses the Restfully, a wrapper of REST requests.
If you operate from outside Grid'5000, you need to use your login and password to be able to connect to the API.
echo "grid5000:
username: your-grid5000-login
password: your-grid5000-password" >> ~/.restclient && chmod 600 ~/.restclient
If you operate from within Grid'5000, the authentication is currently automatically handled for you (since you already had to connect to the frontend via SSH).
Code example
The following code will reserve one /22 subnet and one node.
#!/usr/bin/env ruby
require 'rubygems'
require 'restfully' # gem install restfully
require 'yaml'
config = YAML.load_file(File.expand_path("~/.restclient"))['grid5000']
config[:base_uri] = "https://api.grid5000.fr/sid/sites/rennes/jobs.json"
Restfully::Session.new(config) do |jobs, session|
jobs.submit({:resources => "slash_22=1+nodes=1,walltime=00:30:00", :command => "sleep 500"})
end
From the frontend of a site
Subnet reservation through OAR is similar to normal resource reservation.
To reserve 4 /22 subnets and 2 nodes, just type:
|
frontend:
|
oarsub -l slash_22=4+nodes=2 -I |
You can of-course have more complex request. To obtain 4 /22 on different /19 subnets, you can type:
|
|
frontend:
|
oarsub -l slash_19=4/slash_22=1+nodes=2/core=1 -I |
Working with your subnets
Once you have subnets reserved, you can work with them using the g5k-subnet gem.
Using the Grid'5000 API
Authentication
See Authentication
Getting the g5k-subnet gem
The gemfile is located on a server in Grid'5000 that is not accessible from outside. The simplest way to install the gem is described in the following.
Then, clone it on your local machine:
|
|
localhost:
|
Finally, install it:
|
|
localhost:
|
rake build |
|
|
localhost:
|
gem install g5k-subnets |
Code example
See the API documentation to learn to use the API. The following code will get the subnets of the job 123456789 of Rennes site. Using the Subnet class, we compress the subnets returned by the API (only /21 are returned in the API). You can see different printing options in the cfg definition.
#!/usr/bin/env ruby
require 'rubygems'
require 'restfully' # gem install restfully
require 'yaml'
require 'g5k-subnets'
config = YAML.load_file(File.expand_path("~/.restclient"))['grid5000']
config[:base_uri] = "https://api.grid5000.fr/sid/sites/rennes/jobs/123456789.json"
subnets = nil
Restfully::Session.new(config) do |job, session|
subnets = job['resources_by_type']['subnets']
end
s = Grid5000::Subnets.new(subnets, {:broadcast => false, :netmask => false, :prefix => true, :ips => false, :summarize => true, :gateway => false})
s.print
If the job requested 3 /21 subnets, running this sample code may print:
#get_g5k_subnets_api.rb 10.8.0.0/20 10.8.32.0/21
From inside Grid'5000
The simplest way to get the list of your allocated subnets is to use the g5k-subnets script provided on the head node of the submission. OAR provides a property file ($OAR_RESOURCE_PROPERTIES_FILE) where the resources are described. By default, the script uses this file if no other is specified with the -f option.
# g5k-subnets 10.8.0.0 10.8.8.0
The -p option prints the CIDR format
# g5k-subnets -p 10.8.0.0/21 10.8.8.0/21
Different other printing options are available (-b to display broadcast address, -n to see the netmask, and -a is equivalent to -bnp):
# g5k-subnets -a 10.8.0.0/21 10.11.255.255 255.255.252.0 10.11.255.254 10.8.8.0/21 10.11.255.255 255.255.252.0 10.11.255.254
You can also summarize the subnets into a larger one if they are contiguous:
# g5k-subnets -sp 10.8.0.0/20
However, when not contiguous, the networks are not merged.
# g5k-subnets -p 10.8.0.0/21 10.8.16.0/21 # g5k-subnets -ps 10.8.0.0/21 10.8.16.0/21
Another option available is to get the subnets of any job (running or terminated) using the -j option. This can only be used from a computer where oarstat is usable. Because it uses oarstat that will query the OAR database, it is slower than just reading a property file
# g5k-subnets -j 123456 -p 10.8.0.0/21 10.8.16.0/21
Subnets and properties
|
|
Warning |
|---|---|
If you need to use properties on nodes, you can not use the usual "-p" as demonstrated in the following example: $ oarsub -I -l slash_22=1+nodes=2 -p "cluster = 'parapluie'" # Set default walltime to 3600. There are not enough resources for your request OAR_JOB_ID=-5 # Error: oarsub failed, please verify your request syntax.The "-p" property is expressed on all resources asked, and subnets don't have a "cluster" property set to "parapluie". | |
To reach this objective, you must put the property just in from on the type of resource:
# oarsub -I -l slash_22=1+{"cluster='parapluie'"}nodes=2
[ADMISSION RULE] Set default walltime to 3600.
[ADMISSION RULE] Modify resource description with type constraints
Generate a job key...
OAR_JOB_ID=377169
Interactive mode : waiting...
Starting...
Advanced usage
Using Grid'5000 gateways
Gateways addresses of each site are given here.
You can also get the gateways addresses using the subnet class:
require 'g5k-subnets'
s = Grid5000::Subnets.new(['10.8.0.0/21'], {:gateway => true})
puts s.gateway # will print 10.11.255.254
You can get the gateway with the command too:
# g5k-subnets -gN 10.159.255.254
Obtaining a unique mac address
If you deploy VMs and have reserved subnets, you can obtain unique mac addresses corresponding to the IP.
|
Note |
|---|---|
The DHCP servers respects the mac - ip association as given by g5k-subnets, and will allocate IPs in accordance with the mac addresses. | |
You can display a corresponding mac address with g5k-subnets:
# g5k-subnets -im 10.158.16.1 00:16:3E:9E:10:01 ....
g5k-subnet also provide 2 additional binaries : g5k-ip2mac and g5k-mac2ip:
# g5k-ip2mac 10.158.16.1 00:16:3E:9E:10:01