API
Introduction
Until 2009, Grid'5000 was mainly accessed and operated via shell commands executed from frontend machines. To ease scripting and provide better access to the functionalities of the instrument (Grid'5000), an API has been developed on top of most of the Grid'5000 tools by the development team and is available to users since November 2009.
To achieve ease of use and large accessibility, we use the HTTP protocol (RFC2616). As a consequence of using the HTTP protocol as an application protocol, any HTTP client can be used to query the API: command-line tools (cURL), browsers, and the numerous HTTP libraries available in your favorite programming language.
We have chosen to be as close as possible to the principles of a REST (REpresentational State Transfer) architecture, which recommends to define the semantics of an operation on a resource by means of the standard HTTP operations (e.g. POST, GET, PUT, DELETE) and the standard HTTP status codes (e.g. 200, 401, 404, 500). REST usages include, but are not limited to, implementing CRUD (Create, Read, Update, Delete) semantics over the network. We also implement the concept of "Hypermedia as the Engine of Application State" (HATEOAS), by specifying a set of hyperlinks in all responses returned by the API, which allows a user agent to discover at runtime the set of available resources as well as their semantics and content types, and transition from one resource to another.
Security is ensured by encrypting the traffic using HTTP over SSL/TLS (HTTPS, RFC5246), and requesting login:password credentials using the HTTP Basic Authentication (RFC2617) scheme. A mutual authentication scheme using SSL Client Certificates has been envisioned, but not implemented due to lack of resources.
Finally, fault tolerance and scalability are ensured by distributing all the APIs over the Grid'5000 sites, and using proven HTTP technology such as load balancers and proxies to monitor the application servers and route the traffic to alive servers.
Overview of the Current API Landscape
Platform state and reproducibility
The reference-API data (platform state/description) is stored in a central Grid5000 git repository which contains numerous characteristics about nodes. This repository currently is hosted on github. https://github.com/grid5000/reference-repository
Those characteristics can change over time and affect performances or other sensible elements, thus it is important to keep a reference of the state of the nodes you used at the time of an experiment.
At the moment, there is no automatic association between an experiment and the state of used nodes.
You are advised to retrieve manually and keep the last commit identifier of the git repository for later references.
To do so, you can retrieve the information provided by this URL: https://api.github.com/repos/grid5000/reference-repository/git/refs/heads/master
E.g.:
"ref": "refs/heads/master", "url": "https://api.github.com/repos/grid5000/reference-repository/git/refs/heads/master", "object": { "sha": "fa6830cb904414522929fcea57d2941404e31621", "type": "commit", "url": "https://api.github.com/repos/grid5000/reference-repository/git/commits/fa6830cb904414522929fcea57d2941404e31621" }
And save the "sha" property.
It is not possible to access the data at the time of a past experiment through the API, because the API only serves the latest state.
To access such data, you may access the git repository directly, e.g.:
https://github.com/grid5000/reference-repository/tree/fa6830cb904414522929fcea57d2941404e31621/data/grid5000
Using the API
The API entry-point is (both from within Grid'5000 and outside):
https://api.grid5000.fr/
HTTP
To understand how the API works, it is important to know how the HTTP protocol works. For an in-depth view, please refer to RFC 2616
In short, a client sends a request to the server, with control metadata added as HTTP headers, and the server sends back a response. For instance:
Example of HTTP Request (we Accept any kind of content, and we provide Authorization credentials):
GET/3.0/sites/rennesHTTP/1.1Authorization: Basic xxxUser-Agent: curl/7.21.2 (x86_64-apple-darwin10.4.0) libcurl/7.21.2 OpenSSL/1.0.0a zlib/1.2.5 libidn/1.19 Host: api.grid5000.frAccept: */*
Example of HTTP Response:
HTTP/1.1200 OKDate: Fri, 04 Dec 2020 09:42:50 GMT Server: thin Allow: GET Vary: accept,Accept-Encoding Last-Modified: Thu, 03 Dec 2020 17:31:21 GMT ETag: W/"3b0ebf623e5b58991c056d75fcd37fa9"Content-Type: application/json;chartset=utf-8Cache-Control: max-age=60, public, must-revalidate=true, proxy-revalidate=true, s-maxage=60 Content-Length: 3789 X-Request-Id: a1087546-3d47-4140-b8f4-0c7a0e05fd5f X-Runtime: 0.004494 Via: 1.1 api-server-v3.rennes.grid5000.fr:4444 X-Api-Auth-Type: IDENT X-Api-User-CN: auser X-Remote-Ident: auser X-Kadeploy-User: auser { "compilation_server": false, "description": "Grid5000 Rennes site", "email_contact": "support-staff@lists.grid5000.fr", "frontend_ip": "172.16.111.106", "g5ksubnet": { "gateway": "10.159.255.254", "network": "10.156.0.0/14" }, "ipv6": { "prefix": "2001:660:4406:07", "site_global_kavlan": 16, "site_index": 7 }, "kavlan_ip_range": "10.24.0.0/14", "kavlans": { "1": { "gateway": "192.168.207.254", "network": "192.168.192.0/20" }, "16": { "gateway": "10.27.255.254", "network": "10.27.192.0/18" }, "2": { "gateway": "192.168.223.254", "network": "192.168.208.0/20" }, "3": { "gateway": "192.168.239.254", "network": "192.168.224.0/20" }, "4": { "gateway": "10.24.63.254", "network": "10.24.0.0/18" }, "5": { "gateway": "10.24.127.254", "network": "10.24.64.0/18" }, "6": { "gateway": "10.24.191.254", "network": "10.24.128.0/18" }, "7": { "gateway": "10.24.255.254", "network": "10.24.192.0/18" }, "8": { "gateway": "10.25.63.254", "network": "10.25.0.0/18" }, "9": { "gateway": "10.25.127.254", "network": "10.25.64.0/18" }, "default": { "gateway": "172.16.111.254", "network": "172.16.96.0/20" } }, "latitude": 48.1, "location": "Rennes, France", "longitude": -1.6667, "name": "Rennes", "production": true, "renater_ip": "192.168.4.19", "security_contact": "support-staff@lists.grid5000.fr", "sys_admin_contact": "support-staff@lists.grid5000.fr", "type": "site", "uid": "rennes", "user_support_contact": "support-staff@lists.grid5000.fr", "virt_ip_range": "10.156.0.0/14", "web": "http://www.irisa.fr", "version": "44a8812238067115b482b283b38c44d7d55ec585", "links": [ { "rel": "clusters", "href": "/3.0/sites/rennes/clusters", "type": "application/vnd.grid5000.collection+json" }, { "rel": "network_equipments", "href": "/3.0/sites/rennes/network_equipments", "type": "application/vnd.grid5000.collection+json" }, { "rel": "pdus", "href": "/3.0/sites/rennes/pdus", "type": "application/vnd.grid5000.collection+json" }, { "rel": "servers", "href": "/3.0/sites/rennes/servers", "type": "application/vnd.grid5000.collection+json" }, { "rel": "self", "type": "application/vnd.grid5000.item+json", "href": "/3.0/sites/rennes" }, { "rel": "parent", "type": "application/vnd.grid5000.item+json", "href": "/3.0/" }, { "rel": "version", "type": "application/vnd.grid5000.item+json", "href": "/3.0/sites/rennes/versions/44a8812238067115b482b283b38c44d7d55ec585" }, { "rel": "versions", "type": "application/vnd.grid5000.collection+json", "href": "/3.0/sites/rennes/versions" }, { "rel": "jobs", "type": "application/vnd.grid5000.collection+json", "href": "/3.0/sites/rennes/jobs" }, { "rel": "deployments", "type": "application/vnd.grid5000.collection+json", "href": "/3.0/sites/rennes/deployments" }, { "rel": "vlans", "type": "application/vnd.grid5000.collection+json", "href": "/3.0/sites/rennes/vlans" }, { "rel": "metrics", "type": "application/vnd.grid5000.collection+json", "href": "/3.0/sites/rennes/metrics" }, { "rel": "storage", "type": "application/vnd.grid5000.collection+json", "href": "/3.0/sites/rennes/storage" }, { "rel": "status", "type": "application/vnd.grid5000.item+json", "href": "/3.0/sites/rennes/status" } ] }
Authentication
From outside Grid'5000
- When accessing the API from outside Grid'5000, you MUST send your Grid'5000 credentials (login and password) via the use of the HTTP Basic Authentication mechanism (every browser and the vast majority of HTTP libraries support this mechanism). Example with cURL:
crohr@local-machine:~$ curl -k -u mylogin:mypassword -X POST https://api.grid5000.fr/stable/sites/rennes/jobs -d "command=sleep 100"
- You may also choose to set up an SSH tunnel with one of the access machines available from the Internet, so that the authentication of your requests is handled transparently for you (see below #How to set up an SSH Tunnel).
From within Grid'5000
- From within Grid'5000, you are transparently authenticated IF AND ONLY IF you connect from a frontend or access machine.
crohr@frontend:~$ curl -k -X POST https://api.grid5000.fr/stable/sites/rennes/jobs -d "command=sleep 100"
- Connections from other Grid'5000 machines require setting up an SSH tunnel between your node and a frontend machine (see #How to set up an SSH Tunnel).
Media Type
A Media Type (also called MIME Type, or Content-Type) is an identifier for file formats on the Internet. When a Client (we’ll call it a User-Agent in the rest of the documentation) receives an HTTP response from a Server, it needs to know the content-type of the response to be able to interpret it, and make use of it. For instance, your browser knows how to deal with formats such as text/html (HTML source file), image/jpeg (JPEG Image), application/pdf (PDF Document), etc.
In Grid‘5000 API, we are using two custom media types:
- one to represent a resource item:
application/vnd.grid5000.item+json - one to represent a resource collection (having references to multiple items) :
application/vnd.grid5000.collection+json
Those two media types are based on the JSON format, but using properties with specific semantics that are described below.
A User-Agent that knows about these specifics will then be able to handle that kind of payload, and interact with the Grid‘5000 API.
Properties
Every application/vnd.grid5000.item+json or application/vnd.grid5000.collection+json payload is a JSON object (also called hash or dictionary), which comes with the following mandatory properties:
- links, which is an array of link elements that define the relationship between the current resource and other resources of the API. Each link element is an object, which will have at least
rel,hrefandtypekeys defined.relspecifies the link relationship between the current and the target resource,hrefspecifies the URI of the target resource andtypeif the target resource is a collection or an item (using the media types).
The application/vnd.grid5000.item+json media type has the additional mandatory properties:
- a unique
id (uid)property will be returned, that uniquely identify the item within the collection.
The application/vnd.grid5000.collection+json media type has the additional mandatory properties:
- an
itemsproperty containing an array of item elements will be returned. Each item element is an object that contains a (possibly partial) description of the item. An item element may or may not have a links property. - a
totalproperty, specifying what is the total number of items in the collection. This number is always greater or equal to the number of items returned. This is especially useful for paginating collections. - an
offsetproperty, specifying what is the current offset of the first item that appears in the collection, among the total number of items in the collection. This is especially useful for paginating collections.
Link Relations
Valid link attributes
Each link element has three properties:
relspecifies the relationship between the current resource, and the resource targeted by the linkhrefspecifies the URI at which the target resource can be accessed. This is most of the time an HTTP URI (http:scheme), but it could based on other schemes (xmpp:,mailto:, …)typespecifies the content-type of the target resource
Link Relationships
Some of link relation values for the rel property of a link element that can be found:
self: refers to the current resourceparent: refers to the first parent resource in the resource hierarchysites: refers to the list of Sitesjobs: refers to the list of Jobsvlans: refers to the list of Vlans- …
Depending on the part of the API, some specifics relation values can be found. For example, in the vlan part, a dhcpd relationship can be found to start or stop a dhcp server.
Content Negotiation
When you query a resource, you ask for a representation of an object. Depending on the resource you are requesting, there could be multiple representations (of different content types) available (e.g. XML, JSON, PDF, HTML, etc.). It is the responsibility of the user to indicate which representation (or format) it wants.
To set the format that you accept, the Grid'5000 APIs support two methods:
- you could either explicitly insert the format at the end of the URL, e.g.:
GETentry-point/global-version-id/resource.json
- or, you may prefer to set the Accept HTTP header to the correct mime type, e.g.:
GETentry-point/global-version-id/resourceAccept: application/json
If you put both, the Accept HTTP header will be ignored.
API Versioning
The Grid'5000 API is made up of multiple APIs that may evolve independently from each other. Changes occurring in one API are reflected by a new version number for this particular API. In the long term, scripts or applications that are built upon multiple APIs may have to deal with multiple version numbers for each API, which is not a very enjoyable perspective. That's why the Grid'5000 API is globally versioned: a global version id will map to the specific API version numbers in a transparent manner. All you have to do is to suffix the API entry-point with the global version id of your choice:
https://api.grid5000.fr/global-version-id
See https://api.grid5000.fr for the list of available global version ids. Starting from the 2.0 version, the format of the global version ids for STABLE versions will be as follows:
major.minor
The latest stable version is available under the alias stable.
The unstable version will always be called sid.
Scripts and applications that are programmed against a specific global version id of STABLE quality should never encounter problems due to changes in the APIs, until the official deprecation of the version.
Resources
The resources that can be queried are described in the documentation of each API. The resources are identified by a relative URI, which should be added to the end of the API entry-point to form the URL.
For example, /sites/rennes is a resource exposed by the Reference API. If you would like to get the representation of this resource as it is returned by the 3.0 version of the Grid'5000 API, the URL to enter in your HTTP client would be:
https://api.grid5000.fr/3.0/sites/rennes
As an other example, to get the list of sites, the URL will be:
https://api.grid5000.fr/3.0/sites
For a particular cluster:
https://api.grid5000.fr/3.0/sites/rennes/clusters/paravance/
Versioning
The resources are described in the reference-repository, which is a Git repository. Therefore, the API offers the possibility to list the latest changes in the Grid'5000 infrastructure, or to fetch resources description for a specific version or at a specific time.
List versions
Each URL resources can be suffixed with /versions to get a list of versions (Git commit). For example:
$ curl https://api.grid5000.fr/3.0/sites/rennes/clusters/versions?pretty "total": 392, "offset": 0, "items": [ { "uid": "68fdd053e6767ba2da1edf5105f57ff7b963f91f", "date": "Thu, 03 Dec 2020 09:56:38 GMT", "message": "[all] upgrade postinstall", "author": "Author", "type": "version", "links": [ { "rel": "self", "href": "/3.0/sites/rennes/clusters/versions/68fdd053e6767ba2da1edf5105f57ff7b963f91f", "type": "application/vnd.grid5000.item+json" }, { "rel": "parent", "href": "/3.0/sites/rennes/clusters", "type": "application/vnd.grid5000.item+json" } ] }, { "uid": "5d9b32e3f6f0eb63608274d028994e99ea4c0698", "date": "Fri, 20 Nov 2020 17:20:33 GMT", "message": "[all] Add prometheus metrics to all clusters", "author": "Author", "type": "version",
And following the logic behind the API, each item can be accessed, like https://api.grid5000.fr/3.0/sites/rennes/clusters/versions/68fdd053e6767ba2da1edf5105f57ff7b963f91f
Get resources for a version
To get a resource description for specific version of reference-repository, the version *GET* parameter can be added to the request. For example, to get on old description of the node dahu-29 on Grenoble:
$ curl "https://api.grid5000.fr/3.0/sites/grenoble/clusters/dahu/nodes/dahu-29?pretty&version=0c1396da414cca75faadd7b52fbb4fea11667213"
It is also possible to Get the reference repository at a specific time, using the date or timestamp *GET* parameters.
With a UNIX timestamp:
$ curl "https://api.grid5000.fr/3.0/sites/grenoble/clusters/dahu/nodes/dahu-29.json?pretty×tamp=1606400407"
Status Codes
Every response that you'll get from the APIs will contain a status code. In Grid'5000 we use a subset of the standards HTTP status codes [1]. You may encounter one of the following:
- A 200 status code is returned when the request is successful.
- A 201 status code is returned when the request has been fulfilled and resulted in a new resource being created.
- A 202 status code is returned when the request has been accepted. The real processing may happen later.
- A 304 status code is returned on a conditional GET request [2] when the requested resource has not been modified since the last access.
- A 400 status code is returned when something in the request of the user agent is not correct.
- A 401 status code is returned when the user agent needs to authenticate.
- A 403 status code is returned when the access to the requested resource is forbidden to the currently logged user.
- A 404 status code is returned when a resource does not exist.
- A 405 status code is returned when the user agent uses an HTTP method not supported by the resource.
- A 406 status code is returned when the requested format is not available.
- A 415 status code is returned when the payload of the request uses an unsupported content-type (e.g. application/xml instead of application/json).
- A 422 status code is returned when the payload of the request uses a unsupported content-type, but the data structure is not valid.
- A 500 status code is returned when the server encountered an error.
- A 503 status code is returned when the service is not available.
- A 504 status code is returned if the request took too much time to complete.
The body of responses whose status code is 4xx or 5xx may contain additional information about the error.
Caching
|
Warning |
|---|---|
On a first read, this section can be skipped without any regrets. This section is targeted at advanced users/developers interested in improving the performance of their client applications. | |
As described in the RFC2616:
HTTP is typically used for distributed information systems, where performance can be improved by the use of response caches. The HTTP/1.1 protocol includes a number of elements intended to make caching work as well as possible.
Hence, client applications can (and should) cache the responses so that subsequent requests for the same information use the cached data. The use of caching strategies can dramatically reduce delays and save bandwidth.
That's why most of the responses returned by the Grid'5000 APIs include HTTP headers to support one or both of the caching models: expiration-based or validation-based. If you don't know what this means, read this article: http://tomayko.com/writings/things-caches-do.
Below are the different schemes that can exist when the API returns cacheable responses.
1. no caching
O |-----| -|- <-------internet------> | API | / \ |-----|
2. the API builds the response once and stores it into cache for a certain amount of time.
O |-----| |-----| -|- <-------internet------> |CACHE| <---> | API | / \ |-----| |-----|
3. the client receives the response once and stores it into cache for a certain amount of time.
O |-----| |-----| -|- <---> |CACHE| <-------internet------> | API | / \ |-----| |-----|
4. both the client and the API have a cache in front of them.
O |-----| |-----| |-----| -|- <---> |CACHE| <-------internet------> |CACHE| <---> | API | / \ |-----| |-----| |-----|
Most of the APIs of Grid'5000 will use some kind of caching strategies from their side (scheme 2). It is recommended that client applications include a caching strategy in their implementation (scheme 4): this will save bandwidth, reduce latency and may improve the tolerance of the client to network outages.
Note that only a few HTTP libraries natively support client-side caching (e.g.: httplib2 in Python). For ruby, the rest-client-components gem offer this functionality.
FAQ
"Certificate verification failed"
Your HTTP client may produce a warning or refuses to connect if your client does not trust the certification authority (CA) that has signed our certificate. We are using a widely deployed CA (TERENA), so this should not happen. Please check your configuration before disabling certificate validation.
How to set up an SSH Tunnel
Replace login, machine and site in the following command:
$ ssh -NL 3443:api.grid5000.fr:443login@machine.site.grid5000.fr
For example, if you want to access the API from a Grid'5000 node (within Grid'5000), you might set up an SSH tunnel as follows:
grid5000-node $ ssh -NL 3443:api.grid5000.fr:443 your_user@frontend
On the contrary, from outside Grid'5000 you would set up an SSH tunnel as follows:
your-machine $ ssh -NL 3443:api.grid5000.fr:443 your_user@access.grid5000.fr
As a consequence, the API entry-point, as seen from your machine, will become:
https://localhost:3443
Thus, an example of request would be:
$ curl -k -X POST https://localhost:3443/sid/sites/rennes/jobs -d "command=sleep 100"
When you're done with the API, hit CTRL-C to destroy the tunnel, unless you added the -f flag to run it in the background.
Resources
- cURL, the recommended tool for command-line access to the API.
- RestClient, a Ruby HTTP client.
- Restfully, a Ruby wrapper for REST APIs.
- g5k-campaign, a tool to launch campaigns on Grid'5000.