API: Difference between revisions

From Grid5000
Jump to navigation Jump to search
No edit summary
Line 3: Line 3:
{{Portal|Tutorial}}
{{Portal|Tutorial}}
{{Pages|API}}
{{Pages|API}}
{{TutorialHeader}}


== Introduction ==
== Introduction ==

Revision as of 16:56, 9 December 2020

See also: API | API tutorial | API specification
Note.png Note

This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

Introduction

Until 2009, Grid'5000 was mainly accessed and operated via shell commands executed from frontend machines. To ease scripting and provide better access to the functionalities of the instrument (Grid'5000), an API has been developed on top of most of the Grid'5000 tools by the development team and is available to users since November 2009.

To achieve ease of use and large accessibility, we use the HTTP protocol (RFC2616). As a consequence of using the HTTP protocol as an application protocol, any HTTP client can be used to query the API: command-line tools (cURL), browsers, and the numerous HTTP libraries available in your favorite programming language.

We have chosen to be as close as possible to the principles of a REST (REpresentational State Transfer) architecture, which recommends to define the semantics of an operation on a resource by means of the standard HTTP operations (e.g. POST, GET, PUT, DELETE) and the standard HTTP status codes (e.g. 200, 401, 404, 500). REST usages include, but are not limited to, implementing CRUD (Create, Read, Update, Delete) semantics over the network. We also implement the concept of "Hypermedia as the Engine of Application State" (HATEOAS), by specifying a set of hyperlinks in all responses returned by the API, which allows a user agent to discover at runtime the set of available resources as well as their semantics and content types, and transition from one resource to another.

Security is ensured by encrypting the traffic using HTTP over SSL/TLS (HTTPS, RFC5246), and requesting login:password credentials using the HTTP Basic Authentication (RFC2617) scheme. A mutual authentication scheme using SSL Client Certificates has been envisioned, but not implemented due to lack of resources.

Finally, fault tolerance and scalability are ensured by distributing all the APIs over the Grid'5000 sites, and using proven HTTP technology such as load balancers and proxies to monitor the application servers and route the traffic to alive servers.

Overview of the Current API Landscape

Api Overview.png

Platform state and reproducibility

The reference-API data (platform state/description) is stored in a central Grid5000 git repository which contains numerous characteristics about nodes. This repository currently is hosted on github. https://github.com/grid5000/reference-repository

Those characteristics can change over time and affect performances or other sensible elements, thus it is important to keep a reference of the state of the nodes you used at the time of an experiment.

At the moment, there is no automatic association between an experiment and the state of used nodes.

You are advised to retrieve manually and keep the last commit identifier of the git repository for later references.

To do so, you can retrieve the information provided by this URL: https://api.github.com/repos/grid5000/reference-repository/git/refs/heads/master

E.g.: 

   "ref": "refs/heads/master",
   "url": "https://api.github.com/repos/grid5000/reference-repository/git/refs/heads/master",
   "object": {
       "sha": "fa6830cb904414522929fcea57d2941404e31621",
       "type": "commit",
       "url": "https://api.github.com/repos/grid5000/reference-repository/git/commits/fa6830cb904414522929fcea57d2941404e31621"
   }

And save the "sha" property.

It is not possible to access the data at the time of a past experiment through the API, because the API only serves the latest state.

To access such data, you may access the git repository directly, e.g.: 

https://github.com/grid5000/reference-repository/tree/fa6830cb904414522929fcea57d2941404e31621/data/grid5000

Using the API

The API entry-point is (both from within Grid'5000 and outside): 

 https://api.grid5000.fr/

HTTP

To understand how the API works, it is important to know how the HTTP protocol works. For an in-depth view, please refer to RFC 2616

In short, a client sends a request to the server, with control metadata added as HTTP headers, and the server sends back a response. For instance:

Example of HTTP Request (we Accept any kind of content, and we provide Authorization credentials): 

 GET /3.0/sites/rennes HTTP/1.1
 Authorization: Basic xxx
 User-Agent: curl/7.21.2 (x86_64-apple-darwin10.4.0) libcurl/7.21.2 OpenSSL/1.0.0a zlib/1.2.5 libidn/1.19
 Host: api.grid5000.fr
 Accept: */*

Example of HTTP Response: 

HTTP/1.1  200 OK
Date: Fri, 04 Dec 2020 09:42:50 GMT
Server: thin
Allow: GET
Vary: accept,Accept-Encoding
Last-Modified: Thu, 03 Dec 2020 17:31:21 GMT
ETag: W/"3b0ebf623e5b58991c056d75fcd37fa9"
Content-Type: application/json;chartset=utf-8
Cache-Control: max-age=60, public, must-revalidate=true, proxy-revalidate=true, s-maxage=60
Content-Length: 3789
X-Request-Id: a1087546-3d47-4140-b8f4-0c7a0e05fd5f
X-Runtime: 0.004494
Via: 1.1 api-server-v3.rennes.grid5000.fr:4444
X-Api-Auth-Type: IDENT
X-Api-User-CN: auser
X-Remote-Ident: auser
X-Kadeploy-User: auser

{
 "compilation_server": false,
 "description": "Grid5000 Rennes site",
 "email_contact": "support-staff@lists.grid5000.fr",
 "frontend_ip": "172.16.111.106",
 "g5ksubnet": {
   "gateway": "10.159.255.254",
   "network": "10.156.0.0/14"
 },
 "ipv6": {
   "prefix": "2001:660:4406:07",
   "site_global_kavlan": 16,
   "site_index": 7
 },
 "kavlan_ip_range": "10.24.0.0/14",
 "kavlans": {
   "1": {
     "gateway": "192.168.207.254",
     "network": "192.168.192.0/20"
   },
   "16": {
     "gateway": "10.27.255.254",
     "network": "10.27.192.0/18"
   },
   "2": {
     "gateway": "192.168.223.254",
     "network": "192.168.208.0/20"
   },
   "3": {
     "gateway": "192.168.239.254",
     "network": "192.168.224.0/20"
   },
   "4": {
     "gateway": "10.24.63.254",
     "network": "10.24.0.0/18"
   },
   "5": {
     "gateway": "10.24.127.254",
     "network": "10.24.64.0/18"
   },
   "6": {
     "gateway": "10.24.191.254",
     "network": "10.24.128.0/18"
   },
   "7": {
     "gateway": "10.24.255.254",
     "network": "10.24.192.0/18"
   },
   "8": {
     "gateway": "10.25.63.254",
     "network": "10.25.0.0/18"
   },
   "9": {
     "gateway": "10.25.127.254",
     "network": "10.25.64.0/18"
   },
   "default": {
     "gateway": "172.16.111.254",
     "network": "172.16.96.0/20"
   }
 },
 "latitude": 48.1,
 "location": "Rennes, France",
 "longitude": -1.6667,
 "name": "Rennes",
 "production": true,
 "renater_ip": "192.168.4.19",
 "security_contact": "support-staff@lists.grid5000.fr",
 "sys_admin_contact": "support-staff@lists.grid5000.fr",
 "type": "site",
 "uid": "rennes",
 "user_support_contact": "support-staff@lists.grid5000.fr",
 "virt_ip_range": "10.156.0.0/14",
 "web": "http://www.irisa.fr",
 "version": "44a8812238067115b482b283b38c44d7d55ec585",
 "links": [
   {
     "rel": "clusters",
     "href": "/3.0/sites/rennes/clusters",
     "type": "application/vnd.grid5000.collection+json"
   },
   {
     "rel": "network_equipments",
     "href": "/3.0/sites/rennes/network_equipments",
     "type": "application/vnd.grid5000.collection+json"
   },
   {
     "rel": "pdus",
     "href": "/3.0/sites/rennes/pdus",
     "type": "application/vnd.grid5000.collection+json"
   },
   {
     "rel": "servers",
     "href": "/3.0/sites/rennes/servers",
     "type": "application/vnd.grid5000.collection+json"
   },
   {
     "rel": "self",
     "type": "application/vnd.grid5000.item+json",
     "href": "/3.0/sites/rennes"
   },
   {
     "rel": "parent",
     "type": "application/vnd.grid5000.item+json",
     "href": "/3.0/"
   },
   {
     "rel": "version",
     "type": "application/vnd.grid5000.item+json",
     "href": "/3.0/sites/rennes/versions/44a8812238067115b482b283b38c44d7d55ec585"
   },
   {
     "rel": "versions",
     "type": "application/vnd.grid5000.collection+json",
     "href": "/3.0/sites/rennes/versions"
   },
   {
     "rel": "jobs",
     "type": "application/vnd.grid5000.collection+json",
     "href": "/3.0/sites/rennes/jobs"
   },
   {
     "rel": "deployments",
     "type": "application/vnd.grid5000.collection+json",
     "href": "/3.0/sites/rennes/deployments"
   },
   {
     "rel": "vlans",
     "type": "application/vnd.grid5000.collection+json",
     "href": "/3.0/sites/rennes/vlans"
   },
   {
     "rel": "metrics",
     "type": "application/vnd.grid5000.collection+json",
     "href": "/3.0/sites/rennes/metrics"
   },
   {
     "rel": "storage",
     "type": "application/vnd.grid5000.collection+json",
     "href": "/3.0/sites/rennes/storage"
   },
   {
     "rel": "status",
     "type": "application/vnd.grid5000.item+json",
     "href": "/3.0/sites/rennes/status"
   }
 ]
}

Authentication

From outside Grid'5000

  • When accessing the API from outside Grid'5000, you MUST send your Grid'5000 credentials (login and password) via the use of the HTTP Basic Authentication mechanism (every browser and the vast majority of HTTP libraries support this mechanism). Example with cURL:
 crohr@local-machine:~$ curl -k -u mylogin:mypassword -X POST https://api.grid5000.fr/stable/sites/rennes/jobs -d "command=sleep 100"
  • You may also choose to set up an SSH tunnel with one of the access machines available from the Internet, so that the authentication of your requests is handled transparently for you (see below #How to set up an SSH Tunnel).

From within Grid'5000

  • From within Grid'5000, you are transparently authenticated IF AND ONLY IF you connect from a frontend or access machine.
 crohr@frontend:~$ curl -k -X POST https://api.grid5000.fr/stable/sites/rennes/jobs -d "command=sleep 100"
  • Connections from other Grid'5000 machines require setting up an SSH tunnel between your node and a frontend machine (see #How to set up an SSH Tunnel).

Media Type

A Media Type (also called MIME Type, or Content-Type) is an identifier for file formats on the Internet. When a Client (we’ll call it a User-Agent in the rest of the documentation) receives an HTTP response from a Server, it needs to know the content-type of the response to be able to interpret it, and make use of it. For instance, your browser knows how to deal with formats such as text/html (HTML source file), image/jpeg (JPEG Image), application/pdf (PDF Document), etc.

In Grid‘5000 API, we are using two custom media types:

  • one to represent a resource item: application/vnd.grid5000.item+json
  • one to represent a resource collection (having references to multiple items) : application/vnd.grid5000.collection+json

Those two media types are based on the JSON format, but using properties with specific semantics that are described below.

A User-Agent that knows about these specifics will then be able to handle that kind of payload, and interact with the Grid‘5000 API.

Properties

Every application/vnd.grid5000.item+json or application/vnd.grid5000.collection+json payload is a JSON object (also called hash or dictionary), which comes with the following mandatory properties:

  • links, which is an array of link elements that define the relationship between the current resource and other resources of the API. Each link element is an object, which will have at least rel, href and type keys defined. rel specifies the link relationship between the current and the target resource, href specifies the URI of the target resource and type if the target resource is a collection or an item (using the media types).

The application/vnd.grid5000.item+json media type has the additional mandatory properties:

  • a unique id (uid) property will be returned, that uniquely identify the item within the collection.

The application/vnd.grid5000.collection+json media type has the additional mandatory properties:

  • an items property containing an array of item elements will be returned. Each item element is an object that contains a (possibly partial) description of the item. An item element may or may not have a links property.
  • a total property, specifying what is the total number of items in the collection. This number is always greater or equal to the number of items returned. This is especially useful for paginating collections.
  • an offset property, specifying what is the current offset of the first item that appears in the collection, among the total number of items in the collection. This is especially useful for paginating collections.

Link Relations

Valid link attributes

Each link element has three properties:

  • rel specifies the relationship between the current resource, and the resource targeted by the link
  • href specifies the URI at which the target resource can be accessed. This is most of the time an HTTP URI (http: scheme), but it could based on other schemes (xmpp:, mailto:, …)
  • type specifies the content-type of the target resource
Link Relationships

Some of link relation values for the rel property of a link element that can be found:

  • self: refers to the current resource
  • parent: refers to the first parent resource in the resource hierarchy
  • sites: refers to the list of Sites
  • jobs: refers to the list of Jobs
  • vlans: refers to the list of Vlans
  • …

Depending on the part of the API, some specifics relation values can be found. For example, in the vlan part, a dhcpd relationship can be found to start or stop a dhcp server.

Content Negotiation

When you query a resource, you ask for a representation of an object. Depending on the resource you are requesting, there could be multiple representations (of different content types) available (e.g. XML, JSON, PDF, HTML, etc.). It is the responsibility of the user to indicate which representation (or format) it wants.

To set the format that you accept, the Grid'5000 APIs support two methods:

  • you could either explicitly insert the format at the end of the URL, e.g.:
 GET entry-point/global-version-id/resource.json
  • or, you may prefer to set the Accept HTTP header to the correct mime type, e.g.:
 GET entry-point/global-version-id/resource
 Accept: application/json

If you put both, the Accept HTTP header will be ignored.

API Versioning

The Grid'5000 API is made up of multiple APIs that may evolve independently from each other. Changes occurring in one API are reflected by a new version number for this particular API. In the long term, scripts or applications that are built upon multiple APIs may have to deal with multiple version numbers for each API, which is not a very enjoyable perspective. That's why the Grid'5000 API is globally versioned: a global version id will map to the specific API version numbers in a transparent manner. All you have to do is to suffix the API entry-point with the global version id of your choice: 

 https://api.grid5000.fr/global-version-id

See https://api.grid5000.fr for the list of available global version ids. Starting from the 2.0 version, the format of the global version ids for STABLE versions will be as follows: 

 major.minor

The latest stable version is available under the alias stable.

The unstable version will always be called sid.

Scripts and applications that are programmed against a specific global version id of STABLE quality should never encounter problems due to changes in the APIs, until the official deprecation of the version.

Note.png Note

The current stable version of Grid'5000 API is 3.0.

Resources

The resources that can be queried are described in the documentation of each API. The resources are identified by a relative URI, which should be added to the end of the API entry-point to form the URL.

For example, /sites/rennes is a resource exposed by the Reference API. If you would like to get the representation of this resource as it is returned by the 3.0 version of the Grid'5000 API, the URL to enter in your HTTP client would be: 

 https://api.grid5000.fr/3.0/sites/rennes

As an other example, to get the list of sites, the URL will be: 

 https://api.grid5000.fr/3.0/sites

For a particular cluster: 

https://api.grid5000.fr/3.0/sites/rennes/clusters/paravance/

Versioning

The resources are described in the reference-repository, which is a Git repository. Therefore, the API offers the possibility to list the latest changes in the Grid'5000 infrastructure, or to fetch resources description for a specific version or at a specific time.

List versions

Each URL resources can be suffixed with /versions to get a list of versions (Git commit). For example: 

$ curl https://api.grid5000.fr/3.0/sites/rennes/clusters/versions?pretty
 "total": 392,
 "offset": 0,
 "items": [
   {
     "uid": "68fdd053e6767ba2da1edf5105f57ff7b963f91f",
     "date": "Thu, 03 Dec 2020 09:56:38 GMT",
     "message": "[all] upgrade postinstall",
     "author": "Author",
     "type": "version",
     "links": [
       {
         "rel": "self",
         "href": "/3.0/sites/rennes/clusters/versions/68fdd053e6767ba2da1edf5105f57ff7b963f91f",
         "type": "application/vnd.grid5000.item+json"
       },
       {
         "rel": "parent",
         "href": "/3.0/sites/rennes/clusters",
         "type": "application/vnd.grid5000.item+json"
       }
     ]
   },
   {
     "uid": "5d9b32e3f6f0eb63608274d028994e99ea4c0698",
     "date": "Fri, 20 Nov 2020 17:20:33 GMT",
     "message": "[all] Add prometheus metrics to all clusters",
     "author": "Author",
     "type": "version",

And following the logic behind the API, each item can be accessed, like https://api.grid5000.fr/3.0/sites/rennes/clusters/versions/68fdd053e6767ba2da1edf5105f57ff7b963f91f

Get resources for a version

To get a resource description for specific version of reference-repository, the version *GET* parameter can be added to the request. For example, to get on old description of the node dahu-29 on Grenoble: 

$ curl "https://api.grid5000.fr/3.0/sites/grenoble/clusters/dahu/nodes/dahu-29?pretty&version=0c1396da414cca75faadd7b52fbb4fea11667213"

It is also possible to Get the reference repository at a specific time, using the date or timestamp *GET* parameters. With a UNIX timestamp: 

$ curl "https://api.grid5000.fr/3.0/sites/grenoble/clusters/dahu/nodes/dahu-29.json?pretty&timestamp=1606400407"


Status Codes

Every response that you'll get from the APIs will contain a status code. In Grid'5000 we use a subset of the standards HTTP status codes [1]. You may encounter one of the following:

  • A 200 status code is returned when the request is successful.
  • A 201 status code is returned when the request has been fulfilled and resulted in a new resource being created.
  • A 202 status code is returned when the request has been accepted. The real processing may happen later.
  • A 304 status code is returned on a conditional GET request [2] when the requested resource has not been modified since the last access.
  • A 400 status code is returned when something in the request of the user agent is not correct.
  • A 401 status code is returned when the user agent needs to authenticate.
  • A 403 status code is returned when the access to the requested resource is forbidden to the currently logged user.
  • A 404 status code is returned when a resource does not exist.
  • A 405 status code is returned when the user agent uses an HTTP method not supported by the resource.
  • A 406 status code is returned when the requested format is not available.
  • A 415 status code is returned when the payload of the request uses an unsupported content-type (e.g. application/xml instead of application/json).
  • A 422 status code is returned when the payload of the request uses a unsupported content-type, but the data structure is not valid.
  • A 500 status code is returned when the server encountered an error.
  • A 503 status code is returned when the service is not available.
  • A 504 status code is returned if the request took too much time to complete.

The body of responses whose status code is 4xx or 5xx may contain additional information about the error.

Caching

Warning.png Warning

On a first read, this section can be skipped without any regrets. This section is targeted at advanced users/developers interested in improving the performance of their client applications.

As described in the RFC2616: 

  HTTP is typically used for distributed information systems, where
  performance can be improved by the use of response caches. The
  HTTP/1.1 protocol includes a number of elements intended to make
  caching work as well as possible.

Hence, client applications can (and should) cache the responses so that subsequent requests for the same information use the cached data. The use of caching strategies can dramatically reduce delays and save bandwidth.

That's why most of the responses returned by the Grid'5000 APIs include HTTP headers to support one or both of the caching models: expiration-based or validation-based. If you don't know what this means, read this article: http://tomayko.com/writings/things-caches-do.

Below are the different schemes that can exist when the API returns cacheable responses.

1. no caching 

  O                           |-----|
 -|-  <-------internet------> | API |
 / \                          |-----|

2. the API builds the response once and stores it into cache for a certain amount of time. 

  O                           |-----|       |-----|
 -|-  <-------internet------> |CACHE| <---> | API |
 / \                          |-----|       |-----|

3. the client receives the response once and stores it into cache for a certain amount of time. 

  O         |-----|                         |-----|
 -|-  <---> |CACHE| <-------internet------> | API |
 / \        |-----|                         |-----|

4. both the client and the API have a cache in front of them. 

  O         |-----|                         |-----|       |-----|
 -|-  <---> |CACHE| <-------internet------> |CACHE| <---> | API |
 / \        |-----|                         |-----|       |-----|

Most of the APIs of Grid'5000 will use some kind of caching strategies from their side (scheme 2). It is recommended that client applications include a caching strategy in their implementation (scheme 4): this will save bandwidth, reduce latency and may improve the tolerance of the client to network outages.

Note that only a few HTTP libraries natively support client-side caching (e.g.: httplib2 in Python). For ruby, the rest-client-components gem offer this functionality.

FAQ

"Certificate verification failed"

Your HTTP client may produce a warning or refuses to connect if your client does not trust the certification authority (CA) that has signed our certificate. We are using a widely deployed CA (TERENA), so this should not happen. Please check your configuration before disabling certificate validation.

How to set up an SSH Tunnel

Replace login, machine and site in the following command: 

 $ ssh -NL 3443:api.grid5000.fr:443 login@machine.site.grid5000.fr

For example, if you want to access the API from a Grid'5000 node (within Grid'5000), you might set up an SSH tunnel as follows: 

 grid5000-node $ ssh -NL 3443:api.grid5000.fr:443 your_user@frontend

On the contrary, from outside Grid'5000 you would set up an SSH tunnel as follows: 

 your-machine $ ssh -NL 3443:api.grid5000.fr:443 your_user@access.grid5000.fr

As a consequence, the API entry-point, as seen from your machine, will become: 

 https://localhost:3443

Thus, an example of request would be: 

 $ curl -k -X POST https://localhost:3443/sid/sites/rennes/jobs -d "command=sleep 100"

When you're done with the API, hit CTRL-C to destroy the tunnel, unless you added the -f flag to run it in the background.

Resources

  • cURL, the recommended tool for command-line access to the API.
  • RestClient, a Ruby HTTP client.
  • Restfully, a Ruby wrapper for REST APIs.
  • g5k-campaign, a tool to launch campaigns on Grid'5000.

References

Retrieved from "https://www.grid5000.fr/mediawiki/index.php?title=API&oldid=89597"