API: Difference between revisions

From Grid5000
Jump to navigation Jump to search
(Rewritten the introduction)
Line 5: Line 5:


== Introduction ==
== Introduction ==
At the time of writing, Grid5000 is mainly accessed and operated via shell commands executed from frontend machines. To ease scripting and provide better access to the grid functionalities, an API is currently developed by the Grid5000 Technical Committee. This API consists in a layer on top of most of the Grid5000 tools that will provide stable and well-defined interfaces accessible via the HTTP protocol.


Most of the standardization efforts are achieved by trying to be as close as possible to the principles of a '''REST''' (REpresentational State Transfer) architecture, which recommends to define the semantics of an operation on a '''resource''' by means of the '''standard HTTP operations''' (e.g. GET, PUT, POST, DELETE) and the '''standard HTTP status codes''' (e.g. 200, 401, 404, 500).
Until 2009, Grid5000 was mainly accessed and operated via shell commands executed from frontend machines. To ease scripting and provide better access to the grid functionalities, an API has been developed on top of most of the Grid5000 tools by the Grid5000 Technical Committee and is available to users since november 2009.


The concept of a REST resource must not be confused with the concept of a resource in the context of Grid5000: a REST resource, identified by a URI (e.g. /sites/rennes), represents a specific view into an object, which MAY incidentally be a Grid5000 resource (e.g. node, switch) but not necessarily.
The API main objectives are:
* Accessible from everywhere (from within and outside Grid5000), via an ubiquitous protocol;
* Easy to use;
* Stable and well defined interfaces, i.e. take the most out of standardization;
* Reasonably secure;
* Fault tolerant;
* Scalable;


As a consequence of using the HTTP protocol as an application protocol, any HTTP client can be used to query the API: command-line tools (cURL), browsers and the numerous HTTP libraries available in the vast majority of programming languages.
To achieve ''ease of use'' and ''world-wide accessibility'', we use the '''HTTP protocol''' ([http://www.w3.org/Protocols/rfc2616/rfc2616.html RFC2616]), one of the most well known protocol that is used everyday by millions of users in the biggest distributed system in the World: the Web.
As a consequence of using the HTTP protocol as an application protocol, any HTTP client can be used to query the API: command-line tools (cURL), browsersand the numerous HTTP libraries available in your favorite programming language.
 
Most of the standardization efforts are achieved by trying to be as close as possible to the principles of a [http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm '''REST'''] (REpresentational State Transfer) architecture, which recommends to define the semantics of an operation on a '''resource''' by means of the '''standard HTTP operations''' (e.g. POST, GET, PUT, DELETE) and the '''standard HTTP status codes''' (e.g. 200, 401, 404, 500). REST usages include, but are not limited to, implementing CRUD (Create, Read, Update, Delete) semantics over the network.
 
We also implement the concept of "Hypermedia as the Engine of Application State" (HATEOAS), by specifying a set of hyperlinks in all responses returned by the API, which allow a user agent to discover at runtime the set of available resources as well as their semantics and content types, and transition from one resource to another.
 
''Security'' is ensured by encrypting the traffic using HTTP over SSL/TLS (HTTPS, [http://tools.ietf.org/html/rfc5246 RFC5246]), and requesting login:password credentials using the HTTP Basic Authentication ([http://www.ietf.org/rfc/rfc2617.txt RFC2617]) scheme. A mutual authentication scheme using SSL Client Certificates is on the way.
 
Finally, ''fault tolerance'' and ''scalability'' are ensured by distributing all the APIs over the Grid5000 sites, and using proven HTTP technology such as load balancers and proxies to monitor the application servers and route the traffic to alive servers.


=== Access to the API ===
=== Access to the API ===
Line 31: Line 45:
=== Resources ===
=== Resources ===
The resources that can be queried are described in the documentation of each API.  
The resources that can be queried are described in the documentation of each API.  
The resources are identified by a URI, which should be added to the end of the API entry-point to form the URL.
The resources are identified by a relative URI, which should be added to the end of the API entry-point to form the URL.
For example, <code>/grid5000/sites/rennes</code> is a resource exposed by the Reference API. If you would like to get the representation of this resource as it is returned by the <code>2.0</code> version of the Grid5000 API, the URL to enter in your HTTP client would be:
For example, <code>/grid5000/sites/rennes</code> is a resource exposed by the Reference API. If you would like to get the representation of this resource as it is returned by the <code>2.0</code> version of the Grid5000 API, the URL to enter in your HTTP client would be:
   https://api.grid5000.fr/2.0/grid5000/sites/rennes
   https://api.grid5000.fr/2.0/grid5000/sites/rennes
Line 39: Line 53:
The list of the APIs and their documentation is available at: [[#APIs]].
The list of the APIs and their documentation is available at: [[#APIs]].


=== Format ===
=== Content Negotiation ===
When you query a REST resource, you ask for a representation of an object. Depending on the resource you are requesting, there could be multiple representations available. It is the responsibility of the user to indicate which representation (or format) it accepts.
When you query a REST resource, you ask for a representation of an object. Depending on the resource you are requesting, there could be multiple representations (of different content types) available. It is the responsibility of the user to indicate which representation (or format) it accepts.


To set the format that you accept, you could either explicitly insert the format at the end of the URL, e.g.:
To set the format that you accept, you could either explicitly insert the format at the end of the URL, e.g.:
Line 53: Line 67:
* A 200 status code is returned when the request is successful.  
* A 200 status code is returned when the request is successful.  
* A 201 status code is returned when the request has been fulfilled and resulted in a new resource being created.
* A 201 status code is returned when the request has been fulfilled and resulted in a new resource being created.
* A 304 status code is returned on a conditional GET request [http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3] when the requested resource has not been modified.
* A 202 status code is returned when the request has been accepted. The real processing may happen later.
* A 304 status code is returned on a conditional GET request [http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3] when the requested resource has not been modified since the last access.
* A 400 status code is returned when something in the request of the user agent is not correct.
* A 401 status code is returned when the user agent needs to authenticate.
* A 403 status code is returned when the access to the requested resource is forbidden to the currently logged user.
* A 403 status code is returned when the access to the requested resource is forbidden to the currently logged user.
* A 404 status code is returned when a resource does not exist.  
* A 404 status code is returned when a resource does not exist.  
* A 405 status code is returned when the user agent uses an HTTP method not supported by the resource.
* A 406 status code is returned when the requested format is not available.
* A 406 status code is returned when the requested format is not available.
* A 415 status code is returned when the payload of the request uses an unsupported content-type (e.g. application/xml instead of application/json).
* A 500 status code is returned when the server encountered an error.
* A 500 status code is returned when the server encountered an error.
* A 503 status code is returned when the service is not available.
* A 504 status code is returned if the request took too much time to complete.
The body of responses whose status code is 4xx or 5xx may contain additional information about the error.
The body of responses whose status code is 4xx or 5xx may contain additional information about the error.


Line 136: Line 158:


== Tutorials / Practicals ==
== Tutorials / Practicals ==
* [[API Main Practical]] Resource discovery, Monitoring, Submission, Deployment
* [[API Main Practical]] Resource discovery, Monitoring, Submission, Deployment.
* [[API Metrology Practical]]
* [[API Metrology Practical]] Learn how to collect near to real-time metrics about your nodes.


== FAQ ==
== FAQ ==

Revision as of 10:33, 16 June 2010


Introduction

Until 2009, Grid5000 was mainly accessed and operated via shell commands executed from frontend machines. To ease scripting and provide better access to the grid functionalities, an API has been developed on top of most of the Grid5000 tools by the Grid5000 Technical Committee and is available to users since november 2009.

The API main objectives are:

  • Accessible from everywhere (from within and outside Grid5000), via an ubiquitous protocol;
  • Easy to use;
  • Stable and well defined interfaces, i.e. take the most out of standardization;
  • Reasonably secure;
  • Fault tolerant;
  • Scalable;

To achieve ease of use and world-wide accessibility, we use the HTTP protocol (RFC2616), one of the most well known protocol that is used everyday by millions of users in the biggest distributed system in the World: the Web. As a consequence of using the HTTP protocol as an application protocol, any HTTP client can be used to query the API: command-line tools (cURL), browsers, and the numerous HTTP libraries available in your favorite programming language.

Most of the standardization efforts are achieved by trying to be as close as possible to the principles of a REST (REpresentational State Transfer) architecture, which recommends to define the semantics of an operation on a resource by means of the standard HTTP operations (e.g. POST, GET, PUT, DELETE) and the standard HTTP status codes (e.g. 200, 401, 404, 500). REST usages include, but are not limited to, implementing CRUD (Create, Read, Update, Delete) semantics over the network.

We also implement the concept of "Hypermedia as the Engine of Application State" (HATEOAS), by specifying a set of hyperlinks in all responses returned by the API, which allow a user agent to discover at runtime the set of available resources as well as their semantics and content types, and transition from one resource to another.

Security is ensured by encrypting the traffic using HTTP over SSL/TLS (HTTPS, RFC5246), and requesting login:password credentials using the HTTP Basic Authentication (RFC2617) scheme. A mutual authentication scheme using SSL Client Certificates is on the way.

Finally, fault tolerance and scalability are ensured by distributing all the APIs over the Grid5000 sites, and using proven HTTP technology such as load balancers and proxies to monitor the application servers and route the traffic to alive servers.

Access to the API

The API entry-point is (both from within Grid5000 and outside):

 https://api.grid5000.fr

Versioning

The Grid5000 API is made up of multiple APIs that may evolve independently from each other. Changes occurring in one API are reflected by a new version number for this particular API. In the long term, scripts or applications that are built upon multiple APIs may have to deal with multiple version numbers for each API, which is not a very enjoyable perspective. That's why the Grid5000 API is globally versioned: a global version id will map to the specific API version numbers in a transparent manner. All you have to do is to suffix the API entry-point with the global version id of your choice:

 https://api.grid5000.fr/global-version-id

Or, if you connect via an SSH tunnel:

 https://localhost:3443/global-version-id

See https://api.grid5000.fr for the list of available global version ids (e.g. 1.0-stable). Starting from the 2.0 version, the format of the global version ids for STABLE versions will be as follows:

 major.minor

The unstable version will always be called sid

Scripts and applications that are programmed against a specific global version id of STABLE quality should never encounter problems due to changes in the APIs, until the official deprecation of the version.

Resources

The resources that can be queried are described in the documentation of each API. The resources are identified by a relative URI, which should be added to the end of the API entry-point to form the URL. For example, /grid5000/sites/rennes is a resource exposed by the Reference API. If you would like to get the representation of this resource as it is returned by the 2.0 version of the Grid5000 API, the URL to enter in your HTTP client would be:

 https://api.grid5000.fr/2.0/grid5000/sites/rennes

Or, if you connect via an SSH tunnel:

 https://localhost:3443/2.0/grid5000/sites/rennes

The list of the APIs and their documentation is available at: #APIs.

Content Negotiation

When you query a REST resource, you ask for a representation of an object. Depending on the resource you are requesting, there could be multiple representations (of different content types) available. It is the responsibility of the user to indicate which representation (or format) it accepts.

To set the format that you accept, you could either explicitly insert the format at the end of the URL, e.g.:

 GET entry-point/global-version-id/resource.json

Or, you may prefer to set the Accept HTTP header to the correct mime type, e.g.:

 GET entry-point/global-version-id/resource
 Accept: application/json

If you put both, the Accept HTTP header will be ignored.

Status Codes

A subset of the standards HTTP status codes [1] is returned by the APIs:

  • A 200 status code is returned when the request is successful.
  • A 201 status code is returned when the request has been fulfilled and resulted in a new resource being created.
  • A 202 status code is returned when the request has been accepted. The real processing may happen later.
  • A 304 status code is returned on a conditional GET request [2] when the requested resource has not been modified since the last access.
  • A 400 status code is returned when something in the request of the user agent is not correct.
  • A 401 status code is returned when the user agent needs to authenticate.
  • A 403 status code is returned when the access to the requested resource is forbidden to the currently logged user.
  • A 404 status code is returned when a resource does not exist.
  • A 405 status code is returned when the user agent uses an HTTP method not supported by the resource.
  • A 406 status code is returned when the requested format is not available.
  • A 415 status code is returned when the payload of the request uses an unsupported content-type (e.g. application/xml instead of application/json).
  • A 500 status code is returned when the server encountered an error.
  • A 503 status code is returned when the service is not available.
  • A 504 status code is returned if the request took too much time to complete.

The body of responses whose status code is 4xx or 5xx may contain additional information about the error.

Authentication

From within Grid5000

  • From within Grid5000, you are transparently authenticated IF you connect from a frontend or access machine.
 crohr@frontend:~$ curl -k -X POST https://api.grid5000.fr/sid/grid5000/sites/rennes/jobs -d "command=sleep 100"
  • Connections from other Grid5000 machines requires setting up an SSH tunnel between your node and a frontend machine (see #Setting_up_an_SSH_Tunnel).

From outside Grid5000

  • When accessing the API from outside Grid5000, you MUST send your Grid5000 credentials (login and password) via the use of the HTTP Basic Authentication mechanism (every browser and the vast majority of HTTP libraries support this mechanism). Example with cURL:
 crohr@local-machine:~$ curl -k -u mylogin:mypassword -X POST https://api.grid5000.fr/sid/grid5000/sites/rennes/jobs -d "command=sleep 100"

If you don't want to pass your login and password each time you make a request, create a ~/.netrc file on your LOCAL MACHINE with the following content (don't forget to chmod 600 it):

 machine api.grid5000.fr 
 login mylogin
 password mypassword

Then, cURL will automatically read this file and set up the authentication when you pass the -n option in your commands:

 crohr@local-machine:~$ curl -kn -X POST https://api.grid5000.fr/sid/grid5000/sites/rennes/jobs -d "command=sleep 100"
  • You may also choose to set up an SSH tunnel with one of the access machines available from the Internet, so that the authentication of your requests is handled transparently for you (see below #Setting_up_an_SSH_Tunnel).

Setting up an SSH Tunnel

Replace login, machine and site in the following command:

 $ ssh -NL 3443:api.grid5000.fr:443 login@machine.site.grid5000.fr

For example, if you want to access the API from a Grid5000 node (within Grid5000), you might set up the SSH tunnel as follows:

 bordeaux-node $ ssh -NL 3443:api.grid5000.fr:443 crohr@frontend.bordeaux.grid5000.fr

On the contrary, from outside Grid5000 you would set up an SSH tunnel as follows:

 your-machine $ ssh -NL 3443:api.grid5000.fr:443 crohr@access.bordeaux.grid5000.fr

As a consequence, the API entry-point, as seen from your machine, will become:

 https://localhost:3443

Thus, an example of request would be:

 $ curl -k -X POST https://localhost:3443/sid/grid5000/sites/rennes/jobs -d "command=sleep 100"

When you're done with the API, hit CTRL-C to destroy the tunnel, unless you added the -f flag to run it in the background.

Caching

Warning.png Warning

On a first read, this section can be skipped without any regrets. For those interested in improving the performance of their client applications, read on !

As described in the RFC2616:

  HTTP is typically used for distributed information systems, where
  performance can be improved by the use of response caches. The
  HTTP/1.1 protocol includes a number of elements intended to make
  caching work as well as possible.

Hence, client applications can (and should) cache the responses so that subsequent requests for the same information use the cached data. The use of caching strategies can dramatically reduce delays and save bandwidth.

That's why most of the responses returned by the Grid5000 APIs include HTTP headers to support one or both of the caching models: expiration-based or validation-based. If you don't know what this means, read this article: http://tomayko.com/writings/things-caches-do.

Below are the different schemes that can exist when the API returns cacheable responses.

1. no caching

  O                           |-----|
 -|-  <-------internet------> | API |
 / \                          |-----| 

2. the API builds the response once and stores it into cache for a certain amount of time.

  O                           |-----|       |-----|
 -|-  <-------internet------> |CACHE| <---> | API |
 / \                          |-----|       |-----|

3. the client receives the response once and stores it into cache for a certain amount of time.

  O         |-----|                         |-----|
 -|-  <---> |CACHE| <-------internet------> | API |
 / \        |-----|                         |-----|

4. both the client and the API have a cache in front of them.

  O         |-----|                         |-----|       |-----|
 -|-  <---> |CACHE| <-------internet------> |CACHE| <---> | API |
 / \        |-----|                         |-----|       |-----|

Most of the APIs of Grid5000 will use some kind of caching strategies from their side (scheme 2). It is recommended that client applications include a caching strategy in their implementation (scheme 4): this will save bandwidth, reduce latency and may improve the tolerance of the client to network outages.

Note that only a few HTTP libraries natively support client-side caching (e.g.: httplib2 in Python). For ruby, the rest-client-components gem offer this functionality.

APIs

The list of available APIs and their documentation can be found at: https://api.grid5000.fr

Tutorials / Practicals

FAQ

"Certificate verification failed"

Your HTTP client may produce a warning or refuses to connect if you do not have a version of the Grid5000 Certification Authority (CA) on your machine. Currently though, the certificate infrastructure of Grid5000 is undergoing massive changes and we recommend that you disable the verification during this period.

Resources

References