Ceph

	Warning
	Available to users for testing

Introduction

Ceph is a distributed object store and file system designed to provide excellent performance, reliability and scalability. - See more at: http://ceph.com/

Grid'5000 Deployment

Sites	Size	Configuration	Rados	RBD	CephFS	RadosGW
Rennes	~ 9TB	16 OSDs on 4 nodes

Configuration

Generate your key

In order to access to the object store you will need a Cephx key. See : https://api.grid5000.fr/sid/storage/ceph/ui/

Your key will also available from the frontends :

frontend:

curl -k https://api.grid5000.fr/sid/storage/ceph/auths/jdoe.keyring

[client.jdoe]
  key = AQBwknVUwAPAIRAACddyuVTuP37M55s2aVtPrg==

Note : Replace jdoe by your login.

Create/Update/Delete Ceph pools

Requierement : Generate your key

Manage your Ceph pools from the Grid'5000 Ceph frontend : https://api.grid5000.fr/sid/storage/ceph/ui/

Replication size

• 1 : No replication (not secured, most efficient for write operations)
• n : One primary object + n-1 replicas (more security, less efficient for write operations)

	Note
	See http://ceph.com/papers/weil-thesis.pdf (page 130)

Configure Ceph on clients

On a deployed environment

frennes:

oarsub -I -t deploy

frennes:

kadeploy3 -e ubuntu-x64-1404 -k -f $OAR_NODEFILE

	Note
	This version of ubuntu offers a version of Ceph who works with RBD support in Qemu/KVM.

frennes:

ssh root@node

node:

apt-get install ceph

Create a ceph configuration file /etc/ceph/ceph.conf :

[global]
  mon initial members = ceph0,ceph1,ceph2
  mon host = 172.16.111.30,172.16.111.31,172.16.111.32

Create a ceph keyring file /etc/ceph/ceph.client.jdoe.keyring with your keyring :

[client.jdoe]
  key = AQBwknVUwAPAIRAACddyuVTuP37M55s2aVtPrg==

On the frontend or a node with production environment

	Note
	The ceph version on frontend and production environment is old. Object Store access works, but not the support of RBD in Qemu/KVM.

Create a ceph configuration file ~/.ceph/config :

[global]
  mon initial members = ceph0,ceph1,ceph2
  mon host = 172.16.111.30,172.16.111.31,172.16.111.32

Create a ceph keyring file ~/.ceph/ceph.client.jdoe.keyring with your keyring :

[client.jdoe]
  key = AQBwknVUwAPAIRAACddyuVTuP37M55s2aVtPrg==

Usage

Rados Object Store access

Requierement : Create a Ceph pool • Configure Ceph on client

From command line

	Note
	Add -c ~/.ceph/config to rados CLI when using as user (not root) from the frontend or the production environment.

Put an object into a pool

node:

rados --id jdoe --pool jdoe_pool put <infile> <objectname>

List objects of a pool

node:

rados --id jdoe --pool jdoe_pool ls

Get object from a pool

node:

rados --id jdoe --pool jdoe_pool get <objectname> <outfile>

Remove an object

node:

rados --id jdoe --pool jdoe_pool rm <objectname>

Usage informations

frontend:

rados -c ~/.ceph/config --id pmorillo --pool pmorillo_objectstore df

pool name       category                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB
pmorillo_objectstore -                    1563027            2            0            0           0            0            0          628      2558455
  total used       960300628       295991
  total avail     7800655596
  total space     9229804032

From your application (C/C++, Python, Java, Ruby, PHP...)

See : http://ceph.com/docs/master/rados/api/librados-intro/

RBD (Rados Block Device)

Requierement : Create a Ceph pool • Configure Ceph on client

Create a Rados Block Device

node:

rbd --id jdoe --pool jdoe_pool create <rbd_name> --size <MB>

node:

rbd --id jdoe --pool jdoe_pool ls

node:

rbd --id jdoe --pool jdoe_pool info <rbd_name>

Create filesystem and mount RBD

node:

modprobe rbd

node:

rbd --id jdoe --pool jdoe_pool map <rbd_name>

node:

rbd --id jdoe --pool jdoe_pool showmapped

id pool          image snap device    
1  jdoe_pool <rbd_name>  -    /dev/rbd1 

node:

mkfs.ext4 /dev/rbd1

node:

mkdir /mnt/rbd

node:

mount /dev/rbd1 /mnt/rbd

node:

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        15G  1.6G   13G  11% /
...
/dev/sda5       525G   70M  498G   1% /tmp
/dev/rbd1        93M  1.6M   85M   2% /mnt/rbd

Resize, snapshots, copy, etc...

See :

• http://ceph.com/docs/master/rbd/rados-rbd-cmds/
• http://ceph.com/docs/master/man/8/rbd/#examples
• http://ceph.com/docs/master/rbd/rbd-snapshot/
• rbd -h

QEMU/RBD

Requierement : Create a Ceph pool • Configure Ceph on client

Convert a qcow2 file into RBD

frennes:

scp /grid5000/virt-images/wheezy-x64-base.qcow2 root@node:/tmp

node:

apt-get install qemu qemu-kvm

node:

qemu-img convert -f qcow2 -O raw /tmp/wheezy-x64-base.qcow2 rbd:pool_name/debian7:id=jdoe

node:

rbd --id jdoe --pool pool_name ls

debian7

Start KVM virtual machine from a Rados Block Device

node:

screen kvm -m 1024 -drive format=raw,file=rbd:pool_name/debian7:id=jdoe -nographic