Configuring a Block Device on a Ceph Client

First, using the ceph-admin machine, check if the cluster is in a correct status:

[root@ceph-1 ~]# ceph -s
 cluster 376331d2-4da0-4f41-8040-0cf433148a08
 health HEALTH_OK
 monmap e1: 3 mons at {ceph-1=,ceph-2=,ceph-3=}
 election epoch 10, quorum 0,1,2 ceph-1,ceph-2,ceph-3
 osdmap e73: 9 osds: 9 up, 9 in
 flags sortbitwise,require_jewel_osds
 pgmap v603: 112 pgs, 7 pools, 8833 kB data, 184 objects
 344 MB used, 8832 MB / 9176 MB avail
 112 active+clean
 [root@ceph-1 ~]#

Create the block device

[root@ceph-1 ~]# rbd create myblock --size 200 --image-format 1
 rbd: image format 1 is deprecated
 [root@ceph-1 ~]#

In the above command, myblock is the name of the rbd image. 200 is the size in MB. RBD image must be in format 1. I tried without using that option but was getting “write error: No such device or address” when mapping the RBD device on my client host.

We can check the block device by issuing the following command.

[root@ceph-1 ~]# rbd list
 [root@ceph-1 ~]#

From one of the monitor node, retrieve the client name and key by looking at the /etc/ceph/ceph.client.admin.keyring file.

[root@ceph-1 ~]# cat /etc/ceph/ceph.client.admin.keyring 
 key = AQByd5xYfFmqFBAABrv/q2mUKrQdS2Uo5nVq+g==
 caps mds = "allow *"
 caps mon = "allow *"
 caps osd = "allow *"
 [root@ceph-1 ~]#

On the client host, verify if the kernel supports rbd modules.

[root@client ~]# modprobe rbd
 [root@client ~]#

If it gives you an error, then RBD is not installed. Install kmod-rbd and kmod-libceph packages and reload the RBD module.

Map the RBD device in the client host.

[root@client ~]# echo ",, name=admin,secret=AQByd5xYfFmqFBAABrv/q2mUKrQdS2Uo5nVq+g== rbd myblock" > /sys/bus/rbd/add

The above command will create a new device on the client host.

[root@client ~]# ll /dev/rbd*
 brw-rw---- 1 root disk 252, 0 Feb 16 16:36 /dev/rbd0

Time to format the device.

[root@client ~]# mkfs.ext4 /dev/rbd0 
 mke2fs 1.42.9 (28-Dec-2013)
 Discarding device blocks: done 
 Filesystem label=
 OS type: Linux
 Block size=1024 (log=0)
 Fragment size=1024 (log=0)
 Stride=4096 blocks, Stripe width=4096 blocks
 51200 inodes, 204800 blocks
 10240 blocks (5.00%) reserved for the super user
 First data block=1
 Maximum filesystem blocks=33816576
 25 block groups
 8192 blocks per group, 8192 fragments per group
 2048 inodes per group
 Superblock backups stored on blocks: 
 8193, 24577, 40961, 57345, 73729

Allocating group tables: done 
 Writing inode tables: done 
 Creating journal (4096 blocks): done
 Writing superblocks and filesystem accounting information: done

Create a mount point and mount the device

[root@client ~]# mkdir /mnt/cephblock
 [root@client ~]# mount /dev/rbd0 /mnt/cephblock/

The device is mounted and ready to be used.

[root@client ~]# lsblk
 vda 253:0 0 20G 0 disk 
 └─vda1 253:1 0 20G 0 part /
 vdb 253:16 0 1G 0 disk 
 rbd0 252:0 0 200M 0 disk /mnt/cephblock
 [root@client ~]#
[root@client ~]# df -h
 Filesystem Size Used Avail Use% Mounted on
 /dev/vda1 20G 1.7G 19G 9% /
 devtmpfs 902M 0 902M 0% /dev
 tmpfs 920M 0 920M 0% /dev/shm
 tmpfs 920M 25M 896M 3% /run
 tmpfs 920M 0 920M 0% /sys/fs/cgroup
 tmpfs 184M 0 184M 0% /run/user/0
 /dev/rbd0 190M 1.6M 175M 1% /mnt/cephblock

When needed, you can unmount the filesystem and remove the RBD device using:

echo "0" >/sys/bus/rbd/remove