Create Ceph cluster

I use Ubuntu 22.04 LTS on each node with installed Docker.

Create first node

apt install -yqq cephadm

cephadm bootstrap --cluster-network 10.200.6.0/24 --mon-ip 10.200.7.31 --initial-dashboard-password dashboard-password --dashboard-password-noupdate 

Dashboard URL: https://10.200.7.31:8443/
Login: admin
Password: dashboard-password

Add nodes

Run on new nodes:

apt install -y cephadm

Run on first node:

ssh-copy-id -f -i /etc/ceph/ceph.pub root@10.200.7.32
ceph orch host add ceph2 10.200.7.32 --labels _admin

ssh-copy-id -f -i /etc/ceph/ceph.pub root@10.200.7.33
ceph orch host add ceph3 10.200.7.33 --labels _admin

Remove node

Run on first node:

ceph orch host drain ceph2

ceph orch osd rm status

ceph orch host rm ceph2

Add storage

On each node:

ceph orch apply osd --all-available-devices

Status

Services:

ceph orch ps

Disks stats:

rados df

Pools stats:

ceph osd pool stats

Storage

Pools

ceph osd pool my-pool create

Services

RWG

ceph orch apply rgw rwg.srv --placement='2 label:_admin'
ceph osd pool rwg-pool create
ceph osd pool set rbd-pool compression_algorithm lz4
ceph osd pool application enable rwg-pool rwg 

RBD

ceph osd pool rbd-pool create
ceph osd pool set rbd-pool compression_algorithm lz4
ceph osd pool application enable rbd-pool rbd 

NFS

List clusters:

ceph nfs cluster ls

Details:

ceph nfs cluster info

Simple

ceph orch apply nfs nfs.single.1 --placement='1 label:_admin'

High availability

ceph nfs cluster create nfs.ha.1 'label:_admin count-per-host:1' --ingress --virtual_ip 10.200.7.30 --port 5555