ceph notes

https://access.redhat.com/node/1499443/part-ii-networking-recommendations
https://www.slideshare.net/mirantis/ceph-talk-vancouver-20

Server Thoughts

• Calamari (maybe an openstack vm)
  • Count: 1
  • CPU: 1x Quad Core
  • Memory: 8GB
  • Disk RAID 1 volume: <1TB
  • Disk Controller: ---
  • Networking: 2x 1GBE
• Monitor
  • Count: 2-3
  • CPU: 1x Hex Core
  • Memory: 32GB
  • Disk OS: 2x 240GB SSD?
  • Disk Monitor Store: ?1x 500gb? SSD
  • Disk Controller:
  • Networking: 1x 2Port 10GBE
• OSD
  • Count: 5
  • CPU: 1x Octa Core (
  • Memory: 64GB+ ?
  • Disk OS: 1x 240GB SSD for OS
  • Disk Journaling: 2x 480GB SSD? maybe 2?(verify with Matt journal sizes for disks may need dedicated)
  • Disk Storage: 9x 1.8TB 10k SAS SFF Drives
  • Disk Controller: 2GB Flash, 12gbit sas
  • Networking: 2x 2Port 10GBE (Client Network and Cluster Network each with 20gbps [4 ports])

Compsnets

• OSD - Storage Daemon
• Monitor - Metadata server (should setup 3 for HA)
• Calamari - Admin server

CPU

OSD using EC and not HOST pools use more CPU
Monitors are not CPU Intensive

RAM - more is better

• 1gb per daemon
• OSD deamons need 2gb ram per daemon
• A general rule of thumb is ~1GB of RAM for 1TB of storage space

Storage

• Ceph can operate with heterogeneous systems. homogeneous configurations for the OSDs assigned to a pool is recommended.

  RAID

• If you have systems with RAID controllers, configure them for RAID 0 (JBOD).
• When using Ceph as a backend for OpenStack volumes and images, we recommend a host bus adapter with SAS drives (10-15k rpms) and enterprise-grade SSDs using the same controller for journals.

Networking

• Should support 2 physical networks, 1 for client side, and 1 for CEPH side
• both should be at least 10gbps
  . The public network handles client traffic and communication with Ceph monitors. The cluster (back-side) network handles OSD heartbeats, replication, backfilling and recovery traffic.
• use jumbo frames

Journaling

• There are also file system limitations to consider: btrfs is not quite stable enough for production, but it has the ability to journal and write data simultaneously, whereas XFS and ext4 do not.
• Consider using SSD drive for journaling

OSD

• Running multiple OSDs on a single disk—​irrespective of partitions—​is NOT a good idea.
• Running an OSD and a monitor on a single disk—​irrespective of partitions—​is NOT a good idea.

Monitors

Use SSDs for Monitor stores
The Monitor stores can generate a significant amount of I/O operations, therefore an ideal storage media for the stores are Solid-state Drives (SSDs).
To ensure data integrity during power loss, you must disable all caches or safeguard them by hardware mechanisms like battery backup units or super capacitors coupled with non-volatile stores.