Ceph测试系列（一）：Ceph 应用方案的服务器硬件选型 (2 of 2)

一、前言

redhat官网给出不同应用场景时，其推荐的Ceph集群服务器硬件配置（包括CPU/Memory/Disk/Network）。仅作为服务器配置选择时参考，不做推荐使用。

场景包含如下：

场景：一侧重IOPS的（IOPS低延迟），例如实时性要求高的，但数据量不大的。比如订单的生成。

场景二：侧重Throughput的 （吞吐量优先），吞吐量大，但IOPS延时性要求适当。例如，实时流。

场景三：侧重容量和价格Cost/Capacity的（大存储量），例如大文件的存储。

二、硬件选型标准

1. 这里主要讲IOPS，Throughput, Cost/Capacity 三种不同场景的应用方案在硬件选型时的标准：

 

SOLUTIONS

CPU

RAM

Networking

OSD Media

OSDs

Journal Media

Controller/HBA

Vendor(i.e.)

Example Uses

IOPS-OPTIMIZED

10 cores per NVMe SSD, assuming a 2 GHz CPU.

16GB baseline, plus 2GB per OSD

10 Gigabit Ethernet (GbE) per 12 OSDs (each for client- and cluster-facing networks)

High-performance, high-endurance enterprise NVMe SSDs

Four per NVMe SSD

High-performance, high-endurance enterprise NVMe SSD, co-located with OSDs

Native PCIe bus

SuperMICRO(SYS-5038MR-OSD006P)

Database

THROUGHPUT-OPTIMIZED

0.5 cores per HDD, assuming a 2 GHz CPU

16GB baseline, plus 2GB per OSD

10 GbE per 12 OSDs (each for client- and cluster-facing networks)

7,200 RPM enterprise HDDs

One per HDD

High-endurance, high-performance enterprise serial-attached SCSI (SAS) or NVMe SSDs;

OSD-to-journal ratio: 4-5:1 for an SSD journal, or 12-18:1 for an NVMe journal

Host bus adapter (HBA): just a bunch of disks (JBOD)

SuperMICRO(SSG-6028R-OSD072P)

Streaming media

COST/CAPACITY-OPTIMIZED

0.5 cores per HDD, assuming a 2 GHz CPU

16GB baseline, plus 2GB per OSD

10 GbE per 12 OSDs (each for client- and cluster-facing networks)

7,200 RPM enterprise HDDs

One per HDD

Co-located on the HDD

HBA:JBOD

SuperMICRO(SSG-6048R-OSD216P)

Archive

Reference: https://www.redhat.com/en/resources/resources-red-hat-ceph-storage-hardware-selection-guide-html

 

2. SuperMicro推荐的Ceph服务器机型：

块和文件存储类服务器：OSD 服务器 (CPU负载低)

CategoryBlock and File OSD Node Server SKUSSG-6019P-ACR12L+SSG-5029P-E1CTR12LSSG-6029P-E1CR12LProfileDensity OptimizedCost OptimizedCompute OptimizedForm Factor1U2U2UCPUDual Intel® Xeon® Scalable Processors (8C/16T each)Dual Intel® Xeon® Scalable Processors (12C/24T each)Dual Intel® Xeon® Scalable Processors (8C/16T each)Memory96GB DDR4 RDIMMHBABroadcom 3216 (IT mode)Broadcom 3008 (IT mode)Broadcom 3008 (IT mode)Power600W Redundant Power Supplies800W Redundant Power Supplies1200W Redundant Power SuppliesNICDual-Port 25G Mellanox (SFP+ compatibility) via AOCStorageTypeQtyForm FactorTypeQtyForm FactorTypeQtyForm FactorCachingOptional NVMeUp to 22.5" 7mmOptional NVMe1AICOptional NVMe1AICCapacity6-12 TB 7K RPM123.5" SAS3 HDD6-12 TB 7K RPM123.5" SAS3 HDD6-12 TB 7K RPM123.5" SAS3 HDDDrive (OS)2x 2.5" 7mm SATA SSD2x 2.5"  SATA SSD2x 2.5"  SATA SSD

媒体类媒体流和媒体仓库服务器：OSD服务器 (CPU负载适中)

CategoryMedia OSD Node (File and Object) Server SKUSSG-6029P-E1CR24LSSG-6049P-E1CR45LSSG-6049P-E1CR45L+ProfileMedia StreamingMedia RepositoryForm Factor1U2UCPUDual Intel® Xeon® Scalable Processors (12C/24T each)Dual Intel® Xeon® Scalable Processors (12C/24T each)Memory192GB DDR4 RDIMM384GB DDR4 RDIMMHBABroadcom 3008 (IT mode)Broadcom 3008 (IT mode)(L model) Broadcom 3216 (IT mode)(L+ model)Power1600W Redundant Power Supplies1600W Redundant Power SuppliesNICSIOM-based Dual-Port 25G Mellanox (SFP+ compatibility) + Dual-Port 10G RJ45StorageTypeQtyForm FactorTypeQtyForm FactorCachingOptional NVMeUp to 2AICOptional NVMeUp to 62.5" U.2Capacity6-12 TB 7K RPM243.5" SAS3 HDD6-12 TB 7K RPM453.5" SAS3 HDDDrive (OS)2x 2.5" SATA SSD2x 2.5"  SATA SSD

媒体类存档服务器：OSD服务器(CPU负载稍高)

CategoryMedia OSD Node (File and Object) Server SKUSSG-6049P-E1CR60LSSG-6049P-E1CR60L+ProfileArchive (minimum recommended deployment size of 10+ OSD Nodes)Form Factor4UCPUDual Intel® Xeon® Scalable Processors (14C/28T each)Memory384GB DDR4 RDIMMHBABroadcom 3008 (IT mode)(L model) Broadcom 3616 (IT mode)(L+ model)Power2000W Redundant Power SuppliesNICSIOM-based Dual-Port 25G Mellanox (SFP+ compatibility) + Dual-Port 10G RJ45StorageTypeQtyForm FactorCachingOptional NVMeUp to 62.5" U.2Capacity6-12 TB 7K RPM603.5" SAS3 HDDDrive (OS)2x 2.5" SATA SSD

 Monitor/网关服务器 (CPU负载适中)

CategoryMonitor Node / Gateway Server SKUSYS-1019P-WTRProfileFlash Performance (minimum recommended deployment size of 3 Monitor Nodes)Form Factor1UCPUSingle Intel® Xeon® Scalable Processor (12C/24T each)Memory96GB DDR4 RDIMMHBABroadcom 3008 (IT mode)(L model)Power500W Redundant Power SuppliesNICDual-Port 25G Mellanox (SFP+ compatibility) via AOCStorageTypeQtyForm FactorCachingN/A--CapacityN/A--Drive (OS)2x 2.5" SATA SSD

美光固态高速服务器：(CPU负载高)

CategoryMicron Solid Ready Reference BOM Check with PM for detailsSYS-1028U-MON50MSYS-1029U-OSD100MServer SKUSYS-1028U-TNRT+SYS-1029U-TN10RTProfileMonitor NodeOSD NodeForm Factor1U1UCPUDual Intel® Xeon® Scalable Processors (10C/20T each)Dual Intel® Xeon® Scalable Processors (24C/48T each)Memory128GB DDR4 RDIMM384GB DDR4 RDIMMHBABroadcom 3008 (IT mode)N/APower750W Redundant Power Supplies1000W Redundant Power SuppliesNIC2x Single-Port 50GbE QSFP28 (Mellanox)2x Dual-Port 100GbE QSFP (Mellanox)StorageTypeQtyForm FactorTypeQtyForm FactorCachingN/A--Optional NVMe--CapacityN/A--6.4 TB NVMe10U.2 NVMeDrive (OS)1x SATADOM1x M.2 SATA SSD

三、结论：

可以看到高负载的CPU服务器需求集中在(美光)固态高速服务器 和 媒体类存档服务器(OSD服务器)。由于没有高负载计算过，存储服务器总体来说对CPU要求不高。所以如果你的CPU配置高，CPU负载不高，可以用如下服务器配置优化方案：

配置CEPH的副本模式为 EC（Erase Code）来使用更多的CPU资源，减少磁盘IO，而不是配置Replicated。使用低延迟高吞吐率的NVME SSD 盘 和 增加 磁盘数使用Cache Tier 和 增加Cache size: 使用Cache Tier 和 增加 Cache size如果x86服务器功耗和价格不符合你的需求，可以考虑采购arm服务器，在存储领域，arm服务器的性价比很高。

Refer to: https://www.supermicro.org.cn/en/solutions/red-hat-ceph