You want to start with ceph and you have no idea what are the right drives or disks to use with ceph?
Ceph can use nearly any drive type which is fitting into your servers. Anyway we can reduce the drive types to group of devices.
Drive types
For storing data are tons of solutions out in the market. We will consider only the major relevant devices, which also find consideration in a ceph storage cluster.
Spindle based rotational drives, known as hard disk drives (HDD)
These drives are mostly known by their form factor, which is 2.5″ (small form factor, abbreviated SFF) and 3.5″ (large form factor, abbreviated LFF). Please note, that on discussion with storage guys, you often will hear about SFF or LFF drives. The benefit of these drives is as of end 2020, that they are still much cheaper as SSDs compared to the provided storage in TB.
Benefits of HDDs
- Very high capacities per form factor
- very good per price per TB
- well known and established technology for datacenters
- wide range of additional hardware available to support the specific features of HDDs
- high environmental sustainability, since most parts are build out of metal and have therefore a high recycling rate of the parts
Drawbacks of HDDs
- high energy consumption (compared to SSDs) in standby
- very high energy consumpltion (compared to SSDs) under load
- low IOPS (compared to SSDs)
- since moving parts are involved, the environment have to be very stable to avoid issues with the platter bearings and speed of accessing data
The storage admin’s Performance aspects of HDDs
These drives have at least one rotating disk, from there their name was derived. These disks are named platter and such a drive typically contains a bunch of platter, each platter have at least 2 moving heads, which are reading/writing the upper and lower side in the same time. The benefits of this HDDs are their huge storage capacity which are up to 24TB in one LFF-drive. The drawbacks are the moving parts like the rotating platters and the moving heads. Thus together also results into a high energy usage on stand-by along with a much higher energy usage on reading/writing (compared to SSDs)
- Typical IOPS:
- 60 IOPS for SFF 5.4k rpm
- 90 IOPS for LFF 7.2k rpm
- 100 IOPS for SFF 7.2k rpm
- 160 IOPS for SFF 10k rpm
- 240 IOPS for SFF 15k rpm
Typical bandwidth:
- 60 IOPS for SFF 5.4k rpm
- 90 IOPS for LFF 7.2k rpm
- 100 IOPS for SFF 7.2k rpm
- 160 IOPS for SFF 10k rpm
- 240 IOPS for SFF 15k rpm
- a
- These drives are mostly known by their form factor, which is 2.5″ (small form factors, abbreviated SFF) and 3.5″ (large form factor, abbreviated LFF). Please note, that on discussion with storage guys, you often will hear about SFF or LFF drives. The benefit of these drives is as of end 2020, that they are stil much cheaper as SSDs compared to the provided storage in TB.
These drives have at least one rotating disk, from there their name was derived. These disks are named platter and such a drive typically contains a bunch of platter, each platter have at least 2 moving heads, which are reading/writing the upper and lower side in the same time. The benefits of this HDDs are their huge storage capacity which are up to 24TB in one LFF-drive. The drawbacks are the moving parts like the rotating platters and the moving heads. Thus together also results into a high energy usage on stand-by along with a much higher energy usage on reading/writing (compared to SSDs)
Typical IOPS:- 60 IOPS for SFF 5.4k rpm
- 90 IOPS for LFF 7.2k rpm
- 100 IOPS for SFF 7.2k rpm
- 160 IOPS for SFF 10k rpm
- 240 IOPS for SFF 15k rpm
Typical bandwidth:
- 60 IOPS for SFF 5.4k rpm
- 90 IOPS for LFF 7.2k rpm
- 100 IOPS for SFF 7.2k rpm
- 160 IOPS for SFF 10k rpm
- 240 IOPS for SFF 15k rpm
- Solid state drives
<YG please provide a mockup/picture> These drives have gained a very fast adoption in the market with the adoption to the consumers SATA interface and in the consumer typical HDD form factor (2.5″ with available adapters to 3.5″).
These drives have had from beginning on incredible fast IOPS compared to HDDs and thus resulting in factors of performance improvement typically seen on system boot processes. Also this drives quickly saturated the bandwidth of the given SATA interface, which was 6GBit/s gross (SATA3) and resulting in about 560MBps (4.5Gbit/s) net payload speed for read or write.
You have to define where these drives are integrated. A storage device can be attached to a different wiring system or better to say data-bus. Let us go from inside (often not hot replaceable) to the outside located devices:
- PCIe cards
a very famous – because incredibly fast on release date – is intel’s PCIe card Optane 900 series. This is a PCIe card, which is attached directly to your motherboards PCIe slot. This card cannot be replace during your server is up and running, and even more you have to disassemble the server case to get down to this card to replace it. A hot replace of a defective device is thus not possible. So from the perspective to keep your server up and running, these card seem to be a bad fit. But it is not, you will read this later why. Sadly due to the usage in datacenters or high performance storage based applications, these cards are known to be expensive compared to similar alternatives. - M.2 PCIe cards
This cards have gained especially in the consumer world the major role as the new storage device format. Especially they have the same connection speed as the PCIe cards and are due to the PCB design much smaller and thus easier to produce. This factors helped to make these cards similar priced to the 2.5″ based SATA-SSDs. These things allowed these NVMe drives to gain a very fast adoption to speedy computers and laptops. Also here, like with the PCIe based devices, we have to unscrew the server case and thus we cannot replace this devices in hot operation (during server is running) - PCIe
Lets assume you have a server and this server has now attached somehow drives to it. To define the somehow, we need to understand the differences between DAS, NAS, SAN.
- DAS – direct attached storage
here we have to see also different types of drives.- her
- NAS – network attached storage
Spindle based drives
SATA
SAS
asdfas