How to test HDD & SSD for journal / metadata compatibility?

You have read a lot about fast SSDs which do not perform well in a ceph cluster?
SSDs which die fast in a ceph cluster?
Shall you prefer HDDs over SSDs?
You have no clue how to determine what is true or what is a myth?

To understand this article fully you shall read also this both articles:

• What disk or drive types to use with ceph?
• How ceph is handling atomic persisting of IOs/objects
• Ceph storage devices

Available tools under linux for our tests

Under linux we are pretty happy to have a bunch of tools, which help us to test read in depth our drives. Well known tools are

• fio
  flexible I/O tester
• dd
  disk dump

Controller dependent impacts on your tests

Linux use disk caches, so also do hardware controllers. These caches will cause wrong tests, since we will test the speed of the RAM instead the speed of the drive.

Disable the linux cache

Linux uses for all drives a cache. Here we need first to disable the write-cache of linux on our drive

sudo hdparm -W 0 /dev/hda 0

In the above example we disable the write cache for the drive /dev/hda

Disable HP controller cache

HP controllers have typically RAID array management, which itself requires also controller based caches to optimize the write and read performance. This caches we also need to disable!

sudo hpacucli ctrl slot=2 modify dwc=disable
sudo hpacucli controller slot=2 logicaldrive 1 modify arrayaccelerator=disable

The above command assumes, that your HP controller is in slot 2 and your drive is the number 1:

Using fio

For the more thorough tests we need to use fio. This is the sample command for our tests:

sudo fio --filename=/dev/<yourDevice> --direct=1 --sync=1 --rw=write --bs=4k --numjobs=<numOfConcurrentThreads> --iodepth=1 --runtime=600 --time_based --group_reporting --name=journal-test

Let us dive into the command options:

• –filename
  the device we want to test.
  E.g. /dev/sda or /dev/nvme0n1
• –direct
  here we tell fio to work with O_DIRECT (details see Performance considerations for journal / metadata disks)
• –sync
  here we teill fio to work with O_DSYNC (details see Performance considerations for journal / metadata disks)
• –rw
  what IO pattern to use. In our case we use write for sequential writes as journal/metadata writes are always sequential (details see Performance considerations for journal / metadata disks))
• –bs
  this is the block size. With the blocksize we simulate the object size handed over from a client to our OSD.. In our example here, we are submitting 4K objects.
  4k is is probably a worst case scenario. If you know your workload in terms of block size, you can here modify it to your needs
• –numjobs
  here we simulate the amount of concurrent client accesses to our ceph OSD. number of threads that will be running, think this has ceph-osd daemons writing to the journal
• –iodepth
  we are submitting IO one by one.
• –runtime
  job duration in seconds. Here you shall test different durations to ensure that all optimizations of the HDD/SSD vendor gets overridden to get down to the real performance values of your drive.
  The drive optimizations are typically tiering of different storage types. So this tiers are helping to increase the overall performance. Means something like memory, caches and on SSDs faster SSD cells, as the cells for the final data storage on SSDs.
• –time_based
  if you have a fast drive, the test could finish the test before reaching the “runtime” limit. So this ensures to run the test for the specified runtime. So basically it reruns the operation over an over again until the total runtime is reached.
• –group_reporting
  This tells fio to report a overall value. If you have multiple threads (numjobs) usually fio will report each job independently. But the independent report does not help us here. So we group the results into one overall result with this parameter.
• –name
  name of this run/test

IMPORTANT NOTICE

The above command will overwrite your data on the drive! So be careful to take the correct drive, else you operating system or even your data is getting overwritten!

Tests to saturate your fastest drives

Your disks can be so fast that you need multiple tests to max out (saturate) the disks. We can try this with the runtime or with the numjobs parameters.

Test via long running tests

As named already above you need to test different runtimes with –runtime parameter to saturate the caches or drive tunings. Start with 60 seconds and increase it step wise to 120, 180. My tests are often this:

• 60
  start test
• 180
  second test
• 600
  third test

If the 600 seconds test has nearly the same values as the previous tests, you either have already the saturation or we cannot reach itdo not reach a saturation

Test via parallel running jobs

If you have really fast drives, the above named test will not saturate your drive. To saturate the drive we have to run more parallel threads against the drive with the parameter –numjobs. My tactic here is to try first one thread and to get up by one increment.

• –numjobs=1
• –numjobs=2
• –numjobs=3
• –numjobs=4

Using dd

Sometimes you do not have the fio on you system nor the option to install it. In this case you can use also dd. This is the sample command for our dd tests. First let us generate a random test file and we ensure that the file is synced to disk

sudo dd if=/dev/urandom of=randomtestfile bs=1M count=1024
sudo sync

now lets run the test against our drive

sudo dd if=randomtestfile of=/dev/<yourdrive> bs=4k count=100000 oflag=direct,dsync

Let us dive into the command options:

• if
  the input file, which is either our test file or the random device
• of
  the output file, which is either our test file or our drive
• bs
  this is the block size. With the blocksize we simulate the object size handed over from a client to our OSD.. In our example here, we are submitting 4K objects.
  4k is is probably a worst case scenario. If you know your workload in terms of block size, you can here modify it to your needs
• count
  here we tell the the dd command how many operations to process
• oflag
  here we define the IO operations like O_DIRECT and O_DSYNC (details see Performance considerations for journal / metadata disks)

Community driven performance measurings

You can submit your own measurings also here to let other users reduce their work too, which you also have profited yourself. So please share your details, it costs you only one minute, which helps others to reduce maybe hours of days of senseless analysis.

Drive vendor	Drive vendor (not listed above)	Drive market model name	Drive model code	Drive capaacity in (GB)	Device connection type	Device connection type (not listed above)	Drive firmware version	Bandwidth: 1 thread	IOPS: 1 thread	Bandwidth: 5 threads	IOPS: 5 threads	Bandwidth: 10 threads	IOPS: 10 threads	commands executed for your test	results got from the above command	Notes / Hints to share to the community
Intenso		ThomaspepQQ		Gray Mountain	other			63481545103	40375122477	46805260705	85721554407	84786464085	18432425306	樂貸網｜全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com	樂貸網｜全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com
Intenso		RicardocagTT		Klimmen	other			08863318443	18884346705	42611155108	55342071845	23034627456	38014254332	樂貸網 全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com/	樂貸網 全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com/
Intel		DarrellDexDP		Khujand	other			10357536765	46236517247	51815835281	21167570541	80430528523	87678774225	水微晶玻尿酸 - 八千代醫美集團 https://yachiyo.com.tw/	水微晶玻尿酸 - 八千代醫美集團 https://yachiyo.com.tw/
BIWIN		AndrewpruptOW		Kuwait	other			77266064603	38424584716	54607012234	53706550835	42666253704	28481525522	第一借錢網-借錢,小額借款 https://168cash.com.tw/	第一借錢網-借錢,小額借款 https://168cash.com.tw/
Maxtor		RandyusafeQS		Aarschot	other			45045003217	50230743881	75862817264	88864684128	46802143373	28521516316	品空間----您是SOHO族、團隊創業、新創工作者- https://gouterspace.com/	品空間----您是SOHO族、團隊創業、新創工作者- https://gouterspace.com/
Plextor		EdwardKedUE		Kuwait	other			65211242080	88202524603	12753084548	46377711856	07435805414	63220021121	第一借錢網 https://168cash.com.tw/	第一借錢網 https://168cash.com.tw/
TrekStor		RicardocagTT		Klimmen	other			75738467378	03070162546	27214148534	62688000651	41351377638	82071382833	樂貸網 全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com/	樂貸網 全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com/
Goodram		RicardocagTT		Klimmen	other			04802510457	45645777251	48784246836	66035136820	35057411348	70041507334	樂貸網 全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com/	樂貸網 全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com/
Verico		ThomaspepQQ		Gray Mountain	other			74615557423	42616440131	65531603882	71660301856	01632167284	47034782275	樂貸網｜全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com	樂貸網｜全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com
KingDian		AndrewpruptOW		Kuwait	other			63618124746	70402276445	15067304223	85220863035	34828817321	03201833034	第一借錢網-借錢,小額借款 https://168cash.com.tw/	第一借錢網-借錢,小額借款 https://168cash.com.tw/
Mach		DarrellDexDP		Khujand	other			67336061804	27712161368	86841684453	77804484775	58805737544	58661836215	水微晶玻尿酸 - 八千代醫美集團 https://yachiyo.com.tw/	水微晶玻尿酸 - 八千代醫美集團 https://yachiyo.com.tw/
Emtec		EdwardKedUE		Kuwait	other			66472054262	85822834872	41744616841	86555744282	58710222386	43854464343	第一借錢網 https://168cash.com.tw/	第一借錢網 https://168cash.com.tw/
Philips		RandyusafeQS		Aarschot	other			15246054378	03773175534	27708175660	43787751261	85084644108	15288406247	品空間----您是SOHO族、團隊創業、新創工作者- https://gouterspace.com/	品空間----您是SOHO族、團隊創業、新創工作者- https://gouterspace.com/
SSSTC		RicardocagTT		Klimmen	other			67506111703	00865818320	38127553005	68285421466	06256860404	84282006308	樂貸網 全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com/	樂貸網 全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com/
HP		HarleybleltDW		Cheltenham	other			68060330088	86873142842	52383086560	57424282114	83221428012	80507550060	八千代醫美集團特別聘請來自台大、馬偕、長庚、台北醫學大學等三十多位，擁有豐富醫美經驗的專業醫師及多位女醫生看診 https://yachiyo.com.tw/	八千代醫美集團特別聘請來自台大、馬偕、長庚、台北醫學大學等三十多位，擁有豐富醫美經驗的專業醫師及多位女醫生看診 https://yachiyo.com.tw/
Patriot		HarleybleltDW		Cheltenham	other			25153652488	24203374040	33856575171	36751118845	32836005163	74801761575	八千代醫美集團特別聘請來自台大、馬偕、長庚、台北醫學大學等三十多位，擁有豐富醫美經驗的專業醫師及多位女醫生看診 https://yachiyo.com.tw/	八千代醫美集團特別聘請來自台大、馬偕、長庚、台北醫學大學等三十多位，擁有豐富醫美經驗的專業醫師及多位女醫生看診 https://yachiyo.com.tw/
addlink		ThomaspepQQ		Gray Mountain	other			00163367806	82084472611	65315372666	13587608668	81350331557	14322771166	樂貸網｜全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com	樂貸網｜全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com
G.Skill		AndrewpruptOW		Kuwait	other			38425182743	33867215813	13486085131	02122366674	34583301865	21147207224	第一借錢網-借錢,小額借款 https://168cash.com.tw/	第一借錢網-借錢,小額借款 https://168cash.com.tw/
Xlyne		EdwardKedUE		Kuwait	other			38340000533	00645687137	46384685760	66102727373	12374655120	53616738812	第一借錢網 https://168cash.com.tw/	第一借錢網 https://168cash.com.tw/
Kingston		DarrellDexDP		Khujand	other			74278011563	47571638825	08786360601	08146666650	50462538666	55387017162	水微晶玻尿酸 - 八千代醫美集團 https://yachiyo.com.tw/	水微晶玻尿酸 - 八千代醫美集團 https://yachiyo.com.tw/
Toshiba		RandyusafeQS		Aarschot	other			64524716218	26336727226	67045105701	16820746754	27424111622	23228266705	品空間----您是SOHO族、團隊創業、新創工作者- https://gouterspace.com/	品空間----您是SOHO族、團隊創業、新創工作者- https://gouterspace.com/
KingDian		HarleybleltDW		Cheltenham	other			51818416182	08848325361	31334212043	38410813748	34326571242	52322405185	八千代醫美集團特別聘請來自台大、馬偕、長庚、台北醫學大學等三十多位，擁有豐富醫美經驗的專業醫師及多位女醫生看診 https://yachiyo.com.tw/	八千代醫美集團特別聘請來自台大、馬偕、長庚、台北醫學大學等三十多位，擁有豐富醫美經驗的專業醫師及多位女醫生看診 https://yachiyo.com.tw/
Verbatim		HarleybleltDW		Cheltenham	other			44460584065	04144271332	73274020247	54028303671	84070457842	45226272564	八千代醫美集團特別聘請來自台大、馬偕、長庚、台北醫學大學等三十多位，擁有豐富醫美經驗的專業醫師及多位女醫生看診 https://yachiyo.com.tw/	八千代醫美集團特別聘請來自台大、馬偕、長庚、台北醫學大學等三十多位，擁有豐富醫美經驗的專業醫師及多位女醫生看診 https://yachiyo.com.tw/
Patriot		ThomaspepQQ		Gray Mountain	other			42513837434	67628243044	26150745838	66153085288	82314148821	82262335753	樂貸網｜全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com	樂貸網｜全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com
PNY		EdwardKedUE		Kuwait	other			62430531607	72620235052	56075317885	56177547144	17445223452	13307480425	第一借錢網 https://168cash.com.tw/	第一借錢網 https://168cash.com.tw/
ADATA		DarrellDexDP		Khujand	other			21256422048	68324724785	18322732526	74351458636	57843735043	73578514574	水微晶玻尿酸 - 八千代醫美集團 https://yachiyo.com.tw/	水微晶玻尿酸 - 八千代醫美集團 https://yachiyo.com.tw/
MicroStorage		AndrewpruptOW		Kuwait	other			83370836423	01050616104	01031026177	66373875375	03477215036	22277004018	第一借錢網-借錢,小額借款 https://168cash.com.tw/	第一借錢網-借錢,小額借款 https://168cash.com.tw/
ADATA		RandyusafeQS		Aarschot	other			57337221124	55117533770	55122270526	50341442325	75237774264	05213765053	品空間----您是SOHO族、團隊創業、新創工作者- https://gouterspace.com/	品空間----您是SOHO族、團隊創業、新創工作者- https://gouterspace.com/
Lexar		RicardocagTT		Klimmen	other			08081251677	58683202505	66048061337	77413060521	35337404868	47874251606	樂貸網 全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com/	樂貸網 全台最多身分證借錢、小額借款服務資訊平台推薦 https://yujie365.com/
Silicon		HarleybleltDW		Cheltenham	other			56477850326	27470717333	42255887376	00771167512	02461620714	23782787546	八千代醫美集團特別聘請來自台大、馬偕、長庚、台北醫學大學等三十多位，擁有豐富醫美經驗的專業醫師及多位女醫生看診 https://yachiyo.com.tw/	八千代醫美集團特別聘請來自台大、馬偕、長庚、台北醫學大學等三十多位，擁有豐富醫美經驗的專業醫師及多位女醫生看診 https://yachiyo.com.tw/