AWS Benchmark Details
Wildfire routinely runs benchmarks on AWS cloud servers. We do this for several reasons:
- The hardware is well understood by the public.
- Other users can try their solutions on AWS and see how it works.
It should be noted that AWS is not the "latest" and "fastest" in terms of server hardware.
AWS Instance
The test EC2 instance was run as a spot instance. Various runs were in Seoul, Singapore, Oregon, and even Stockholm. WildFire has AWS CLI scripts that start spot instances where they are available, and at the lowest cost.
- i4i.metal
- Dual Socket Xeon Platinum 8375C
- 32 cores with 64 threads (per socket)
- 1024 GB RAM
- 8 x 3750 GB NVMe SSDs
AWS Limitations:
AWS servers have some limitations:
- The hardware is locked down.
- The hardware's performance (especially SSDs) can vary somewhat from instance to instance and boot to boot.
This is not to denegrate AWS. They have real security requirements for the .metal servers. Just imagine if bad actors could plant malware in a NIC or in an SSD's firmware.
The Benchmark
ESS for this test uses:
- Rocky Linux 9.4
- WildFire QC "profile" test
- Read/write mix: 100/0 , 70/30 , 50/50 , 30/70 , 0/100
- 0% , 30%, and 60% block compressibility
- "go routines" at q > 200
- Block sizes of 64K , 32K , 16K, 8K , 4K
- O_DIRECT
Test Program
These tests are run with an in-house benchmark routine written in GO. These generally track FIO results, excepth that FIO reports slightly higher IOPS at small block sizes. We use GO because it gives us a quick "profile" for a device in a single operation.
Test Duration
The test is short, so bare MD arrays using "MDADM RAID" can produce very inconsistent results. This appears to be the SSDs and not MD RAID. SSDs when presented with random writes need to do internal garbage collection almost immediately. This in turn can impact the overall performance of the SSD, even for pure reads.
Test Results
All of these results are the combined read/write MB/sec measured results. All tests were run RAID-0, RAID-5, and RAID-6. All arrays were created with identical parameters. Compression for stock RAID was set at 0% as bare SSDs and bare MD RAID are not impacted by block compressibility.
The MB/s below the compressibility precentage is the fill rate prior to random block read/write tests. Each test pre-fills the test LBA range with blocks at the specified fill level. This pre-fill is 1MB blocks at q=16 and tends to saturate the media easily.
ECR RAID-0
0% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
15913MB/s 64KB 17583 17169 16964 17066 16658
32KB 13830 14667 15333 16228 16861
16KB 9024 10759 12277 14030 16750
8KB 5540 7109 8626 11052 12851
4KB 3108 4179 5372 7464 7239
30% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
21253MB/s 64KB 18803 19826 20679 21360 12736
32KB 13541 15499 17289 19458 12365
16KB 8474 10599 12627 15762 14777
8KB 4854 6501 8255 11241 10384
4KB 2720 3746 4923 7151 7295
60% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
26881MB/s 64KB 18006 20956 26057 31769 20391
32KB 11104 14692 18594 25917 22219
16KB 7812 10603 13543 19366 16154
8KB 5224 7163 9344 13444 10251
4KB 2818 3982 5327 8017 7362
ECR RAID-5
0% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
14051MB/s 64KB 17501 16294 15657 15282 14579
32KB 13830 14037 14251 14529 14836
16KB 8930 10345 11440 12723 14637
8KB 5476 6856 8233 10285 12075
4KB 3084 4105 5210 7107 7842
30% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
18706MB/s 64KB 18744 19131 19393 19348 11731
32KB 13474 14962 16276 17639 11247
16KB 8449 10375 12121 14671 13632
8KB 4823 6385 8020 10638 10042
4KB 2703 3698 4829 6917 7005
60% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
25886MB/s 64KB 16934 20422 24904 28965 18615
32KB 11057 14327 17902 24071 21117
16KB 8035 10488 13357 18382 15546
8KB 5201 7054 9169 12939 10177
4KB 2796 3907 5241 7838 7121
ECR RAID-6
0% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
12046MB/s 64KB 17343 15244 14371 13705 12919
32KB 13832 13464 13224 13159 12510
16KB 8971 10033 10841 11573 12553
8KB 5493 6740 7930 9550 11504
4KB 3100 4088 5117 6816 7671
30% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
15833MB/s 64KB 18733 18311 17894 17531 10797
32KB 13479 14450 15368 16129 11052
16KB 8473 10146 11558 13543 13069
8KB 4859 6333 7777 10112 9936
4KB 2710 3692 4761 6703 7047
60% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
25290MB/s 64KB 16187 19355 22990 26324 18394
32KB 11068 13921 17158 22405 20639
16KB 8097 10200 12839 17377 15410
8KB 5216 7040 9009 12472 10064
4KB 2807 3913 5203 7658 7182
Stock RAID-0
0% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
17018MB/s 64KB 1692 11422 11393 10535 9210
32KB 323 3612 8007 9767 8931
16KB 139 735 3087 8711 8425
8KB 393 1663 3857 7030 7591
4KB 76 234 719 3304 5688
Stock RAID-5
0% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
2571MB/s 64KB 2901 604 393 272 197
32KB 370 714 440 310 228
16KB 124 1014 635 476 342
8KB 432 1529 939 689 500
4KB 80 1292 803 592 430
Stock RAID-6
0% comp BlkSz RW = 100/0 90/10 80/20 70/30 60/40 50/50 40/60 30/70 20/80 10/90 0/100
2007MB/s 64KB 1713 543 336 255 177
32KB 414 651 390 282 193
16KB 129 827 494 374 266
8KB 385 1000 615 447 324
4KB 73 884 557 399 293 Copyright © 2024 EasyCo LLC dba WildFire-Storage