Mirror seeing half the write IOPS on one disk than the other, is this normal?
Mirror seeing half the write IOPS on one disk than the other, is this normal?
I’m syncoiding from my normal RAIDz2 to a backup mirror made of 2 disks. I looked at zpool iostat and I noticed that one of the disks consistently shows less than half the write IOPS of the other: capacity operations bandwidth pool ...
I'm syncoiding from my normal RAIDz2 to a backup mirror made of 2 disks. I looked at zpool iostat and I noticed that one of the disks consistently shows less than half the write IOPS of the other:
capacity operations bandwidth
pool alloc free read write read write
------------------------------------ ----- ----- ----- ----- ----- -----
storage-volume-backup 5.03T 11.3T 0 867 0 330M
mirror-0 5.03T 11.3T 0 867 0 330M
wwn-0x5000c500e8736faf - - 0 212 0 164M
wwn-0x5000c500e8737337 - - 0 654 0 165M
This is also evident in iostat:
f/s f_await aqu-sz %util Device
0.00 0.00 3.48 46.2% sda
0.00 0.00 8.10 99.7% sdb
The difference is also evident in the temperatures of the disks. The busier disk is 4 degrees warmer than the other. The disks are identical on paper and bought at the same time.
Is this behaviour expected?
It might be that the data to both disks saturates a common link before the second disk reaches full iops capability, and thus the driver then writes at full speed on one disk and at half speed on the other, for twice as long.
I put the low IOPS disk in a good USB 3 enclosure, hooked to an on-CPU USB controller. Now things are flipped:
capacity operations bandwidth pool alloc free read write read write ------------------------------------ ----- ----- ----- ----- ----- ----- storage-volume-backup 12.6T 3.74T 0 563 0 293M mirror-0 12.6T 3.74T 0 563 0 293M wwn-0x5000c500e8736faf - - 0 406 0 146M wwn-0x5000c500e8737337 - - 0 156 0 146MYou might be right about the link problem.
Looking at the B350 diagram, the whole chipset is hooked via PCIe 3.0 x4 link to the CPU. The other pool (the source) is hooked via USB controller on the chipset. The SATA controller is also on the chipset so it also shares the chipset-CPU link. I'm pretty sure I'm also using all the PCIe links the chipset provides for SSDs. So that's 4GB/s total for the whole chipset. Now I'm probably not saturating the whole link, in this particular workload, but perhaps there's might be another related bottleneck.
I'm not fully familiar with the overheads associated with all things going on on a chipset, but it's not unreasonable to think that this workload, plus whatever the chipset has to do (hardware management tasks mostly), as well as the CPU's other tasks on similar interfaces that might saturate the IO die/controller, would influence this.
B350 isn't a very fast chipset to begin with, and I'm willing to bet the CPU in such a motherboard isn't exactly current-gen either. Are you sure you're even running at PCIe 3.0 speeds too? There are 2.0 only CPUs available for AM4.
Usually means a failing drive in my experience.
Interesting. SMART looks pristine on both drives. Brand new drives - Exos X22. Doesn't mean there isn't an impending problem of course. I might try shuffling the links to see if that changes the behaviour on the suggestions of the other comment. Both are currently hooked to an AMD B350 chipset SATA controller. There are two ports that should be hooked to the on-CPU SATA controller. I imagine the two SATA controllers don't share bandwidth. I'll try putting one disk on the on-CPU controller.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:
Fewer Letters More Letters NVMe Non-Volatile Memory Express interface for mass storage PCIe Peripheral Component Interconnect Express PSU Power Supply Unit SATA Serial AT Attachment interface for mass storage SSD Solid State Drive mass storage ZFS Solaris/Linux filesystem focusing on data integrity
6 acronyms in this thread; the most compressed thread commented on today has 6 acronyms.
[Thread #958 for this sub, first seen 8th Sep 2024, 16:15] [FAQ] [Full list] [Contact] [Source code]