Yesterday I was running some benchmarks against a new MySQL server configuration. As part of my testing I wanted to see how things looked with ZFS as the back-end. So I loaded up some SSDs and attempted to create a ZFS pool. Zpool spit out a “device busy” error when I tried to create my pool leading to a confused and bewildered matty. After a bit of tracing I noticed that mdadm was laying claim to my devices:
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : active (auto-read-only) raid5 sde[0] sdc[2] sdb[1]
1465148928 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
Previously I did some testing with mdadm and it dawned on me that the headers may still be resident on disk. Sure enough, they were:
$ mdadm -E /dev/sdb
/dev/sdb:
Magic : a92b4efc
Version : 0.90.00
UUID : dc867613:8e75d8e8:046b61bf:26ec6fc5
Creation Time : Tue Apr 28 19:25:16 2009
Raid Level : raid5
Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
Array Size : 1465148928 (1397.27 GiB 1500.31 GB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 127
Update Time : Fri Oct 21 17:18:22 2016
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : d2cbaad - correct
Events : 4
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 16 1 active sync /dev/sdb
0 0 8 64 0 active sync /dev/sde
1 1 8 16 1 active sync /dev/sdb
2 2 8 32 2 active sync /dev/sdc
I didn’t run ‘mdadm –zero-superblock’ after my testing so of course md thought it was still the owner of these devices. After I zero’ed the md super block I was able to create my pool without issue. Fun times in the debugging world. :)