[andriytk]

Currently, when more than nparity disks get faulted during the rebuild, only first nparity disks would go to faulted state, and all the remaining disks would go to degrated state. When a hot spare is attached to that degraded disk for rebuild creating the spare mirror, only that hot spare is getting rebuilt, but not the degraded device. So when later during scrub some other attached draid spare happens to map to that spare, it will end up with cksum error.

Moreover, if the user clears the degraded disk from errors, the data won't be resilvered to it, hot spare will be detached almost immediately and the data that was resilvered only to it will be lost.

Solution: write to all mirrored devices during rebuild, similar to traditional/healing resilvering, but only if we can verify the integrity of the data, or when it's the draid spare we are writing to, in which case we are writing to a reserved spare space, and there is no danger to overwrite any good data.

The argument that writing only to rebuilding draid spare vdev is faster than writing to normal device doesn't hold since, at a specific offset being rebuilt, draid spare will be mapped to a normal device anyway.

How Has This Been Tested?

1. Create a pool with draid3:4s vdev and populate it with some data.
2. Fail 4 devices (zpool offline -f), zed would start automatic sequential resilver, 3 disks would be faulted while one disk would be degraded (since nparity is 3).
3. Wait for resilver and scrub to complete.

There should be no cksum errors after scrub.

redundancy_draid_degraded2 automation test is added also to cover the scenario.

I've run all the redundancy tests in the loop:

$ ./scripts/zfs-tests.sh -T redundancy -I 100
...
Results Summary
PASS	 1893
FAIL	   7

Running Time:	07:19:22
Percent passed:	99.6%

Those 7 failures are all from redundancy_draid_spare1 test, which is a known issue.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

Closes #18420.

[behlendorf]

Using the parity columns when available to verify the block was rebuilt correctly is a great idea. The approach in this PR makes a lot of sense to me as a fix for this corner case. Just a couple inline comments.