zfs_write() not making progress after source data destroyed

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	12.9
Kernel Version	Debian 6.1.128-1 (2025-02-07) x86_64
Architecture	x86_64
OpenZFS Version	2.3.0 (OpenZFS proper)

Describe the problem you're observing

I am writing a small standalone version of a stress testing tool used by one of our customers. I want to include this version in the test suite. I was working on it locally on my laptop, which runs OpenZFS, and it managed to crash and hang inside zfs_write() in a rather surprising way! I know why it did, but I'm not sure what a good fix is, and so this bug report.

The program is here: https://github.com/robn/fsync_stress/blob/main/fsync_stress.c. Note that it's usable, but unfinished. I wouldn't have posted it this early except for this bug report!

The program creates lots of threads, and each one loops, calling (effectively): open()->write()->fsync()->close(), over and over again, reporting which step of the process it got to and what the error was. The idea is that a driver program elsewhere will break the pool in interesting ways, and then we can analyse the program's behaviour in response.

The important part is that the main thread creates a 16M buffer and fills it with random bytes, and then passes it down to each thread, so that they don't have to set up their own data for the write.

So! There was a bug in my "wait for threads" logic, and the main thread exited early. The worker threads were killed as expected, then the program hung, and wouldn't be killed:

✦ $ ./fsync_stress
OK: filenum=6 state=CLOSE size=3601403 err=0
OK: filenum=35 state=CLOSE size=305332 err=0
OK: filenum=31 state=CLOSE size=365499 err=0
OK: filenum=26 state=CLOSE size=657581 err=0
OK: filenum=51 state=CLOSE size=1223345 err=0
E: condvar wait failed: Success
^C

^F

^Z

^C^C
^Z^Z^C^C

Looking closer, the main thread was a zombie, but one of its work threads were somewhere still running:

✦ $ pgrep -aw fsync_stress
3872095 [fsync_stress] <defunct>
3872187 ./fsync_stress

top would show that thread pegging a core at 100%. I was unable to do anything interesting with it though: it wouldn't respond to signals, wouldn't allow ptrace() to attach, and so on.

However, it was doing something. It's IO accounting showed write_bytes rapidly rising:

$ sudo cat /proc/3872187/io
rchar: 16781201
wchar: 732256439
syscr: 11
syscw: 289
read_bytes: 4948480
write_bytes: 409197530900661
cancelled_write_bytes: 0

Eventually I went for a system-wide trace, which showed enough interesting function names to show where we probably were stuck.

A little reading in zfs_write() led here:

		ssize_t tx_bytes;
		if (abuf == NULL) {
			tx_bytes = zfs_uio_resid(uio);
			zfs_uio_fault_disable(uio, B_TRUE);
			error = dmu_write_uio_dbuf(sa_get_db(zp->z_sa_hdl),
			    uio, nbytes, tx);
			zfs_uio_fault_disable(uio, B_FALSE);
#ifdef __linux__
			if (error == EFAULT) {
				zfs_clear_setid_bits_if_necessary(zfsvfs, zp,
				    cr, &clear_setid_bits_txg, tx);
				dmu_tx_commit(tx);
				/*
				 * Account for partial writes before
				 * continuing the loop.
				 * Update needs to occur before the next
				 * zfs_uio_prefaultpages, or prefaultpages may
				 * error, and we may break the loop early.
				 */
				n -= tx_bytes - zfs_uio_resid(uio);
				pfbytes -= tx_bytes - zfs_uio_resid(uio);
				continue;
			}
#endif

My very strong suspicion is that the main thread died it's memory was destroyed, but the thread was in the syscall with a handle on that memory region as the source data. Because it had gone away, reading it would cause a fault. However this code is capturing EFAULT and retrying.

Seems provable with a trace:

$ sudo bpftrace -e 'i:s:10 { exit() } kr:dmu_write_uio_dbuf { @s[kstack] = count(); @r[retval] = count() }'
Attaching 2 probes...


@r[0]: 1
@r[14]: 570165
@s[
    zfs_write+1303
    zpl_iter_write+254
    vfs_write+562
    ksys_write+107
    do_syscall_64+85
    entry_SYSCALL_64_after_hwframe+110
]: 570166

So that's the problem. I don't know this code very well though, so I'm not sure what the fix is though.

My first thought was that this should have a check to see if it made no progress, but I think the assumption here is that we never would have even got here if zfs_uio_prefaultpages() had done its job, so maybe it's not failing as it should be?

Or maybe we should have taken some refcount on the input pages to prevent it being destroyed before we were done?

Or maybe something else.

Incidentally, I managed to save my computer by noticing the quota checks at the top of the loop. I set a user quota to one byte less than the current usage, and it immediately exited with EDQUOT. Phew.

@robn

@robn I think pfbytes reset in case of the EFAULT is incorrect. Would we reset it to 0, we would call (and likely fail) another zfs_uio_prefaultpages(). Now we implicitly trust previously prefetched data, which appear to be wrong. Even though it feels wrong for kernel to free process memory before all threads exit.

@amotin

Confirmed on OpenZFS 2.4.2 — reproduced via Samba (not a stress harness), with a rangelock-cascade severity note

This is the same defect as above, zfs_write() not making progress after source data destroyed. Adding a real-world reproduction on a newer release, plus one observation about blast radius that the original report didn't surface, since the stress-harness threads there were killed before it could show.

Environment


Host	Proxmox VE
Kernel	`7.0.2-6-pve`
OpenZFS	`2.4.2-pve1`
Arch	x86_64
Workload	`smbd` (Samba) write into a dataset, inside an LXC container

What happened

A single smbd worker thread spun at 100% CPU in R state inside a pwrite64() for ~3 days, unkillable. Its thread-group leader was a zombie (<defunct>). This is the #17129 shape exactly: the owning smbd connection process tore down (client disconnect / connection cleanup) while an async-I/O worker was mid-pwrite, and the userspace buffer backing the write was freed/unmapped. The worker is now stuck on a source buffer that can never be faulted in again, and the EFAULT-retry path in zfs_write() loops forever instead of returning the error.

Runtime confirmation

dmu_write_uio_dbuf() returns EFAULT (14) on essentially every call; one full tx create/assign/write/commit cycle runs ~150k times/sec from this one thread:

# bpftrace -e 'kprobe:dmu_tx_assign,kprobe:dmu_write_uio_dbuf { @[probe]=count(); } interval:s:3 { exit(); }'
@[kprobe:dmu_write_uio_dbuf]: 457882
@[kprobe:dmu_tx_assign]:      461659

# bpftrace -e 'kretprobe:dmu_write_uio_dbuf { @ret[retval]=count(); } interval:s:3 { exit(); }'
@ret[0]:  8
@ret[14]: 463355

perf record -g: 99.74% in zfs_write, the cost being the per-iteration dbuf-array churn (dmu_buf_hold_array_by_dnode → dbuf_hold → dbuf_hold_impl → dbuf_create in, dbuf_rele → dbuf_destroy → spl_kmem_cache_free out).

sysrq l kernel stack of the spinning thread:

RIP: spl_kmem_cache_free+0x5 [spl]
  dbuf_destroy
  dbuf_rele_and_unlock
  dbuf_rele
  dmu_write_uio_dnode+0x209 [zfs]
  dmu_write_uio_dbuf [zfs]
  zfs_write+0x5b8 [zfs]
  zpl_iter_write
  vfs_write
  __x64_sys_pwrite64

The EFAULT branch is unchanged between zfs-2.3.0 (where #17129 was filed) and zfs-2.4.2 — the loop still does n -= tx_bytes - zfs_uio_resid(uio); pfbytes -= ...; continue;, and the per-chunk re-prefault stays gated on if (pfbytes < nbytes). So when zero bytes are consumed, neither n nor pfbytes move, the page is never re-prefaulted, and the continue skips the tx_bytes == 0 backstop.

Severity note: it is not just one stuck thread

The original report's worker threads were killed, so the rangelock impact wasn't visible. In a live multi-client workload it is significant: zfs_write() holds the file's RL_WRITER rangelock across the entire (now infinite) chunk loop. With a pending block-size grow that lock covers the whole file [0, UINT64_MAX] (cf. #18046). Every other reader/writer of that file then blocks indefinitely in zfs_rangelock_enter() / cv_wait_common. On this system one stuck write parked 46 other smbd threads in cv_wait_common, driving load average to ~47 while CPU/IO PSI stayed near zero (condvar sleepers, not IO stalls). So a single unfaultable buffer can wedge all access to a file, and the unkillable syscall blocks the owning process from exiting (the zombie leader).

Proposed fix

Handle zero forward progress on the Linux EFAULT path rather than looping. Echoing @amotin's note that the pfbytes handling is wrong here: either break with EFAULT when no bytes were consumed (extend the existing tx_bytes == 0 backstop to this path), and/or re-prefault unconditionally before retrying. Returning EFAULT to userspace is correct for a buffer that cannot be faulted in IMO.

Linux: fix zfs_write() infinite loop on unfaultable buffer

NAS-142537 / 26.0.0-RC.1 / Update ZFS 2.4 to zfs-2.4.4 (by ixhamza) (#…

NAS-142538 / 25.10.7 / None / Update ZFS 2.3 to zfs-2.3.9 (by ixhamza) (

Linux: fix zfs_write() infinite loop on unfaultable buffer

Navigation Menu

Repository navigation

`zfs_write()` not making progress after source data destroyed #17129

System information

Describe the problem you're observing

Environment

What happened

Runtime confirmation

Severity note: it is not just one stuck thread

Proposed fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

zfs_write() not making progress after source data destroyed #17129

Description

System information

Describe the problem you're observing

Activity

amotin commented on Mar 10, 2025

seasox commented on Jun 7, 2026

Environment

What happened

Runtime confirmation

Severity note: it is not just one stuck thread

Proposed fix

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`zfs_write()` not making progress after source data destroyed #17129