[anodos325]

This commit adds handling for the STATX_CHANGE_COOKIE so that we can properly allow NFS clients to track file changes and manage client-side caches.

If knfsd does not have STATX_CHANGE_COOKIE in statx result then it will synthesize the NFS change_info4 structure and related change4id values algorithmically based on the ctime value of the file. Since internally ZFS is using ktime_get_coarse_real_ts64() for the timestamp calculation here it introduces the possibility that the change will not increment the change4id of directories / files causing a failure in the client to invalidate its attr cache (among other things). See RFC 8881 Section 10.8 for discussion of how clients may implement name and directory caching.

Notable in this commit is that we are not initializing the inode->i_version to the znode->z_seq number. The reason for this is that we're intentionally not setting SB_I_VERSION. This indicates that the filesystem manages its own i_version and so it is not populated in the generic_fillattr.

The following compares tight loop of setattr over NFSv4 protocol while tracing kprobe:nfsd4_change_attribute.

Before change:

inode, change_attribute
4723, 7590032215978780890
4723, 7590032215978780890
4723, 7590032215978780890
4723, 7590032215982780865
4723, 7590032215982780865

After change:

inode, change_attribute
7602, 7590032992517123951
7602, 7590032992517123952
7602, 7590032992517123953
7602, 7590032992517123954
7602, 7590032992517123955

Motivation and Context

Description

How Has This Been Tested?

Testing this requires an NFS client and Linux + OpenZFS NFS server.

On client I mounted export and ran the following python script (where /mnt/nfs is the nfs4 mount):

import os


FILE = '/mnt/nfs/foo/bar.txt'
f = open(FILE, 'w')

while True:
    os.fchmod(f.fileno(), 0o777)

On server I used the following bpftrace script:

#include <linux/stat.h>
BEGIN {
    printf("ino, cookie\n");
}

kprobe:nfsd4_change_attribute {
    $instat = (struct kstat *)arg0;
    @ino[tid] = $instat->ino;
}

kretprobe:nfsd4_change_attribute {
    printf("%lu, %lu\n", @ino[tid], retval);
}

kretprobe:nfsd4_change_attribute {
    delete(@ino[tid]);
}

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[anodos325]

For posterity, the issue was highlighted by broadcom / vmware support when investigating an issue with one of their customers:

The host’s NFS 4.1 client checks for changeID to figure out if it needs to flush or update its cache. This relates to getAttr/setAttr operations or responses. If the current changeID received is greater than the previous changeID stored in the cache, it updates its entry in the cache with the current response’s value. We suspect the storage server is not updating the changeID.