cat fast path (splice on linux) does not work for 0 size files

[jendap]

Cat has fast path using splice system call. Unfortunately it does not work for many "file systems" reporting wrong file size.

In particular I have fuse file system returning incorrect size (zero or otherwise). It should be fine since one should treat file size on linux as a hint and read until EOF. That is what gnu cat does. It is also what uutils do in slow path! The fast path calling splice will stop at whatever file size is returned in stat. It is inconsistent and break stuff. Can we disable it please?

BTW: It works for cat /proc/self/stat which also report 0 size. The reason is probably for it somewhere in splice.c. But we're not going to change syscall behavior anyway. Calling the syscall in rust code looks fine, right?

BTW: I can see workaround if you really want to keep the splice around. But it is already quite a lot of code to do the splice. Is it worth even more hacks?

[jendap]

I can't replicate it now myself. Something spooky with my kernel? :) Next time I make sure to open issue only after replicating on least two machines. Sorry to bother you. Thanks for uutils 👍

[jendap]

Here is reproducible code - at least on Ubuntu 25.10 with 6.17 kernel (even fresh virtual machine):

#define FUSE_USE_VERSION 31

#include <fuse.h>
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <errno.h>

static void *bugfs_init(struct fuse_conn_info *conn, struct fuse_config *cfg) {
    cfg->direct_io = 1;
    return NULL;
}

static int bugfs_getattr(const char *path, struct stat *stbuf, struct fuse_file_info *fi) {
    if (strcmp(path, "/") == 0) {
        stbuf->st_mode = S_IFDIR | 0755;
        stbuf->st_nlink = 2;
    } else {
        stbuf->st_mode = S_IFREG | 0444;
        stbuf->st_size = 3;
    }
    return 0;
}

static int bugfs_read(const char *path, char *buf, size_t size, off_t offset, struct fuse_file_info *fi) {
    char *data = "foo bar baz";
    size_t len = strlen(data);
    if (offset < len) {
        if (offset + size > len)
            size = len - offset;
        memcpy(buf, data + offset, size);
        return size;
    } else {
        return 0;
    }
}

static struct fuse_operations bugfs_operations = {
    .init = bugfs_init,
    .getattr = bugfs_getattr,
    .read = bugfs_read,
};

int main(int argc, char *argv[]) {
    if (argc != 2) {
        fprintf(stderr, "Usage: %s <mount_point>\n", argv[0]);
        return 1;
    }

    return fuse_main(argc, argv, &bugfs_operations, NULL);
}

Compile and run:

sudo apt install gcc pkgconf libfuse-dev

gcc -o bugfs bugfs.c `pkg-config fuse3 --cflags --libs`

mkdir mnt_test
./bugfs mnt_test

See the problem:

cat mnt_test/file
# 'foo'

# there is a number of options to print something extra in cat - any of them will trigger slow path
cat -e mnt_test/file
# 'foo bar baz'

gnucat mnt_test/file
# 'foo bar baz'

The problem is with 'direct_io = 1'. As far as I know it is the only way to create something like /proc filesystem. Such fs typically return st_size = 0 because it is expensive to compute and/or different every time. Read() does the real work (not fixed string like this toy example).

The slow path work (read() in loop until EOF). The fast path on linux using splice does not work. It is probably a bug in kernel. But uutils should be consistent with itself and with gnu utils, right?

[oms-ger]

System: Ubuntu 25.10; x86-64; Linux kernel 6.15

Incompatible and hazardous behavior confirmed. Affects all file content in FUSE filesystems with file size reported by stat() & Co smaller than size of actual / delivered content.

In this example, file size by stat() & Co (inode attributes) is hardcoded to 13. Actual generated & delivered content: 409 bytes.

Relevant syscalls of GNU coreutils cat:

.691637 openat(AT_FDCWD, "my_file.txt", O_RDONLY) = 3 <0.000100>
.691783 fstat(3, {st_mode=S_IFREG|0400, st_size=500, ...}) = 0 <0.000033>
.691850 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000015>
.691946 read(3, "---\nprocess_name"..., 262144) = 409 <0.000106>
.692086 write(1, "---\nprocess_name"..., 409) = 409 <0.000033>
.692161 read(3, "", 262144)     = 0 <0.000044>
.692305 close(3)                = 0 <0.000059>

Summary:

1. file gets opened
2. fstat() returns size 13
3. read() returns 409 bytes ✅
4. write() writes 409 bytes
5. 2nd read() returns 0 (EOF)
6. takes ca. 0.7 ms

Relevant syscalls of uutils coreutils cat:

.622391 statx(AT_FDCWD, "my_file.txt", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0400, stx_size=13, ...}) = 0 <0.000123>
.622576 openat(AT_FDCWD, "my_file.txt", O_RDONLY|O_CLOEXEC) = 3 <0.000074>
.622699 fstat(3, {st_mode=S_IFREG|0400, st_size=13, ...}) = 0 <0.000033>
.622774 fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0xe), ...}) = 0 <0.000015>
.622822 pipe2([4, 5], 0)        = 0 <0.000021>
.622875 splice(3, NULL, 5, NULL, 131072, 0) = 13 <0.000115>
.623024 splice(4, NULL, 1, NULL, 13, 0) = 13 <0.000021>
.623073 splice(3, NULL, 5, NULL, 131072, 0) = 0 <0.000026>
.623133 close(5)                = 0 <0.000020>
.623181 close(4)                = 0 <0.000021>
.623229 close(3)                = 0 <0.000051>

Summary:

1. statx() returns size 13
2. file gets opened
3. fstat() returns size 13
4. splice() forwards only 13 (instead of all 409 delivered) bytes 💥
5. 2nd splice() call finds 0 more data (EOF); 396 bytes lost
6. takes ca. 0.9 ms

Sequence of operations invoked in FUSE server (user space) by uutils cat:

1. lookup() (multiple times)
2. statx() (returns size = 13 bytes)
3. open()
4. read() (returns 409 bytes)
5. flush() + release() ("close()")

Setting or unsetting FUSE filesystem flags FUSE_SPLICE_READ, FUSE_SPLICE_MOVE, FUSE_SPLICE_WRITE makes no difference. In Linux kernel fs/ they're not really used as yet.

So there is nothing a FUSE driver can do after open() to convince splice() of another size.

I use an own FUSE fs written in #[no_std] Rust using https://github.com/jmillikin/rust-fuse btw.

Assessment

This affects primarily synthetic files, i.e. dynamically generated content. Yet I suppose it also affects regularly stored files in FUSE, if their contents change between openat() and fstat() call. This would constitute a TOCTOU vulnerability in all FUSE context, whenever splice() is used!

Linux kernel's splice() behavior is to blame. It uses pre-buffered size value instead of determining how much data is available at call time. If that's meaning of FUSE_SPLICE_READ & Co flags, it seems to have not been implemented in many years.

I appreciate efforts to improve efficiency by using splice() instead of read() / write(). Yet this poses:

• an incompatibility with GNU coreutils
• likely a vulnerability
• slightly worse performance, esp. on smaller files (arguably most common use case for cat?), esp. due to more syscalls. (Many more syscalls overall in uutils than in GNU coreutils btw.) Esp. on FUSE. It always uses its read() / write() userspace API anyway.

Recommendations

• disable or remove write_fast_using_splice() by default
• any of...
  • require an explicit new command arg to use it when available, like -S or -F (or --splice-mode or --fastpath)
  • or a corresponding environment variable config
  • or determine whether source file is located on any FUSE filesystem; never use splice() then
• add note about incompatibility and possible TOCTOU vulnerability in splice() mode used in Linux FUSE filesystems to documentation

[sylvestre]

likely a vulnerability

How?

[oms-ger]

likely a vulnerability

How?

Think of a file that contains some information like "required_user: bob". May be used for some authorization, by a (not very sophisticated) SW that uses 'cat' command to extract data. For files residing on a FUSE filesystem, 'cat' could deliver just "required_user:". I.e. empty value. Making SW believe no authorization at all were required. Open gates for attackers.

Or the inverse: file contains list of authorized users, yet empty file gets delivered. So authorized users like bob are suddenly locked out. Various less obvious, trickier and more realistic scenarios of attacks conceivable.

Please learn about TOCTOU kind of vulnerabilities for more. It may also apply to FUSE filesystems with non-synthetic file sizes, i.e. backed by concrete files. Namely if file gets changed (extended) between fstat() and splice() call by another process. Like with NFS or CIFS filesystems, this may happen on a remote machine.

More obvious, on FUSE filesystems it's a violation of 1st listed key objective of uutils. I needed to install GNU coreutils on Ubuntu 25.10 for my SW to keep working. Hopefully FTTB only.

As said, (performance) improvements are appreciated. Yet not if they come at expense of incompatibility / vulnerability by default. For arguably most use cases (smaller files), benefit of slice() over read() seems negligible at best. If not worse. Saving on much higher number of syscalls by uutils in general would certainly be more effective performance optimization.

[jendap]

This is NOT a security issue. It is a bug. It is not even here in coreutils. They are actually rather nice!

This is a kernel bug.

The question is: Whom of us can send a patch to kernel?

[oech3]

Does #10835 (comment) work as a workaround?