ext4.Remove() Spotted dirty metadata buffer

[gustavo-iniguez-goya]

Hi,

When using ext4.Remove() to delete a file, a warning message is printed out to dmesg:

[41531.073709] JBD2: Spotted dirty metadata buffer (dev = sda1, blocknr = 1581072). There's a risk of filesystem corruption in case of system crash.

or:

Oct 09 19:47:28 deb13-malware kernel: JBD2: Spotted dirty metadata buffer (dev = sda1, blocknr = 1).
Oct 09 19:47:28 deb13-malware kernel: JBD2: Spotted dirty metadata buffer (dev = sda1, blocknr = 0).

Then when rebooting the machine, the filesystem needs to be fixed:

/dev/sda1: One or more block group descriptor checksums are invalid. FIXED.
/dev/sda1: Group descriptor 48 cheksum is 0xb8a8, should be 0xbe45. FIXED.
Directory inode 392449, block #0: directory passes check but fails checksum.
FIXED.
Unattached inode 393851


/dev/sda1: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY.

(...)
(initramfs) fsck

unattached inode 392449
Connect to /lost&found? y
inode 392449 ref count is 2, should be 1. Fix(y)? y
Pass 5: Checking group summary information
Block bitmap differences: +(1624055--1624313)
Fix(y)? y
free blocks count wrong for group #48 (26239, counted=24167)
Fix(y)? y

(...)

After reboot again, this message is printed to the journal:

[  212.492933] systemd-journald[284]: File /var/log/journal/86c66472ba9f4933a2263c6af3af4bb7/user-1001.journal corrupted or uncleanly shut down, renaming and replacing.

OS: Debian 13
Kernel: 6.12.48+deb13-amd64

/dev/sda1 details

dumpe2fs 1.47.2 (1-Jan-2025)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          e570496f-7942-4866-a370-e5e0a065793a
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1512560
Block count:              6041088
Reserved block count:     302054
Overhead clusters:        138981
Free blocks:              4413887
Free inodes:              1351798
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8176
Inode blocks per group:   511
Flex block group size:    16
Filesystem created:       Fri Aug  1 11:48:45 2025
Last mount time:          Thu Oct  9 19:57:16 2025
Last write time:          Thu Oct  9 19:57:16 2025
Mount count:              1
Maximum mount count:      -1
Last checked:             Thu Oct  9 19:54:56 2025
Check interval:           0 (<none>)
Lifetime writes:          61 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      604fb342-fee0-4680-8ab9-2dfe7418d827
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0x842a5acc
Journal features:         journal_incompat_revoke journal_64bit journal_checksum_v3
Total journal size:       128M
Total journal blocks:     32768
Max transaction length:   32768
Fast commit length:       0
Journal sequence:         0x00071dd4
Journal start:            12693
Journal checksum type:    crc32c
Journal checksum:         0xb06c0062

ext4.Remove() test that deletes /mnt/test.log (not empty)

package main

import (
    "fmt"

    "github.com/diskfs/go-diskfs"
    "github.com/diskfs/go-diskfs/filesystem/ext4"
)

func main() {
    disk, err := diskfs.Open(
        "/dev/sda1",
        diskfs.WithOpenMode(diskfs.ReadWrite),
    )
    if err != nil {
        return
    }
    defer disk.Close()

    fs, err := disk.GetFilesystem(0)
    if err != nil {
        return
    }

    ext4fs, ok := fs.(*ext4.FileSystem)
    if !ok {
        return
    }
    defer ext4fs.Close()

    err = ext4fs.Remove("/mnt/test.log")
    if err != nil {
        fmt.Printf("%s\n", err)
    }
}

[deitch]

Hi, thanks for this.

First of all, /dev/sda1 is not mounted when you run this, is it? That almost is guaranteed to bring corruption, since both the kernel (via its ext4 driver) and this are trying to write to the same device.

I would be interested in whether or not we can recreate it using a filesystem on a disk image. Have you tried that? Create an ext4 filesystem on, say, a 50MB or 100MB disk image using mkfs.ext4, mount it, load it up with lots of files and directories, umount it, then run this program, and, when finished, mount it again and see?

[deitch]

I was able to do it. I used the ext4.img generated by the tests in this repo under filesystem/ext4/testdata/. I copied the file over, and then changed your sample above to access ext4.img and remove /foo/subdirfile.txt. I got the following:

$ sudo mount /tmp/ext4.img /mnt
mount: /mnt: mount(2) system call failed: Structure needs cleaning.
       dmesg(1) may have more information after failed mount system call.

And then the dmesg:

[21179019.049868] EXT4-fs (loop0): ext4_check_descriptors: Checksum for group 0 failed (54844!=27298)
[21179019.051275] EXT4-fs (loop0): group descriptors corrupted!

I suspect group descriptors not properly updated, but we should be able to check that.

[gustavo-iniguez-goya]

In my case /dev/sda1 was mounted 0:) Actually, I didn't even think about it.

Glad you were able to reproduce it. If you need any test from my side just let me know.

[deitch]

@gustavo-iniguez-goya check out #309, it solves that particular issue (and some others on the way). There still is an issue with the directory entry being written linearly even when an htree. That is the next thing to solve.

[gustavo-iniguez-goya]

thank you @deitch !

The behaviour has changed with latest fix. When removing the file now it doesn't print warnings to dmesg.

After rebooting the machine, this message is printed out to the tty and the machine does not start:

EXT4-fs error (device sda1): __ext4_find_entry:1656: inode #392449: comm systemd-fstab-g: checksuming directory block 0

which is probably what you mentioned about the directory issue.

After rebooting the machine again, fsck fixes automatically the inconsistency:

/dev/sda1: recovering journal-....
/dev/sda1: contains a file system with errors, check forced.
Directory inode 392449, block #0: directory passes checks but fails checksum.
FIXED.oading.
Unattached zero-length inode 394550. CLEARED.
/dev/sda1: 169013/1512560 files (0.3% non-contiguous), 1791722/6041088 blocks

[deitch]

Correct. The group descriptors now are correct, but the directory entry checksums are incomplete. Getting there.