platform: use F_FULLFSYNC on macOS for SyncFile data durability, fixes #9383#9592
platform: use F_FULLFSYNC on macOS for SyncFile data durability, fixes #9383#9592mr-raj12 wants to merge 1 commit intoborgbackup:1.4-maintfrom
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 1.4-maint #9592 +/- ##
=============================================
+ Coverage 81.92% 81.94% +0.01%
=============================================
Files 38 38
Lines 11306 11307 +1
Branches 1781 1781
=============================================
+ Hits 9263 9265 +2
Misses 1456 1456
+ Partials 587 586 -1 ☔ View full report in Codecov by Sentry. |
…borgbackup#9383 On macOS, os.fsync() only flushes to the drive's write cache, not to persistent storage. True durability requires fcntl(fd, F_FULLFSYNC). Without it a power loss can silently discard data that SyncFile considered safely persisted. Add fdatasync() and sync_dir() to darwin.pyx using F_FULLFSYNC with an os.fsync() fallback for network filesystems that do not support it. Import them in platform/__init__.py so they override the base implementations on macOS. Backport of borgbackup#9385 to 1.4-maint. Signed-off-by: Mrityunjay Raj <mr.raj.earth@gmail.com>
mr-raj12
force-pushed
the
backport-9385-to-1.4-maint
branch
from
b186f07 to
58e1c0f
Compare
| from ..platformflags import ( | ||
| is_win32, | ||
| is_linux, | ||
| is_freebsd, | ||
| is_darwin, | ||
| is_cygwin, | ||
| is_haiku, | ||
| ) |
There was a problem hiding this comment.
there is still a black change.
|
I ran the tests on macOS (ARM), they passed. |
| On macOS, os.fsync() only flushes to the drive's write cache. | ||
| fcntl F_FULLFSYNC flushes to persistent storage. |
There was a problem hiding this comment.
The first sentence is a bit weird. It "only" flushes the drive's write cache. What else is there to flush that FULLSYNC does?
There was a problem hiding this comment.
fsync() on macOS
fsync() (and Python's os.fsync()) asks the OS kernel to flush dirty data from its page cache to the drive. On macOS, however, fsync() does not guarantee that data has been committed to stable storage on the physical device. It only ensures the data has been handed off to the drive's own write buffer/cache.
If the drive has a volatile write cache (most HDDs and many SSDs do), a power loss after fsync() returns can still result in data loss or corruption, because the drive firmware hasn't necessarily flushed its internal cache to the platters/NAND.
F_FULLFSYNC (macOS-specific)
F_FULLFSYNC is a macOS-specific fcntl command that goes one step further: it issues a hardware flush command (like FLUSH CACHE in ATA or SYNCHRONIZE CACHE in SCSI/NVMe) to the drive, forcing it to commit all buffered writes from the drive's own cache to persistent storage.
There was a problem hiding this comment.
so, os.fsync is only a OS-level flush, while fcntl F_FULLSYNC does a hw-level write buffer flush additionally.
2 participants
Backport of #9385 to
1.4-maint.SyncFile.sync()callsplatform.fdatasync(self.fd), which on macOS resolves toos.fsync().fsync()only flushes to the drive's write cache as it does not guarantee data reaches persistent storage. A power loss could silently lose datathat
SyncFilebelieved was safely persisted.The fix was acknowledged with a TODO in
src/borg/platform/base.py:TODO: Use F_FULLSYNC on macOS.This backport implements the platform-specific override suggested in #9383: add
fdatasync()andsync_dir()toplatform/darwin.pyxusingfcntl(fd, F_FULLFSYNC)with anos.fsync()fallback for network filesystems that do not supportF_FULLFSYNC.Changes
src/borg/platform/darwin.pyxfdatasync()usingfcntl F_FULLFSYNCwithos.fsync()fallback; addsync_dir()using the samesrc/borg/platform/__init__.pyfdatasync,sync_dirfrom.darwinin the darwin blocksrc/borg/platform/base.pysrc/borg/testsuite/platform.pydocs/changes.rstChecklist
1.4-maint(maintenance branch as change is only applicable here as master is already fixed via platform: use F_FULLSYNC on macOS for SyncFile data durability, fixes #9383 #9385)