the patch is small: one line, plus a comment. it changes no logic, and it is upstream now. the interesting part is not the line. it is the three wrong answers I had to throw away before it.
this is the story of a KCSAN report that looked like a missing lock, turned out to be a deliberate lockless read, and still needed fixing - because "we did it on purpose" is not something a data-race detector can infer from the source.
what syzbot reported
on 2026 june 19, syzbot filed a report titled:
BUG: KCSAN: data-race in __percpu_counter_limited_add / percpu_counter_add_batch
the object is an s64: sbinfo->used_blocks, a per-CPU counter on a
tmpfs superblock. two plain accesses hit the same 8 bytes, with no common lock between them:
write to 0xffff88812a80e9d0 of 8 bytes by task 11484 on cpu 0:
percpu_counter_add_batch+0x101/0x120 lib/percpu_counter.c:107
percpu_counter_sub include/linux/percpu_counter.h:277
shmem_inode_unacct_blocks mm/shmem.c:263
shmem_recalc_inode+0x143/0x1f0 mm/shmem.c:466
shmem_undo_range+0xb20/0xb60 mm/shmem.c:1272
shmem_evict_inode / evict / iput / filename_unlinkat / __x64_sys_unlink
read to 0xffff88812a80e9d0 of 8 bytes by task 16836 on cpu 1:
__percpu_counter_limited_add+0xc4/0x3f0
percpu_counter_limited_add include/linux/percpu_counter.h:77
shmem_inode_acct_blocks+0xf4/0x230 mm/shmem.c:232
shmem_get_folio_gfp+0x5d3/0xd90 mm/shmem.c:2502
shmem_write_begin+0xfc/0x1f0 mm/shmem.c:3239
generic_perform_write / shmem_file_write_iter / __kernel_write_iter
the write is on tmpfs's free path: truncate or unlink, through
shmem_recalc_inode() and shmem_inode_unacct_blocks(), into
percpu_counter_sub(). the read is on the allocation path: writing into a file,
through shmem_get_folio() and shmem_inode_acct_blocks(), into
percpu_counter_limited_add(). same superblock, same counter, one side under
fbc->lock and the other side not.
the two lines in question, in lib/percpu_counter.c:
fbc->count += count + amount; /* under fbc->lock */
...
if (abs(count + amount) <= batch &&
((amount > 0 && fbc->count + unknown <= limit) || /* no lock */
(amount < 0 && fbc->count - unknown >= limit))) {
syzbot had no reproducer, just the splat. that is usually a hint that the path is a little unusual.
what KCSAN actually detects
KCSAN is a dynamic data-race detector built on compiler instrumentation. it is not a static
analyzer, and it is not KASAN. in fact KCSAN depends on !KASAN; you pick
one sanitizer for a given build. KASAN would never see this report anyway, because nothing here is
out of bounds - it is an 8-byte load racing an 8-byte store on the same address.
the mechanism matters for everything that follows. for each plain access it instruments, KCSAN runs a short state machine:
- check. is there a watchpoint on this address? if there is, and at least one side is a write, report. this is the common case, and it is where this report came from.
- setup. otherwise, with a low probability, arm a watchpoint on the address and remember the current value. most accesses are never sampled at all - that is the "watchpoint-based sampling" part, and it is why a real race can take a while to surface.
- delay. if a watchpoint was armed, stall for a short random time, to widen the window in which the other CPU can touch the address.
- re-check. read the value again; if it changed under us, report a race of unknown origin.
how often step 2 fires and how long step 3 stalls are tunable at runtime, which is the first thing to reach for when a report is rare:
/sys/module/kcsan/parameters/skip_watch # accesses to skip before arming another watchpoint
/sys/module/kcsan/parameters/udelay_task # microseconds to stall after arming one
/sys/kernel/debug/kcsan # "on" / "off": the whole detector, if built in but disabled
it also explains the two report shapes. a normal report prints both stacks, because both sides were instrumented. the other shape - "race at unknown origin" - is printed when the watchpoint saw the value change but the conflicting access was not instrumented at all, so there is no second stack to show.
marked atomic accesses - READ_ONCE(), WRITE_ONCE(),
atomic_*, and friends - are instrumented too, but only to check for a
watchpoint. they never set one.
that asymmetry is the whole trap. annotating one side of a race with READ_ONCE() does
not silence KCSAN: the plain access still sets a watchpoint, and the marked access still walks
into it and reports. and it is not wrong to report it - the two accesses really are unordered.
why the race initially looks suspicious
two threads touching an s64 without a common lock is the textbook definition of a data
race, and it is easy to stop reading there and go hunting for the missing lock. worse, the value is
used to enforce a limit:
if (... fbc->count + unknown <= limit)
a tmpfs mount can carry a size= limit, which becomes sbinfo->max_blocks.
if a stale read lets a page allocation through, could a tmpfs exceed the size the user asked for?
that sounds like a real bug, not a false positive.
so the question is not "which lock did someone forget". it is "is this read allowed to be approximate, and by how much". that requires reading the counter's design, not just the backtrace.
how percpu_counter works
the structure is a compromise between accuracy and cache-line contention:
struct percpu_counter {
raw_spinlock_t lock;
s64 count; /* the global part */
s32 __percpu *counters; /* one s32 per CPU */
};
percpu_counter_add_batch()adds to this CPU's locals32. when the local value would reachbatch, it takesfbc->lock, folds the local amount into the globalcount, and zeroes the local slot.percpu_counter_sum()takes the lock and adds the globalcountplus every CPU's local slot.- plain reads are deliberately cheap and approximate:
percpu_counter_read()just loads the globalcount, with no lock at all.
so the global count lags the true total by whatever is parked in the per-CPU slots.
each slot is bounded by batch, so the total in flight is bounded by
unknown = batch * num_online_cpus(). that is the slack term.
percpu_counter_limited_add(limit, amount) is the interesting one: it adds
amount only if the resulting total is still within limit. it is a
three-stage fallback, each stage more accurate and more expensive than the last:
- fast path, no lock. if this CPU's slot stays within
batchand the global count plus the slack is clearly inside the limit, add to the local slot and return. this is the path the report hits. - locked estimate. otherwise take
fbc->lockand re-test the limit against the global count, again with theunknownslack. if the answer is unambiguous either way, stop here - no per-CPU summation needed. - exact sum. only if the slack straddles the limit does it walk the CPUs
(
for_each_cpu_or(cpu, cpu_online_mask, cpu_dying_mask)), add every slot, and decide on the true total.
/* stage 1: skip taking the lock when safe */
if (abs(count + amount) <= batch &&
((amount > 0 && gcount + unknown <= limit) ||
(amount < 0 && gcount - unknown >= limit))) {
this_cpu_add(*fbc->counters, amount);
return true;
}
/* stage 2: take the lock; skip the full sum when the slack does not straddle the limit */
raw_spin_lock(&fbc->lock);
count = fbc->count + amount;
...
/* stage 3: still ambiguous - sum every CPU's slot and decide on the exact total */
that shape is what makes the lockless read defensible as an approximation rather than a guess: the
fast path may only skip the lock when the answer is unambiguous even after the worst-case slack is
added in, so the error it can introduce is bounded by unknown and nothing more.
why the read is intentionally lockless
used_blocks is touched on every tmpfs page allocation and free. taking
fbc->lock on each one would serialize allocation against free, which is exactly what
per-CPU counters exist to avoid. so the fast path reads fbc->count without the lock,
on purpose.
is that read "safe because it is conservative"? no. my first version of the comment said something like that, and it was wrong. the honest statement is narrower:
- a concurrent flush moves value between the global count and a per-CPU slot;
- but other CPUs' locked slow paths also add to the global count while the fast path adds to a slot;
- so a stale low read is not a lower bound on anything. it can let the fast path proceed where the locked slow path would have refused.
what limits the damage is the per-CPU slack. the abs(count + amount) <= batch guard
keeps this CPU's slot within batch across the update, and every other slot is bounded
by batch too, so the total cannot run ahead by more than unknown. the
read is a cheap approximation used to skip the lock when there is clearly room, not a hard
guarantee.
that is the part a quick read of the diff gets wrong, and it is worth writing down in the code so the next person does not "fix" the read by taking the lock:
/*
* Lockless on purpose: gcount may be stale, so this is only an
* approximation, bounded by the per-cpu slack ("unknown").
*/
gcount = data_race(READ_ONCE(fbc->count));
the local is named gcount, not total, because it is the global part -
total already means the summed value including the per-CPU slots everywhere else in
this file. naming matters when the whole point is not to mislead the next reader.
why READ_ONCE() isn't enough for KCSAN
this is the trap that cost me the most time. READ_ONCE(fbc->count) looks like the
fix: it tells the compiler this is a single, deliberate load. but to KCSAN it is a marked
atomic access, and marked accesses only ever check watchpoints, they never create them. so:
- the plain compound store
fbc->count += ...sets a watchpoint and stalls inside its critical section; - the
READ_ONCE()read on the other CPU looks for a watchpoint, finds it, and reports.
if anything, the READ_ONCE() made the code look more careful without silencing
anything. the tool is not asking "did you tell the compiler"; it is asking "do you acknowledge
this race as intended".
there is a second trap: READ_ONCE() on an s64 is not a 64-bit atomic
load on 32-bit. it compiles to a single volatile access of that type, and on a 32-bit target that
can still be two loads. it stops the compiler from refetching or caching the value, which is what
percpu_counter_read_positive() already relied on, but it does not give you atomicity -
that is what atomic64_t is for.
why data_race(READ_ONCE()) is the correct annotation
data_race() is the only thing that actually removes an access from KCSAN's view: it
disables instrumentation around the expression, so the access neither sets nor consumes a
watchpoint. and include/linux/compiler.h documents exactly this shape:
- accesses to a lock-protected variable stay plain;
- deliberate lockless accesses to that same variable use
data_race(); - if the access must also be atomic, use
data_race(READ_ONCE(x)).
keeping the writers plain matters. if you sprinkle WRITE_ONCE() on the locked
writes, you make them marked, and a future genuinely-buggy lockless write - the kind you actually
want KCSAN to catch - gets harder to see. annotate the one intentional side, leave the rest
visible.
the final hunk:
count = __this_cpu_read(*fbc->counters);
/*
* Lockless on purpose: gcount may be stale, so this is only an
* approximation, bounded by the per-cpu slack ("unknown").
*/
gcount = data_race(READ_ONCE(fbc->count));
/* Skip taking the lock when safe */
if (abs(count + amount) <= batch &&
((amount > 0 && gcount + unknown <= limit) ||
(amount < 0 && gcount - unknown >= limit))) {
no logic change. the syzbot report was a claim about the C memory model, and the claim was true; the race simply was not a bug. what was missing was saying so in a way the tool understands.
testing and verification
two separate things need evidence: that the code does not change, and that KCSAN goes quiet for this pair.
object code. a defconfig build, gcc 16.2.1, before and after, comparing the
disassembly of one function with objdump -d - same compiler, same config, so the patch
is the only variable. __percpu_counter_limited_add() keeps the same size
(0x23b), still contains the same four fbc->count loads, and has
identical branch targets. the .text differs in 45 of 2171 bytes, all of it register
allocation and compare-operand order. as a control, a comment-only rebuild is byte-identical - so
"no functional change intended" is not just a phrase.
KCSAN A/B. build one kernel with CONFIG_KCSAN=y and
CONFIG_KASAN=n (the two sanitizers are mutually exclusive - you pick one per build),
but leave KCSAN off at boot and switch it on from init by writing on to
/sys/kernel/debug/kcsan. the first attempt used KCSAN_STRICT +
KCSAN_WEAK_MEMORY + early enable and never finished booting in 420 s under KVM;
deferring it keeps boot sane, and the stall knob (udelay_task) is what widens the
window enough for a 120 s run to mean something.
the workload is a 120 s stress on one size=64M tmpfs in an 8 vCPU QEMU/KVM guest:
threads doing write() and unlink()/ftruncate()
concurrently, which is exactly shmem_inode_acct_blocks against
shmem_inode_unacct_blocks. the base kernel produces the report almost immediately:
BUG: KCSAN: data-race in __percpu_counter_limited_add / percpu_counter_add_batch
write to ... by percpu_counter_add_batch+0xcc/0x150
shmem_inode_unacct_blocks / shmem_recalc_inode / shmem_undo_range
shmem_setattr / do_ftruncate / __x64_sys_ftruncate
read to ... by __percpu_counter_limited_add+0x8a/0x2e0
shmem_inode_acct_blocks / shmem_get_folio_gfp / shmem_write_begin
generic_perform_write / shmem_file_write_iter / vfs_write
base: 38 reports of that exact title, the first one 0.3 s after the workers started. with the
annotation: 0 percpu_counter reports in the same 120 s. unrelated races
(d_make_discardable/dentry_kill, osq_lock) still appear,
which is a useful control - the detector was definitely awake, it just had nothing to say about
this pair.
KCSAN is sampling, so "0 reports" is strong evidence, not proof. but 38 hits in a third of a second on one side, and nothing in two minutes on the other, is about as clean as a reproduction gets.
tag choices in the commit message
the patch went to akpm and linux-mm, cc Hugh Dickins - who wrote the original
_limited_add() - and Baolin Wang. two tag choices are worth calling out:
Reported-by:carries the syzbot address, and checkpatch wantsCloses:immediately after it with the bug URL, so the report is linked rather than just thanked.- there is deliberately no
Fixes:tag. this annotates an intentional race; it is not a regression and it does not fix a crash. aFixes:tag invitesstable/AUTOSELbackports and frames the change as a bug fix, which is the wrong frame. checkpatch does warn "perhaps it also needs a 'Fixes:' tag" - that warning is expected and was left in on purpose. if a maintainer wants the tag, they will ask.
takeaways
- a KCSAN splat is a statement about the C memory model, not automatically a bug. some races are intentional.
- intentional means "annotate", not "ignore": the next reader deserves to know why, and the tool needs to be told.
READ_ONCE()/WRITE_ONCE()do not silence KCSAN;data_race()does. usedata_race(READ_ONCE(x))when you need both.- annotate narrowly. keep the locked accesses plain so real lockless bugs stay visible.
- be precise about lockless approximations. "conservative" was wrong; "bounded by the per-CPU slack" is right.