# SIGSTOP/SIGCONT File Corruption: abort_all_rpcs Causes Double Writes

## Summary

When a process is stopped with SIGSTOP during a `write()` system call, the
Hurd's signal machinery aborts the in-flight RPC and returns EINTR to the
client. However, the server (ext2fs) has already completed the write and
advanced the file pointer. When the client retries the write after SIGCONT,
the data is written again at a different offset, causing file corruption
(gaps, duplicate data, or misaligned content).

This is a fundamental design flaw in the interaction between the interruptible
RPC mechanism and server-side I/O completion semantics.

## Experimental Confirmation

Tested on Debian GNU/Hurd amd64 (GNU Mach 1.8, Hurd 0.9, March 2026 image).

A test program writes 400 sequential 256KB blocks (104,857,600 bytes) to a
file via `write()` with offset=-1 (current file pointer). A second script
sends 500 rapid SIGSTOP/SIGCONT cycles to the writer.

**Control run** (no SIGSTOP): File is exactly 104,857,600 bytes, all values
in sequence. No corruption.

**SIGSTOP run**: File is 105,906,176 bytes (+1,048,576 = 4 × 256KB extra).
Analysis shows 404 blocks instead of 400, with 4 duplicate blocks:

```
Block 193: FirstVal=12582912  DUPLICATE (double-write!)
Block 225: FirstVal=14614528  DUPLICATE (double-write!)
Block 305: FirstVal=19791872  DUPLICATE (double-write!)
Block 385: FirstVal=24969216  DUPLICATE (double-write!)
```

Each duplicate block contains identical data to its predecessor — the same
256KB was written twice because the client retried the RPC after the server
had already completed it. The extra 1,048,576 bytes = 4 × 256KB exactly
matches the 4 double-writes.

**GDB server-side tracing** was attempted (attaching to a separate ext2fs
translator instance and setting a breakpoint on `diskfs_S_io_write`) but
proved impractical: breakpoint commands pause all ext2fs threads during I/O,
and combined with SIGSTOP/SIGCONT pressure on the client, this deadlocks
the system. The corruption pattern analysis above provides equivalent
evidence — the identical duplicate blocks can only be explained by the
server completing the write, advancing the file pointer, then the client
retrying the same data at the new position.

## 1. Detailed Trace: SIGSTOP to abort_all_rpcs to EINTR to Retry

### Step 1: Thread calls write()

The call chain is:

```
write()                                     [glibc/sysdeps/mach/hurd/write.c:26]
  -> __write_nocancel()                     [glibc/sysdeps/mach/hurd/write_nocancel.c:24]
    -> _hurd_fd_write()                     [glibc/hurd/fd-write.c:24]
      -> __io_write(port, buf, nbytes, -1)  [MIG-generated stub]
        -> _hurd_intr_rpc_mach_msg()        [glibc/hurd/intr-msg.c:29]
          -> INTR_MSG_TRAP()                [glibc/sysdeps/mach/hurd/x86_64/intr-msg.h:24]
            -> syscall (mach_msg_trap)
```

The `__write_nocancel` calls `_hurd_fd_write` with offset=-1
(write_nocancel.c:27), meaning "use current file pointer". The fd-write
wrapper calls `__io_write(port, buf, *nbytes, offset, &wrote)` which sends
an IO_write RPC to ext2fs via the interruptible message trap.

The thread sets `ss->intr_port = m->header.msgh_remote_port` (intr-msg.c:98)
before entering the syscall, marking this RPC as interruptible. The thread
then blocks in `mach_msg_trap` waiting for the server reply.

### Step 2: SIGSTOP arrives

The signal thread in `_hurd_internal_post_signal()` (hurdsig.c) processes
SIGSTOP as `act = stop` (line 821-823). It calls:

```c
void suspend (void)                         [hurdsig.c:636]
{
    __proc_dostop (port, _hurd_msgport_thread);   // stop all threads
    abort_all_rpcs (signo, &thread_state, 1);     // abort in-flight RPCs
    __proc_mark_stop (port, signo, detail->code); // mark process stopped
}
```

### Step 3: abort_all_rpcs iterates over all threads

`abort_all_rpcs()` (hurdsig.c:498) iterates over all sigstates. For each
thread (except the signal thread), it calls:

```c
reply_ports[nthreads] = _hurdsig_abort_rpcs(ss, signo, 1,
                                             state, &state_changed, NULL);
```

### Step 4: _hurdsig_abort_rpcs processes the blocked thread

In `_hurdsig_abort_rpcs()` (hurdsig.c:387):

1. It calls `abort_thread()` to abort the thread's kernel context (line 406).
   After this, the thread's `mach_msg_trap` is interrupted by the kernel.

2. It examines the thread's PC. Since the thread was blocked in `mach_msg_trap`,
   the PC is at `_hurd_intr_rpc_msg_in_trap` and `SYSRETURN` is
   `MACH_RCV_INTERRUPTED` (the send succeeded, but the receive was interrupted).
   This matches the condition at line 427-431.

3. Since the RPC was already sent (the write request reached ext2fs), the
   signal thread sends `__interrupt_operation(intr_port, timeout)` to tell the
   server to cancel the operation (line 445-446).

4. **Critical:** Back in `abort_all_rpcs` (line 531-539), if the reply port
   is non-NULL (meaning `interrupt_operation` succeeded), it forces:
   ```c
   state->basic.SYSRETURN = EINTR;
   state_changed = 1;
   ```
   This overwrites the thread's return value to EINTR.

5. `abort_all_rpcs` then waits for replies on all interrupted reply ports
   (lines 551-568), consuming the server's reply messages.

6. The `suspend()` function proceeds to call `__proc_mark_stop()` and the
   process is stopped.

### Step 5: What happened on the server side (ext2fs)

Meanwhile, `diskfs_S_io_write()` (hurd/libdiskfs/io-write.c:24) received and
processed the IO_write RPC:

1. For offset=-1 (append mode or current-position mode), it reads the
   current file pointer: `off = cred->po->filepointer` (line 55).

2. It performs the actual write via `_diskfs_rdwr_internal()` (line 83).

3. **It advances the file pointer:** `cred->po->filepointer += nwritten`
   (line 88).

4. The MIG stub sends a reply with `*amt = nwritten` (success, N bytes written).

The `interrupt_operation` that arrives at the server (via
`ports_S_interrupt_operation` in libports/interrupt-operation.c:27) calls
`ports_interrupt_rpcs()` which calls `hurd_thread_cancel()` on server threads.
But `diskfs_S_io_write` runs to completion without checking any cancellation
flag -- it holds `np->lock` the entire time and performs a straightforward
write. The interrupt has no effect on the already-executing write.

The server's reply message is consumed by `abort_all_rpcs` at line 556-558,
so the client thread never sees it.

### Step 6: SIGCONT and retry

When SIGCONT arrives, `resume()` (hurdsig.c:653) resumes all threads. The
writing thread returns from `INTR_MSG_TRAP` with `err = EINTR` (as forced
by `abort_all_rpcs`).

In `_hurd_intr_rpc_mach_msg()` (intr-msg.c), the EINTR case (line 271-293):

- For SIGSTOP, `ss->intr_port` handling is key. At hurdsig.c:483:
  ```c
  if (! signo || !(_hurd_sigstate_actions(ss)[signo].sa_flags & SA_RESTART))
      ss->intr_port = MACH_PORT_NULL;
  ```
  SIGSTOP's default action is `stop`, not `handle`, so this code path in
  `_hurdsig_abort_rpcs` is what controls whether `intr_port` gets cleared.
  Since SIGSTOP is signo=19 and has SA_RESTART in the default sigaction
  (line 71: `actions[0].sa_flags = SA_RESTART`), `intr_port` is NOT cleared.

- Back in intr-msg.c line 274: `if (ss->intr_port != MACH_PORT_NULL)` -- this
  is true (intr_port was not cleared). So the code enters the restart path
  (lines 280-287):
  ```c
  __mig_dealloc_reply_port (rcv_name);
  m->header.msgh_local_port = rcv_name = __mig_get_reply_port ();
  m->header.msgh_bits = msgh_bits;
  // Restore original message header fields
  goto message;  // RETRY THE ENTIRE RPC
  ```

**The full IO_write RPC is re-sent** with the original data buffer. The server
receives this as a brand-new write request.

### Step 7: The duplicate write

The server processes the retried IO_write. For offset=-1:
- `off = cred->po->filepointer` -- but the file pointer was already advanced
  by the first (completed) write! So this write goes to `original_offset + N`
  instead of `original_offset`.
- The data is written again at the new position.
- The file pointer advances again: `filepointer += nwritten`.

**Result:** The same data is written twice -- once at the correct position,
once at an advanced position. For sequential writes (e.g., tar output), this
creates gaps filled with zeros or duplicate data blocks, corrupting the file.

## 2. Why the Retry Writes to the Wrong Position

The core problem is a split between client and server state:

| Aspect | Client believes | Server reality |
|--------|----------------|----------------|
| First write | Was interrupted, 0 bytes written | Completed, N bytes written |
| File pointer | At original position | Advanced by N bytes |
| Retry | Resends same data | Writes at new position |

The client retries because:
1. `abort_all_rpcs` forces `SYSRETURN = EINTR` regardless of server completion
2. `abort_all_rpcs` consumes the server's success reply (line 556-558)
3. The EINTR path in `intr-msg.c` restores the original message and retries

The server writes at the wrong position because:
1. `diskfs_S_io_write` always completes atomically once entered
2. The file pointer (`cred->po->filepointer`) was advanced by the first write
3. The retry uses offset=-1, which reads the (now-advanced) file pointer

## 3. What POSIX Requires vs What the Hurd Does

### POSIX Requirements (IEEE Std 1003.1)

For `write()` interrupted by a signal:

> If write() is interrupted by a signal after it successfully writes some
> data, it shall return the number of bytes written.

> If write() is interrupted by a signal before it writes any data, it
> shall return -1 with errno set to [EINTR].

Key principle: **If any bytes were written before interruption, the call must
return a short write count, not EINTR.** The application can then decide
whether to retry for the remaining bytes.

### What the Hurd Does

The Hurd violates POSIX in two ways:

1. **Returns EINTR after bytes were written:** The server completed the full
   write, but the client sees EINTR. POSIX says this must not happen.

2. **Implicit retry causes double-write:** Worse, the SA_RESTART mechanism
   causes an automatic retry, so the application never even sees EINTR. It
   sees a successful write of N bytes -- but 2N bytes were actually written,
   with the second N at the wrong offset.

The root cause is that the Hurd's interruptible RPC mechanism has no way to
communicate partial completion. The `interrupt_operation` + EINTR protocol
is all-or-nothing: either the RPC completed (and the reply arrives normally)
or it was interrupted (and the client retries). There is no "completed but
interrupted before the reply was delivered" state.

## 4. Whether Other RPCs Are Affected

### io_read (libdiskfs/io-read.c)

**Yes, affected.** `diskfs_S_io_read` (io-read.c:23) has the same pattern:
- For offset=-1: `off = cred->po->filepointer` (line 48)
- Advances file pointer: `cred->po->filepointer += *datalen` (line 103)
- No cancellation check

A SIGSTOP during read would cause the same data to be read twice, with the
second read at an advanced position. For a pipe or sequential reader, this
means skipped data.

### io_seek (libdiskfs/io-seek.c)

Seek modifies the file pointer but doesn't advance it incrementally, so a
retry would produce the same result. **Not affected.**

### General pattern

Any RPC that:
1. Uses the file pointer (offset=-1)
2. Has side effects (writes data, advances pointer)
3. Runs to completion without checking cancellation

...is vulnerable. This includes `io_write` and `io_read` at minimum.

RPCs with explicit offsets (e.g., `pwrite`/`pread` which pass a real offset
instead of -1) would write to the same position on retry, so the double-write
would be idempotent. **Not corrupting for explicit-offset operations.**

## 5. Connection to Haskell Build Corruption

Samuel mentioned the Haskell build corruption (amd64 only, affects shared
libraries at specific offsets) might be related to signal delivery.

### Assessment

**Likely related mechanism, different trigger.** The Haskell corruption:

- Affects `sbuild` builds, which use job control (process groups, SIGTSTP/SIGCONT)
- `sbuild` may send SIGSTOP/SIGCONT to build processes for resource management
- Even without explicit SIGSTOP, any signal that triggers `abort_all_rpcs`
  followed by RPC retry can cause the same double-write pattern
- The corruption at specific offsets in shared libraries is consistent with
  duplicate write blocks: a section of the ELF file gets written twice, with
  the second copy shifted by the size of the first write

However, the `abort_all_rpcs` function is only called in three places:
1. `suspend()` -- for SIGSTOP/SIGTSTP/SIGTTIN/SIGTTOU (stop signals)
2. Fatal signals (term/core) -- process is dying, doesn't matter
3. Indirectly via the `handle` case for signal delivery to individual threads

For the `handle` case (hurdsig.c:1039), individual thread RPC abort uses
`_hurdsig_abort_rpcs` which only clears `intr_port` when SA_RESTART is not
set (line 483). If SA_RESTART is set (the default, line 71), the RPC is
retried -- same bug.

So **any signal delivered to a thread doing a write, if SA_RESTART is set
(the default), can trigger this bug.** This could include SIGCHLD from child
process completion during a build, SIGALRM from timers, etc.

For the Haskell case specifically:
- Build systems use lots of child processes, generating SIGCHLD
- If SIGCHLD has SA_RESTART (default), and arrives during a write, the write
  is retried after completion
- But the `handle` case goes through `_hurdsig_abort_rpcs` which only retries
  if `intr_port` is still set -- and for `handle`, `intr_port` is cleared
  when SA_RESTART is not set. When SA_RESTART IS set, the code at
  intr-msg.c:350 `if (ss->intr_port != MACH_PORT_NULL)` is true, so it
  retries the RPC. **Same bug.**

**Verdict: The Haskell corruption is very plausibly caused by the same
mechanism, triggered by ordinary signal delivery (SIGCHLD, etc.) rather than
SIGSTOP specifically.**

## 6. Proposed Fixes

### Fix A: Server returns short write on interrupt (Not viable)

Modify `diskfs_S_io_write` to check for interruption and return EINTR
**before** performing the write, or return the byte count if the write
already completed.

The problem: `diskfs_S_io_write` doesn't check `ports_self_interrupted()`
at all. The `interrupt_operation` sets a cancel flag, but `io_write` ignores
it. However, this fix wouldn't help because the write has already completed
by the time `interrupt_operation` arrives. The server processes the write
atomically and always sends a success reply.

### Fix B: Client-side awareness of completed RPCs (Recommended)

The real fix needs to be in `abort_all_rpcs`. When it receives the server's
reply (line 556-558), it should check if the reply indicates success. If
the RPC completed successfully, the thread's state should NOT be set to
EINTR -- instead, the reply should be delivered to the thread normally.

**Proposed change to `~/glibc/hurd/hurdsig.c`, `abort_all_rpcs()` around
line 550-568:**

```c
/* Wait for replies from all the successfully interrupted RPCs.  */
while (nthreads-- > 0)
    if (reply_ports[nthreads] != MACH_PORT_NULL)
      {
        error_t err;
        /* Use a larger buffer to receive the full reply.  */
        struct {
          mach_msg_header_t head;
          mach_msg_type_t type;
          int retcode;
        } reply;
        err = __mach_msg (&reply.head,
                          MACH_RCV_MSG|MACH_RCV_TIMEOUT, 0,
                          sizeof reply,
                          reply_ports[nthreads],
                          _hurd_interrupted_rpc_timeout,
                          MACH_PORT_NULL);
        switch (err)
          {
          case MACH_MSG_SUCCESS:
            /* The server replied successfully. The RPC completed.
               We should NOT force EINTR on this thread -- the reply
               contains the real result. Stash the reply for the thread
               to pick up when it resumes. */
            /* TODO: mechanism to deliver this reply to the thread */
            break;
          case MACH_RCV_TIMED_OUT:
          case MACH_RCV_TOO_LARGE:
            break;
          default:
            assert_perror (err);
          }
      }
```

The difficulty: there is no current mechanism to stash a reply and deliver it
to the resumed thread. The thread's `mach_msg` call has been aborted and will
return EINTR; there's no way to inject a different return value with the reply
data.

### Fix C: Don't abort RPCs for SIGSTOP (Simplest, most targeted)

Since SIGSTOP suspends all threads anyway, there's no need to abort their
RPCs. The threads are suspended by the kernel; their RPCs will simply wait.
When the process is continued, the threads resume and the RPCs complete
normally.

**Proposed change to `~/glibc/hurd/hurdsig.c`, `suspend()` at line 636:**

```c
void suspend (void)
{
    __USEPORT (PROC,
     ({
       __mutex_lock (&_hurd_siglock);
       __proc_dostop (port, _hurd_msgport_thread);
       __mutex_unlock (&_hurd_siglock);
       /* Do NOT call abort_all_rpcs() for stop signals.
          The threads are suspended; their RPCs will complete
          when resumed.  Aborting them causes double-writes
          because the server may have already completed the
          operation.  */
       reply ();
       __proc_mark_stop (port, signo, detail->code);
     }));
    _hurd_stopped = 1;
}
```

**Tradeoffs:**
- Pro: Simple, targeted fix. Doesn't affect other signal handling.
- Pro: SIGSTOP/SIGTSTP are the clear triggers for Mike's bug report.
- Con: Server threads handling the RPCs will also be suspended (they're in
  the same task... wait, no -- ext2fs is a separate task). Actually, ext2fs
  runs as a separate translator process, so its threads are NOT stopped by
  `__proc_dostop`. The server will complete the RPC and send a reply. The
  client thread, being kernel-suspended, will resume the `mach_msg` receive
  when continued. This should work correctly.
- Con: `abort_all_rpcs` also serves to ensure no thread holds locks that
  could deadlock the signal thread. But for SIGSTOP, the signal thread
  doesn't need to run user handlers -- it just needs to call
  `__proc_mark_stop`. The only concern is if `reply()` needs the siglock
  or other resources held by a stopped thread. The `__proc_dostop` call
  already stops all threads, so `abort_all_rpcs` is called after they're
  stopped. Removing it should be safe for SIGSTOP.

**Risk:** If a stopped thread holds a lock needed by the signal thread
(e.g., the dtable lock), the signal thread could deadlock. But this risk
already exists -- `abort_all_rpcs` doesn't release locks, it only aborts
RPCs. The lock issue is orthogonal.

### Fix D: Make the retry aware of server-side completion

Modify `_hurd_intr_rpc_mach_msg` to not retry write RPCs, or to check
whether the server already advanced state. This is impractical because the
interruptible RPC layer is generic and has no knowledge of IO semantics.

### Fix E: Server-side idempotency for IO_write

Make `diskfs_S_io_write` detect and handle retried writes. This would
require tracking the last write operation per-port and recognizing duplicates.
Complex and fragile.

### Recommended approach

**Fix C** (don't abort RPCs for SIGSTOP) is the simplest and most correct
fix. It addresses the root cause: there is no reason to abort RPCs when
stopping a process. The server (ext2fs) is a separate process that continues
running; the client threads are kernel-suspended and will naturally resume
their `mach_msg` receives when continued.

For the broader problem (signal delivery during writes with SA_RESTART),
**Fix B** is needed but requires more design work to stash and replay
server replies. An interim approach: modify `_hurdsig_abort_rpcs` to not
set `intr_port = MACH_PORT_NULL` for IO_write RPCs when the server reply
has already been received. But again, the reply consumption happens in
`abort_all_rpcs`, not `_hurdsig_abort_rpcs`, and only for the SIGSTOP path.

For the individual-thread signal delivery case (`handle` at hurdsig.c:1039),
`_hurdsig_abort_rpcs` calls `__interrupt_operation` and then waits for the
reply in the trampoline code. The same double-write bug applies there: if
the server already completed the write before `interrupt_operation` arrives,
the reply will indicate success, but the thread is set up to re-enter the
RPC after the signal handler returns (when SA_RESTART is set). **This is
the likely mechanism for the Haskell corruption.**

A comprehensive fix would need to:
1. Receive the server reply in the signal handling path
2. If the reply indicates success (not EINTR), arrange for the thread to
   see the success reply rather than retrying the RPC
3. This likely requires changes to the sigcontext/sigreturn mechanism to
   restore the thread to a "reply received" state rather than "re-send RPC"
   state

## 7. File:Line References

| File | Line(s) | Description |
|------|---------|-------------|
| `~/glibc/hurd/hurdsig.c` | 387-488 | `_hurdsig_abort_rpcs()` -- aborts individual thread's RPC |
| `~/glibc/hurd/hurdsig.c` | 427-431 | MACH_RCV_INTERRUPTED check -- detects in-flight RPC |
| `~/glibc/hurd/hurdsig.c` | 445-446 | `__interrupt_operation()` call to server |
| `~/glibc/hurd/hurdsig.c` | 483-484 | `ss->intr_port = MACH_PORT_NULL` -- controls retry |
| `~/glibc/hurd/hurdsig.c` | 498-569 | `abort_all_rpcs()` -- iterates all threads |
| `~/glibc/hurd/hurdsig.c` | 531-539 | Forces `SYSRETURN = EINTR` and consumes reply |
| `~/glibc/hurd/hurdsig.c` | 550-568 | Consumes server replies (discards them!) |
| `~/glibc/hurd/hurdsig.c` | 636-651 | `suspend()` -- called for SIGSTOP |
| `~/glibc/hurd/hurdsig.c` | 821-823 | SIGSTOP default action is `stop` |
| `~/glibc/hurd/hurdsig.c` | 919-921 | `suspend()` called for stop action |
| `~/glibc/hurd/hurdsig.c` | 65-76 | Default sigaction: SA_RESTART for all signals |
| `~/glibc/hurd/intr-msg.c` | 29-393 | `_hurd_intr_rpc_mach_msg()` -- interruptible RPC |
| `~/glibc/hurd/intr-msg.c` | 98 | `ss->intr_port` set before RPC |
| `~/glibc/hurd/intr-msg.c` | 271-293 | EINTR handling -- retry if `intr_port` set |
| `~/glibc/hurd/intr-msg.c` | 307-325 | MACH_RCV_INTERRUPTED -- retry receive |
| `~/glibc/hurd/intr-msg.c` | 342-381 | Server EINTR reply -- retry if `intr_port` set |
| `~/glibc/sysdeps/mach/hurd/x86_64/intr-msg.h` | 24-74 | INTR_MSG_TRAP assembly |
| `~/glibc/sysdeps/mach/hurd/write_nocancel.c` | 24-29 | `__write_nocancel()` -- passes offset=-1 |
| `~/glibc/hurd/fd-write.c` | 24-46 | `_hurd_fd_write()` -- calls `__io_write` |
| `~/hurd/libdiskfs/io-write.c` | 24-99 | `diskfs_S_io_write()` -- server write handler |
| `~/hurd/libdiskfs/io-write.c` | 51-56 | offset=-1 handling: reads filepointer |
| `~/hurd/libdiskfs/io-write.c` | 87-88 | File pointer advance after write |
| `~/hurd/libdiskfs/io-read.c` | 24-110 | `diskfs_S_io_read()` -- same pattern, also affected |
| `~/hurd/libdiskfs/io-read.c` | 47-48 | offset=-1: reads filepointer |
| `~/hurd/libdiskfs/io-read.c` | 102-103 | File pointer advance after read |
| `~/hurd/libports/interrupt-operation.c` | 27-44 | `ports_S_interrupt_operation()` |
| `~/hurd/libports/interrupt-rpcs.c` | 25-43 | `ports_interrupt_rpcs()` -- calls hurd_thread_cancel |
| `~/hurd/libports/interrupted.c` | 30-51 | `ports_self_interrupted()` -- never checked by io_write |