Hi all, We (Kinetic Engineering) run the IgH EtherCAT master under RTAI 5.3 on Linux 4.19.266 x86_64: a hard real-time cyclic task at 1 kHz beside several soft real-time tasks, including EoE, with mutex synchronisation between them. Getting that combination reliable took work on both sides of the fence - master-side patches and a substantial RTAI patch series - and we're sharing both, plus the findings that don't need a patch at all. Both archives exceed what the list's mail filters comfortably accept, so they are shared as download links below:
## 1. `2026-07-12-Etherlab-patchset.zip` - the master patchset + our addendum Download (7.4 MB): https://app.box.com/s/ejyfi35kvp7kiweii6iwfoh1ba62ji9t This is the complete, ready-to-use patchset: the unofficial patchset 20190904 (Gavin Lambert) on IgH default-branch rev 33b922 (+ stable-1.5 through 0c011d), with our additions appended after that set in the `series` file under "Kinetic specific patches". `README2.md` describes each addition, and `CHANGELOG.md` carries the history. Highlights for RT-framework users: - **`features/printk_deferred/0001`** - deferred printk for `EC_*INFO/ERR/WARN/DBG`: RT frameworks (RTAI/Xenomai/PREEMPT_RT) need master log output to never synchronously take `console_lock` from RT context - a one-line warning through a slow console driver is otherwise a multi-millisecond stall. - **`features/eoe-defer-netif_rx_ni/0001`** - EoE: defer `netif_rx_ni()` outside `master_sem`. Inline network-stack processing (bridging, ARP) under `master_sem` produces unpredictable hold times that stall the cyclic path; with an EoE port in a bridge this is severe. - **`base/0037-replace-linux-rtmutex-with-locks.h`** - RTAI defines its own `rt_mutex` symbols conflicting with `linux/rtmutex.h`; route master locking through the patchset's `locks.h` abstraction so the configured RT system supplies the lock type. - **`features/eoe-queued-guard/0001`** + **`features/eoe-mailbox-correctness/0001..0006`** - EoE/mailbox correctness set: ETG.1000.6-compliant mailbox writes, VoE/FoE retry fixes, working-counter reporting, EoE carrier tracking the slave's AL state, and a library-side `ENOPROTOOPT` fix for `eoe_is_open`. - Plus general correctness and device patches: DC-register read retry, overlapped-PDO datagram sizing, register-0x0012 alias handling, two ccat driver fixes, SII-file feature follow-ups, and diagnosis/JSON-XML extensions for the command-line tool. ## 2. `2026-07-12-RTAI-patchset.zip` - the RTAI 5.3 patch series Download (0.5 MB): https://app.box.com/s/lp3qccatcfhlx7l6tnhi8rvnqq9w7wny RTAI 5.3's scheduler, semaphore, and priority-inheritance internals could not run this workload reliably (the UBI ownership-steal on the master mutex, plus several latent SMP and mode-transition defects that EoE traffic exposes). The folder carries our patch series against RTAI 5.3: three small baseline fixes (0001-0003), an out-of-tree build fix (0004), and a three-layer restructuring - 0010 (the architectural rewrite; adds `MUT_SEM` and single-trap hard-first semaphore primitives), 0011 (defensive tripwires: log + recover, never fault, zero overhead until a bug fires), 0012 (wide-coverage, light-touch diagnostic instrumentation with a `/proc/rtai/bugreport` endpoint). `README.md` in the folder is the orientation; `COVER-LETTER.md` and `ARCHITECTURE.readme` carry the full design narrative, and each patch has a `.readme` with an embedded validation-status block. The series is validated on real SMP and UP x86_64 hardware: production-representative test batteries, >13 h and >26 h EtherCAT-DC + EoE soaks at 1 kHz, and a 16.6 h monitored uniprocessor soak. The folder's `linux/` subdirectory holds `hal-linux-4.19.266-cip86-x86-25.patch` - the complete RTAI-flavoured ipipe/hal kernel patch for linux-4.19.266 x86, in its post-series form (it includes the two exported RTAI notify hooks the series adds, and is byte-identical to what the RTAI tree carries at `base/arch/x86/patches/` after the series is applied). It is provided standalone for kernel-side readers; it applies to the kernel, not to the RTAI tree. ## Why posted here, as-is We're posting both patch sets to this list as-is rather than as contributions to a project repository: - The **master patches** are based on the unofficial patchset 20190904, not the current project head, so they are not directly mergeable into the master's git project. Anyone wanting one of them upstream will need to rebase it - most are small and self-contained. - The **RTAI series** no longer has an upstream home to send it to: RTAI's own channels show no recent maintenance activity. This community is where EtherCAT-on-RTAI deployments live - we know of at least one other EtherCAT project that has hit the same EoE-under-RTAI failure classes this series resolves. ## Findings for anyone running the master under an RT framework (no patch required) 1. **`ecrt_master_deactivate()` must be treated as a blocking call** even though the API requires RT context: its teardown path reaches `kthread_stop()` → `schedule()`. On an RT framework where the caller is a hard task, that in-service block is fatal unless the framework can demote the task at the blocking instant (stock RTAI/i-pipe relaxes only at syscall/trap boundaries). We handled this framework-side - an i-pipe notify at the top of `schedule()` lets RTAI relax the task (see the RTAI series). EtherLab-side no change is required, but the API documentation could usefully state "this call may sleep; hard-RT callers need framework support for in-service blocking, or must deactivate from a context their framework may demote." 2. **Serialise app-context master access via a hard-first mutex acquire.** The master mutex ("masterSem") pattern - one mutex serialising the cyclic exchange, app-context master calls, and EoE - has a hidden failure mode under RT frameworks: if the app-side acquire runs while the caller is still soft (Linux-scheduled), the cyclic hard task can block behind a holder whose forward progress is gated by the Linux scheduler. Priority inheritance does not close this (it defends the holder against other RT contenders but cannot make Linux run a soft thread sooner), so the cyclic path inherits Linux-timescale delays - invisible to up-front jitter accounting, surfacing as rare large spikes at contention. This class is easy to underrate precisely because it is unmeasured: cyclictest-style jitter histograms never show it - the histogram stays clean until a contention coincidence lands a spike on the Linux scheduling timescale (milliseconds, far beyond anything a jitter budget contemplates). Our RTAI series adds composite primitives that make the acquire hard-first and atomic (one LXRT trap in: goHard + priority raise + acquire; one trap out), so the entire hold window runs hard-RT and the cost becomes bounded, up-front, and visible in the jitter budget instead of a hidden tail: int saved_prio, old_hard; if (rt_sem_wait_hard(masterSem, MASTER_PRIO, &saved_prio, &old_hard) < RTE_BASE) { /* ecrt_* master calls: exec_slave_requests, SDO/FoE admin, EoE housekeeping */ rt_sem_signal_hard(masterSem, saved_prio, old_hard); } (It also cuts the per-bracket LXRT trap count from 6-10 to 2.) The primitives are RTAI-series-specific, but the pattern - never let a soft/preemptible context hold the mutex the cyclic path contends - applies to running the master under any RT framework. CAVEAT: do NOT use the hard-first bracket around the master's shutdown/deactivation calls, which run after hard-RT operation has ended and which sleep (`ecrt_master_deactivate` - see finding 1): re-hardening immediately before a sleeping call manufactures the very in-service block finding 1 describes. Acquire the mutex soft on those paths - with the cyclic exchange stopped, the soft-holder concern is gone. For comparison, the dual-kernel peers document this case as a hazard rather than solving it: Xenomai 3 Cobalt raises `SIGDEBUG_MIGRATE_PRIOINV` when a primary-mode thread is found sleeping on a mutex owned by a secondary-mode thread (https://doc.xenomai.org/v3/html/xeno3prm/group__cobalt__api__thread.html, `pthread_setmode_np` reason codes), and EVL's mutex documentation warns that switching to in-band mode while holding a lock "is wrong since this would introduce a priority inversion" (https://v4.xenomai.org/core/user-api/mutex/). The composite acquire makes the hazard unrepresentable by construction. 3. **`ecrt_master_rt_slave_requests(master, true)` + app-context `exec_slave_requests`** is a sound pattern for running slave-request processing in parallel with the cyclic exchange - validated at 1 kHz under RTAI with the serialisation handled master-side, no `master_sem` participation from the app path. 4. **Log output from RT context**: see the `printk_deferred` patch above - recommended for any RT-framework deployment. 5. **EL6601 gateway-slave observation** (informational): under sustained CoE-burst load we observed a slave-side mailbox wedge persisting across AL `INIT→PREOP→OP` cycles, clearing only on slave power-cycle. Master-side review found no cause; reported for community awareness. ## Authorship disclosure These patches and this message were developed with Claude (Anthropic) - initially with Claude Opus 4.6-4.8, subsequently reviewed and finished with Claude Fable 5 - under our close direction and monitoring. We have read through the redux patch (0010) and the EoE master patches ourselves; the harden (0011) and diag (0012) layers - the optional, removable instrumentation layers - have not been hand-reviewed line by line. Assurance rests on the empirical record documented in the readmes: instrument-diffed real-hardware test batteries (tripwire counters, ring records, whole-log fault scans - not absence-of-crash claims) and multi-hour soaks, with every witness channel deliberately provoked at least once. Review accordingly. ## Offered as-is This is a one-off publication: we developed these patches for our own product and are sharing them in case they're useful to others. We're not in a position to actively maintain or iterate them for the community, so the documentation is written to stand on its own - the per-patch readmes and `ARCHITECTURE.readme` in the RTAI folder carry the full design rationale, deviation registers, and test evidence an adopter would need. Regards, Graeme Foot [email protected]<mailto:[email protected]>
-- Etherlab-users mailing list [email protected] https://lists.etherlab.org/mailman/listinfo/etherlab-users
