fdcavalcanti opened a new issue, #16337:
URL: https://github.com/apache/nuttx/issues/16337
### Description / Steps to reproduce the issue
Hi all,
The performance tests using IPERF are failing after the PR #16231. This can
be easily replicated using the `sta_softap` defconfig on any of the following
targets: ESP32, ESP32S3, ESP32C3 and ESP32C6.
The test connects to WiFi hotspot succesfully, but fails when `iperf -c
<ip_addr>` is called. It affects RISC-V and Xtensa devices, as shown above.
I have tracked the problem using `git bisect` and arrived at this point:
`sched/wqueue: Change dq to list.` (`9dbb9b49`)
# Steps to reproduce
Example using ESP32S3:
- Checkout to `9dbb9b49`
- `./tools/configure.sh esp32s3-devkit:sta_softap`
- Enable: `DEBUG_ASSERTIONS`, `SCHED_BACKTRACE`
- make and flash
- Connect to WiFi using `wapi essid`, `wapi psk` and `renew`
- Call `iperf -c <server_ip>`
# Error Log
```
nsh> renew wlan0
nsh> iperf -c 10.42.0.1
IP: 10.42.0.182
mode=tcp-client sip=10.42.0.182:5001,dip=10.42.0.1:5001, interval=3, time=30
[CPU1] xtensa_user_panic: User Exception: EXCCAUSE=001d task: lpwork
[CPU1] dump_assert_info: Current Version: NuttX 10.4.0 9dbb9b49c6 May 7
2025 09:34:26 xtensa
[CPU1] dump_assert_info: Assertion failed user panic: at file:
common/xtensa_assert.c:190 task(CPU1): lpwork process: Kernel 0x42013690
[CPU1] up_dump_register: PC: 420348c2 PS: 00060d32
[CPU1] up_dump_register: A0: 82034988 A1: 3fcb0e80 A2: 3fcab6e0
A3: 3fc9a08c
[CPU1] up_dump_register: A4: 420258b0 A5: 3fc9a054 A6: 00000000
A7: 3fcb0e80
[CPU1] up_dump_register: A8: 820348be A9: 00000000 A10: 00000000
A11: 00000096
[CPU1] up_dump_register: A12: 3fcab708 A13: 00060320 A14: fffffff2
A15: 3fcb0e60
[CPU1] up_dump_register: SAR: 0000001f CAUSE: 0000001d VADDR: 00000000
[CPU1] up_dump_register: LBEG: 40056fc5 LEND: 40056fe7 LCNT: ffffffff
[CPU1] dump_stackinfo: User Stack:
[CPU1] dump_stackinfo: base: 0x3fcb0070
[CPU1] dump_stackinfo: size: 00004032
[CPU1] dump_stackinfo: sp: 0x3fcb0e80
[CPU1] stack_dump: 0x3fcb0e60: 00000000 00000000 00000000 00000000 820349a7
3fcb0eb0 3fcab6e0 3fc9a08c
[CPU1] stack_dump: 0x3fcb0e80: 00000000 00060d20 00000096 3fcb7998 00000001
3c0b01a0 3fc9faf0 3fcb0e90
[CPU1] stack_dump: 0x3fcb0ea0: 82025920 3fcb0ed0 00000000 3fc9a08c 420258b0
3fc9a054 00000096 3fcb0eb0
[CPU1] stack_dump: 0x3fcb0ec0: 82025c21 3fcb0ef0 3fc9a054 cccccccd 420258b0
3fc9a054 00000096 3fcb0ed0
[CPU1] stack_dump: 0x3fcb0ee0: 82043dc9 3fcb0f10 3fc9bf2c 3fc9a054 00000003
3fc996c8 00000000 3fcb0ef0
[CPU1] stack_dump: 0x3fcb0f00: 82043508 3fcb0f40 3fc9bf2c 3fc9a054 82017c49
3fcb0f40 3fcabaf8 00060d20
[CPU1] stack_dump: 0x3fcb0f20: 42025b00 00000028 3c0b5b1c 3fcb0f10 820435bd
3fcb0f60 3fc9bf2c 420949f8
[CPU1] stack_dump: 0x3fcb0f40: 3fc9bf2c 00000002 00000054 3fcb0f40 82046ad5
3fcb0f80 3fc9bf2c 42046f78
[CPU1] stack_dump: 0x3fcb0f60: 3fc9a054 00000000 00000029 3fcb0f60 82046b1c
3fcb0fa0 3fc9be3c 000000f0
[CPU1] stack_dump: 0x3fcb0f80: 3fca12b0 00000002 fffffff2 3fcb0f80 82013708
3fcb0fc0 3fc9be3c 3fcaff58
[CPU1] stack_dump: 0x3fcb0fa0: 3fc9bf2c 3fc9bf2c 3fca12b0 3fcb0fa0 82014500
3fcb0fe0 00000000 00060120
[CPU1] stack_dump: 0x3fcb0fc0: 3fc90e38 00060120 3fca1ea4 3fcb0fc0 00000000
3fcb1010 00000002 42013690
[CPU1] stack_dump: 0x3fcb0fe0: 3fcab718 42046afc 3fc9be3c 00000000 3fcab6e0
3fcab710 3fcab708 3fcb0fe0
[CPU1] stack_dump: 0x3fcb1000: 00000000 3fcb1030 00000000 00000000 00000000
00000000 00000000 3fcb1010
[CPU1] stack_dump: 0x3fcb1020: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[CPU1] sched_dumpstack: backtrace| 2: 0x42039aac 0x40379a9d 0x40379d40
0x40379699 0x40379176 0x403750a7 0x420348c2 0x42034988
[CPU1] sched_dumpstack: backtrace| 2: 0x420349a7 0x42025920 0x42025c21
0x42043dc9 0x42043508 0x420435bd 0x42046ad5 0x42046b1c
[CPU1] sched_dumpstack: backtrace| 2: 0x42034988
[CPU1] dump_fatal_info: Dump CPU0: PAUSED
[CPU1] up_dump_register: PC: 42093c54 PS: 00060b20
[CPU1] up_dump_register: A0: 8201304c A1: 3fcab680 A2: 00000000
A3: 3fc90c00
[CPU1] up_dump_register: A4: 3fc90e38 A5: 00060b20 A6: 00000004
A7: 3fcab680
[CPU1] up_dump_register: A8: 00000000 A9: 3fcc17a0 A10: 3fcc0728
A11: 3fc98f28
[CPU1] up_dump_register: A12: 8037a140 A13: 3fc98f08 A14: 00000000
A15: 3fcc1630
[CPU1] up_dump_register: SAR: 00000020 CAUSE: 3fcab680 VADDR: 00000000
[CPU1] up_dump_register: LBEG: 400556d5 LEND: 400556e5 LCNT: fffffffe
[CPU1] dump_stackinfo: User Stack:
[CPU1] dump_stackinfo: base: 0x3fcaaaf0
[CPU1] dump_stackinfo: size: 00003056
[CPU1] dump_stackinfo: sp: 0x3fcab680
[CPU1] stack_dump: 0x3fcab660: 3f016400 00000000 00000000 00002000 803751f6
3fcab6a0 00000006 3fc90da0
[CPU1] stack_dump: 0x3fcab680: 3c0b037e 00000001 00000004 3fcab660 803752e2
3fcab6e0 00000000 00000002
[CPU1] stack_dump: 0x3fcab6a0: 0003fe28 3fcaf00c 00000000 00000002 3fc90c00
00000004 00000002 000000f0
[CPU1] stack_dump: 0x3fcab6c0: 00000004 3fc90da0 3fc90da0 3fcab6a0 80045c04
3fceb550 80000000 3c0b0000
[CPU1] stack_dump: 0x3fcab6e0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[CPU1] sched_dumpstack: backtrace| 0: 0x4201304c 0x403751f6 0x403752e2
0x40045c04 0x40043ab9 0x40034c48 0x40000000
[CPU1] dump_tasks: PID GROUP CPU PRI POLICY TYPE NPX STATE EVENT
SIGMASK STACKBASE STACKSIZE COMMAND
[CPU1] dump_tasks: ---- --- 0 --- -------- ------- --- -------
---------- ---------------- 0x3fc98768 2048 irq
[CPU1] dump_tasks: ---- --- 1 --- -------- ------- --- -------
---------- ---------------- 0x3fc98f68 2048 irq
[CPU1] dump_task: 0 0 0 0 FIFO Kthread - Running
0000000000000000 0x3fcaaaf0 3056 CPU0 IDLE
[CPU1] dump_task: 1 0 1 0 FIFO Kthread - Assigned
0000000000000000 0x3fcaf1e8 3048 CPU1 IDLE
[CPU1] dump_task: 2 0 1 100 RR Kthread - Running
0000000000000000 0x3fcb0070 4032 lpwork 0x3fcab6e0 0x3fcab710
[CPU1] dump_task: 3 3 0 100 RR Task - Waiting
Semaphore 0000000000000000 0x3fcb14d8 8136 nsh_main
[CPU1] dump_task: 4 0 0 255 RR Kthread - Waiting
Semaphore 0000000000000000 0x3fcb3890 704 spiflash_op 0x3fcb336c
[CPU1] dump_task: 5 0 1 255 RR Kthread - Waiting
Semaphore 0000000000000000 0x3fcb3c70 704 spiflash_op 0x3fcb336c
[CPU1] dump_task: 6 0 0 223 RR Kthread - Waiting
Semaphore 0000000000000000 0x3fcb6570 4048 rt_timer
[CPU1] dump_task: 7 0 0 253 RR Kthread - Waiting MQ
empty 0000000000000000 0x3fcb7ca0 6624 wifi
[CPU1] dump_task: 9 9 0 100 RR Task - Waiting
Semaphore 0000000000000000 0x3fcc0ba8 4024 iperf -c 10.42.0.1
[CPU1] dump_task: 10 9 0 100 RR pthread - Waiting
Mutex:2 0000000000000000 0x3fcc5f78 4072 iperf_traffic 0x4203e964
0x3fcc1a00
[CPU1] sched_dumpstack: backtrace| 0: 0x4201304c 0x403751f6 0x403752e2
0x40045c04 0x40043ab9 0x40034c48 0x40000000
[CPU1] sched_dumpstack: backtrace| 1: 0x420130d4 0x40000000 0x40000000
[CPU1] sched_dumpstack: backtrace| 2: 0x42039aac 0x403798b1 0x42033bcd
0x40379e11 0x40379699 0x40379176 0x403750a7 0x420348c2
[CPU1] sched_dumpstack: backtrace| 2: 0x42034988 0x420349a7 0x42025920
0x42025c21 0x42043dc9 0x42043508 0x420435bd 0x42046ad5
[CPU1] sched_dumpstack: backtrace| 2: 0x42034988
[CPU1] sched_dumpstack: backtrace| 3: 0x42017f94 0x42033fc2 0x42034002
0x42020779 0x4201deb0 0x4201ea38 0x4201eb08 0x4201c3e4
[CPU1] sched_dumpstack: backtrace| 3: 0x4201c250 0x4201c210 0x42017ed0
0x42014510 0x40000000 0x40000000
[CPU1] sched_dumpstack: backtrace| 4: 0x42017f94 0x403792a9 0x42014500
0x40000000 0x40000000
[CPU1] sched_dumpstack: backtrace| 5: 0x42017f94 0x403792a9 0x42014500
0x40000000 0x40000000
[CPU1] sched_dumpstack: backtrace| 6: 0x42017f94 0x42013eac 0x4201be22
0x42014500 0x40000000 0x40000000
[CPU1] sched_dumpstack: backtrace| 7: 0x42092e34 0x42092ed0 0x42047f20
0x4037eaac 0x42014500 0x40000000 0x40000000
[CPU1] sched_dumpstack: backtrace| 9: 0x42017f94 0x42013eac 0x42033afd
0x4203f140 0x4203c809 0x42017ed0 0x42014510 0x40000000
[CPU1] sched_dumpstack: backtrace| 9: 0x40000000
[CPU1] sched_dumpstack: backtrace|10: 0x42017f94 0x42017b8d 0x42017e24
0x42026a54 0x42026ab0 0x42026af1 0x42026bb8 0x42026855
[CPU1] sched_dumpstack: backtrace|10: 0x42041010 0x42040155 0x4203f998
0x4203f9de 0x4203ebb4 0x4203e8d0 0x4203e9a0 0x42037110
[CPU1] sched_dumpstack: backtrace|10: 0x42044fd4 0x40000000 0x40000000
```
And the output from `btdecode`:
```
$ ./tools/btdecode.sh esp32s3 bt
Backtrace for task 2:
0x42039aac: sched_dumpstack at sched_dumpstack.c:71
0x40379a9d: dump_running_task at assert.c:666
0x40379d40: dump_assert_info at assert.c:717
(inlined by) _assert at assert.c:898
0x40379699: xtensa_user_panic at xtensa_assert.c:191 (discriminator 1)
0x40379176: xtensa_user at ??:?
0x403750a7: _xtensa_user_handler at xtensa_user_handler.S:190
0x420348c2: work_queue_period_wq at kwork_queue.c:169
0x42034988: work_queue_wq at kwork_queue.c:248
0x420349a7: work_queue at kwork_queue.c:254
0x42025920: tcp_update_timer at tcp_timer.c:257
0x42025c21: tcp_timer at tcp_timer.c:821
0x42043dc9: tcp_poll at tcp_devpoll.c:104
0x42043508: devif_poll_tcp_connections at devif_poll.c:658
(inlined by) devif_poll_connections at devif_poll.c:857
0x420435bd: devif_poll at devif_poll.c:1054
0x42046ad5: wlan_dopoll at esp_wlan.c:1001 (discriminator 2)
0x42046b1c: wlan_txavail_work at esp_wlan.c:1121
0x42034988: work_queue_wq at kwork_queue.c:248
Backtrace for task 0:
0x4201304c: nx_start at nx_start.c:778 (discriminator 1)
0x403751f6: __esp32s3_start at esp32s3_start.c:457 (discriminator 1)
0x403752e2: __start at ??:?
0x40045c04: ?? ??:0
0x40043ab9: ?? ??:0
0x40034c48: ?? ??:0
0x40000000: ?? ??:0
Backtrace dump for all tasks:
Backtrace for task 9:
0x42017f94: nxsem_wait at sem_wait.c:168
0x42013eac: nxsem_wait_uninterruptible at sem_wait.c:252 (discriminator 1)
0x42033afd: pthread_join at pthread_join.c:149 (discriminator 3)
0x4203f140: iperf_start at iperf.c:963 (discriminator 4)
0x4203c809: iperf_main at iperf_main.c:320
0x42017ed0: nxtask_startup at task_startup.c:72
0x42014510: nxtask_start at task_start.c:72
0x40000000: ?? ??:0
0x40000000: ?? ??:0
Backtrace for task 7:
0x42092e34: file_mq_timedreceive_internal at mq_receive.c:195
0x42092ed0: file_mq_receive at mq_receive.c:488
0x42047f20: esp_queue_recv at esp32s3_wifi_adapter.c:1634
0x4037eaac: ppTask at ??:?
0x42014500: nxtask_start at task_start.c:99
0x40000000: ?? ??:0
0x40000000: ?? ??:0
Backtrace for task 6:
0x42017f94: nxsem_wait at sem_wait.c:168
0x42013eac: nxsem_wait_uninterruptible at sem_wait.c:252 (discriminator 1)
0x4201be22: rt_timer_thread at esp32s3_rt_timer.c:530
0x42014500: nxtask_start at task_start.c:99
0x40000000: ?? ??:0
0x40000000: ?? ??:0
Backtrace for task 5:
0x42017f94: nxsem_wait at sem_wait.c:168
0x403792a9: spi_flash_op_block_task at esp32s3_spiflash.c:948
0x42014500: nxtask_start at task_start.c:99
0x40000000: ?? ??:0
0x40000000: ?? ??:0
Backtrace for task 4:
0x42017f94: nxsem_wait at sem_wait.c:168
0x403792a9: spi_flash_op_block_task at esp32s3_spiflash.c:948
0x42014500: nxtask_start at task_start.c:99
0x40000000: ?? ??:0
0x40000000: ?? ??:0
Backtrace for task 3:
0x42017f94: nxsem_wait at sem_wait.c:168
0x42033fc2: nxsched_waitpid at sched_waitpid.c:165
0x42034002: waitpid at sched_waitpid.c:618
0x42020779: nsh_builtin at nsh_builtin.c:166
0x4201deb0: nsh_execute at nsh_parse.c:545
0x4201ea38: nsh_parse_command at nsh_parse.c:2813
0x4201eb08: nsh_parse at nsh_parse.c:2923
0x4201c3e4: nsh_session at nsh_session.c:248
0x4201c250: nsh_consolemain at nsh_consolemain.c:81
0x4201c210: nsh_main at nsh_main.c:82
0x42017ed0: nxtask_startup at task_startup.c:72
0x42014510: nxtask_start at task_start.c:72
0x40000000: ?? ??:0
0x40000000: ?? ??:0
Backtrace for task 2:
0x42039aac: sched_dumpstack at sched_dumpstack.c:71
0x403798b1: dump_backtrace at assert.c:451
0x42033bcd: nxsched_foreach at sched_foreach.c:69 (discriminator 2)
0x40379e11: dump_fatal_info at assert.c:769
(inlined by) _assert at assert.c:905
0x40379699: xtensa_user_panic at xtensa_assert.c:191 (discriminator 1)
0x40379176: xtensa_user at ??:?
0x403750a7: _xtensa_user_handler at xtensa_user_handler.S:190
0x420348c2: work_queue_period_wq at kwork_queue.c:169
0x42034988: work_queue_wq at kwork_queue.c:248
0x420349a7: work_queue at kwork_queue.c:254
0x42025920: tcp_update_timer at tcp_timer.c:257
0x42025c21: tcp_timer at tcp_timer.c:821
0x42043dc9: tcp_poll at tcp_devpoll.c:104
0x42043508: devif_poll_tcp_connections at devif_poll.c:658
(inlined by) devif_poll_connections at devif_poll.c:857
0x420435bd: devif_poll at devif_poll.c:1054
0x42046ad5: wlan_dopoll at esp_wlan.c:1001 (discriminator 2)
0x42034988: work_queue_wq at kwork_queue.c:248
Backtrace for task 1:
0x420130d4: nx_idle_trampoline at nx_smpstart.c:88 (discriminator 1)
0x40000000: ?? ??:0
0x40000000: ?? ??:0
Backtrace for task 0:
0x4201304c: nx_start at nx_start.c:778 (discriminator 1)
0x403751f6: __esp32s3_start at esp32s3_start.c:457 (discriminator 1)
0x403752e2: __start at ??:?
0x40045c04: ?? ??:0
0x40043ab9: ?? ??:0
0x40034c48: ?? ??:0
0x40000000: ?? ??:0
Backtrace for task 10:
0x42017f94: nxsem_wait at sem_wait.c:168
0x42017b8d: nxmutex_lock at lib_mutex.c:257
0x42017e24: nxrmutex_restorelock at lib_mutex.c:1108
0x42026a54: net_restorelock at net_lock.c:212
0x42026ab0: _net_timedwait at net_lock.c:115
0x42026af1: net_sem_timedwait_uninterruptible at net_lock.c:368
0x42026bb8: net_bufpool_timedalloc at net_bufpool.c:110
0x42026855: devif_callback_alloc at devif_callback.c:287
0x42041010: psock_setup_callbacks at tcp_connect.c:100
(inlined by) psock_tcp_connect at tcp_connect.c:368
0x42040155: inet_connect at inet_sockif.c:1366
0x4203f998: psock_connect at connect.c:147
0x4203f9de: connect at connect.c:243
0x4203ebb4: iperf_tcp_client at iperf.c:791
0x4203e8d0: iperf_run_client at iperf.c:475
0x4203e9a0: iperf_run_tcp_client at iperf.c:831
(inlined by) iperf_task_traffic at iperf.c:858
0x42037110: pthread_startup at pthread_create.c:61
0x42044fd4: pthread_start at pthread_create.c:149
0x40000000: ?? ??:0
0x40000000: ?? ??:0
```
### On which OS does this issue occur?
[OS: Linux]
### What is the version of your OS?
Linux Mint 22.1
### NuttX Version
master
### Issue Architecture
[Arch: risc-v], [Arch: xtensa]
### Issue Area
[Area: Kernel], [Area: Drivers], [Area: Networking]
### Host information
_No response_
### Verification
- [x] I have verified before submitting the report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@nuttx.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
