[Bug]: [TloraPager] RotaryEncoder crash

[polarikus]

Category

Other

Hardware

Other

Is this bug report about any UI component firmware like InkHUD or Meshtatic UI (MUI)?

• Meshtastic UI aka MUI colorTFT
• InkHUD ePaper
• OLED slide UI on any display

Firmware Version

2.7.15

Description

Description

The T-LoRa Pager crashes with a LoadProhibited exception when using the rotary encoder. The crash occurs in GPIO interrupt handlers that attempt to access PSRAM through non-ISR-safe functions.

Steps

1. Wait for the transition to sleep mode:
   INFO | ??:??:?? 113 [PowerFSM] setup LORA_DIO1 (GPIO14) with wakeup by gpio interrupt
2. Press the encoder not just once, but press and hold it for a long time

Relevant log output

Core  1 register dump:
PC      : 0x40385273  PS      : 0x00060934  A0      : 0x80382687  A1      : 0x3fc9f010
A2      : 0x3fcf7bcc  A3      : 0xb33fffff  A4      : 0x0000cdcd  A5      : 0x00060d23
A6      : 0x00060d23  A7      : 0x0000abab  A8      : 0x0000cdcd  A9      : 0xffffffff
A10     : 0x00060b23  A11     : 0x00000003  A12     : 0x4037593d  A13     : 0x3fc9f130
A14     : 0x02cf7bcc  A15     : 0x00ffffff  SAR     : 0x00000019  EXCCAUSE: 0x00000006
EXCVADDR: 0x00000000  LBEG    : 0x40056f5c  LEND    : 0x40056f72  LCOUNT  : 0xffffffff
Core  1 was running in ISR context:
EPC1    : 0x421c02a3  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x40385273

stack:
0x40385270: esp_ptr_external_ram at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/soc/include/soc/soc_memory_types.h:115
  \-> inlined by: spinlock_acquire at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_hw_support/include/soc/spinlock.h:99
  \-> inlined by: xPortEnterCriticalTimeout at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port/xtensa/port.c:301
0x40382684: vPortEnterCritical at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port/xtensa/include/freertos/portmacro.h:578
  \-> inlined by: xQueueGenericSendFromISR at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:1082
0x420bc375: InputBroker::requestPollSoon(InputPollable*) at /Users/igordanilov/Dev/C++/meshtastic-firmware/src/input/InputBroker.cpp:26
0x420bc389: RotaryEncoderImpl::init()::{lambda()#1}::_FUN() at /Users/igordanilov/Dev/C++/meshtastic-firmware/src/input/RotaryEncoderImpl.cpp:33
  \-> inlined by: _FUN at /Users/igordanilov/Dev/C++/meshtastic-firmware/src/input/RotaryEncoderImpl.cpp:33
0x42002681: __onPinInterrupt at /Users/igordanilov/.platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-gpio.c:159
0x403758d9: gpio_isr_loop at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/driver/gpio.c:417
0x403758fe: gpio_intr_service at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/driver/gpio.c:434
0x40379fd9: _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1118
0x4037593a: gpio_ll_clear_intr_status_high at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/hal/esp32s3/include/hal/gpio_ll.h:150
  \-> inlined by: gpio_intr_service at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/driver/gpio.c:444
0x420fcefd: esp_vApplicationIdleHook at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/freertos_hooks.c:63
0x40382fcc: prvIdleTask at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4099


Core  0 register dump:
PC      : 0x40385371  PS      : 0x00060134  A0      : 0x803828d5  A1      : 0x3fcbda60
A2      : 0x00000001  A3      : 0x3fcb58b4  A4      : 0x3fcb58b4  A5      : 0x00060123
A6      : 0x00060120  A7      : 0x00000001  A8      : 0x00000001  A9      : 0x00000001
A10     : 0x00060123  A11     : 0x00000000  A12     : 0x00060120  A13     : 0x00000000
A14     : 0x02cf7bcc  A15     : 0x00ffffff  SAR     : 0x00000008  EXCCAUSE: 0x00000006
EXCVADDR: 0x00000000  LBEG    : 0x40056f5c  LEND    : 0x40056f72  LCOUNT  : 0xffffffff

stack:
0x4038536e: xPortEnterCriticalTimeout at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port/xtensa/port.c:312
0x403828d2: vPortEnterCritical at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port/xtensa/include/freertos/portmacro.h:578
  \-> inlined by: xQueueReceive at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:1400
0x420bc1e9: InputBroker::pollSoonWorker(void*) at /Users/igordanilov/Dev/C++/meshtastic-firmware/src/input/InputBroker.cpp:73

Guru Meditation Error: Core  1 panic'ed (Interrupt wdt timeout on CPU1).

Core  1 register dump:
PC      : 0x4038a50f  PS      : 0x00020034  A0      : 0x8213ddbe  A1      : 0x3fc9ee20
A2      : 0x00020023  A3      : 0xa5a5a5a5  A4      : 0x8213ddbe  A5      : 0x00060025
A6      : 0xfffbfff0  A7      : 0x00000046  A8      : 0x3fca280c  A9      : 0x00000001
A10     : 0x60000000  A11     : 0x00000001  A12     : 0x0000000a  A13     : 0x3fc9ee0c
A14     : 0x00000001  A15     : 0x3fcbdae0  SAR     : 0x0000000a  EXCCAUSE: 0x00000006
EXCVADDR: 0x00000000  LBEG    : 0x400570e8  LEND    : 0x400570f3  LCOUNT  : 0x00000000
Core  1 was running in ISR context:
EPC1    : 0x421c02a3  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x4038a50f

stack:
0x4038a50c: xthal_window_spill at ??:?
0x4213ddbb: esp_core_dump_replace_sp at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/espcoredump/include_core_dump/port/xtensa/esp_core_dump_port_impl.h:67
  \-> inlined by: esp_core_dump_setup_stack at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/espcoredump/src/core_dump_common.c:86
  \-> inlined by: esp_core_dump_write at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/espcoredump/src/core_dump_common.c:160
0x4213e3c1: esp_core_dump_to_flash at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/espcoredump/src/core_dump_flash.c:342
0x420fdd92: esp_panic_handler at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c:352
0x420fe092: panic_handler at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/panic_handler.c:188
0x40377fa9: panicHandler at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/panic_handler.c:213
0x40377960: xt_highint4 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/soc/esp32s3/highint_hdl.S:119

[mverch67]

This should be fixed in 2.7.16 by #8746 .

[polarikus]

@mverch67

The problem is not in duplicated events from the encoder, but in an attempt to access uninitialized or freed memory from within an ISR.

According to the stack trace I provided, the crash occurs on line 33 of the new encoder class (with ISR):
0x420bc389: RotaryEncoderImpl::init()::{lambda()#1}::_FUN() at /Users/igordanilov/Dev/C++/meshtastic-firmware/src/input/RotaryEncoderImpl.cpp:33
In the commit you provided, the old class with GPIO listener (RotaryEncoderInterruptImpl1) is disabled and replaced precisely by RotaryEncoderImpl (the crash occurs on its line 33).

Most likely, the problem is in RotaryEncoderImpl, specifically in the lambda function on line 33:
auto interruptHandler = { inputBroker->requestPollSoon(interruptInstance); };

The static variable interruptInstance is not protected from data races.

I haven't been able to reproduce it on 2.7.16 yet, but as I described in the issue – there are no clear reproduction steps, it will take some time.

[mverch67]

But your stack trace looks exactly the same as in #8565 (comment) which was caused by the double registration of the ISR handler.

[polarikus]

@mverch67 The crash was saved on version 2.7.17 (built from develop), here's the stack trace:

Guru Meditation Error: Core  1 panic'ed (Interrupt wdt timeout on CPU1). 

Core  1 register dump:
PC      : 0x400559cc  PS      : 0x00060734  A0      : 0x820bce74  A1      : 0x3fc9cfe0  
A2      : 0x00000001  A3      : 0x3fc9d00c  A4      : 0x00000000  A5      : 0x00000000  
A6      : 0x3fceed48  A7      : 0x3fceed94  A8      : 0x80385235  A9      : 0x400559cc  
A10     : 0x00060721  A11     : 0x00000001  A12     : 0x00060521  A13     : 0x00000005  
A14     : 0x00060723  A15     : 0x00000001  SAR     : 0x00000019  EXCCAUSE: 0x00000006  
EXCVADDR: 0x00000000  LBEG    : 0x40056f5c  LEND    : 0x40056f72  LCOUNT  : 0xffffffff  
Core  1 was running in ISR context:
EPC1    : 0x421c0b7f  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x400559cc


Backtrace: 0x400559c9:0x3fc9cfe0 0x420bce71:0x3fc9d000 0x420bcea9:0x3fc9d030 0x42002665:0x3fc9d050 0x403758d9:0x3fc9d070 0x403758fe:0x3fc9d090 0x40379fd9:0x3fc9d0b0 0x4037593a:0x3fcebc90 0x40385402:0x3fcebca0 0x4211f0a2:0x3fcebcc0 0x42063a2b:0x3fcebd10 0x420eac75:0x3fcebd30 0x42061431:0x3fcebd50 0x420bd41d:0x3fcebd70 0x420db6a7:0x3fcebd90 0x42061081:0x3fcebdb0 0x420bde97:0x3fcebdd0 0x4200ab9f:0x3fcebdf0

stack:
0x420bce71: InputBroker::requestPollSoon(InputPollable*) at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/input/InputBroker.cpp:25
0x420bcea9: RotaryEncoderImpl::init()::{lambda()#1}::_FUN() at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/input/RotaryEncoderImpl.cpp:33
  \-> inlined by: _FUN at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/input/RotaryEncoderImpl.cpp:33
0x42002665: __onPinInterrupt at /Users/igordanilov/.platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-gpio.c:159
0x403758d9: gpio_isr_loop at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/driver/gpio.c:417
0x403758fe: gpio_intr_service at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/driver/gpio.c:434
0x40379fd9: _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1118
0x4037593a: gpio_ll_clear_intr_status_high at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/hal/esp32s3/include/hal/gpio_ll.h:150
  \-> inlined by: gpio_intr_service at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/driver/gpio.c:444
0x40385402: vPortClearInterruptMaskFromISR at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port/xtensa/include/freertos/portmacro.h:571
  \-> inlined by: vPortExitCritical at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port/xtensa/port.c:332
0x4211f0a2: esp_light_sleep_start at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_hw_support/sleep_modes.c:894
0x42063a2b: doLightSleep(unsigned long long) at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/sleep.cpp:454
0x420eac75: _ZL6lsIdlev$lto_priv$622 at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/PowerFSM.cpp:107
0x42061431: Fsm::run_machine() at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/.pio/libdeps/tlora-pager/arduino-fsm/Fsm.cpp:142
0x420bd41d: concurrency::PowerFSMThread::runOnce() at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/PowerFSMThread.h:22
0x420db6a7: concurrency::OSThread::run() at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/concurrency/OSThread.cpp:85
0x42061081: ThreadController::runOrDelay() at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/.pio/libdeps/tlora-pager/ArduinoThread/ThreadController.cpp:59
0x420bde97: loop() at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/main.cpp:1624
0x4200ab9f: loopTask(void*) at /Users/igordanilov/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp:50


Core  0 register dump:
PC      : 0x40383f51  PS      : 0x00060334  A0      : 0x8038281f  A1      : 0x3fcf7510  
A2      : 0x3fcad84c  A3      : 0x00000000  A4      : 0x3fcad814  A5      : 0x00060323  
A6      : 0x00060320  A7      : 0x00000001  A8      : 0x00000000  A9      : 0x00000002  
A10     : 0x00060320  A11     : 0x00000000  A12     : 0x00000000  A13     : 0x00000000  
A14     : 0x3fcf7940  A15     : 0x00000000  SAR     : 0x00000008  EXCCAUSE: 0x00000006  
EXCVADDR: 0x00000000  LBEG    : 0x40056f5c  LEND    : 0x40056f72  LCOUNT  : 0xffffffff  


Backtrace: 0x40383f4e:0x3fcf7510 0x4038281c:0x3fcf7530 0x420bccb1:0x3fcf7570

stack:
0x40383f4e: xTaskGetSchedulerState at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4802
0x4038281c: xQueueReceive at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:1391
0x420bccb1: InputBroker::pollSoonWorker(void*) at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/input/InputBroker.cpp:6


ELF file SHA256: a5160cb8b36c7bb4

Guru Meditation Error: Core  1 panic'ed (Interrupt wdt timeout on CPU1). 

Core  1 register dump:
PC      : 0x4038a50f  PS      : 0x00020834  A0      : 0x8213e69a  A1      : 0x3fc9cdf0  
A2      : 0x00020823  A3      : 0xa5a5a5a5  A4      : 0x8213e69a  A5      : 0x00060825  
A6      : 0xfffbfff0  A7      : 0x00000046  A8      : 0x3fca078c  A9      : 0x00000001  
A10     : 0x60000000  A11     : 0x00000001  A12     : 0x0000000a  A13     : 0x3fc9cddc  
A14     : 0x00000001  A15     : 0x3fcf7570  SAR     : 0x0000000a  EXCCAUSE: 0x00000006  
EXCVADDR: 0x00000000  LBEG    : 0x400570e8  LEND    : 0x400570f3  LCOUNT  : 0x00000000  
Core  1 was running in ISR context:
EPC1    : 0x421c0b7f  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x4038a50f


Backtrace: 0x4038a50c:0x3fc9cdf0 0x4213e697:0x3fc9ce00 0x4213ec9d:0x3fc9ce20 0x420fe602:0x3fc9ce40 0x420fe902:0x3fc9ceb0 0x40377fa9:0x3fc9cf00 0x40377960:0x3fc9cf20 0x00040022:0x3fc9cfe0 |<-CORRUPTED

stack:
0x4038a50c: xthal_window_spill at ??:?
0x4213e697: esp_core_dump_replace_sp at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/espcoredump/include_core_dump/port/xtensa/esp_core_dump_port_impl.h:67
  \-> inlined by: esp_core_dump_setup_stack at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/espcoredump/src/core_dump_common.c:86
  \-> inlined by: esp_core_dump_write at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/espcoredump/src/core_dump_common.c:160
0x4213ec9d: esp_core_dump_to_flash at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/espcoredump/src/core_dump_flash.c:342
0x420fe602: esp_panic_handler at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c:352
0x420fe902: panic_handler at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/panic_handler.c:188
0x40377fa9: panicHandler at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/panic_handler.c:213
0x40377960: xt_highint4 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/port/soc/esp32s3/highint_hdl.S:119


Core  0 register dump:
PC      : 0x40383f51  PS      : 0x00060334  A0      : 0x8038281f  A1      : 0x3fcf7510  
A2      : 0x3fcad84c  A3      : 0x00000000  A4      : 0x3fcad814  A5      : 0x00060323  
A6      : 0x00060320  A7      : 0x00000001  A8      : 0x00000000  A9      : 0x00000002  
A10     : 0x00060320  A11     : 0x00000000  A12     : 0x00000000  A13     : 0x00000000  
A14     : 0x3fcf7940  A15     : 0x00000000  SAR     : 0x00000008  EXCCAUSE: 0x00000006  
EXCVADDR: 0x00000000  LBEG    : 0x40056f5c  LEND    : 0x40056f72  LCOUNT  : 0xffffffff  


Backtrace: 0x40383f4e:0x3fcf7510 0x4038281c:0x3fcf7530 0x420bccb1:0x3fcf7570

stack:
0x40383f4e: xTaskGetSchedulerState at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4802
0x4038281c: xQueueReceive at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:1391
0x420bccb1: InputBroker::pollSoonWorker(void*) at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/input/InputBroker.cpp:69




ELF file SHA256: a5160cb8b36c7bb4

[Xaositek]

Just experienced this in 2.7.17 myself after navigating through BaseUI and a device uptime of around 3 hours.

Guru Meditation Error: Core 1 panic'ed (Interrupt wdt timeout on CPU1).

Core 1 register dump:
PC : 0x420188f8 PS : 0x00060134 A0 : 0x8037588d A1 : 0x3fc9cf10
A2 : 0x00000000 A3 : 0x00000000 A4 : 0x00000001 A5 : 0x00000004
A6 : 0x3fcf7314 A7 : 0x80000001 A8 : 0x80375868 A9 : 0x420188f8
A10 : 0x3fca8d54 A11 : 0x00000003 A12 : 0x00060023 A13 : 0x80000000
A14 : 0x00000000 A15 : 0x00ffffff SAR : 0x00000019 EXCCAUSE: 0x00000006
EXCVADDR: 0x00000000 LBEG : 0x40056f5c LEND : 0x40056f72 LCOUNT : 0xffffffff
Core 1 was running in ISR context:
EPC1 : 0x42183943 EPC2 : 0x00000000 EPC3 : 0x00000000 EPC4 : 0x420188f8

Backtrace: 0x420188f5:0x3fc9cf10 0x4037588a:0x3fc9cf30 0x40379f65:0x3fc9cf50 0x403758c6:0x3fcf7160 0x420c0afd:0x3fcf7180 0x40382f1c:0x3fcf71a0

Core 0 register dump:
PC : 0x4038282a PS : 0x00060534 A0 : 0x8208c3dc A1 : 0x3fcf7c80
A2 : 0x3fcf191c A3 : 0x3fcf7ccc A4 : 0x3fcf1968 A5 : 0x00000005
A6 : 0x00000000 A7 : 0x80000001 A8 : 0x80382825 A9 : 0x3fcf7c40
A10 : 0x00000001 A11 : 0x3fcaddac A12 : 0x3fcaddac A13 : 0x00060523
A14 : 0x00060520 A15 : 0x00000001 SAR : 0x00000008 EXCCAUSE: 0x00000006
EXCVADDR: 0x00000000 LBEG : 0x40056f5c LEND : 0x40056f72 LCOUNT : 0xffffffff

Backtrace: 0x40382827:0x3fcf7c80 0x4208c3d9:0x3fcf7cc0

ELF file SHA256: 56d7e9d9655adb1f

Guru Meditation Error: Core 1 panic'ed (Interrupt wdt timeout on CPU1).

Core 1 register dump:
PC : 0x4038a45f PS : 0x00020a34 A0 : 0x82100f32 A1 : 0x3fc9cd20
A2 : 0x00020a23 A3 : 0xa5a5a5a5 A4 : 0x82100f32 A5 : 0x00060a25
A6 : 0xfffbfff0 A7 : 0x00000046 A8 : 0x3fca062c A9 : 0x00000001
A10 : 0x60000000 A11 : 0x00000001 A12 : 0x0000000a A13 : 0x3fc9cd0c
A14 : 0x00000001 A15 : 0x3fcf7cc0 SAR : 0x0000000a EXCCAUSE: 0x00000006
EXCVADDR: 0x00000000 LBEG : 0x400570e8 LEND : 0x400570f3 LCOUNT : 0x00000000
Core 1 was running in ISR context:
EPC1 : 0x42183943 EPC2 : 0x00000000 EPC3 : 0x00000000 EPC4 : 0x4038a45f

Backtrace: 0x4038a45c:0x3fc9cd20 0x42100f2f:0x3fc9cd30 0x42101535:0x3fc9cd50 0x420c1992:0x3fc9cd70 0x420c1c92:0x3fc9cde0 0x40377f35:0x3fc9ce30 0x403778ec:0x3fc9ce50 0x00040022:0x3fc9cf10 |<-CORRUPTED

Core 0 register dump:
PC : 0x4038282a PS : 0x00060534 A0 : 0x8208c3dc A1 : 0x3fcf7c80
A2 : 0x3fcf191c A3 : 0x3fcf7ccc A4 : 0x3fcf1968 A5 : 0x00000005
A6 : 0x00000000 A7 : 0x80000001 A8 : 0x80382825 A9 : 0x3fcf7c40
A10 : 0x00000001 A11 : 0x3fcaddac A12 : 0x3fcaddac A13 : 0x00060523
A14 : 0x00060520 A15 : 0x00000001 SAR : 0x00000008 EXCCAUSE: 0x00000006
EXCVADDR: 0x00000000 LBEG : 0x40056f5c LEND : 0x40056f72 LCOUNT : 0xffffffff

Backtrace: 0x40382827:0x3fcf7c80 0x4208c3d9:0x3fcf7cc0

ELF file SHA256: 56d7e9d9655adb1f

Re-entered core dump! Exception happened during core dump!
Rebooting...
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0xc (RTC_SW_CPU_RST),boot:0x2b (SPI_FAST_FLASH_BOOT)
Saved PC:0x420c1b9e
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x4bc
load:0x403c9700,len:0xbd8
load:0x403cc700,len:0x2a0c
entry 0x403c98d0
[ 1299][I][esp32-hal-psram.c:96] psramInit(): PSRAM enabled
[ 1317][I][esp32-hal-i2c.c:75] i2cInit(): Initialising I2C Master: sda=3 scl=2 freq=100000
WARNING file:mesh_interface.py _sendToRadio line:1209 Not sending packet because protocol use is disabled by noProto

[polarikus]

Found a reproducible case, added to the description.
My thoughts:
I think that after transitioning to sleep mode, if multiple encoder presses are made:

The first press triggers the wake-up procedure

For subsequent presses, we attempt to execute xQueueSendFromISR in the InputBroker::requestPollSoon(InputPollable *pollable) method while the system is still waking up. During wake-up, memory may be inaccessible, causing a crash:

soc_memory_types.h:115

inline static bool IRAM_ATTR esp_ptr_external_ram(const void *p) {
#if SOC_SPIRAM_SUPPORTED
    return ((intptr_t)p >= SOC_EXTRAM_DATA_LOW && (intptr_t)p < SOC_EXTRAM_DATA_HIGH);
#else
    return false; // SoC has no external RAM
#endif
}

I'll try to find a solution, and if I find one, I'll submit a PR.

[polarikus]

@mverch67

To test the theory, I did the following:

1. In main-esp32.cpp:

#if SOC_RTCIO_HOLD_SUPPORTED && SOC_PM_SUPPORT_EXT_WAKEUP
    const uint64_t gpioMask = (1ULL << (config.device.button_gpio ? config.device.button_gpio : BUTTON_PIN) |
        (1ULL << (moduleConfig.canned_message.inputbroker_pin_press ?: ROTARY_A)));
#endif

2. In RotaryEncoderImpl:

void RotaryEncoderImpl::deInit() {
    detachInterrupt(moduleConfig.canned_message.inputbroker_pin_a);
    detachInterrupt(moduleConfig.canned_message.inputbroker_pin_b);
    detachInterrupt(moduleConfig.canned_message.inputbroker_pin_press);
    LOG_INFO("RotaryEncoder deInit pins(%d, %d, %d), events(%d, %d, %d)", moduleConfig.canned_message.inputbroker_pin_a,
         moduleConfig.canned_message.inputbroker_pin_b, moduleConfig.canned_message.inputbroker_pin_press, eventCw, eventCcw,
         eventPressed);
}

3. In PowerFSM:

static void lsExit()
{
    rotaryEncoderImpl->init();
    LOG_POWERFSM("State: lsExit");
}

static void onEnter()
{
    rotaryEncoderImpl->init();
    LOG_POWERFSM("State: onEnter");
    if (screen)
        screen->setOn(true);
    setBluetoothEnable(true);
}

static void sdsEnter()
{
    LOG_POWERFSM("State: SDS");
    rotaryEncoderImpl->deInit();
    // FIXME - make sure GPS and LORA radio are off first - because we want close to zero current draw
    doDeepSleep(Default::getConfiguredOrDefaultMs(config.power.sds_secs), false, false);
}

static void lsEnter()
{
    rotaryEncoderImpl->deInit();
    LOG_POWERFSM("lsEnter begin, ls_secs=%u", config.power.ls_secs);
    if (screen)
        screen->setOn(false);
    secsSlept = 0; // How long have we been sleeping this time
    // LOG_INFO("lsEnter end");
}

After this, the crash disappeared. This means we need to:

1. Use ESP32 wakeup from sleep via interrupts
2. Disable interrupts in RotaryEncoderImpl when going to sleep

My solution for testing the theory isn't suitable for production.

I propose implementing SleepObserver/SleepManager. SleepManager would notify all devices (such as GPS, Encoder, LoraModule) that they need to perform certain actions before transitioning to sleep and upon waking up. What do you think? I can take on the implementation for the specific case with RotaryEncoderImpl. If a similar implementation is needed for GPS (I saw it in TODO), we can create a separate issue, and when time permits, I'll work on it as well.

[polarikus]

Regarding the stack trace from @Xaositek: It doesn't look like my issue; seems to be a problem with interrupts in a different place:

stack:
0x420188f5: ChaCha::hashCore(unsigned int*, unsigned int const*, unsigned char) at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/.pio/libdeps/tlora-pager/Crypto/ChaCha.cpp:273
0x4037588a: concurrency::NotifiedWorkerThread::notifyFromISR(int*, unsigned int, bool) at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/concurrency/NotifiedWorkerThread.cpp:53
0x40379f65: _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1087
0x403758c6: gpio_isr_loop at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/driver/gpio.c:416
0x420c0afd: graphics::UIRenderer::drawNavigationBar(OLEDDisplay*, OLEDDisplayUiState*) at /Users/igordanilov/Dev/C++/meshtastic-fw-my-fork/src/graphics/draw/UIRenderer.cpp:1318
0x40382f1c: prvDeleteTLS at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4721
-> inlined by: prvDeleteTLS at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4719

[mverch67]

I propose implementing SleepObserver/SleepManager. SleepManager would notify all devices (such as GPS, Encoder, LoraModule) that they need to perform certain actions before transitioning to sleep and upon waking up. What do you think?

The way to go would be to use the existing notifyLightSleep / notifyLightSleepEnd observables as already used by e.g. ButtonThread and MUI. I.e. RotaryEncoderImpl needs to subscribe and implement the callbacks.

[polarikus]

Предлагаю реализовать SleepObserver/SleepManager. SleepManager будет уведомлять все устройства (такие как GPS, кодировщик, LoraModule) о необходимости выполнения определенных действий перед переходом в спящий режим и после пробуждения. Что вы думаете по этому поводу?

Правильным решением было бы использовать существующие наблюдаемые объекты notifyLightSleep / notifyLightSleepEnd , которые уже используются, например, в ButtonThread и MUI. То есть RotaryEncoderImpl должен подписаться на них и реализовать функции обратного вызова.

Thanks for the tip! I'll take care of this.