Guru Meditation Error: Core 0 panic'ed (Interrupt wdt timeout on CPU0) (IDFGH-398)

[sjoteppagol]

Environment

• Development Kit: [ESP32-Wrover-Kit]
• Kit version (for WroverKit/PicoKit/DevKitC): [v1|v2|v3|v4]
• Core (if using chip or module): [ESP-WROOM-32D]
• IDF version (git rev-parse --short HEAD to get the commit id.): 94ec3c8
• Development Env: [Make]
• Operating System: [Windows]
• Power Supply: [external 3.3]

Problem Description

The device reboots with error message.

Guru Meditation Error: Core 0 panic'ed (Interrupt wdt timeout on CPU0)
Register dump:
PC : 0x400d2132 PS : 0x00060b34 A0 : 0x8008a298 A1 : 0x3ffd9f60
A2 : 0x00000008 A3 : 0x00000000 A4 : 0x00000001 A5 : 0x3ffda578
A6 : 0x00000000 A7 : 0x00000000 A8 : 0x3ffc566c A9 : 0x3ffc5650
A10 : 0x00000000 A11 : 0x00000001 A12 : 0x43422c0c A13 : 0x00000001
A14 : 0x00000000 A15 : 0x3ffc05c0 SAR : 0x00000000 EXCCAUSE: 0x00000005
EXCVADDR: 0x00000000 LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00000000

Backtrace: 0x400d2132:0x3ffd9f60 0x4008a295:0x3ffd9f80

//Detailed problem description goes here.
This type of guru meditation repeatedly occurred and back trace result doesn't seems to point out any application related code: Kinldy find the backtrace result below.

$ xtensa-esp32-elf-addr2line -pfiaC -e build/ecowater.elf 0x400d2132:0x3ffd9f60 0x4008a295:0x3ffd9f80
0x400d2132: esp_vApplicationIdleHook at C:/ESP_V3.0/msys32/home/sjoteppagol/esp-idf-v3.0/components/esp32/freertos_hooks.c:85
0x4008a295: prvIdleTask at C:/ESP_V3.0/msys32/home/sjoteppagol/esp-idf-v3.0/components/freertos/tasks.c:3529

Expected Behavior

The device should work normally.

Actual Behavior

The device is set up to run continuously but it is observed that same type of guru meditation happened 3-5 times in whole day.

Steps to repropduce

1. Run system for 6 to 8 hours
2. At some point system resets with message Guru Meditation Error: Core 0 panic'ed(Interrupt wdt timeout on CPU0)
3. The occurrence of guru meditation is un-predictable but it is observed that if the system continuously running for over 6 to 8 hours at least one time it experienced Guru meditation reset.

// It helps if you attach a picture of your setup/wiring here.

Code to reproduce this issue

// If your code is longer than 30 lines, GIST is preferred.

Debug Logs

Debug log goes here, should contain the backtrace, as well as the reset source if it is a crash.
Please copy the plain text here for us to search the error log. Or attach the complete logs but leave the main part here if the log is *too* long.

Other items if possible

• sdkconfig file (attach the sdkconfig file from your project folder)
• elf file in the build folder (note this may contain all the code details and symbols of your project.)
• coredump (This provides stacks of tasks.)
  $ xtensa-esp32-elf-addr2line -pfiaC -e build/ecowater.elf 0x400d2132:0x3ffd9f60 0x4008a295:0x3ffd9f80
  0x400d2132: esp_vApplicationIdleHook at C:/ESP_V3.0/msys32/home/sjoteppagol/esp-idf-v3.0/components/esp32/freertos_hooks.c:85
  0x4008a295: prvIdleTask at C:/ESP_V3.0/msys32/home/sjoteppagol/esp-idf-v3.0/components/freertos/tasks.c:3529

[gerekon]

@sjoteppagol Does the code you specified (empty app_main) re-produces the problem? I think this is just copy-pasted from problem's description template.
It's hard to determine the root cause having such little amount of info. Now I can say that your problem can be caused by the code which accidentally disables interrupts and does not re-enable them till system watchdog fires.
Could you provide your code for testing, sdkconfig, Elf file?

[sjoteppagol]

@gerekon The code specified (empty app_main) was part of problem description template and accidently left like that, sorry for that. I wont be able to share code, but I will try to reproduce this issue and share you both sdkconfig, elf file.

[Alvin1Zhang]

@sjoteppagol Could you help share if any updates for this issue? Thanks.

[sjoteppagol]

I could not able to re-produce this issue.
Thanks

[sjoteppagol]

I am able to reproduce this issue, kept device running over night and in 12 hours device suffered 38 times from Guru Meditation: Core 0 panic'ed (Interrupt wdt timeout on CPU0). Could you please suggest a way to overcome this issue.
Kindly find the attached sdkconfiglong, elf and Back trace result.

Backtrace_result.txt
sdkconfig.txt
Gurumediation_testing.zip

[sjoteppagol]

From the above log, I see one patter where the “ESP AWS Task Will Re-Init for New Connection” followed by “Shadow Connect” results in some program hung hence the WDT is getting triggered…. May be some programming sequence is not tolerated by some stack.!!

[gerekon]

@sjoteppagol Do you run your project on bare ESP32-Wrover-Kit or you are using additional HW connected to it?

If no extra HW needed:
What actions should be performed to reproduce your problem? Is it enough just to flash your ELF file and leave the device working for 12 hours? Could you provide you partition table binary also?

Could you try with CONFIG_INT_WDT_CHECK_CPU1=0 in sdkconfig?

[sjoteppagol]

Hello @gerekon ,
Thank you for your response, it was helpful.
As per your suggestion made changes in sdkconfig, not seen this issue till now.
Could you please explain what this option actually do!?
Whether it will impact my other functionality ?

[gerekon]

@sjoteppagol CONFIG_INT_WDT_CHECK_CPU1 controls checks for stalled OS on CPU1. It produces watchdog panic when OS does not tick on CPU1, e.g. when interrupts are disabled for a long period of time (by default 300 ms).
Since it is not possible for us to reproduce your situation can I ask you to perform some additional verification?
There is new feature added in IDF by 624828c. It makes IDF printing core dumps for both CPUs upon WDT timeout. It can be cherry-picked cleanly onto your 94ec3c8.
Can you pickup that changes and re-try your tests with enabled CONFIG_INT_WDT_CHECK_CPU1=1? It can help to detect if the problem in IDF or in your code.

[Alvin1Zhang]

@sjoteppagol Thanks for reporting, would you please help share if any updates for this issue? Thanks.

[ion-girloanta]

Got the same error when
portENTER_CRITICAL_ISR
without
portEXIT_CRITICAL_ISR

[gerekon]

@ion-girloanta This is incorrect. Every portENTER_CRITICAL_ISR should be followed by portEXIT_CRITICAL_ISR. portENTER_CRITICAL_ISR disables interrupts and this leads to WDT triggering. This is intended behavior, otherwise w/o interrupts system can be non-functional and unresponsive.

[enricop]

#2542

[ole00]

I had the same issue. In my case the watchdog was triggered by poisoning the heap to Comprehensive level (I suspect the poisoning took too long) . When the level was lowered to Basic (no poisoning) or the timeout set via CONFIG_INT_WDT_CHECK_CPU1 (as suggested above by gerekon) was increased to 600 the issue disappeared.

[copleston]

Echoing what @ole00 said, I found that Comprehensive heap debugging was causing the errors described above.

[SoucheSouche]

Closing issue after 3 years of inactivity. If the problem still occurs, please refer to fatal errors documentation.