Re-enable cpu cache. Guru Meditation Error: Core 0 panic'ed (Cache disabled but cached memory region accessed). (IDFGH-5520)

[RuslanSpartak]

Hello, esp32 sdk developers.

1. ENV: platformio, esp-idf 4.3, custom development kit
2. I have strange behaviour with sdk version 4.3 (stable).
   When I don't place I2C driver in IRAM (Core 1) then I had sometimes read problems on I2C bus, but generally other stuff is working. When I want to add it in IRAM, I constantly receive "Re-enable cpu cache. Guru Meditation Error: Core 0 panic'ed (Cache disabled but cached memory region accessed). " So seems like wifi driver (Core 0) not working or somehow I2C placed in IRAM influent on wifi and other stuff.
3. Re-enable cpu cache.
   Guru Meditation Error: Core 0 panic'ed (Cache disabled but cached memory region accessed).
4. on sdk version 4.2.1 all works as expected.

Core 0 register dump:
PC : 0x401f80b0 PS : 0x00060034 A0 : 0x80081f11 A1 : 0x3ffc0260
A2 : 0x00000000 A3 : 0x00000000 A4 : 0x3ffbf728 A5 : 0x00002000
A6 : 0x00000026 A7 : 0x3ffd5ef0 A8 : 0x80081e44 A9 : 0x3ffee0c4
A10 : 0x3ffbf728 A11 : 0x00001000 A12 : 0x3ffed5c9 A13 : 0x3ff53000
A14 : 0x00000020 A15 : 0xffffcffb SAR : 0x0000001f EXCCAUSE: 0x00000007
EXCVADDR: 0x00000000 LBEG : 0x40001609 LEND : 0x4000160d LCOUNT : 0x00000000

Backtrace:0x401f80ad:0x3ffc0260 0x40081f0e:0x3ffc0290 0x40083f75:0x3ffc02c0 0x40096d55:0x3ffbc2a0 0x4009cb6d:0x3ffbc2d0 0x4009cb8e:0x3ffbc2f0 0x400856d3:0x3ffbc370 0x401566a1:0x3ffbc3b0 0x40153e1b:0x3ffbc3e0 0x40154989:0x3ffbc400 0x40154d11:0x3ffbc420 0x40153111:0x3ffbc480 0x40153685:0x3ffbc4d0 0x40153ca9:0x3ffbc530 0x401f889d:0x3ffbc550 0x40152ca7:0x3ffbc570 0x40152ccd:0x3ffbc5a0 0x40112f9b:0x3ffbc5c0 0x400d4e11:0x3ffbc600 0x402016dd:0x3ffbc630 0x40096612:0x3ffbc650

[RuslanSpartak]

So maybe it is not wifi issue but I2C issue, as using i2c_driver_install(i2c_port, conf.mode, I2C_MASTER_RX_BUF_DISABLE, I2C_MASTER_TX_BUF_DISABLE, (ESP_INTR_FLAG_IRAM | ESP_INTR_FLAG_LEVEL3)); cause rebooting problem, and if using i2c_driver_install(i2c_port, conf.mode, I2C_MASTER_RX_BUF_DISABLE, I2C_MASTER_TX_BUF_DISABLE, 0) works but cost me reading issues.

[RuslanSpartak]

I use optimization for size. I check even with -O0. It wasn't working. So any ideas why, when I want to use ISR in IRAM (I2C) I had such problem?

[alamaral]

I'm seeing the same issue, but on pretty much all the examples like nvs_rw_value that call nvs_flash_init().
Have you figured anything out? I was using the latest code, then went back to 4.3.1. I just now tried v4.2.1 to see if that would work, and it does...

I've had stuff running in the past, with older versions of esp-idf, but the latest stuff just seems badly broken (or something changed that I haven't figured out). In any case, I've worked with several different ESP-32's from different vendors, trying to get audio to work, but the esp-adf versions they provide are ancient... I'm quickly becoming disenchanted with this development environment...

[o-marshmallow]

Hello @RuslanSpartak ,
I tried reproducing your problem with i2c_self_test example, specifying ESP_INTR_FLAG_IRAM | ESP_INTR_FLAG_LEVEL3 as flags, but I was unable to get your error. I tried with both master and v4.3.
Can you please provide a code snippet to reproduce your issue?

[alamaral]

Probably. I just used the defaults. I've since gone back to using an older version and it just worked.

[AxelLin]

@RuslanSpartak
Any chance to decode the backtrace? e.g. by running "idf monitor".

[RuslanSpartak]

On 4.2.1 it is working. On 4.3 - not.  18 листопада 2021, 04:03:56, від "Axel Lin" ***@***.***>: I'm seeing the same issue, but on pretty much all the examples like nvs_rw_value that call nvs_flash_init(). Have you figured anything out? I was using the latest code, then went back to 4.3.1. I just now tried v4.2.1 to see if that would work, and it does... @alamaral Just wondering if you compile with -O0? Your report looks like the same as #7835 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[RuslanSpartak]

I use platformio ide. So can't decode backtrace. 18 листопада 2021, 05:36:52, від "Axel Lin" ***@***.***>: @RuslanSpartak Any chance to decode the backtrace? e.g. by running "idf monitor". — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[o-marshmallow]

@RuslanSpartak In that case, you can decode the backtrace manually using objdump on your ELF file.
The backtrace is a list of PC:SP, so you you get rid of the SP, you get the first PC that triggers the crash and then all the call stack.
So if you have time you can use xtensa-esp32-elf-objdump -d your_elf.elf to generate the assembly code of you file and then look for the PCs given in the backtrace

[RuslanSpartak]

Dump file looks like:
Exception Cause: Not found

0x40078000: ?? ??:0
0x40080400: _init at ??:?
0x40080604: ?? ??:0
0x401d537c: i2c_hal_update_config at ??:?
0x400849a9: esp_timer_impl_get_counter_reg at C:\Users\Ruslan.platformio\packages\framework-espidf\components\esp_timer\src/esp_timer_impl_lac.c:126 (discriminator 2)
0x400849b1: esp_timer_impl_get_counter_reg at C:\Users\Ruslan.platformio\packages\framework-espidf\components\esp_timer\src/esp_timer_impl_lac.c:127 (discriminator 1)
0x401d5379: i2c_hal_enable_slave_rx_it at C:\Users\Ruslan.platformio\packages\framework-espidf\components\hal/i2c_hal.c:170
0x40083192: i2c_isr_handler_default at C:\Users\Ruslan.platformio\packages\framework-espidf\components\driver/i2c.c:492
0x400851c1: _xt_medint3 at C:\Users\Ruslan.platformio\packages\framework-espidf\components\freertos\port\xtensa/xtensa_vectors.S:1274
0x40097870: spi_flash_ll_set_read_mode at C:\Users\Ruslan.platformio\packages\framework-espidf\components\hal\esp32\include/hal/spi_flash_ll.h:277
(inlined by) spi_flash_hal_configure_host_io_mode at C:\Users\Ruslan.platformio\packages\framework-espidf\components\hal/spi_flash_hal_common.inc:141
0x4009ca7a: spi_flash_chip_generic_config_host_io_mode at C:\Users\Ruslan.platformio\packages\framework-espidf\components\spi_flash/spi_flash_chip_generic.c:437
0x4009ca9e: spi_flash_chip_generic_read at C:\Users\Ruslan.platformio\packages\framework-espidf\components\spi_flash/spi_flash_chip_generic.c:194
0x40086923: esp_flash_read at C:\Users\Ruslan.platformio\packages\framework-espidf\components\spi_flash/esp_flash_api.c:663
0x4013a06a: esp_partition_read at C:\Users\Ruslan.platformio\packages\framework-espidf\components\spi_flash/partition.c:410
0x401379ba: nvs::NVSPartition::read(unsigned int, void*, unsigned int) at C:\Users\Ruslan.platformio\packages\framework-espidf\components\nvs_flash\src/nvs_partition.cpp:45
(inlined by) nvs::NVSPartition::read(unsigned int, void*, unsigned int) at C:\Users\Ruslan.platformio\packages\framework-espidf\components\nvs_flash\src/nvs_partition.cpp:39
0x401384d9: nvs::Page::readEntry(unsigned int, nvs::Item&) const at C:\Users\Ruslan.platformio\packages\framework-espidf\components\nvs_flash\src/nvs_page.cpp:797
0x4013880f: nvs::Page::findItem(unsigned char, nvs::ItemType, char const*, unsigned int&, nvs::Item&, unsigned char, nvs::VerOffset) at C:\Users\Ruslan.platformio\packages\framework-espidf\components\nvs_flash\src/nvs_page.cpp:841
(inlined by) nvs::Page::findItem(unsigned char, nvs::ItemType, char const*, unsigned int&, nvs::Item&, unsigned char, nvs::VerOffset) at C:\Users\Ruslan.platformio\packages\framework-espidf\components\nvs_flash\src/nvs_page.cpp:804
0x40136b1a: nvs::Storage::init(unsigned int, unsigned int) at C:\Users\Ruslan.platformio\packages\framework-espidf\components\nvs_flash\src/nvs_storage.cpp:112
0x40137af2: nvs::NVSPartitionManager::init_custom(nvs::Partition*, unsigned int, unsigned int) at C:\Users\Ruslan.platformio\packages\framework-espidf\components\nvs_flash\src/nvs_partition_manager.cpp:99
0x40137bcf: nvs::NVSPartitionManager::init_partition(char const*) at C:\Users\Ruslan.platformio\packages\framework-espidf\components\nvs_flash\src/nvs_partition_manager.cpp:61
0x40135f49: nvs_flash_init_partition at C:\Users\Ruslan.platformio\packages\framework-espidf\components\nvs_flash\src/nvs_api.cpp:135
0x400f9682: LoadConfiguration at C:\Users\Ruslan\Documents\parana-esp32-firmware/components\Spinn\HAL\Src/spinn_configuration.cpp:39
0x400d461f: app_main at C:\Users\Ruslan\Documents\parana-esp32-firmware/src/main.cpp:978
0x401dce37: main_task at C:\Users\Ruslan.platformio\packages\framework-espidf\components\freertos\port/port_common.c:133 (discriminator 2)

[AxelLin]

@o-marshmallow
Any finding in #7249 (comment)

[raphael-bmec-co]

I'm also seeing this issue. Is there any feedback?

[AxelLin]

I'm also seeing this issue. Is there any feedback?

@raphael-bmec-co
Which esp-idf version are you using (git describe --tags) ?

@o-marshmallow
I'm wondering if this fix is related: #8422 (the fix is not yet available in any stable branches)

[raphael-bmec-co]

@AxelLin I'm actually using the Arduino core. I have pinned down the issue now and I am not 100% sure it is related but here it is for interest: espressif/arduino-esp32#5751 (comment)

[o-marshmallow]

@AxelLin @RuslanSpartak
This backtrace is very intriguing to me, it seems like i2c_hal_enable_slave_rx_it called the function esp_timer_impl_get_counter_reg but that's not possible, i2c_hal_enable_slave_rx_it is implemented to call i2c_ll_slave_enable_rx_it only.

The other thing, which explains the error is that i2c_hal_update_config is placed in .flash.text (which starts at 0x400d0020 on the ESP32). But the cache was disabled, I guess by the SPI flash driver.

@AxelLin So yes, I would say it is related to #8422, we should not rely on flash functions in ISR because ISR are/may be in IRAM.

[raphael-bmec-co]

If that is the case then there may be some connection to this too: espressif/arduino-esp32#5882 (comment)

[o-marshmallow]

A patch is on the way for v4.3, it is currently under review

[raphael-bmec-co]

@o-marshmallow Thank you for the update. Do you have any idea what the pathway and timelines are for the patch making its way into the Arduino Core?

[o-marshmallow]

Hi @raphael-bmec-co ,
Unfortunately I don't know the answer as Arduino core is another project

[RuslanSpartak]

@AxelLin @RuslanSpartak This backtrace is very intriguing to me, it seems like i2c_hal_enable_slave_rx_it called the function esp_timer_impl_get_counter_reg but that's not possible, i2c_hal_enable_slave_rx_it is implemented to call i2c_ll_slave_enable_rx_it only.

The other thing, which explains the error is that i2c_hal_update_config is placed in .flash.text (which starts at 0x400d0020 on the ESP32). But the cache was disabled, I guess by the SPI flash driver.

@AxelLin So yes, I would say it is related to #8422, we should not rely on flash functions in ISR because ISR are/may be in IRAM.

It is true. I found it a long time ago, but forget to mention. Hope that new releases will be with fix.

[raphael-bmec-co]

@o-marshmallow you mentioned you will be patching 4.3. Will you also be patching 4.4? The latest versions of the Arduino core are built against 4.4 and the issue is also seen there.

[RuslanSpartak]

@o-marshmallow you mentioned you will be patching 4.3. Will you also be patching 4.4? The latest versions of the Arduino core are built against 4.4 and the issue is also seen there.

It was tested with 4.3 and I had there that issue. I didn't check it with v4.4, but if developers didn't change anything in ISR of I2C than issue should remain.

[o-marshmallow]

Hi @raphael-bmec-co @RuslanSpartak,

Good news, both 4.3 and 4.4 have been patched internally, they will be merged to Github once the internal CI passes

[raphael-bmec-co]

Awesome! Looking forward to being up and running again soon.

[RuslanSpartak]

Did we have updates already in v4.3?

[o-marshmallow]

@RuslanSpartak The fix has been merged to release/v4.3: a793553

[RuslanSpartak]

So you want to say that fix is ready. As I know problem is in calling i2c_ll_update that implement not in IRAM and for esp32 have just ; as implementation. So I have the same cache problem as before. So want to know if I take release v4.3.2 the fix is also present there?

[songruo]

@RuslanSpartak No, tag v4.3.2 does not have this fix. The fix is merged to branch release/v4.3. And it will be included in the next release of v4.3.3.

[RuslanSpartak]

I use platformio. Hope it will support v4.3.3

[raphael-bmec-co]

@o-marshmallow can you confirm that V4.4 will also be patched still? The Arduino core is now based on V4.4 not V4.3.

[songruo]

@raphael-bmec-co The fix has already been merged to release/v4.4: 0022954

[raphael-bmec-co]

@songruo thank you. I'm a little confused because this fix was merged 9 days ago but just 5 days ago @o-marshmallow said the patch was under internal review. Anyway seems to be the same code so I will give it a go.

[raphael-bmec-co]

@o-marshmallow crash is still occurring exactly as before. I will repeat the test code and stack trace here:

#include <WiFi.h>
#include <BLEDevice.h>
#include <BLEUtils.h>
#include <BLEServer.h>

// Set to 1 to accelerate the crash. Set to zero to see a typical use case.
#define ACCELERATED_CRASH 1

// BLE defines.
#define SERVICE_UUID        "4fafc201-1fb5-459e-8fcc-c5c9c331914b"
#define CHARACTERISTIC_UUID "beb5483e-36e1-4688-b7f5-ea07361b26a8"

void setup() {
  // Comment this line and the crash does not happen.
  heap_caps_malloc_extmem_enable(0);

  // Serial.
  Serial.begin(115200);
  Serial.println("Starting BLE...");

  // Minimum BLE simplified from example.
  BLEDevice::init("Long name works now");
  BLEServer *pServer = BLEDevice::createServer();
  BLEService *pService = pServer->createService(SERVICE_UUID);
  BLECharacteristic *pCharacteristic = pService->createCharacteristic(
      CHARACTERISTIC_UUID,
      BLECharacteristic::PROPERTY_READ |
          BLECharacteristic::PROPERTY_WRITE
  );

  pCharacteristic->setValue("Hello World says Neil");
  pService->start();
  BLEAdvertising *pAdvertising = BLEDevice::getAdvertising();
  pAdvertising->addServiceUUID(SERVICE_UUID);
  BLEDevice::startAdvertising();
  Serial.println("Started BLE");

  // Minimum WiFi.
  Serial.println("Starting WiFi connection attempt...");
  WiFi.begin("missing network", "password");

  Serial.print("Available heap:");
  Serial.println(ESP.getFreeHeap());
}

void loop() {

  // This causes a crash to happen faster and may be useful for debugging.
  // However the crash will happen even without this block.
  if(ACCELERATED_CRASH){
    while(!WiFi.isConnected()){
      log_i("Attempting connection...");
      WiFi.begin("missing network", "password");
      delay(100);
    }
  }

  delay(1000);
  Serial.println("Loop");
}

Guru Meditation Error: Core 0 panic'ed (Cache disabled but cached memory region accessed).

Core 0 register dump:
PC : 0x4008484e PS : 0x00060235 A0 : 0x80084920 A1 : 0x3ffbe98c
A2 : 0x3f80da7c A3 : 0xbad00bad A4 : 0x00060023 A5 : 0x40092978
A6 : 0x003fffff A7 : 0x3ffd378c A8 : 0x3ffbbd6c A9 : 0x3ffbea7c
A10 : 0x00000010 A11 : 0x3ffc04e0 A12 : 0x00000001 A13 : 0x00000002
A14 : 0x3ffc43e8 A15 : 0x3f800000 SAR : 0x00000010 EXCCAUSE: 0x00000007
EXCVADDR: 0x00000000 LBEG : 0x40090710 LEND : 0x40090726 LCOUNT : 0x00000000

Backtrace:0x4008484b:0x3ffbe98c |<-CORRUPTED
#0 0x4008484b:0x3ffbe98c in esp_timer_stop at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/e
sp-idf/components/esp_timer/src/esp_timer.c:192

Seems like there is something else going on?

[songruo]

@raphael-bmec-co I checked your code, and I think the problem is heap_caps_malloc_extmem_enable(0);. Not sure why you need to do this, but this line made all the heap allocation happens in external memory, including the esp_timer_handle_t which is malloced in esp_timer_create. So the reason why you had an error in esp_timer_stop is because you are trying to access external memory to get the esp_timer handler when cache is disabled.

My thought is your problem is unrelated to this I2C issue. If you have any further question regarding the heap_caps_malloc_extmem_enable, please open a new issue for discussion.

[raphael-bmec-co]

Hi @songruo thank you for looking into this.

The Arduino Core contributors also pointed out this likelihood and that there was a default value of 4096. Unfortunately, using 4096 still used up too much heap for me to use SSL and BLE. That said, I have been using a value of 64 and this seems stable so far.

Thanks for your assistance.

[RuslanSpartak]

will we have fix for sdk version 4.4.1i (2c_ll_update problem when it is in IRAM)?

[RuslanSpartak]

will we have fix for sdk version 4.4.1i (2c_ll_update problem when it is in IRAM)?

Are you going to add that fix in general? or only at some sdk versions?

[mahavirj]

Are you going to add that fix in general? or only at some sdk versions?

Fix is available in v4.4.2: 0022954

[mahavirj]

FYI: we have added fix for this in esp_timer too: c5decf2 along with test configuration for SPIRAM malloc threshold as 0. Closing this issue

[RuslanSpartak]

Are you going to add that fix in general? or only at some sdk versions?

Fix is available in v4.4.2: 0022954

I checked in platformio sdk v4.4.2 and bug is still there.

[RuslanSpartak]

Any updates on that?

[RuslanSpartak]

Hello everybody.
Can anybody add fix for I2C in IRAM for platformio sdk 5.3.0 - (esp-idf 4.4.3).
We need to add IRAM_ATTR to one of function that is called from ISR.

[Suresh6060]

I am getting the same issue on ESP32.
I use GpxMatrix.h to display info on HUB75 Led matrix and LittleFS to save data.

As long as matrix is not initiated , LittleFS functions nicely. But after display initiated and if LittleFS is used again, ESP32 raises the above error. Is there any way to re-enable cache or prevent GpxMatrix from disabling it or any other way , like CONFIG settings to overcome this issue ?