samd: RGBMatrix causing DMA hangs

[dhalbert]

This is a distillation of #6205; thanks to the commenters in that issue. The use of RGBMatrix is causing a DMA hang when used in conjunction with ESP32SPI. This line is looping forever:
https://github.com/adafruit/samd-peripherals/blob/d3b20192cf94fdea68a412596e082108ba5ebbf0/samd/dma.c#L207

        while ((dma_transfer_status(SHARED_TX_CHANNEL) & 0x3) == 0) {}

RGBMatrix uses ProtoMatter, which does significant work to send data to the RGB matrix in an ISR. If RGBMatrix is not used, the hang does not happen.

Test program:

import board
import busio
from digitalio import DigitalInOut
import displayio
import adafruit_requests as requests
from adafruit_esp32spi import adafruit_esp32spi
import adafruit_esp32spi.adafruit_esp32spi_socket as socket
from adafruit_display_text.bitmap_label import Label
from adafruit_matrixportal.matrix import Matrix
from adafruit_bitmap_font import bitmap_font

from secrets import secrets

esp32_cs = DigitalInOut(board.ESP_CS)
esp32_ready = DigitalInOut(board.ESP_BUSY)
esp32_reset = DigitalInOut(board.ESP_RESET)

esp = adafruit_esp32spi.ESP_SPIcontrol(board.SPI(), esp32_cs, esp32_ready, esp32_reset)

requests.set_socket(socket, esp)
esp.connect_AP(secrets["ssid"], secrets["password"])
print("Connected")

font = bitmap_font.load_font("lib/5x7.bdf")
led_board = Matrix().display
parent_group = displayio.Group()
textbox1 = Label(font=font, anchor_point=(0.0,0.0), anchored_position= (0,0))
parent_group.append(textbox1)
led_board.show(parent_group)

counter = 0
while True:
    r = requests.get("http://192.168.1.222:8000")
    textbox1.text = r.text
    counter += 1
    print(f'{counter} fetches')

Server that returns random-length results, for use with the test program above:

from flask import Flask, request
import logging
import random
import sys

app = Flask(__name__)

@app.get('/')
def get_response():
    return 'x' * random.randrange(750, 1500)

app.logger.setLevel(logging.INFO)
app.run(host="0.0.0.0", port=8000)

In general, within a few hundred fetches, the hang occurs, though the occurrence is otherwise random.

(Note that when RGBMatrix is not used, an ESP32SPI web fetch can still fail, due to an interrmittent communication error with the ESP32. However, that does not cause a hang. I will characterize and write up that bug in another issue.)

[dhalbert]

I've done a lot of testing to narrow this down, and also tried some potential fixes or workarounds, none of which have yet succeeded.

Display:
The problem does not seem to occur when using a parallel bus, I2C, or SPI display.

Priority:
By default, ProtoMatter runs at priority 0, and peripherals run at priority 7. I tried making ProtoMatter run at a lesser priority than peripherals (prio 3 vs 2). That did not help.

Bit depth:
The default color bit depth for ProtoMatter is 2. I tried 1 and 3; neither helped. 1 should take less time.

Overclocking:
I tried overclocking the SAMD51 from 120 MHz (default) to 200 MHz, to spend less time in the ISR. That did not solve the problem either.

no SPI DMA:
I disabled SPI use of DMA, and as one might expect, the problem goes away.

[tannewt]

I wonder if there is a DMA status register or memory protection register that can give a clue to what is happening.

Are you on USB while doing these tests? USB is also on that same memory bus.