1M UART Baud rate issue (IDFGH-7366)

[naragururaj]

We are using ESP32 wrover for UART and BLE. Max baud rate that we can achieve with no data corruption is 115200 on the UART between 2 serial device. As per the spec I understand we can go upto 1M. Are there any known limitation?

[naragururaj]

Data from host processor
MSG 1 - 55 aa 11 00 fd 0c 00 30 28 20 20 30 29 20 12 a0 16 28 29 30 15 be ef
MSG 2 - 55 aa 11 00 fd 0c 00 30 28 20 20 30 29 20 12 a0 16 28 29 30 15 be ef

Data recvd on ESP
MSG 1 - 55 aa 11 00 fd 0c 00 30 28 20 20 30 29 20 be ef 55 aa 11 00 fd 0c 00

SDK version - 4.4
Issue - Missing bytes on UART 1 while running at 1Mb baud(marked in Bold Italic)

[barkov00]

@naragururaj Hi! I also use uart on 1 megabit and ran into the same problems.
Try to set uart_set_rx_full_threshold(..., 30) instead of 120 (default value). This will help reduce data loss, but will not fix the problem entirely.
Sometimes uart_read_bytes() missing all count of requested data. This may be due to the global disabling of interrupts in other parts of the framework. This is not due to a hardware or software circular buffer overflow, these flags are not set when such an error occurs.
Also, uart_flush_input() may produce garbage (doesn't clean completely) in input buffer, use uart_get_buffered_data_len() and then uart_read_bytes(...,..., bytes_available) instead of uart_flush_input()
ESP32 UART don't have DMA - this is the main reason for unstable operation at high bitrates.
This issue make in all version of frameworks.

And finally solution: use small MCU with DMA unit for UART communication at 1 MBit/s with reliable protocol with acknowledgment between ESP32 and MCU,

[naragururaj]

Hi @barkov00, Thanks for getting back to me with the response. If 1Mb has data loss, then can you please confirm 460,800 rate has no or minimal loss or better if you could suggest what is the highest optimal rate that we can use?

[barkov00]

@naragururaj i dont try this bitrate. can you make tests and report about results here?

[alisitsyn]

Hi @naragururaj, @barkov00,

Thank you for this report.
The 2M bit support in ESP-IDF UART diver is confirmed by loop back communication unit test.

I think firstly the issue should be confirmed on both sides using the same testing approach. I propose to use the uart_echo example on ESP32 and start the python script below to check communication. The echo example baud rate should be changed here.
uart_echo.zip

The script to check errors can be started on the host machine and can communicate with echo example using a simple USB to TTL adapter.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import time
import secrets
import serial
import logging
from typing import List


serial_port = '/dev/ttyUSB1'  # <<<<<<<<<<<<<<<<<<< Requires to change physical port
serial_baudrate = 115200 * 8   # <<<<<<<<<<< Change the speed (shall match the echo example baud rate)
blocksize = 1024 # max 1024 in uart_echo example
serial_timeout = 10
    
ser = serial.Serial(
    port = serial_port, 
    baudrate = serial_baudrate, 
    parity = serial.PARITY_NONE, 
    stopbits = serial.STOPBITS_ONE, 
    bytesize = serial.EIGHTBITS,
    xonxoff = False,
    rtscts = False,
    dsrdtr = False,
    timeout = serial_timeout
)

logging.basicConfig(filename='baud_test.log', level=logging.DEBUG, format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S:')

print('\t'.join([
    'block',
    'size',
    'baud',
    'bit per sec:',
    'kbps (KB/s)',
]))

seconds = []
for i in range(1, 100_000):
    data = secrets.token_bytes(blocksize)
    start_time = time.time()
    if ser.write(data) == len(data):
        read_data = ser.read(len(data))
        if read_data == data:
            seconds.append(time.time() - start_time)
            bps = 2 * 8 * len(data) * len(seconds) / sum(seconds)
            upto = bps / serial_baudrate
            kbps = bps / 8 / 1000
            if i % 10 == 0:
                logging.debug(f'Block {i}, is correct.') 
                print('\t'.join([
                    f'{i}',
                    f'{blocksize}',
                    f'{serial_baudrate}',
                    f'{bps:.2f}',
                    f'{kbps:.3} KB/s',
                ]))
            continue
        else:
            err_list = [j for j in range(0, min(len(data), len(read_data))) if read_data[j] != data[j]]
            errors = 0
            for idx in err_list:
                errors += 1
            print('Block %d, Errors: %d' % (i, errors)) 
            logging.debug('\t'.join([f'Block {i}, Error in index: ',
                " ".join('{:d}:({:02x},{:02x})'.format(idx, read_data[idx], data[idx]) for idx in err_list)])) 

The above script and posted modified echo example allow verification of received data. In case of failure, it prints all buffer indexes with expected and received data.

Please try this on your side to confirm the errors on different baud rates. Please send your baud_test.log file produced by the script.
Thank you.

[barkov00]

@alisitsyn , Hi! This test works nice. It seems, when using other modules (WiFi, TCP stack) with USART it sometimes cause data loss.
Can higher priority interrupts within the SDK block the execution of a USART interrupt for a while?

[alisitsyn]

@barkov00,

There are several aspects that may apply to this behavior:

1. If the UART ISR is placed into flash, its execution can be deferred mainly when your code uses flash access APIs which disable CPU cache. This may cause FIFO overflow and the loss of data kind of errors. In this case, placing the interrupt into IRAM is recommended using the CONFIG_UART_ISR_IN_IRAM kconfig value.
2. The active objects in your high-speed UART application should be configured appropriately to avoid task blocking by other higher priority tasks used by components in your application. When blocking happens the UART task may not be able to read the ring buffer on time which also can cause data loss. Most of the components do allow configuring the task priority and affinity options.
3. The UART driver is universal and complex enough and this may cause delays of processing in some cases. If you need the high-speed processing of your data it is recommended to use a UART-HAL and custom interrupt-based driver instead of a regular one and place the interrupt into IRAM. This option is supported by API. A simple example of using custom UART interrupt.

Can higher priority interrupts within the SDK block the execution of a USART interrupt for a while?

The other components with their interrupts and tasks may apply to this behavior but it is more like an application architecture/configuration issue. I propose focusing on these points and doing more investigation in this area.

ESP32 UART don't have DMA - this is the main reason for unstable operation at high bitrates.

The DMA support in the driver is just another approach for driver realization. This option has been investigated in UHCI drivers (esp32s3, esp32s2, and esp32c3) but cannot be used as a panacea for this kind of issues.

[naragururaj]

Hi @alisitsyn,

Sorry for the late reply.

• My application does not use any of the flash api's. It is just reading the data from another MCU over UART lines and passing it to the BLE stack for dispatching the data into the air.
• Application includes only 1 task whose job is to read data from the UART buffer and just do step 1. For handling concurrent issue there is a binary semaphore being used. May be it is the culprit. Will dig deeper to find the root cause.
• Thank you for the HAL demo. If step 2 doesn't yield positive results then will consider adopting the HAL implementation.

[alisitsyn]

Hi @naragururaj,

Your application algorithm looks ok to me. I think you need to check several scenarios to handle the data and using of HAL is required in high speed applications. We still at least need some test approach and test results to confirm if it is espressif software/hardware issue. I have some automated test results that show that UART can work on 1Mbit reliably. However my setup with just 40 * 512 byte frames on 1152000 might be very different with yours.

[Alvin1Zhang]

Thanks for reporting, sorry for the slow turnaround, fix is available at f352469, feel free to reopen.