Performance issue with serial port

[jestebanes]

Hardware:

Board: Own design based on ESP-WROOM-32
Core Installation/update date: 25/aug/2017
IDE name: Arduino IDE
SDK version: v3.0-dev-270-g69646f6d
Flash Frequency: 80Mhz
Upload Speed: 512000

Description:

I'm working on a serial protocol with ACK that I used with esp8266 without troubles

The trouble is that esp32 serial ports with HardwareSerial are too slow for me.
In my design the esp32 works as master in a multimaster environment similar to RS485, I send a command to a slave and the slave send me an ACK, I wait for the response command from salve and I must send an ACK to slave in a window time of 500uS after the slave response, but in the measures I made with my entire system running I'm not able to send the ACK in less than 4 o 5 mS.

In the sketch I send, I'm working at 19200 bauds, I used a serial sniffer to assess the time. Before : is the elapsed time in milliseconds, in the first line is the command sent by the master and the slave ACK, in the second line is the response of the slave, and in the third line the master ACK and a new response from the slave.
The transmission of the 7 bytes last 3.6mS and the ACK is sent in 2.4mS, this is one of the better times I got because there isn't running the BLE or the Wifi stack

P300254927:021D58C292421A
P300254938:03C059C202CD5D
P300254944:CA03C059C202CD5D

Another issue, if I use the serial2.flush command the ESP32 send rubbish to the bus, no matter before or after the write command

Sketch:

#include "HardwareSerial.h"

unsigned long maxTimeout = 20; //20 mS timeOut
int speed;

HardwareSerial Serial2(2);

void setTimeOut(unsigned long n){
maxTimeout = n;
}

unsigned long getTimeOut(void){
return maxTimeout;
}

int serial_read(uint8_t *dataStream, int size){
int count = 0;
unsigned long timeOut = getTimeOut() + millis(); //Actual time

while((millis() < timeOut) && (count < size)){// Wait until data recived and no timeout
if(Serial2.available()){
dataStream[count]=Serial2.read();
count++;
}
yield();
}
return count;
}

void setSerial(int bus_speed, byte parity){ //TODO velocity changes hangs on
speed = bus_speed;
//Serial2.flush(); //wait to the send buffer
delay(2); //wait for the last char
if(parity){
Serial2.begin(bus_speed, SERIAL_8E1);
}else{
Serial2.begin(bus_speed, SERIAL_8N1);
}
yield();
}

int serial_read_validator(uint8_t *dataStream, int size, uint8_t *validator){
int count = 0;
unsigned long timeOut = getTimeOut() + millis(); //Actual time

while((millis() < timeOut) && (count < size)){// Wait until data recived and no timeout 200mS
if(Serial2.available()){
dataStream[count]=Serial2.read();
if(count<4){
if(dataStream[count] == validator[count]){
count++;
}else{
count = 0;
}
}else{
count++;
}
}
yield();
}
return count;
}

void setup() {
Serial2.begin(19200, SERIAL_8N1);
}

void loop() {
uint8_t comm1[]={0x02, 0x1D, 0x58, 0xC2, 0x92, 0x42};
uint8_t val[]={0x02, 0x1D, 0x58, 0xC2};
uint8_t resp[128];
uint8_t ack[]={0xCA};
int readed;

Serial2.write(comm1, 6);
//Serial2.flush();
readed = serial_read_validator(resp, 7, val);
if((readed == 7) && (resp[6]==0x1A)){
serial_read(resp, 7);
Serial2.write(ack, 1);
}
delay(100);
}

[jestebanes]

I tried, the low-level functions in uart.h with esp-idf and I've got the same result.
An interesting additional trouble is, with low level functions of uart.h compiling in arduino IDE write o read doesn't work.

the esp-idf code I try is a variant of esp-idf example to send a command using my protocol, the timing is worst than in the previous arduino version, is like the microcontroller always wait for the 20mS, even the data is already on the bus.

/* Uart Events Example

This example code is in the Public Domain (or CC0 licensed, at your option.)

Unless required by applicable law or agreed to in writing, this
software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied.
*/
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_system.h"
#include "nvs_flash.h"
#include "driver/uart.h"
#include "freertos/queue.h"
#include "esp_log.h"
#include "soc/uart_struct.h"

/**

• This is a example exaple which echos any data it receives on UART1 back to the sender, with hardware flow control
• turned on. It does not use UART driver event queue.
• • port: UART1
• • rx buffer: on
• • tx buffer: off
• • flow control: on
• • event queue: off
• • pin assignment: txd(io4), rxd(io5), rts(18), cts(19)
    */

#define ECHO_TEST_TXD (17)
#define ECHO_TEST_RXD (16)
#define ECHO_TEST_RTS (18)
#define ECHO_TEST_CTS (19)

#define BUF_SIZE (1024)

//an example of echo test with hardware flow control on UART1
static void echo_task()
{
int len;
const int uart_num = UART_NUM_2;
uart_config_t uart_config = {
.baud_rate = 19200,
.data_bits = UART_DATA_8_BITS,
.parity = UART_PARITY_DISABLE,
.stop_bits = UART_STOP_BITS_1,
.flow_ctrl = UART_HW_FLOWCTRL_DISABLE,
.rx_flow_ctrl_thresh = 122,
};
//Configure UART1 parameters
uart_param_config(uart_num, &uart_config);
//Set UART1 pins(TX: IO4, RX: I05, RTS: IO18, CTS: IO19)
uart_set_pin(uart_num, ECHO_TEST_TXD, ECHO_TEST_RXD, ECHO_TEST_RTS, ECHO_TEST_CTS);
//Install UART driver (we don't need an event queue here)
//In this example we don't even use a buffer for sending data.
uart_driver_install(uart_num, BUF_SIZE * 2, 0, 0, NULL, 0);

uint8_t data[]={0x02, 0x1D, 0x58, 0xC2, 0x92, 0x42};
uint8_t rec[20];
uint8_t ack[]={0xCA};
while(1) {
	uart_write_bytes(uart_num, (const char*) data, 6);
    //Read data from UART
    len = uart_read_bytes(uart_num, rec, 7, 20 / portTICK_RATE_MS);
	len = uart_read_bytes(uart_num, rec, 7, 20 / portTICK_RATE_MS);
	uart_write_bytes(uart_num, (const char*) ack, 1);
	vTaskDelay(50);
}

}

void app_main()
{
//A uart read/write example without event queue;
xTaskCreate(echo_task, "uart_echo_task", 1024, NULL, 10, NULL);
}

[jestebanes]

Still working on it
I found an improvement in serial performance
Before every writes preceded for a read, the ESP32 waits a minimum time indicated in UART_TX_IDLE_NUM, if this register is forced to zero the idle time is reduced from my initials 4 or 5 milliseconds to 500uS to 1.8mS, not enough for me but an improvement.

I suggest introducing in esp32-hal-uart.c at the end of function uartBegin the following instruction

uart->dev->idle_conf.tx_brk_num = 0;

[piaoyezju]

uart->dev->idle_conf.rx_idle_thrhd = 1;
uart->dev->idle_conf.tx_idle_num = 1;
I add this at the end of function uartBegin, it works fine

[jestebanes]

Hi!
There is one thing, you need to be careful changing the machine registers only into the protected section.
My final version, not perfect but works 90% of times (line 217 of esp32-hal-uart.c)

if ( uart->dev->conf0.stop_bit_num == TWO_STOP_BITS_CONF) {
    uart->dev->conf0.stop_bit_num = ONE_STOP_BITS_CONF;
    uart->dev->rs485_conf.dl1_en = 1;
}

uart->dev->idle_conf.tx_idle_num = 0;
uart->dev->idle_conf.tx_brk_num = 0;
uart->dev->idle_conf.rx_idle_thrhd = 0;
uart->dev->conf1.rxfifo_full_thrhd = 1;
uart->dev->conf1.rx_tout_thrhd = 1;
UART_MUTEX_UNLOCK();

[Pthuggin]

@jestebanes I am running into a problem that is exactly what you have been describing. I need to read a message header in over the UART Rx and then respond if it is the correct header in the next frame. The read byte / write byte operation should take a total of 1ms.

The best performance that I can achieve with the ESP-IDF is 6ms. Do you have any new thoughts on the matter?

[jestebanes]

Hi @Pthuggin, as I said before with the proposed change in the esp32-hal-uart.c begin function I get a response time less than one millisecond after the packet is completely received most of the times, but it is randomly, I need to check the response and restart the process when fail.
This is because the "real time" system under the Arduino core, but we don't have the source and we can not change the priority of the different process.
Another thing that improves the response time is disconnecting the WiFi while the serial communication is going on.
If I have an update in the future I will post here
Best regards

[Pthuggin]

@jestebanes I have solved this issue within the ESP-IDF framework. I can always respond in <250 us. If you are interested in the solution still let me know.

[Akskaypohankar]

@Pthuggin Please I desperately need your solution I need to receive nearly 2kb of data via serial and process it at the same time.

[Pthuggin]

The solution that works for me is not perfect, but it does the job right now. I modified the ESP-IDF functions uart_driver_install() and uart_rx_intr_handler_default() to process data as it is clocked in. This was done by setting the RX FIFO full threshold to one byte of data and enabling the full interrupt.

[Akskaypohankar]

i tried that too i changed RX fifo size to 512(maybe its too much ) but my device (sparkfun things ) started rebooting so i reversed to 128

[Pthuggin]

No, the RX FIFO size doesn't matter. That can stay at default. The overhead of the standard calls is too slow to process serial data quickly. Thus, you have to do it in the interrupt. The first step is to have the ESP32 generate an interrupt on each data byte. This is done by changing:

uart_intr_config_t uart_intr = {
.intr_enable_mask = UART_RXFIFO_FULL_INT_ENA_M
| UART_RXFIFO_TOUT_INT_ENA_M
| UART_FRM_ERR_INT_ENA_M
| UART_RXFIFO_OVF_INT_ENA_M
| UART_BRK_DET_INT_ENA_M
| UART_PARITY_ERR_INT_ENA_M
| UART_GLITCH_DET_INT_ENA_M,
.rxfifo_full_thresh = 1,
.rx_timeout_thresh = UART_TOUT_THRESH_DEFAULT,
.txfifo_empty_intr_thresh = UART_EMPTY_THRESH_DEFAULT
};

This struct within the driver install function must be updated to set .rxfifo_full_thresh = 1. This will cause the unit to generate an interrupt everytime a single byte of data is received. The overall RX FIFO size doesn't matter. The amount of data in the FIFO before an interrupt does matter.

[Akskaypohankar]

can you share your code to understand the whole process?

[Akskaypohankar]

http://www.dobitaobyte.com.br/tratar-interrupcoes-na-uart-com-esp32/
i found this a lot better (with some modifications ) I can now receive 4000 bytes via serial

[Adminius]

hey. i have the same error. i have more than 2ms between received bytes :/
any "official" solution?

[kleinmantara]

Will there be a solution without hacking into the SDK?

BTW: The more I work with the ESP32, the more I realize how much works fine on the ESP8266.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[stale]

[STALE_DEL] This stale issue has been automatically closed. Thank you for your contributions.