Skip to content
This repository has been archived by the owner on Jul 4, 2024. It is now read-only.

Receive more than 480 channels #1

Open
luksal opened this issue Dec 9, 2019 · 28 comments
Open

Receive more than 480 channels #1

luksal opened this issue Dec 9, 2019 · 28 comments
Assignees
Labels
bug Something isn't working needs testing Something that needs to be tested

Comments

@luksal
Copy link
Owner

luksal commented Dec 9, 2019

Because of clearing the uart queue after detected break, the last channels are not received

@luksal luksal added the bug Something isn't working label Dec 9, 2019
@luksal luksal self-assigned this Dec 9, 2019
@macdroid53
Copy link

macdroid53 commented Dec 19, 2019

Any idea how to fix this?

I have a fully working, non-arduino, esp-idf, freertos version of my code running now and can confirm nothing above chan 480.

I can experiment...just not sure where to start. Shouldn't it suspend checking for the break until after it receives 512 bytes?

I monitored the event.size when in DMX_DATA mode and it is always 120. So it appears it gets a thinks it sees a break after reading 120x4, then flushes.
(some math: 88us break + (480 slots * 44us/slot) = 21.208ms) Assuming a max break to break time of 1.2s, it shouldn't be seeing a break for another 1.1second.

So what is it seeing as the early break...or am I not thinking right about what is going on?

@luksal
Copy link
Owner Author

luksal commented Dec 19, 2019

I currently have no spare time to work on this issues. But I know why this happens and how to fix this 😅🙈

@macdroid53
Copy link

Well, I've got the time...I'm retired. So I'm willing to help if you give some direction.

@luksal luksal added the needs testing Something that needs to be tested label Dec 20, 2019
@macdroid53
Copy link

I put your fix into my code. freertos just reboots every time it hits the new code.

So I checked the values of event.size and current_rx_addr at that point.
event.size = 120
current_rx_addr = 360

So, when it reads that data at that point the total is still 480.

I started reading the uart.c code. It appears the 120 bytes event.size reports (both in the 'case UART_DATA:' section and in the 'case UART_BREAK:' section in dmx.c) comes from the define in uart.c '#define UART_FULL_THRESH_DEFAULT (120)'.

Does that make sense?
Is something else is triggering the break detection?
Or is the full threshold being hit?

@luksal
Copy link
Owner Author

luksal commented Dec 21, 2019

Mhh this fix was just an idea and i had no time to test it...
At which line exactly did freertos reboot?

@luksal
Copy link
Owner Author

luksal commented Dec 21, 2019

So maybe I have to move the receive code to be full interrupt driven

@macdroid53
Copy link

I saw the "needs testing" flag, that's why I had at it. ;)

Hmm...interrupt maybe.BUt, looking at the low level routines in uart.c (and at the ESP32 tech manual) it looks like the driver is doing interrupts on the 120 bytes avail in the uart buffer.) But, seems weird that it's getting a data signal at every 120 bytes, then gets a break signal at 480 bytes (i.e. 360+ 120) and that means it still needs to find 32 more bytes to get 512 DMX slots.

Below is what I took from your update. If the line in bold (and it's associated code) is uncommented the ESP32 panic reboots repeatedly. With it commented out and the preceding print statements are uncommented, I can see the state event size, and count of bytes delivered so far. Note, I also added prints to see if ther was any other kind of uart error happening. These never print, so I don't think there are overruns, etc. happening.

                case UART_BREAK:
                    // break detected
                    // check if there are bytes left in the queue
                    //printf("dmx_state: %d\n", dmx_state);
                    //printf("event.size: %d\n", event.size);
                    //printf("current_rx_addr: %d\n", current_rx_addr);
                    **_/*if(dmx_state == DMX_DATA && event.size > 0 && current_rx_addr > 0)_**
                    {
                        uart_read_bytes(DMX_UART_NUM, dtmp, event.size, portMAX_DELAY);
                        xSemaphoreTake(sync_dmx, portMAX_DELAY);
                        // copy received bytes to dmx data array
                        for(int i = 0; i < event.size; i++)
                        {
                            dmx_data[current_rx_addr++] = dtmp[i];
                        }
                        xSemaphoreGive(sync_dmx);
                    }*/
                    // clear queue und flush received bytes                    
                    uart_flush_input(DMX_UART_NUM);
                    xQueueReset(dmx_rx_queue);
                    dmx_state = DMX_BREAK;
                    
		        break;                
                case UART_FRAME_ERR:
                    printf("frame\n");
                    break;
                case UART_PARITY_ERR:
                    printf("parity\n");
                    break;
                case UART_BUFFER_FULL:
                    printf("buff\n");
                    break;
                case UART_FIFO_OVF:
                    printf("fifo\n");
                    break;
                default:
                case UART_BREAK:
                    // break detected
                    // check if there are bytes left in the queue
                    //printf("dmx_state: %d\n", dmx_state);
                    //printf("event.size: %d\n", event.size);
                    //printf("current_rx_addr: %d\n", current_rx_addr);
                    /*if(dmx_state == DMX_DATA && event.size > 0 && current_rx_addr > 0)
                    {
                        uart_read_bytes(DMX_UART_NUM, dtmp, event.size, portMAX_DELAY);
                        xSemaphoreTake(sync_dmx, portMAX_DELAY);
                        // copy received bytes to dmx data array
                        for(int i = 0; i < event.size; i++)
                        {
                            dmx_data[current_rx_addr++] = dtmp[i];
                        }
                        xSemaphoreGive(sync_dmx);
                    }*/
                    // clear queue und flush received bytes                    
                    uart_flush_input(DMX_UART_NUM);
                    xQueueReset(dmx_rx_queue);
                    dmx_state = DMX_BREAK;
                    
		        break;                
                case UART_FRAME_ERR:
                    printf("frame\n");
                    break;
                case UART_PARITY_ERR:
                    printf("parity\n");
                    break;
                case UART_BUFFER_FULL:
                    printf("buff\n");
                    break;
                case UART_FIFO_OVF:
                    printf("fifo\n");
                    break;
                default:

@luksal
Copy link
Owner Author

luksal commented Dec 21, 2019

So what does the stack trace say?

@macdroid53
Copy link

Repeats this:

`Rebooting...
␛[0;32mI (11) boot: ESP-IDF 3.30300.190916 2nd stage bootloader␛[0m
␛[0;32mI (11) boot: compile time 14:53:46␛[0m
␛[0;32mI (11) boot: Enabling RNG early entropy source...␛[0m
␛[0;32mI (15) boot: SPI Speed : 40MHz␛[0m
␛[0;32mI (20) boot: SPI Mode : DIO␛[0m
␛[0;32mI (24) boot: SPI Flash Size : 4MB␛[0m
␛[0;32mI (28) boot: Partition Table:␛[0m
␛[0;32mI (31) boot: ## Label Usage Type ST Offset Length␛[0m
␛[0;32mI (39) boot: 0 nvs WiFi data 01 02 00009000 00006000␛[0m
␛[0;32mI (46) boot: 1 phy_init RF data 01 01 0000f000 00001000␛[0m
␛[0;32mI (53) boot: 2 factory factory app 00 00 00010000 00100000␛[0m
␛[0;32mI (61) boot: End of partition table␛[0m
␛[0;32mI (65) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x09358 ( 37720) map␛[0m
␛[0;32mI (87) esp_image: segment 1: paddr=0x00019380 vaddr=0x3ffbdb60 size=0x01ed4 ( 7892) load␛[0m
␛[0;32mI (90) esp_image: segment 2: paddr=0x0001b25c vaddr=0x40080000 size=0x00400 ( 1024) load␛[0m
␛[0;32mI (94) esp_image: segment 3: paddr=0x0001b664 vaddr=0x40080400 size=0x049ac ( 18860) load␛[0m
␛[0;32mI (110) esp_image: segment 4: paddr=0x00020018 vaddr=0x400d0018 size=0x16e48 ( 93768) map␛[0m
␛[0;32mI (143) esp_image: segment 5: paddr=0x00036e68 vaddr=0x40084dac size=0x03464 ( 13412) load␛[0m
␛[0;32mI (154) boot: Loaded app from partition at offset 0x10000␛[0m
␛[0;32mI (154) boot: Disabling RNG early entropy source...␛[0m
␛[0;32mI (155) cpu_start: Pro cpu up.␛[0m
␛[0;32mI (159) cpu_start: Application information:␛[0m
␛[0;32mI (163) cpu_start: Project name: Electrol_New_Brains-7␛[0m
␛[0;32mI (170) cpu_start: App version: 1.0.0␛[0m
␛[0;32mI (174) cpu_start: Compile time: Dec 18 2019 14:54:01␛[0m
␛[0;32mI (180) cpu_start: ELF file SHA256: 0000000000000000...␛[0m
␛[0;32mI (186) cpu_start: ESP-IDF: 3.30300.190916␛[0m
␛[0;32mI (192) cpu_start: Starting app cpu, entry point is 0x40081318␛[0m
␛[0;32mI (193) cpu_start: App cpu up.␛[0m
␛[0;32mI (203) heap_init: Initializing. RAM available for dynamic allocation:␛[0m
␛[0;32mI (209) heap_init: At 3FFAE6E0 len 0000F480 (61 KiB): DRAM␛[0m
␛[0;32mI (216) heap_init: At 3FFC0D38 len 0001F2C8 (124 KiB): DRAM␛[0m
␛[0;32mI (222) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM␛[0m
␛[0;32mI (228) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM␛[0m
␛[0;32mI (235) heap_init: At 40088210 len 00017DF0 (95 KiB): IRAM␛[0m
␛[0;32mI (241) cpu_start: Pro cpu start user code␛[0m
␛[0;32mI (259) cpu_start: Chip Revision: 1␛[0m
␛[0;32mI (260) cpu_start: Starting scheduler on PRO CPU.␛[0m
␛[0;32mI (0) cpu_start: Starting scheduler on APP CPU.␛[0m
DMX start address: 476
Setup MCPWM...
UART Init...
␛[0;32mI (1561) uart: queue free spaces: 20␛[0m
Start reading...
Start bumping...
DMX 476 : 255
DMX 477 : 255
DMX 478 : 255
DMX 479 : 255
DMX 476 : 0
DMX 477 : 0
DMX 478 : 74
DMX 479 : 69
DMX 478 : 0
DMX 479 : 0
DMX 476 : 255
DMX 477 : 255
DMX 478 : 255
DMX 479 : 255
Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC : 0x400838d8 PS : 0x00060930 A0 : 0x800d1028 A1 : 0x3ffb5540
A2 : 0x0a624965 A3 : 0x00000000 A4 : 0x00000000 A5 : 0x00000000
A6 : 0x00000258 A7 : 0x00000000 A8 : 0x80083c94 A9 : 0x3ffb5520
A10 : 0x3ffb0bec A11 : 0x00000000 A12 : 0x000002cf A13 : 0x3f406692
A14 : 0x00000001 A15 : 0x00000000 SAR : 0x00000000 EXCCAUSE: 0x0000001c
EXCVADDR: 0x0a6249a5 LBEG : 0x4000c2e0 LEND : 0x4000c2f6 LCOUNT : 0x00000000

ELF file SHA256: 0000000000000000000000000000000000000000000000000000000000000000

Backtrace: 0x400838d8:0x3ffb5540 0x400d1025:0x3ffb5580 0x400832b9:0x3ffb55b0

Rebooting...`

@luksal
Copy link
Owner Author

luksal commented Dec 21, 2019

Now you need to decode the Backtrace: https://github.com/me-no-dev/EspExceptionDecoder
(it works even when not using Arduino, because you can load your elf file into it)

@macdroid53
Copy link

macdroid53 commented Dec 22, 2019

After much digging in the bowels of the internet...I discovered the magic incantation to put in the platformio.ini file to get necessary output to the elf file for the decoder to do something.

That done, it provides:

PC: 0x400838e0: xQueueGenericSend at /home/mac/.platformio/packages/framework-espidf/components/freertos/queue.c line 720
EXCVADDR: 0x000a6289

Decoding stack results
0x400838e0: xQueueGenericSend at /home/mac/.platformio/packages/framework-espidf/components/freertos/queue.c line 720
0x400d1025: uart_event_task at src/dmx.c line 158
0x400832c1: vPortTaskWrapper at /home/mac/.platformio/packages/framework-espidf/components/freertos/port.c line 143

uart_event_task at src/dmx.c line 158 is:

                    {
                        uart_read_bytes(DMX_UART_NUM, dtmp, event.size, portMAX_DELAY);
                        xSemaphoreTake(sync_dmx, portMAX_DELAY);
                        // copy received bytes to dmx data array
                        for(int i = 0; i < event.size; i++)
                        {
                            dmx_data[current_rx_addr++] = dtmp[i];
                        }
158>>>>>        xSemaphoreGive(sync_dmx);
                    }

@luksal
Copy link
Owner Author

luksal commented Dec 22, 2019

What does your high water mark says?

@macdroid53
Copy link

High water mark? Where?

@macdroid53
Copy link

So, spent a lot of quality time with this today... ;)
I think this is what is going on. Just putting my thoughts here for posterity.

after the break detect, it reads 120 bytes for each UART_DATA event.
(i.e. the event is triggered when the fifo gets to 120 chars.)
because after 480, it only sees 32 bytes.
these bytes are never read, because the 120 threshold is never hit; event not triggered
so the, next break detect comes along
I think when the break is detected, the buffer is cleared (though I can't find where it does this in uart.c), but, event.size is never changed (it still says 120)
so, attempting to read 120 (or just 32) bytes fails.

@luksal
Copy link
Owner Author

luksal commented Dec 22, 2019

Maybe the uart_get_buffered_data_len and uart_read_bytes functions could give some results here?

@macdroid53
Copy link

macdroid53 commented Dec 22, 2019

uart_get_buffered_len is how I came to my conclusions above. But it returns 0 once the break event has triggered.

@luksal
Copy link
Owner Author

luksal commented Dec 22, 2019

I created an Issue at the ESP-IDF repository: espressif/esp-idf#4537

@macdroid53
Copy link

I posted in the esp forum trying to see if anyone had any ideas, but haven't seen any replies. So that is a good idea.

@benpeoples
Copy link

I found this when googling for solutions to BREAK detection (since the ESP UART seems designed specifically for using BREAK as an end-of-message marker).

In one of the threads related to this, someone was suggesting using BREAK to know when new data is in the buffer, which is a problem because it's acceptable for a DMX transmitter to only send one packet per second, which would create a 1 second lag. So you just need to figure out how to flush that buffer.

However, for BREAK detection, Sparkfun has a brilliant solution in their library: https://github.com/sparkfun/SparkFunDMX

They turn the pin into a GPIO, watch for it to go low for 88us (using interrupts), and then switch the port over to a UART to receive the data.

@luksal
Copy link
Owner Author

luksal commented Apr 10, 2020

@benpeoples unfortunately the detection of the break did not work properly and reliably. As a result, there were jumping channels which made it impossible to use. See also Issue 3 in the SparkFun repository sparkfun/SparkFunDMX#3

@benpeoples
Copy link

ARGH. That's frustrating. I'd hoped it would do it.

I did a deep dive yesterday into the low level workings of the ESP32 UART. I think it's possible to get BRK_DET to successfully tell you that you have a DMX BREAK (in time), but it (the UART) is really not well designed to do this and I'm concerned that it's going to be unreliable.

When I started my current project, it was pre-ESP32 and I was using an external SPI UART... I guess I'm heading back to that. =) Good luck!

@Volvox0815
Copy link

Volvox0815 commented Oct 9, 2020

You can try to reconfig the IRQ after uart_driver_install() to throw an event after every byte. Of course there are a lot of events thrown then (one after every byte), so you get every single byte one by one until next break occurs.
I think for only DMX reception this can be a workaround if the ESP is not too overloaded with other tasks.

uart_intr_config_t uart_intr;
uart_intr.intr_enable_mask = UART_RXFIFO_FULL_INT_ENA_M
						| UART_FRM_ERR_INT_ENA_M
						| UART_RXFIFO_OVF_INT_ENA_M
						| UART_BRK_DET_INT_ENA_M;
uart_intr.rxfifo_full_thresh = 1;
uart_intr.rx_timeout_thresh = 10;
uart_intr_config(your_uart_port, &uart_intr);

@macdroid53
Copy link

This may be the case for DMX use with LED strings. Not sure how well that solution will work when using it for PWM control of multiple channels of the 60/50 hertz duty cycle of a triac dimmer.

@Volvox0815
Copy link

I just tested it (not with Arduino, but this should not matter);
With simply copying the dmx input to a dmx output it worked with that solution. Perhaps you can give your PWM task a higher priority than the dmx if not using the hardware PWM?

Another solution (with the default interrupt at 120 bytes) I just tried is to modify the uart.c in the uart_rx_intr_handler_default(void *param).
You can add the reception of the remaining bytes where the break flag is handled.
If have an older Version of the uart.c, but the logic behind should be similar on newer versions of the IDF.

...
else if(uart_intr_status & UART_BRK_DET_INT_ST_M) {
            /**MOD BEGIN*/
            rx_fifo_len = uart_reg->status.rxfifo_cnt;
            for(buf_idx = 0; buf_idx < rx_fifo_len; buf_idx++) {
                p_uart->rx_data_buf[buf_idx] = uart_reg->fifo.rw_byte;
            }
            if(xRingbufferSendFromISR(p_uart->rx_ring_buf, p_uart->rx_data_buf, rx_fifo_len, &HPTaskAwoken) == pdTRUE) {
                p_uart->rx_buffered_len += rx_fifo_len;
            }
            /**MOD END*/
            uart_reg->int_clr.brk_det = 1;
            uart_event.type = UART_BREAK;
        } 
...

with this you can handle an additional receive call in the dmx.cpp "case UART_BREAK:" in the uart_event_task:
...
size_t buffered_size;
uint8_t dmx_data[513];
size_t current_rx_addr; //where we are currently in our DMX array

...
 case UART_BREAK:
    uart_get_buffered_data_len(DMX_UART_NUM, &buffered_size);
   if(buffered_size > 0){
       uint16_t maxread = 513-current_rx_addr;
       uint16_t readbytes = uart_read_bytes(your_uart, &(dmx_data[current_rx_addr]), buffered_size > maxread ? maxread : buffered_size, portMAX_DELAY);
  	current_rx_addr += readbytes;
  }
    uart_flush(DMX_UART_NUM);
    xQueueReset(dmx_rx_queue);
    current_rx_addr = 0; //set index to start
    dmx_state = DMX_BREAK;
break;
...

I hope I have modified every variable name correctly in the snippets to work with the given dmx.cpp, because I am using my own implementation, but basically with the same workflow.

@luksal
Copy link
Owner Author

luksal commented Oct 9, 2020

You can try to reconfig the IRQ after uart_driver_install() to throw an event after every byte. Of course there are a lot of events thrown then (one after every byte), so you get every single byte one by one until next break occurs.
I think for only DMX reception this can be a workaround if the ESP is not too overloaded with other tasks.

uart_intr_config_t uart_intr;
uart_intr.intr_enable_mask = UART_RXFIFO_FULL_INT_ENA_M
						| UART_FRM_ERR_INT_ENA_M
						| UART_RXFIFO_OVF_INT_ENA_M
						| UART_BRK_DET_INT_ENA_M;
uart_intr.rxfifo_full_thresh = 1;
uart_intr.rx_timeout_thresh = 10;
uart_intr_config(your_uart_port, &uart_intr);

Why not configure the UART to throw an event after 128 or even 512 bytes? then we should get all channels and have very little CPU overhead

@Volvox0815
Copy link

Yes, that was also my first thought, but some DMX Controllers only sent the slots needed and not the full 512 slot frame (I have a simple 24 fader controller that only sends out the first 24 slots).. In this case we do not recognize the whole sent frame if the controller sends something between the n*128 bytes.
But if we know the DMX sender that should be a suitable solution

@macdroid53
Copy link

If you know the sender that is fine. And, you restrict, your receiver to that the first X number of slots, fine, but that is rarely the case if what you are developing for use in professional venue.

The DMX specification is to send all 512 slots and the receiver is supposed to pick the ones that match its address setting.

Professional receivers for stage lighting and such can, typically, be addressed to any one or more address in the 512 range. So it needs to get all 512 frames, extract one or more address frames from anywhere in the 512 packet. It also, should be able to determine if the universe. So, the low level driver can't ignore part of the packet.

And, must respond to values changing in each of it's addressed frames so as not to get choppy fades or jumps from one level to another. Or worse lose a frame and see it as 0, setting the PWM so as it appears to flicker or pulse.

Some of this can, and should, be managed in higher level code, but the low level driver needs to reliably receive the complete frame.

@benpeoples
Copy link

benpeoples commented Oct 10, 2020

Just to chime in, it's also VERY important that you check the start code. There are many things other than the NSC now, so you need to be able to get the break, check the start code after the BREAK, and then start counting slots until you get to yours-- it's perfectly fine to have a state machine that just throws away packets one by one as they come in (that is, you don't have to buffer the full 512 slots).

And I agree that you don't want to ever rely on there being 512 slots. Standard does not specify a minimum number of slots, and I've actually done some lovely work by sending devices 12 slot universes at some ridiculous framerate.

Also, I've come across devices (specifically addressable DMX pixels) that are perfectly happy to have a start address above 512. While I don't think there's any requirement that you support this in the standard, not having your software fall over if it keeps getting bytes after the 512th slot would be good for reliability.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working needs testing Something that needs to be tested
Projects
None yet
Development

No branches or pull requests

4 participants