Feature: fragmentation/reassembly #13
Conversation
Strip trailing spaces. Replace tabs with spaces.
The frame consists of a header, the payload size and a message payload. This structure is only a helper struct for lib internals, to be able to use dynamic payload sizes. Over the air only the header and the payload (without the payload size) are transmitted.
Define MAX_FRAME_SIZE as a makro with value of '32' (Bytes).
Changes in code:
- internally use the introuduced RF24NetworkFrame type to describe and manage header and payload.
- keep the write and read API for backward compatibility.
- make needed changes in some functions to be able to use the RF24NetworkFrame and circular buffer.
- extend the circular buffer class with some functions
- front()
- isEmpty()
- isFull()
- update copyrith
TODO:
- check if the code is supported in arduino
- check for memory lecks
- check for memory space usage (arduino)
- implement automatic fragmentation and reassembly of large payloads
- two independent tx and rx (circular buffer) FIFOs
- clean up code
- remove not need parts
FIXME: problems with reassembly data structure
Fragmentation is transparent to caller application.
TODO:
- multicast fragmentation
- test with multiple nodes in a larger network
- use a circular queue for the frame_queue data structure
- use a LRUCache data structure for the frame reassembly
- lower memory consumption
- arduino porting
- cleanup code
- remove redundant and innecessary code
- Based on IPv4 fragmentation https://tools.ietf.org/html/rfc791.
- Tested only on Raspberry Pi, may not work in Arduinos due to memory usage
Tested only with two raspberris using NRF24L01+ 10 meters appart.
- The lib is capable to detect if the message payload is bigger than frame_size-sizeof(Header)
and split the payload into chuncks and send them separately.
- This commit introduces two new flag types:
- NETWORK_MORE_FRAGMENTS
- NETWORK_LAST_FRAGMENT
- As of IPv4 fragmention this flags are used to mark the frames to notify the receiver if more
fragments will come or this is the last fragment.
- The previous _write() function to transmit the frames is used.
- Changes in payload size management were made to allow dynamic payload sizes without the need
to define a field in the header.
- Introduce a new struct type "RF24NetworkFrame" for better frame handling inside the lib.
This type encapsulates the header, the current size of the payload and a (fixed size) array
containing the payload.
- The frame_buffer data structure was changes in favor of a std::queue for better handling.
- Make some changes renaming variables and defining the values.
- Use a std::map with double key (header.to_from,header.id) to identify the incoming fragments,
allowing the reassembly of multiple transmission simultaneously.
- The reassembly of frames hapens in the enqueue(frame) function.
Based on the fragment_id and header.id the fragments are handled diffently and in the end
the completely assembled frames are queued in the frame_queue presenting them to the calling
application.
- Introduce a new debug flag to print fragmentation debug messages.
- Use definitions for static values like frame size or payload max size.
- Strip trailing spaces and fix some indentation issues.
…e payloads.
Fragmentation is transparent to calling application.
TODO:
- multicast fragmentation
- test with multiple nodes in a larger network
- use a circular queue for the frame_queue data structure
- use a LRUCache data structure for the frame reassembly
- lower memory consumption
- arduino porting
- cleanup code
- remove redundant and unnecessary code
- Based on IPv4 fragmentation https://tools.ietf.org/html/rfc791.
- Tested only on Raspberry Pi, may not work in Arduinos due to memory usage
Tested only with two raspberries using NRF24L01+ 10 meters apart.
- The lib is capable to detect if the message payload is bigger than frame_size-sizeof(Header)
and split the payload into chucks and send them separately.
- This commit introduces two new flag types:
- NETWORK_MORE_FRAGMENTS
- NETWORK_LAST_FRAGMENT
- As of IPv4 fragmentation this flags are used to mark the frames to notify the receiver if more
fragments will come or this is the last fragment.
- The previous _write() function to transmit the frames is used.
- Changes in payload size management were made to allow dynamic payload sizes without the need
to define a field in the header.
- Introduce a new struct type "RF24NetworkFrame" for better frame handling inside the lib.
This type encapsulates the header, the current size of the payload and a (fixed size) array
containing the payload.
- The frame_buffer data structure was changes in favor of a std::queue for better handling.
- Make some changes renaming variables and defining the values.
- Use a std::map with double key (header.to_from,header.id) to identify the incoming fragments,
allowing the reassembly of multiple transmission simultaneously.
- The reassembly of frames happens in the enqueue(frame) function.
Based on the fragment_id and header.id the fragments are handled differently and in the end
the completely assembled frames are queued in the frame_queue presenting them to the calling
application.
- Introduce a new debug flag to print fragmentation debug messages.
- Use definitions for static values like frame size or payload max size.
- Strip trailing spaces and fix some indentation issues.
…F24Network into feature/fragmentation_v2
|
I wrote an application to be able to send IP packets across the RF24Network using a TUN/TAP interface. |
|
This RF24toTUN library is really very cool. I just spent quite a while messing with it, and I have it carrying an SSH session over RF24 radios, with an Arduino node routing the traffic between the nodes too. With the debugging disabled, it seems reasonably quick as well. A few minor things I noticed or modified: Just want to mess around with it a bit more and do a bit more testing before merging the pull request, but it seems like it is working for most or all scenarios so far. |
Merge Pull request from https://github.com/reixd Feature(s): Extended support for very long payloads with fragmentation/reassembly
Fixes for #13 - modify txTimeout value - detect 0 payload length due to corrupt dynamic payload - detect and clear un-finished fragmented payloads - detect and clear frames that exceed MAX_PAYLOAD_SIZE (due to duplicate payloads, etc) - add auto-adjusted delay after writes to account for timing issues - adjust includes in examples to prevent compile errors
|
FYI, I just added some fixes that improve performance and stability, and address some related memory issues etc. that were causing some weird behaviour and the odd crash. Now playing very nicely with the RF24toTUN library also.
*Edit to add: Some of these are just workarounds, I believe some sort of proper error checking (CRC, etc) is neccessary for full featured fragmentation handling, to prevent corrupt payloads. |
|
Great! :D
|
|
What do you think: it would be possible to use the RF24Nework lib with fragmentation in an arduino UNO with an Ethernet shield? That would be great ;) |
|
Hehe, I think it would be possible, but your transfer speeds would take a bit of a nosedive using the SD card as a memory buffer. This adds a whole bunch of additional possibilities, I'm just having fun with the concept of a really long-range wireless IP link for something like $5-7 in components, even if it will only handle 5-6KB/S using one hop/routing node. |
|
I have a mayor goal with this stuff: use the cheap NRFs and Raspi/Arduino to build a decentralized from the Internet not dependent (IPv4,v6 capable) network. Maybe with solar panels and pringles wifi antennas (aka "cantenna"). All of this on the rooftop of buildings. Use cases:
|
|
Don't even get me started! Keep up the great work, I look forward to seeing the RF24toTUN library being developed further, and hope you realize your goals and then some! |
Fully functional fragmentation and reassembly implementation for large payloads.
Fragmentation is transparent to calling application.
TODO:
- multicast fragmentation
- test with multiple nodes in a larger network
- use a circular queue for the frame_queue data structure
- use a LRUCache data structure for the frame reassembly
- lower memory consumption
- arduino porting
- cleanup code
- remove redundant and unnecessary code
Tested only with two raspberries using NRF24L01+ 10 meters apart.
and split the payload into chucks and send them separately.
- NETWORK_MORE_FRAGMENTS
- NETWORK_LAST_FRAGMENT
fragments will come or this is the last fragment.
to define a field in the header.
This type encapsulates the header, the current size of the payload and a (fixed size) array
containing the payload.
allowing the reassembly of multiple transmission simultaneously.
Based on the fragment_id and header.id the fragments are handled differently and in the end
the completely assembled frames are queued in the frame_queue presenting them to the calling
application.