11 Zinx Decoder

1. LengthField Frame Decoding

Zinx provides encoding and decoding capabilities for binary data frames during the data transmission process. With Zinx's TCP stream transmission, developers don't need to worry about packet fragmentation and packet reassembly issues because Zinx handles them internally.

Zinx's binary encoding and decoding algorithm adopts the LengthField algorithm. Developers need to understand the basic parameters of LengthField and can configure the parameters based on their own data frame format in their specific business scenario. The following are the parameter explanations for LengthField:

The code for Zinx's LengthField frame decoding algorithm can be found in this link. The FrameDecoder is a decoder that dynamically splits received binary data frames based on the value of the length field in the message. It is particularly useful when you decode binary messages with an integer header field representing the message body or the entire message length. ziface.LengthField has many configurable parameters, allowing it to decode messages with length fields in any format, which is commonly seen in proprietary client-server protocols.

LengthField is a general-purpose message encoding and decoding protocol that encodes the length of a message as part of the message header during the encoding process. During the decoding process, it first reads the message header, extracts the message length from it, and then reads the subsequent message content based on the length. In the LengthField protocol, four key parameters need to be configured:

• LengthFieldOffset: Represents the offset of the length field in the message header, i.e., the number of bytes between the length field and the start of the message header. Typically, the length field is placed before the message header, so this value is usually a non-negative integer.

• LengthFieldLength: Represents the number of bytes occupied by the length field. Usually, this value is a positive integer less than or equal to 8, as real-world messages typically don't exceed 8 bytes.

• LengthAdjustment: Represents the adjustment needed for the message length. After reading the length field, if any adjustment needs to be made to the length, it can be done through this parameter. For example, if the message length includes the length of the message header, then LengthAdjustment should be set to a negative value so that the actual length of the message can be correctly calculated when reading the message body.

• InitialBytesToStrip: Represents the number of bytes to skip during decoding, i.e., the number of bytes to skip the message header. Typically, this value equals the length of the message header because during decoding, it's necessary to first read the message header and parse the length field, and then skip the message header to read the message body.

• MaxFrameLength: Specifies the maximum frame length that the decoder can handle. When using a frame decoder such as LengthFieldBasedFrameDecoder, if the received frame exceeds MaxFrameLength, the decoder will throw an exception and close the connection. The purpose of this parameter is to limit the processing capability of the decoder to avoid risks such as memory overflow or denial of service attacks. This parameter should be set appropriately based on the specific application's protocol and data transmission method to ensure that the decoder can handle all valid frames without being affected by malicious or erroneous data. It's important to note that the value of MaxFrameLength should be determined according to the specific application's protocol and data transmission method. For example, if large files are being transmitted, a larger value for MaxFrameLength is needed to allow larger frames. On the other hand, if small text messages are being transmitted, a smaller value for MaxFrameLength can be set to improve security and reliability.

1.1 LengthField Configuration Example 1

"Using a 2-byte length field at offset 0, without stripping the message header."

In this example, the value of the length field is 12 (0x0C), which represents the length of the message "HELLO, WORLD". By default, the decoder assumes that the length field represents the number of bytes following the length field. Therefore, a simple parameter configuration can be used for decoding.

LengthFieldOffset = 0
LengthFieldLength = 2
LengthAdjustment = 0
InitialBytesToStrip = 0 (not stripping the message header)

The format of the message before and after decoding is as follows:

Before Decoding (14 bytes)         After Decoding (14 bytes)
+--------+----------------+      +--------+----------------+
| Length | Actual Content |----->| Length | Actual Content |
| 0x000C | "HELLO, WORLD" |      | 0x000C | "HELLO, WORLD" |
+--------+----------------+      +--------+----------------+

1.2 LengthField Configuration Example 2

"Using a 2-byte length field at offset 0, and stripping the message header."

You may want to strip the length field by specifying the "InitialBytesToStrip" parameter. In this example, we specify "2," which is the same as the length of the length field, to remove the first two bytes.

LengthFieldOffset = 0
LengthFieldLength = 2
LengthAdjustment = 0
InitialBytesToStrip = 2 (equal to the length of the Length field)

The format of the message before and after decoding is as follows:

Before Decoding (14 bytes)         After Decoding (12 bytes)
+--------+----------------+      +----------------+
| Length | Actual Content |----->| Actual Content |
| 0x000C | "HELLO, WORLD" |      | "HELLO, WORLD" |
+--------+----------------+      +----------------+

1.3 LengthField Configuration Example 3

"Using a 2-byte length field at offset 0, without stripping the message header, where the length field represents the length of the entire message."

In most cases, the length field only represents the length of the message body, as shown in the previous examples. However, in some protocols, the length field represents the length of the entire message, including the message header. In such cases, we need to specify a non-zero LengthAdjustment. Since in this example the length value in the message is always 2 bytes longer than the length of the message body, we set LengthAdjustment to -2 to compensate.

LengthFieldOffset = 0
LengthFieldLength = 2
LengthAdjustment = -2 (length of the Length field)
InitialBytesToStrip = 0

The format of the message before and after decoding is as follows:

Before Decoding (14 bytes)         After Decoding (14 bytes)
+--------+----------------+      +--------+----------------+
| Length | Actual Content |----->| Length | Actual Content |
| 0x000E | "HELLO, WORLD" |      | 0x000E | "HELLO, WORLD" |
+--------+----------------+      +--------+----------------+

1.4 LengthField Configuration Example 4

"5-byte header containing a 3-byte length field, without stripping the header."

This message is a simple variation of the first example, with an additional header value added before the message. The LengthAdjustment is once again set to 0 because the decoder always considers the length of the pre-defined data when calculating the frame length.

LengthFieldOffset = 2 (equal to the length of Header 1)
LengthFieldLength = 3
LengthAdjustment = 0
InitialBytesToStrip = 0

The format of the message before and after decoding is as follows:

Before Decoding (17 bytes)         After Decoding (17 bytes)
+----------+----------+----------------+      +----------+----------+----------------+
| Header 1 | Length   | Actual Content |----->| Header 1 | Length   | Actual Content |
| 0xCAFE   | 0x00000C | "HELLO, WORLD" |      | 0xCAFE   | 0x00000C | "HELLO, WORLD" |
+----------+----------+----------------+      +----------+----------+----------------+

1.5 LengthField Configuration Example 5

"5-byte header containing a 3-byte length field, without stripping the header."

This is an advanced example that demonstrates the scenario where there is additional header information between the length field and the message body. You need to specify a positive LengthAdjustment to allow the decoder to account for the extra header in the frame length calculation.

LengthFieldOffset = 0
LengthFieldLength = 3
LengthAdjustment = 2 (equal to the length of Header 1)
InitialBytesToStrip = 0

The format of the message before and after decoding is as follows:

Before Decoding (17 bytes)         After Decoding (17 bytes)
+----------+----------+----------------+      +----------+----------+----------------+
| Length   | Header 1 | Actual Content |----->| Length   | Header 1 | Actual Content |
| 0x00000C | 0xCAFE   | "HELLO, WORLD" |      | 0x00000C | 0xCAFE   | "HELLO, WORLD" |
+----------+----------+----------------+      +----------+----------+----------------+

1.6 LengthField Configuration Example 6

"4-byte header with a 2-byte length field at offset 1, stripping the first header field and length field."

This example combines all the previous examples. There is a pre-defined header before the length field and an additional header after the length field. The pre-defined header affects the LengthFieldOffset, while the additional header affects the LengthAdjustment. We also specify a non-zero InitialBytesToStrip to strip the length field and pre-defined header from the frame. If you don't want to strip the pre-defined header, you can specify an InitialBytesToSkip of 0.

LengthFieldOffset = 1 (length of HDR1)
LengthFieldLength = 2
LengthAdjustment = 1 (length of HDR2)
InitialBytesToStrip = 3 (length of HDR1 + length field)

The format of the message before and after decoding is as follows:

Before Decoding (16 bytes)                    After Decoding (13 bytes)
+------+--------+------+----------------+    +------+----------------+
| HDR1 | Length | HDR2 | Actual Content |--> | HDR2 | Actual Content |
| 0xCA | 0x000C | 0xFE | "HELLO, WORLD" |    | 0xFE | "HELLO, WORLD" |
+------+--------+------+----------------+    +------+----------------+

1.7 LengthField Configuration Example 7

"2-byte length field at offset 1 in a 4-byte header, stripping the first header field and length field, where the length field represents the entire message length."

Let's make some modifications to the previous example. The only difference from the previous example is that the length field represents the entire message length, similar to the third example. We need to account for the lengths of HDR1 and Length in the LengthAdjustment. Note that we don't need to consider the length of HDR2 because the length field already includes the length of the entire header.

LengthFieldOffset = 1
LengthFieldLength = 2
LengthAdjustment = -3 (length of HDR1 + length field, negative value)
InitialBytesToStrip = 3

The format of the message before and after decoding is as follows:

Before Decoding (16 bytes)                    After Decoding (13 bytes)
+------+--------+------+----------------+    +------+----------------+
| HDR1 | Length | HDR2 | Actual Content |--> | HDR2 | Actual Content |
| 0xCA | 0x0010 | 0xFE | "HELLO, WORLD" |    | 0xFE | "HELLO, WORLD" |
+------+--------+------+----------------+    +------+----------------+

2. Zinx Integrated Decoders

2.1 IDecoder

Zinx provides the IDecoder decoding interface:

type IDecoder interface {
	IInterceptor
	GetLengthField() *LengthField
}

The IDecoder interface extends the IInterceptor interface. Therefore, to implement a Decoder instance, the following methods need to be implemented:

GetLengthField() *LengthField  // Allows Zinx to obtain the LengthField parameters of the current Decoder
Intercept(IChain) IcResp       // Decoding action of the Decoder in the interceptor

2.2 TLV Decoder (Big Endian Byte Order)

Zinx currently uses the TLV (Tag-Length-Value) protocol as the default message protocol, and it uses the big endian byte alignment.

Note: The TLV Decoder (Big Endian Byte Order) is the default frame parsing method in Zinx.

2.2.1 Message Format

+---------------+---------------+---------------+
|     Tag       |    Length     |     Value     |
| uint32(4byte) | uint32(4byte) |   n byte      |
+---------------+---------------+---------------+

Tag:    uint32 type, 4 bytes in size
Length: uint32 type, 4 bytes in size, represents the length of the Value
Value:  n bytes in size

2.2.2 Configuration Parameters for LengthField

lengthFieldOffset   = 4            (Length is located at byte index 4) Offset of the length field
lengthFieldLength   = 4            (Length is 4 bytes in size) Number of bytes occupied by the length field
lengthAdjustment    = 0            (Length only represents the length of the Value, the program reads only the specified length of bytes and stops. If there are additional 2 bytes for CRC after Value, then it would be +2. If Length indicates the total length of Tag + Length + Value, then it would be -8.)
initialBytesToStrip = 0            (This 0 indicates that the complete protocol content, including Tag + Length + Value, is returned. If you only want to return the Value content by removing the 4 bytes of Tag and 4 bytes of Length, then it would be 8.) Number of bytes to be stripped from the decoded frame initially
maxFrameLength      = 2^32 + 4 + 4 (Since Length is of type uint32, 2^32 represents the maximum length of Value. Additionally, Tag and Length each occupy 4 bytes.)

2.2.3 TLV-Decoder Implementation Code

Interceptor implementation:

const TLV_HEADER_SIZE = 8 // Represents the length of an empty TLV packet

type TLVDecoder struct {
	Tag    uint32 // Message type
	Length uint32 // Message length
	Value  []byte // Message content
}

func NewTLVDecoder() ziface.IDecoder {
	return &TLVDecoder{}
}

func (tlv *TLVDecoder) GetLengthField() *ziface.LengthField {

	return &ziface.LengthField{
		MaxFrameLength:      math.MaxUint32 + 4 + 4,
		LengthFieldOffset:   4,
		LengthFieldLength:   4,
		LengthAdjustment:    0,
		InitialBytesToStrip: 0,
	}
}

func (tlv *TLVDecoder) decode(data []byte) *TLVDecoder {
	tlvData := TLVDecoder{}
	// Get T
	tlvData.Tag = binary.BigEndian.Uint32(data[0:4])
	// Get L
	tlvData.Length = binary.BigEndian.Uint32(data[4:8])
	// Determine the length of V
	tlvData.Value = make([]byte, tlvData.Length)

	// Get V
	binary.Read(bytes.NewBuffer(data[8:8+tlvData.Length]), binary.BigEndian, tlvData.Value)

	return &tlvData
}

func (tlv *TLVDecoder) Intercept(chain ziface.IChain) ziface.IcResp {

	// 1. Get the IMessage from Zinx
	iMessage := chain.GetIMessage()
	if iMessage == nil {
		// Proceed to the next layer in the chain
		return chain.ProceedWithIMessage(iMessage, nil)
	}

	// 2. Get the data
	data := iMessage.GetData()

	// 3. If the data is less than the header size, proceed to the next layer
	if len(data) < TLV_HEADER_SIZE {
		return chain.ProceedWithIMessage(iMessage, nil)
	}

	// 4. Decode TLV
	tlvData := tlv.decode(data)

	// 5. Set the decoded data back to the IMessage, Zinx's Router requires MsgID for addressing
	iMessage.SetMsgID(tlvData.Tag)
	iMessage.SetData(tlvData.Value)
	iMessage.SetDataLen(tlvData.Length)

	// 6. Proceed to the next layer with the decoded data
	return chain.ProceedWithIMessage(iMessage, *tlvData)
}

2.3 HTLVCRC-Decoder (Big-Endian Byte Order)

2.3.1 Message Format

+--------+--------+---------+--------+--------+
| Header |  Code  | DataLen |  Data  |  CRC   |
| 1 byte | 1 byte | 1 byte  | N bytes| 2 bytes|
+--------+--------+---------+--------+--------+

Sample Data:
Header   Code    DataLen     Body                      CRC
A2       10       0E       0102030405060708091011121314 050B

Explanation:
  DataLen (len) is 14 (0E), which represents the length of the Body only.

2.3.2 LengthField Configuration Parameters

lengthFieldOffset   = 2      (The index of len is 2, counting from 0) Offset of the length field
lengthFieldLength   = 1      (len is 1 byte) Length of the length field in bytes
lengthAdjustment    = 2      (len only represents the length of the Body, the program will only read len bytes and end, but there are still 2 bytes for CRC, so it is +2)
initialBytesToStrip = 0      (This 0 represents the complete protocol content. If you don't want A2, then it would be 1) Number of bytes to strip from the decoded frame
maxFrameLength      = 255 + 4 (Start Code, Code, CRC) (len is 1 byte, so the maximum length is the maximum value of an unsigned 1-byte integer)

2.3 HTLVCRC-Decoder (Big-Endian Byte Order)

2.3.3 Implementation Code

const HEADER_SIZE = 5

type HtlvCrcDecoder struct {
	Head    byte   // Header
	Funcode byte   // Code
	Length  byte   // Data Length
	Body    []byte // Data
	Crc     []byte // CRC Checksum
	Data    []byte // Original Data
}

func NewHTLVCRCDecoder() ziface.IDecoder {
	return &HtlvCrcDecoder{}
}

func (hcd *HtlvCrcDecoder) GetLengthField() *ziface.LengthField {
	return &ziface.LengthField{
		MaxFrameLength:      math.MaxInt8 + 4,
		LengthFieldOffset:   2,
		LengthFieldLength:   1,
		LengthAdjustment:    2,
		InitialBytesToStrip: 0,
	}
}

func (hcd *HtlvCrcDecoder) decode(data []byte) *HtlvCrcDecoder {
	datasize := len(data)

	htlvData := HtlvCrcDecoder{
		Data: data,
	}

	// Parse Header
	htlvData.Head = data[0]
	htlvData.Funcode = data[1]
	htlvData.Length = data[2]
	htlvData.Body = data[3 : datasize-2]
	htlvData.Crc = data[datasize-2 : datasize]

	// CRC Check
	if !CheckCRC(data[:datasize-2], htlvData.Crc) {
		zlog.Ins().DebugF("CRC check failed %s %s\n", hex.EncodeToString(data), hex.EncodeToString(htlvData.Crc))
		return nil
	}

	return &htlvData
}

func (hcd *HtlvCrcDecoder) Intercept(chain ziface.IChain) ziface.IcResp {
	// 1. Get Zinx IMessage
	iMessage := chain.GetIMessage()
	if iMessage == nil {
		// Proceed to the next layer in the chain
		return chain.ProceedWithIMessage(iMessage, nil)
	}

	// 2. Get Data
	data := iMessage.GetData()

	// 3. If the read data is less than the header size, proceed to the next layer
	if len(data) < HEADER_SIZE {
		return chain.ProceedWithIMessage(iMessage, nil)
	}

	// 4. HTLV+CRC decoding
	htlvData := hcd.decode(data)

	// 5. Set the decoded data back to IMessage. Zinx's Router needs MsgID for addressing.
	iMessage.SetMsgID(uint32(htlvData.Funcode))

	// 6. Proceed with the decoded data to the next layer
	return chain.ProceedWithIMessage(iMessage, *htlvData)
}

2.4 LTV-Decoder (Little-Endian Byte Order)

2.4.1 Message Format

+----------------+----------------+---------------+
|    Length      |     Tag        |     Value     |
| uint32(4 bytes)| uint32(4 bytes)|    n bytes    |
+----------------+----------------+---------------+
Length: uint32, 4 bytes, indicating the length of the Value
Tag: uint32, 4 bytes
Value: n bytes

2.4.2 LengthField Configuration Parameters

lengthFieldOffset   = 0            (Length field starts at index 0) Offset of the length field
lengthFieldLength   = 4            (Length field is 4 bytes) Number of bytes occupied by the length field
lengthAdjustment    = 4            (Length field only represents the length of the Value. Since the Value follows the Tag, we need to offset by 4 bytes to reach the start address of the Value. So, the value is set to +4. If the Length field represents the total length of Tag+Length+Value, then the value would be -4.)
initialBytesToStrip = 0            (0 indicates returning the complete protocol content Tag+Length+Value. If you only want to return the Value content, excluding the 4 bytes of the Tag and 4 bytes of the Length, then this value would be 8. Number of bytes to strip from the decoded frame initially.
maxFrameLength      = 2^32 + 4 + 4 (Since the Length field is uint32, 2^32 represents the maximum length of the Value. Additionally, Tag and Length each occupy 4 bytes.) Maximum length of the frame

2.4 LTV-Decoder Implementation Code

const LTV_HEADER_SIZE = 8 // Size of the LTV header

type LTV_Little_Decoder struct {
	Length uint32 // Length of the message
	Tag    uint32 // Type of the message
	Value  []byte // Content of the message
}

func NewLTV_Little_Decoder() ziface.IDecoder {
	return &LTV_Little_Decoder{}
}

func (ltv *LTV_Little_Decoder) GetLengthField() *ziface.LengthField {
    
	// Use the default TLV framing method
	return &ziface.LengthField{
		MaxFrameLength:      math.MaxUint32 + 4 + 4,
		LengthFieldOffset:   0,
		LengthFieldLength:   4,
		LengthAdjustment:    4,
		InitialBytesToStrip: 0,
	}
}

func (ltv *LTV_Little_Decoder) decode(data []byte) *LTV_Little_Decoder {
	ltvData := LTV_Little_Decoder{}

	// Get the length (L)
	ltvData.Length = binary.LittleEndian.Uint32(data[0:4])
	// Get the tag (T)
	ltvData.Tag = binary.LittleEndian.Uint32(data[4:8])
	// Determine the length of the value (V)
	ltvData.Value = make([]byte, ltvData.Length)

	// Get the value (V)
	binary.Read(bytes.NewBuffer(data[8:8+ltvData.Length]), binary.LittleEndian, ltvData.Value)

	return &ltvData
}

func (ltv *LTV_Little_Decoder) Intercept(chain ziface.IChain) ziface.IcResp {
	// 1. Get the IMessage from Zinx
	iMessage := chain.GetIMessage()
	if iMessage == nil {
		// Proceed to the next layer in the chain
		return chain.ProceedWithIMessage(iMessage, nil)
	}

	// 2. Get the data
	data := iMessage.GetData()

	// 3. If the data length is less than the LTV header size, proceed to the next layer
	if len(data) < LTV_HEADER_SIZE {
		// Proceed to the next layer in the chain
		return chain.ProceedWithIMessage(iMessage, nil)
	}

	// 4. Decode the LTV message
	ltvData := ltv.decode(data)

	// 5. Set the decoded data back to the IMessage. Zinx's Router needs the MsgID for addressing.
	iMessage.SetDataLen(ltvData.Length)
	iMessage.SetMsgID(ltvData.Tag)
	iMessage.SetData(ltvData.Value)

	// 6. Proceed with the decoded data to the next layer
	return chain.ProceedWithIMessage(iMessage, *ltvData)
}

3. Using Custom Decoder

Developers can also define their own custom decoder by referring to the implementations of the TLV-Decoder and HTLVCRC-Decoder. Then, they can set their custom decoder for the server or client using the SetDecoder(IDecoder) method.

For example, in the case of a server, let's assume that the custom decoder is HTLVCRC-Decoder. You can set the custom decoder for the server as follows:

func main() {
	// Create a server instance
	s := znet.NewServer()

	// Set the HTLVCRC-Decoder for processing HTLVCRC protocol data
	s.SetDecoder(zdecoder.NewHTLVCRCDecoder())
    
	s.AddRouter(0x10, &router.HtlvCrcBusinessRouter{}) // Simulate data with funcode field 0x10
	s.AddRouter(0x13, &router.HtlvCrcBusinessRouter{}) // Simulate data with funcode field 0x13

	// Start the server
	s.Serve()
}

The corresponding business logic is as follows:

type HtlvCrcBusinessRouter struct {
	znet.BaseRouter
}

func (this *HtlvCrcBusinessRouter) Handle(request ziface.IRequest) {

	// Get the MsgID
	msgID := request.GetMessage().GetMsgID()
	zlog.Ins().DebugF("Call HtlvCrcBusinessRouter Handle %d %s\n", msgID, hex.EncodeToString(request.GetMessage().GetData()))

	resp := request.GetResponse()
	if resp == nil {
		return
	}

	// Get the decoded data
	htlvData := resp.(zdecoder.HtlvCrcDecoder)

	zlog.Ins().DebugF("Performing business logic for msgid=0x10 with data: %+v\n", htlvData)
}