Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] SIGABRT in SRT:SndQ thread #2836

Open
jeandube opened this issue Dec 18, 2023 · 0 comments
Open

[BUG] SIGABRT in SRT:SndQ thread #2836

jeandube opened this issue Dec 18, 2023 · 0 comments
Labels
[core] Area: Changes in SRT library core Type: Bug Indicates an unexpected problem or unintended behavior
Milestone

Comments

@jeandube
Copy link
Collaborator

Describe the bug
SRT group sender/listener asserts in SRT:SndQ thread

To Reproduce
System under test: SRT group sender/listener asserts when breaking backup link
eth0: main link (weight=1), eth1: backup link (default weight)
test cycle: network connector of backup link unplug/wait-1-2 secs/replug,
May takes many cycles depending on setup and peer.
MX4E -> SRT GW: 1-2 cycles (100% reproducible)
MX4e -> mxphub: 20-30 cycles (hard to reproduce)
libSRT 1.5.3, no encryption, 2 distinct network adapters used on each peer.

App running in gdb with a modified libsrt.so.1.5.3 with SRT_ASSERT remapped in our own _DoAssert() function (using SIGSEGV)
In gdb session below, assert (len <=cap) cap=1332 and len=18446744073551542776 (-158008840).

[15/12/2023` 10:36:41.730] ASSERT occurred in setLength(/home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/packet.cpp:261).
Expression = "len <= cap"
Process = "sessmgrd"
Thread ID = 19106 "SRT:SndQ:w2"
[Fri Dec 15 10:36:41 2023,sessmgrd:19061] threadcheck.c(__ShowThreadsInfo:1088): THREADCHECK: *THREAD "SRT:SndQ:w2"[19106] at worker(queue.cpp:526), Changes=6, Iterations=6, IsPaused=0, Changed=0s ago.
Thread 32 "SRT:SndQ:w2" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 19106]
0x0000007ff7ebd430 in _DoAssert (szExpression=0x7ff7d34258 "len <= cap",
szFunction=0x7ff7d343b8 <srt::CPacket::setLength(unsigned long, unsigned long)::FUNCTION> "setLength",
szFileName=0x7ff7d341f8 "/home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/packet.cpp", iLineNumber=261) at dbgtrace.c:292
292 dbgtrace.c: No such file or directory.
(gdb) [ 849.409384] axienet_phy_ethtool_get_link_ksettings:2657 ior off:180 1 b
bt
#0 0x0000007ff7ebd430 in _DoAssert (szExpression=0x7ff7d34258 "len <= cap",
szFunction=0x7ff7d343b8 <srt::CPacket::setLength(unsigned long, unsigned long)::FUNCTION> "setLength",
szFileName=0x7ff7d341f8 "/home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/packet.cpp", iLineNumber=261) at dbgtrace.c:292
#1 0x0000007ff7cecda8 in srt::CPacket::setLength (this=0x7f1fffe8c8,
len=18446744073551542776, cap=1332)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/packet.cpp:261
#2 0x0000007ff7c827cc in srt::CSndBuffer::readData (this=0x63e7b0,
w_packet=..., w_srctime=..., kflgs=0, w_seqnoinc=@0x7f1fffe514: 0)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/buffer_snd.cpp:318
#3 0x0000007ff7caec54 in srt::CUDT::packUniqueData (this=0x638e38,
w_packet=...)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/core.cpp:9627
#4 0x0000007ff7cae7ec in srt::CUDT::packData (this=0x638e38, w_packet=...,
w_nexttime=..., w_src_addr=...)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/core.cpp:9514
#5 0x0000007ff7cf4790 in srt::CSndQueue::worker (param=0x7f7807f5f0)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/sr---Type to continue, or q to quit---up
tcore/queue.cpp:585
#6 0x0000007ff7facfd8 in ?? () from /lib/libpthread.so.0
#7 0x0000007f9ac827c8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) up
#1 0x0000007ff7cecda8 in srt::CPacket::setLength (this=0x7f1fffe8c8,
len=18446744073551542776, cap=1332)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/packet.cpp:261
261 /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/packet.cpp: No such file or directory.
(gdb) p cap
$1 = 1332
(gdb) p len
$2 = 18446744073551542776
(gdb) up
#2 0x0000007ff7c827cc in srt::CSndBuffer::readData (this=0x63e7b0,
w_packet=..., w_srctime=..., kflgs=0, w_seqnoinc=@0x7f1fffe514: 0)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/buffer_snd.cpp:318
318 /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/buffer_snd.cpp: No such file or directory.
(gdb) [ 933.451703] axienet_phy_ethtool_get_link_ksettings:2657 ior off:180 1 3
[ 933.458395] axienet_phy_ethtool_get_link_ksettings:2677 ior off:1004 b b
p w_packet
$3 = (srt::CPacket &) @0x7f1fffe8c8: {static MAX_TIMESTAMP = 4294967295,
static TIMESTAMP_MASK = 4294967295, m_nHeader = {inarray = {4294967295, 0,
0, 0}}, m_PacketVector = {{ = {iov_base = 0x7f1fffe8c8,
iov_len = 16}, }, { = {iov_base = 0x641c98,
iov_len = 0}, }}, m_extra_pad = 0,
m_data_owned = false, m_DestAddr = {{sin = {sin_family = 0, sin_port = 0,
sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"},
sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {
__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {
0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}, sa = {sa_family = 0,
sa_data = '\000' <repeats 13 times>}}, len = 28}, m_zCapacity = 0,
m_iSeqNo = @0x7f1fffe8c8, m_iMsgNo = @0x7f1fffe8cc,
m_iTimeStamp = @0x7f1fffe8d0, m_iID = @0x7f1fffe8d4,
m_pcData = @0x7f1fffe8e8, static HDR_SIZE = 16, static UDP_HDR_SIZE = 28,
static SRT_DATA_HDR_SIZE = 44, static ETH_MAX_MTU_SIZE = 1500,
static SRT_MAX_PAYLOAD_SIZE = 1456}
(gdb)

***later session to grab m_pCurrBlock
bt
#0 0x0000007ff7ebd430 in _DoAssert (szExpression=0x7ff7d34258 "len <= cap",
szFunction=0x7ff7d343b8 <srt::CPacket::setLength(unsigned long, unsigned long)::FUNCTION> "setLength",
szFileName=0x7ff7d341f8 "/home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/packet.cpp", iLineNumber=261) at dbgtrace.c:292
#1 0x0000007ff7cecda8 in srt::CPacket::setLength (this=0x7f1fffe8c8,
len=18446744073551542776, cap=1332)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/packet.cpp:261
#2 0x0000007ff7c827cc in srt::CSndBuffer::readData (this=0x63e7b0,
w_packet=..., w_srctime=..., kflgs=0, w_seqnoinc=@0x7f1fffe514: 0)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/buffer_snd.cpp:318
#3 0x0000007ff7caec54 in srt::CUDT::packUniqueData (this=0x638e38,
w_packet=...)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/core.cpp:9627
#4 0x0000007ff7cae7ec in srt::CUDT::packData (this=0x638e38, w_packet=...,
w_nexttime=..., w_src_addr=...)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/srtcore/core.cpp:9514
#5 0x0000007ff7cf4790 in srt::CSndQueue::worker (param=0x7f7807f5f0)
at /home/jdube/sandbox/makito2_project/components/vendors/haisrt/srt.git/sr---Type to continue, or q to quit---
tcore/queue.cpp:585
#6 0x0000007ff7facfd8 in ?? () from /lib/libpthread.so.0
#7 0x0000007f9ac827c8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) p m_pCurrBlock
$4 = (srt::CSndBuffer::Block *) 0x6491a0
(gdb) p *m_pCurrBlock
$5 = {m_pcData = 0x641c98 "", m_iLength = -158008840, m_iMsgNoBitset = 0,
m_iSeqNo = 0, m_tsOriginTime = {m_timestamp = 0}, m_tsRexmitTime = {
m_timestamp = 0}, m_iTTL = 64, m_pNext = 0x6491e0}
(gdb)

@jeandube jeandube added the Type: Bug Indicates an unexpected problem or unintended behavior label Dec 18, 2023
@maxsharabayko maxsharabayko added this to the v1.6.0 milestone Dec 18, 2023
@maxsharabayko maxsharabayko added the [core] Area: Changes in SRT library core label Aug 23, 2024
@maxsharabayko maxsharabayko modified the milestones: v1.5.4, v1.6.0 Aug 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[core] Area: Changes in SRT library core Type: Bug Indicates an unexpected problem or unintended behavior
Projects
None yet
Development

No branches or pull requests

2 participants