P25 Transmissions Bleeding Together #701
Comments
|
@robotastic I've tried to track this down, and it seems it's sometimes not recognizing the terminator (TDU), and that is causing transmissions to occasionally blend. Can you double check this block at some point? I keep coming back to it, and it wondering if you are purposely only tagging "terminate" if the amt_produce<= 0. |
|
I've had no issues with transmission bleeding into other recordings from one talk group into another now that I've moved over to a much more powerful AMD 5800 based system from the Raspberry Pis. I think it's very likely that given that the system is CPU starved it will continue recording on a frequency that is reused from another talk group when the terminate is late to the processed and is racing against other in flight recordings. So I think there is a data race here. |
|
Just to clarify, this issue doesn't present itself as recordings from two different talk groups bleeding together. This is just two transmissions from the same talk group by two different radios, being recorded together is a single "transmission" within a call. You probably wouldn't even notice this happening unless you're logging and comparing the "Call Grants" and actual recorded transmissions, or you're able to watch the SUIDs like a hawk. I could be proven wrong, and it could be a race condition, or resource issue. But given that the audio, the terminations, and the next audio is all on the same voice channel (same recorder) and that it's clearly picking up the audio but missing the 20ish TDUs in between leads me to believe there's a bug to look into. |
|
Thanks for digging into this @tadscottsmith - it is VERY possible there is either a subtle bug in there OR some of the assumptions I made are not universal and are just local to systems I have tested on. The reason I hadn't been including TDU units where there is some audio being produced is because the audio would get dropped because the Termination Flag if true. (I have to double check this). I was always seeing lots of TDUs with amt_produced = 0 . It could be though that there is low signal sometimes and the TDUs are lost... or maybe on some systems there are no blank TDUs... or maybe when a system gets really busy the buffers back up, so the AMT produced could grow even though there is a TDU there. I can take a look into it. let me know if you find clues! |
|
[@Dewey3] If you want to give it a shot, this branch has been updated to better handle things on my P25 P1 system. However, I don't have a P25 P2 system close, so I can't test against one. |
I just tried it, but unfortunately, I didn't see a difference on the P25 P2-TDMA system. Thank you for trying! |
|
Has it ever worked on a 4.x version? I just want to double check, since you'd called out 4.3 on the other thread. |
|
I added a couple a couple of commits. I'm kind of flying blind just digging through code, but I noticed the p25_frame_assembler_impl.cc wasn't ever trying to update the source for P25P2. If you have a minute, give it a shot and see if it makes anything better (or way way worse...). https://github.com/tadscottsmith/trunk-recorder/tree/p25-tx-update |
I want to say no it has not, BUT, that is from a memory that doesn't work like it used to. With that said, I have to add that I actually came into the v4 game late because I thought there was a problem with the recordings, and it wasn't until just about 3 or 4 months ago that I realized the problem was my fault. I had my v3 gains set at something like 7 for digital and 7 for analog and the raspy sound of the over-modulation in v4 with my v3 gains did not click on what to do until earlier this year. |
|
Thanks again! It's close to nighttime retirement time for me, so I'll give this a shot when I return home from work tomorrow. |
I just gave it a shot and unfortunately, no change. Just for a matter of clarification, the way I'm doing the checking is by watching the UID live in Rdio-Scanner after TrunkRecorder sends the transmissions. The system is a very busy system, and a lot of the transmissions are back-to-back-to-back-to-back. When the rapid transmissions happen, most times the same UID will remain on the Rdio-Scanner screen, but occasionally, the UID will fail to populate at all, For the sake of comparison, when I pop my TrunkRecorder v3 card back into the RPi, everything returns to working as it should with the UID following the actual transmitting UID no matter how fast the transmissions come behind each other. I am able to verify the correctness of the live UIDs buy following the same transmissions on the SDS100, SDS200, and Unitrunker. |
|
Bummer! Well, if anyone has a spare box around near a P25P2 system, let me know. Otherwise, without being able to print out some debugging statements to troubleshoot, I am not sure how much more I can dig into for P25P2. I know on P25P1 it seemed that either intentionally or unintentionally the Control Channel wasn't always processing grant message as fast as the voice channel, and it seems to work a little better to trust the voice channel once the recorder has started. |
|
When I get home today, I'll see if I can build a quick testbed RPi for the 2A8 system that you can ssh into and I'll leave up for the weekend. |
|
I won't be able to do anything this weekend, but I can probably find some time next week! We can touch base after the weekend. |
|
I get spotty reception of the PG County system. I can try running to see if I can spot something. There is also a chance the pipeline for getting those IDs to Rdio could have been messed up too.
If I have enough SDRs I maybe able to run 3.x side by side with 4.x and look for differences.
… On Jun 17, 2022, at 11:50 AM, tadscottsmith ***@***.***> wrote:
I won't be able to do anything this weekend, but I can probably find some time next week! We can touch base after the weekend.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.
|
Thanks Luke. You gave me another idea for one thing I didn't think about before. With v4, I am sending the recordings to Rdio on another RPi via plugin. Since the plugin did not exist in v3, I run both on the receiving Rpi ingesting the call into the local copy of Rdio via DIrwatch, then sending downstream to the main Rpi. I need to throw Rdio on the local Rpi when running v4, then ingest the calls locally as I do with v3 to see if there is a difference. Thanks again... I'll probably give that a try this evening since I will not be in a rush to build the testbed RPi. |
|
@robotastic I've got a lot of extra logging turned on, but here's an example on P25 P1 where the master branch wouldn't correctly update the SRCID. The second transmission (below) either doesn't receive or fails to process any TDUs. This means that the second and third transmissions (320643 and 312993) would be recorded together and would only have one SRCID (320643). What seems to be working pretty well is to end the transmission and let it start a new transmission any time that the voice channel and current call have different sources. I think you had most of the code already stubbed out and commented. I'm wondering if something similar is occurring in P25P2. |
|
I'm also curious about this line not having an equivalent for p2tdma? I tried to build it out blind in my branch above, but it may not have worked. |
|
I just completed the trial of v4 into a local copy of Rdio via Rdio's Dirwatch, and I'm still missing a lot of the UIDs when the transmissions are right behind each other. |
|
What is the JSON configuration needed for that level of logging? @tadscottsmith? |
It's just a custom branch I've been using for debugging with a bunch of extra log statements. Nothing that can be changed in the config file as of yet, but I could certainly put in a PR with some additional call logging. |
|
@tadscottsmith that is a really interesting catch, where the source changes but there is no TDU message. How is your reception? is it possible that those "packets" were just dropped for some reason? I was thinking of having the transmissions switch if the source does... I just wasn't sure if that would create new problems if I have some weird errors with the src decoding. It might be worth trying to re-enable that. If I am remembering things right, Phase 2 audio still gets put through the Phase 1 decoder first, and that is where the SRC ID is captured.... I should go check though |
|
I'm not seeing a way to PM on Github so I don't know how to send the credentials. I do have a duplicate Prince George's Co, MD 2A8 system up and running on a spare RPi that is ready for either of you to take a look at via SSH and/or Rdio-Scanner and fiddle with if needed. |
|
@robotastic I added some additional logging, and it appears the TDUs are being received, although this issue seems to happen when there are just a short number of TDUs before the next HDU/LDU. Here's the full log of a transmission from two different sources, 117020 and 116007. You can see here that there are only 5 TDUs in a very brief period of time, and the p25_frame_assembler_impl::general_work() ignores them, because it apparently is never able to get to a state of AMT_PRODUCE == 0 while terminate_call == true. And here you can see at the end of the second "transmission", there were 8 TDUs over a longer period of time and p25_frame_assembler_impl::general_work() appears to be able to process all they way through amt_produced and the terminate tag is correctly applied and sent to the transmission_sink. I'm still learning this flow, so I figured I'd log what I have so far, and maybe something will stand out to you. |
|
Nice detective work - I am going to have to dig into your fork so I can better match up the log messages with what is happening. One interesting thing I saw: If I am reading this right, it looks like the Control Channel, channel grant message comes in before the TDU... which sort of shows the difference in processing time between the two. I will dig in more |
|
PS - I did try to put together a flow chart on call handling... not sure it makes things clearer: https://github.com/robotastic/trunk-recorder/blob/master/docs/notes/CALL-HANDLING.md |
|
This branch is very similar to your master, except that it has a ton of extra logging: To further complicate things, it appears there is a "hang time", or "teardown time" (approximately 2 seconds) in which there is never a grant on the control channel and the source is only updated via the voice channel. I'm trying to find actual documentation on it, but here's an example (no Grant for 154241)... |
|
SO! It looks like the voice data that is sent along with the TDU, may actually be silence. This means it should be safe to terminate if we see any TDU come in... I think. I am going to look at the specs some more. There is also a good chance that there are differences in implementations.
|
|
That is super weird that there was no Channel Grant for the new Source. I wonder if it came in as a Channel Update message vs a Channel Grant? The update messages do not have a Src ID in them. Still - I thought all new units transmitting got a channel grant. |
|
Luke, you're w - a - a - a - y - y beyond my education and pay scale now; however, is there something you can do to compare the last few versions of v3 to the latest versions? I still run v3 and do not see any problems between the transmissions. |
@Dewey3 Are you in the Discord? If you message me, I can take a look and see if I see anything obvious. |
|
If I'm reading this correctly, it seems like if the system is setup in message trunking mode and using the direct call continuation method, that there could be multiple sources (tranmissions) on the traffic channel without additional explicit grants on the control channel. As long as the response was within the hang time, it would just continue sending grant updates. Pages 74-76 explain (diagram is for control channel method and there is no diagram for direct method). |
|
Interesting!! that is a great find @tadscottsmith. It could be something like that. I wonder why the old system was working though? It was only getting the Src IDs from the control channel, and those only come on the Grant messages, not the Updates. Are the transmission really short? Like there wouldn't be time for the Src ID to come over the Voice Channel? |
|
Still - There should be TDUs on the voice channel... but maybe in this mode there are less of them or they always have silent voice samples attached. I will go remove the check on the TDUs and just have them all be equal. |
Thanks much @robotastic and @tadscottsmith for all that you do! As I mentioned, I don't know about the inner workings of trunking or TrunkRecorder, so I unfortunately can't comment there. However, I can say like TrunkRecorder v3, I don't think Unitrunker misses the data for any changes and transmissions on the same PG Co, MD P25P2-TDMA system, and I'm pretty sure that it gets its information from only the control channel. |
|
To be fair - the inner workings of Trunk Recorder are a bit of a mess! Thanks for putting up with it! That was a great clue - it looks like I had actually changed the code so Src IDs were not being sent in from the Voice Channel. I just fixed that in the latest build. |
|
@tadscottsmith I also changed it so that the TDU flag will end a transmission even if there were audio samples that came in with it. |
|
Sounds great. Thanks for the updates! I'm out for a few days but will do some testing later this week or next. Enjoy the holiday! Perhaps @Dewey3 can pull the latest version and see if it's able to keep up on his busy system. |
|
@robotastic I took a look and can’t see any reason that the changes would impact the TGID. What indications were you seeing that the TGID was messed up? I can certainly dig deeper and find out. |
|
@tadscottsmith yea... I didn't see anything either, it was super weird! It could have been some weird compilation thing and it was building against an old library version or something. I did find a fix, I think. The PTT packet gets sent every time someone keys their radio. It can be used to separate the transmissions... and it also has the SRC ID for the next transmission!! I was testing on the same system as @Dewey3 and I think it is working. Give the Master branch a try. I am curious though on where you got those extra message types from, that you added in on your branch? |
|
They were in older versions of OP25. I went back to your v3.3.7 and it seems like OP25 used to update the SRC on many more message types, although mostly they are just idle messages or group voice channel user update messages. You should probably be safe just using the SRC from the voice frames, but I have noticed in my logging that occasionally it seems there's a few frames of voice with a SRC of 0 before the voice channel starts sending the real source. |
|
oh! That is genius! I didn't catch that it had changed on the OP25 side. Let me see if I can re-create what was happening, but use a test branch!:) |
|
Thanks @robotastic - I built this version, but was only able to run it for 5 or 10 minutes today. Looking at a couple of the json files that contain multiple transmissions, that 5 or 10 minutes looks promising, so I can't wait until tomorrow to give it the full test. Just a side note... when I complete a build, I like to create an image file of the fresh build, so I will do a full clean up of the OS with
|
|
There is still an occasional miss, but they are far and few... this version is working miles better at catching the SRCID, in fact, I sometimes see the new ID popup on Rdio a second or two before the transmission actually starts (could be the timing for the call grant). Concerning the packages that got removed with this version, it looks like that killed the "compressWav" function. While I have compressWav set to true, I have never seen anything other than a wav file. That's certainly NOT a problem since I can always rebuild electing to not remove those packages, or let Rdio do the compression for me, the latter is what I am doing for now. |
There seems to be an issue with transmissions occasionally bleeding together within a call. From the attached call audio and JSON, you can clearly hear that there should be four distinct transmissions. All four of the transmissions are recorded correctly, but for some reason the last two seem to be combined by trunk-recorder. There are only three transmissions listed in the JSON .
2191 [SRCID 151586]: "2191."
DISP [SRCID 151004]: "Go ahead."
2191 [SRCID 151586]: "Begin Oak Street Ramp."
DISP [SRCID 151004]: "Copy."
It seems like this Grant correctly starts a third transmission, but does not decode/respond to a TDU before the next transmission begins.
I'm assuming that this Grant then updates the SRCID, but does not end the previous transmission and start a new one.
I realize this could theoretically be a signal issue, but it is happening on multiple systems, and evenly distributed across frequencies, so it leads me to believe there may be a certain condition that is causing it. It seems this is happening on about 2% of calls.
3580-1654728397_858737500-call_1521.zip
The text was updated successfully, but these errors were encountered: