Issue after reconnecting to MAC Sequoia

[Aodake]

After installing the newest agent via binary (since the mpkg version didn’t work), we connected to the computer and granted all the necessary permissions. However, after reconnecting to the Intel Mac running macOS 15.0, we encountered a black screen, with only limited cursor movement in the top left corner.

The issue temporarily resolved when we turned off the screen recording permission and then allowed it again while attempting to connect. However, the problem reappears once we disconnect and try to reconnect.

[si458]

this is linked to this also #6398 #6398 (comment)
the is problems with Sequoia so meshagent physically doesnt work at all anymore!
DO NOT USE Sequoia until we have found/identified whats wrong
apple have caused nothing but problems since Sequoia was released,
even in the news there are reports things dont work like browsers and web proxies, even wifi issues

PS: i have already seen these issues myself in testing on a VM on my m1 mac mini,
this is why i NEVER upgrade straight away

[Aodake]

@si458 The thing is our customer bought new Mac and it came wit the update sadly. But we gonna wait for the bug fixes since its only one PC affected.

[si458]

@Aodake wow you are lucky indeed!
i literally have no idea whats happening with it,

if you run the binary from the command line with -connect and dont close the terminal,
then allow all permissions etc for terminal
it works no problems, no questions asked,

its just the minute you put it into the background with launchctl it just doesnt work?
or rather it works the very first time you run the application!
but after ur first disconnect, it then never works again, it is very strange?

it even seems to be disconnecting and reconnecting also very often which seems weird too?

im more likely to believe its an apple problem than an meshagent problem as i said before
the news i heard is people where having nothing but problems since it was released
even web browsers where not working either!?
and security applications physically dont work either!

will update this post when i find the articles im talking about

[silversword411]

Apple's always breaking shit and making things worse with every release. 1 month only screen recording is a PITA.

Give them feedback, I know I have
https://www.apple.com/feedback/

[DaanSelen]

Hopefully this gets fixed soon, everything works except the Desktop view.

[si458]

@DaanSelen sadly the agent doesnt work properly with Sequoia when running in background mode

HOWEVER i have noticed that the app works all the time if you use the binary app in interactive mode!
meshagent -connect
but oversally this means the app would have to be running 24/7 on the desktop in terminal 👎

so without a developer who specialises in mac applications and knows C++ and can look through our code and work out whats wrong, it wont work correctly

[johnnyq]

Everyone keep an eye on 15.1 that will coming out late this month, just maybe it will fix itself then.

[si458]

@johnnyq I'm waiting for the ipsw to drop, and then I can test in utm

[si458]

So downloaded the 15.1 RC2 ipsw from apple, set up vm, and nope, it still doesn't work :(

If anything, it's worse!

You get no prompts for screen recording or mouse control at all.

It doesn't even add the application into the security options in settings so u cant just flick the toggle to say give it permission, u have to manually locate the file with finder then drag it into the list and click the toggle :(

[johnnyq]

Thanks for testing that out @si458 and continuing the MeshCentral legacy for the community.

Maybe i'm too hopeful, but still hoping maybe Apple will see this and relax their security a bit for third party opensource applications in time for 15.1 release.

I have a few Apple clients, so if things don't pan out maybe I can put them on Rust desk (Haven't tried yet) for the time being until a fix arrives.

[gookkis]

Hi @si458 , recently I test at OS 15.1 Mac Mini have that issue too. I guess it caused meshagent security and permissions not have bundle identifier so caused bugs if running at background like access table at TCC problem…

[si458]

@gookkis i havent tried building a new agent yet tho, thats on my to-do list,
as we do have new code in which pops up asking for permissions etc,
but i still dont think that will fix the remote control only working once :(

[gookkis]

@Aodake wow you are lucky indeed! i literally have no idea whats happening with it,

if you run the binary from the command line with -connect and dont close the terminal, then allow all permissions etc for terminal it works no problems, no questions asked,

its just the minute you put it into the background with launchctl it just doesnt work? or rather it works the very first time you run the application! but after ur first disconnect, it then never works again, it is very strange?

it even seems to be disconnecting and reconnecting also very often which seems weird too?

im more likely to believe its an apple problem than an meshagent problem as i said before the news i heard is people where having nothing but problems since it was released even web browsers where not working either!? and security applications physically dont work either!

will update this post when i find the articles im talking about

Hi @si458 recenlty I try to use nohup for running meshagent for hide terminal, then it solved the problem. But at System and Privacy it's show Terminal apps request for permission ScreenCapture, Accesibility, and Full Disk Write. It's mean there problem at daemons of meshagent when put background with launchctl. Maybe it caused of meshagent binary don't have bundle identifier like Terminal or other apps?

[si458]

@gookkis good spot! i will try have alook next week, im abit swomped at the moment, thanks for the hint!

[gookkis]

@gookkis good spot! i will try have alook next week, im abit swomped at the moment, thanks for the hint!

I found some duplicate process of meshagent too, when there incoming remote access. Maybe it can help solve this issue...

[gookkis]

Here other clue again @si458

[pierrecorsini]

Same issue here on 15.1, the installation seems to work but when trying to access the remote desktop, the permission popup appears again (while it has already been granted previously).

[silversword411]

https://developer.apple.com/documentation/screencapturekit

[si458]

@pierrecorsini, if u have given the permission previously, you might need to remove the app from the list, then let it add it back in and then enable it,
It's a weird bug I've spotted with Mac for some reason?

[gookkis]

Are this because the meshagent running as daemons? Can you @si458 try to create app running at boot but daemons have a bundle identifier?

[si458]

@gookkis i dont quite know what you mean by bundle identifier or even how to get that information?
the app works on 14.x no problems, but sadly 15.x broke it and i dont know why

the other problem ive spotted is when you first run sudo ./meshagent -install
the first connect works! no problems!
but the minute you disconnect it never works again
also if you restart your mac, its also not starting up either, so i dont know why its doing that?
any suggestions/docs would be great!

[gookkis]

I mean meshagent have an identifier like com.meshagent and run at Applications folder, i mean meshagent have a .app standar macos Application not only as binary. because when i running it via terminal permission of screencaputer will recognized as Terminal.app and meshagent work (like it need run not as daemons) maybe this is a clue?

[si458]

@gookkis no the is no .app version, its a binary file only
i did notice that thinking about it, at one point when u run the sudo ./meshagent -install
it popped up in the security as terminal im not sure why?
as it doesnt do it on 14.x ?
but anyways even if you drag and drop the binary into the security panel of screen recording, accessibilty and full file access,
it still doesnt work after the first connect, and ask explained it doesnt start up either after a reboot?
so im not sure what apple have changed in relation to the LaunchDaemon ?

[gookkis]

when meshagent is installed at then we run via terminal with command "sudo path/to/meshagent" and it terminal keep open i can remote it but at setting security and privacy will recognized as Terminal ask for permission. I set it and Terminal will restart and call command again, i can remote my device.

[jakobzudrell]

I compared the trace of a system with Sonomna (which works fine) and Sequoia and this is the part that stood out to me on the sequoia system.

Interestingly enough, if I connect to the Desktop (same as I'm working on) it works the first time (as mentioned by you @si458) but after that it stays black, however the mouse seems to be captured as it jumps to the edge of the screen as soon as I enter the remote desktop area. At first sight this corresponds to the trace above which also shows errors related to RPScreenRecorder.

To me it doesn't look like it's a permission issue with tccd but rather something that happens later when actually accessing the screen recorder API.

[si458]

@jakobzudrell, thanks for ur findings.
There seems to be a mixed array of problems,
like @gookkis explained, it's seems to be showing terminal in the security panel rather than meshagent
Gotta love apple, and there changes

[jakobzudrell]

Yeah, seems like a lot comes together 😅

I can confirm the behavior with the Terminal in security panel, however only when I actually run the agent interactively from the Terminal - this does not surprise me, I think this is just how Apple handles permissions. Experienced the same when setting up restic backup on my PC. When run from Terminal, you need to allow Full Disk access to the Terminal. As opposed to running as a daemon, you can allow full disk access directly to the executable.

In my case above, the Agent was run via LaunchDaemon so the permissions were set for meshagent binary directly.

But yeah, I'll keep looking and checking in here when I find something - however a lot of concepts are new to me here (e.g. when i read tccd in the logs I first have to read up on that to even know what it does 😀), so not very efficient but I'm doing my best...

Quick question aside: Which VM app are you guys using on MacOS? I have set up a Sequoia VM using UTM so I don't have to always test on my own system but some operations (seems mostly stuff to login/user profiles, e.g. login, sudo, ...) take ages to load (on M3 Air)

[si458]

@jakobzudrell I use UTM as well on my M1 mac mini (it's only a base model 8GB/256GB)
I use the virtualised option and not emulated, but seems quick enough to test with!
I just make sure i don't update my mac mini to sequoia otherwise I can't control it with my meshagent haha

[jakobzudrell]

checking back in here... sadly no groundbreaking news :/

So far I tried to get an understanding on (1) how the agent interacts with the system upon startup, new connection, etc. and (2) where the respective functions reside in code. Furthermore, I've observed behavior that I am confident to say is reproducible, at least on my personal device and test VM (which finally works ;));

1. The LaunchDaemon (or root daemon) works fine on my systems. I have a persistent connection to the server and can connect to Terminal and Files - both of those processes do not require any user context.
2. As soon as a new Desktop session is spun up, the agent process forks itself into a process running as the logged on user, to access their desktop. Now this is where it gets interesting:
   2.a. The first time this is done, before any permissions are granted, this usually works fine for me ('works' referring to, I can control the remote mouse/keyboard and also see the remote desktop)
   2.b. Subsequent desktop sessions result in a black screen shown in the MeshCentral webui. However, if I move the cursor within that area, the remote cursor seems to be jumping right to that point where I entered the screen (this can be tested, when doing a desktop session to the same machine), indicating that there is happening at least some communication.
   2.c. After some time, the user context agent process then just crashes.

My current struggles however are not about the agent not working properly, but having a hard time debugging the "user" process of the agent during runtime, which in my opinion is the point of interest for now. I managed to hook into the processes stdout using dtrace but this isn't very reliable and also a hassle because the pid of the forked process is not really predictable.

It would be nice to just tell the user process to log to a file it has permissions to and then add some verbose logging to get familiar with what is going on during that user session. But at this point I am a little limited by my lack of experience with C projects of that size. I will of course try to find a solution to this, but if anyone around here has some more experience with stuff like that, let me know :)

[si458]

@jakobzudrell if u can get the agent to connect to meshcentral, the is a debug option which should output logs to the meshcentral console window (you must keep it open in another window and watch it)

I can't fully remember what the command was but u can check all the console values by running help in the Web ui

[sln-guy]

Having same issues.

Will try my hand at debugging to console as mentioned by @si458.

[si458]

@sln-guy thank u! @jakobzudrell discovered that it's getting an error in relation to the screen capture!
But from what I've read in docs etc nothing changed for permissions so it should just work, so I'm confused why it might be crashing

[jakobzudrell]

hey, sorry for the infrequent responses from my side - having some upcoming exams in the pipe so I have little spare time left.

Sadly, I haven't been successful with the log output to the debug console... Also, when looking at the code of the "user context" agent, it doesn't seem to do a lot of logging, however there seems to be a function KvmDebugLog (found in meshcore/KVM/MacOS/mac_kvm.c) that probably was used for debugging at some point. If we manage to get this to work, we may be able to write detailed logs to a file and get a better understanding on where the agent dies exactly.

Also one thing we may consider thinking about: CGDisplayCreateImage, which seems to be the utilized function that actually takes a screenshot of the display, then sending it through stdin/stdout to the "main" agent process has been deprecated for quite some time now. While it is unlikely to me that this is causing our problems (screenshare works the first time), it probably makes sense to keep our eyes open for any notices from Apple, and maybe at some point switch to the new ScreenCaptureKit, but first things first :P

[jakobzudrell]

ok, I managed to get KvmDebugLog in mac_kvm.c to write to a file at /tmp/mshlog.txt. Feel free to use my fork as a base for this - but keep in mind that I've been playing with the code a lot so it all is a little messed up...
Maybe @sln-guy this helps you do some troubleshooting as well :)

[si458]

@jakobzudrell i can't see any new commits or branches in ur fork? Would be interesting to see what the kvmlog shows!

[jakobzudrell]

@jakobzudrell i can't see any new commits or branches in ur fork? Would be interesting to see what the kvmlog shows!

Yeah, would've made sense to push the changes... 😀 So the main file that I worked on was the mac_kvm.c which also had the KvmDebugLog function.

[jakobzudrell]

Also keep in mind to check permissions on the /tmp/mshlog.txt file. If the loginscreen kvm child writes to it, owner is root, so the actual user session kvm child won't be able to write to it

[jakobzudrell]

I compared the trace of a system with Sonomna (which works fine) and Sequoia and this is the part that stood out to me on the sequoia system.

Interestingly enough, if I connect to the Desktop (same as I'm working on) it works the first time (as mentioned by you @si458) but after that it stays black, however the mouse seems to be captured as it jumps to the edge of the screen as soon as I enter the remote desktop area. At first sight this corresponds to the trace above which also shows errors related to RPScreenRecorder.

To me it doesn't look like it's a permission issue with tccd but rather something that happens later when actually accessing the screen recorder API.

I just performed another check on this;
These messages are only shown when the child agent process crashes. The first time (initial connection, when no permissions have been granted in the past), when everything works fine these messages are not shown. So maybe it is worth focusing on that, trying to understand what these mean

[jakobzudrell]

Ok I think I can confidently say, that the point of failure is the following line in mac_kvm.c;

CGImageRef image = CGDisplayCreateImage(screen_num);

Again, granting the permissions initially works, after that, the above function always returns NULL. This results in the g_shutdown flag being set which ultimately leads to the child process to exit.

[si458]

@jakobzudrell oh dear that function is deprecated! and removed in 15!
this might explain why it doesnt work!
thank you so much for discovering this!
https://developer.apple.com/documentation/coregraphics/1455691-cgdisplaycreateimage

[jakobzudrell]

Also one thing we may consider thinking about: CGDisplayCreateImage, which seems to be the utilized function that actually takes a screenshot of the display, then sending it through stdin/stdout to the "main" agent process has been deprecated for quite some time now. While it is unlikely to me that this is causing our problems (screenshare works the first time), it probably makes sense to keep our eyes open for any notices from Apple, and maybe at some point switch to the new ScreenCaptureKit, but first things first :P

Yup, I think the new way to go is the ScreenCaptureKit... If there is someone to take over this task, then I think it would definitely be the best to directly start working on that, hopefully resolving all the issues with CGDisplayCreateImage.

I started playing around with the ScreenCaptureKit but for some reason I wasn't even able to compile after including the ScreenCaptureKit.h framework 😀 I'll keep working on that and checking in here, but I'm definitely not the one for a quick fix - too little experience :)

[si458]

@jakobzudrell you need to include the framework within the makefile script
https://github.com/Ylianst/MeshAgent/blob/39a927f25ed232ddd826ff6ea716e674a2f9aef7/makefile#L755-L758

[jakobzudrell]

I did, but something down the road seems to go wrong:

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSObjCRuntime.h:612:1: error: expected identifier or '('
  612 | @class NSString, Protocol;

[si458]

@jakobzudrell
https://developer.apple.com/documentation/macos-release-notes/macos-15-release-notes#ScreenCaptureKit
prusa3d/PrusaSlicer#13423
wxWidgets/wxWidgets#24724

yep we need to switch now to using ScreenCaptureKit 👎
so need to get a sample working first then we can try and work out how to implement it into the meshagent

[DarkxPunk]

I decided to upgrade my headless server before reading any of this. Glad it looks like we're on the right path. Hopefully a fix soon. Guess I will stick with VNC for now.