Thoughts about FFmpeg UWP/Desktop Builds

[softworkz]

I didn't say the following earlier, because I wanted to wait until the situation has cleared up a bit and would be understood at least roughly. Now that this is achieved:

If you really really really really want to have just a single FFmpeg build for UWP and Desktop, then it could probably, maybe, perhaps and possibly be done as follows:

We're talking about like 12 functions only which are causing the dependency on the one app-local lib.
If we want to be able to use the UWP build for both, uwp and win32 targets, then it would go only by eliminating those dependencies from the ffmpeg dlls. The first and easiest thought would be to statically link these functions into the ffmpeg libs.
But those functions are: memset, memmove, strchr and the likes. If we would (even manage to) statically link them into the ffmpeg libs, then we would end up with a dependency on kernel32.dll in exchange - which is probably the most forbidden dependency you can even have ;-)

But it might be possible like this:

• You make a copy of the vcxproj and name it ffbridge or something
• Then remove all midl stuff and code, but keep the configs
• Import those 12 functions and create identical exports for them (maybe aliased)
• Then build for UWP and WinUI for all platforms, so you get 8 dlls
• Remove the project from the solution and Store those dlls. They won't need to be rebuilt (ideally never)
• Now it might get a bit tricky and some fiddling will be required: You need to convince ffmpeg to link to those newly created libs (the UWP versions) on build - instead of the MSVCRT140_APP.dll

As a result, you would have a single set of ffmpeg dlls for each platform and only exchange the ffbridge.dll depending on whether it's for UWP or WinUI.

This sounds simple, but I'm sure there will a lot of pitfalls in the way. Maybe it doesn't work out, at all. I have no motivation to even think about it. For what anyway? For the 30 NB output size per platform? For the 10 minutes which it takes to build another version of ffmpeg?

Our ffmpeg build (+MPV) takes 50min for only x86 and x64 - it's no big deal - and here it's just 10 minutes. I would even wait for 100 minutes rather than messing with ffmpeg's build system...

[brabebhin]

This sounds simple,

No, it doesn't, I can see several ways this can go wrong.

Our ffmpeg build (+MPV) takes 50min for only x86 and x64 - it's no big deal - and here it's just 10 minutes. I would even wait for 100 minutes rather than messing with ffmpeg's build system...

Well, TBH, we already have build scripts for desktop. And for desktop we could do stuff like encoders, which aren't allowed in appcontainer.
So even if we ignore the whole dll shenanigans, it would still provide more flexibility in the future to have separated builds.
On the other hand it is more overhead for us.

[softworkz]

This sounds simple,

No, it doesn't, I can see several ways this can go wrong.

LOL, didn't my sentence continue saying just the same ("but I'm sure there will a lot of pitfalls in the way.")?

Well, TBH, we already have build scripts for desktop.

Yes, but they are not working. My PR fixes that.

I needed to make some adjustments to the sub-repos as well. If there's interest, I can submit PRs for those fixes.

[brabebhin]

PR would be useful indeed. Thanks.

[softworkz]

Sorry, that was a bit misinformation. I had made many changes first to h264 and h265 - until I realized that those aren't even needed.
Then I had many more errors which I started to compensate by making adjustments, until I had found out that it was because I had vcpkg installed - AND - totally unexpected for me - is active in all projects once you install it. And since I had done some ffmpeg build experiments a while ago, vcpkg had the same libs but of different versions and so the header files mixed up and collided.

The three repos with changes have all the same and single change:

You probably don't need that. The effective fixes for the Desktop build are already in my PR in Build-FFmpeg.ps1 and FFmpegConfig.sh.

[lukasf]

I don' think that this approach is helpful. If we really need a Desktop FFmpeg build, we just make one.

What really makes me wonder: These 12 functions are all threading related. Nothing in there is a critical or forbidden API. I really don't understand why they are not allowed. If this was a file open or something like that, I would understand (for apps, they would need to insert app permission checks). But for threading? It's strange. Could it not be that these are unintended, false alerts? Things like that have happened in the past. For me, "DesktopCompatible" still sounds like the result should be compatible with desktop, while still targeting AppStore.

Besides that, if we really want/have to go for a Desktop build, we absolutely must make sure that such a build is WACK compatible. And with the solution you have done @softworkz, I have doubts about that. What you do is not a StoreApp build with DesktopCompatible. This is a full-blown Win32 Desktop build. I am really not sure if such a build could be uploaded to Windows Store.

As I said, I failed to create a package of the WinUI sample. If you have a running WinUI app where you can build packages, have you tried running WACK on such an app, using your Desktop ffmpeg build?

[lukasf]

By the way, I tried to find out the difference which DesktopCompatible makes. But the compiler and linker command lines seem to be 100% identical. I don't understand how the DesktopCompatible flag works on compiler/linker level.

[softworkz]

I have seen it. It's in one of the props or targets files. What it does is switching a lib to link with. You can find it with some patience and a good file searching tool :-)

(I didn't take a note of the file path, I'm afraid)

[softworkz]

I don' think that this approach is helpful. If we really need a Desktop FFmpeg build, we just make one.

I just mentioned it because you seemed to be so keen on avoiding a separate build ;-)

These 12 functions are all threading related. Nothing in there is a critical or forbidden API. I really don't understand why they are not allowed.

I'm not sure which 12 functions you have seen, but the one I'm referring to are not about threading - they are about memory manipulation:

And those are quite sensitive in terms of security so I do not wonder that they are restricted.

For me, "DesktopCompatible" still sounds like the result should be compatible with desktop, while still targeting AppStore.

Yes, that's what it does. It's compatible with desktop and AppStore. But AppStore for Desktop, not for devices, i.e. not for xbox and not for HoloLens.

Besides that, if we really want/have to go for a Desktop build, we absolutely must make sure that such a build is WACK compatible. And with the solution you have done @softworkz, I have doubts about that. What you do is not a StoreApp build with DesktopCompatible. This is a full-blown Win32 Desktop build. I am really not sure if such a build could be uploaded to Windows Store.

After the failure of the store as it had been planned originally, the failure of the "Windows runs everywhere" strategy and the failure of the UWP reception at the developer side, they fundamentally changed strategy: The created DesktopBridge with which you can even bring a Windows Forms app into the store. They dropped all restrictions to get apps into the store - but ONLY for desktop apps in the store.
They kept UWP for protecting the xbox. The intermediary "UWP Desktop" strategy was quickly dropped to be replaced with WinAppSDK.
The artifacts from the past are remaining: UWP apps are very restricted to get into the store (no matter whether desktop or devices) and WinAppSDK apps are not. But they can't run on devices.

Those super-strict store restrictions exist for UWP only not for Win AppSDK.

[softworkz]

The test is still running - but I can already guarantee you that it won't fail that test...

...and that's because this test is not even included for WinUI3/WinAppSDK apps. In fact, there's quite a different set of tests for each case:

Left WinAppSDK, right UWP

[brabebhin]

My understanding is that win32 apps like windows forms still need the runfulltrust capability to run when packaged for the store. That's what is allowing the restricted API.

I might be out of date on this.

[softworkz]

My understanding is that win32 apps like windows forms still need the runfulltrust capability to run when packaged for the store. That's what is allowing the restricted API.

That's a very good point actually. Interestingly, both the C# and the C++ VS templates for WinUI3 are declaring runFullTrust in the appxmanifest.

So I researched on whether it's even possible to distribute a WinUI3 app without runFullTrust and the answer is: yes it is.
The way to do this is not supported by the tooling and non-obvious like so many things in this context.

For reference, the procedure is as follows:

• Open the Package.appxmanifest
• Remove rescap from the xml namespaces
• Remove rescap from IgnorableNamespaces
• Remove rescap:runFullTrust from the Capabilities element
• Then change the EntryPoint attribute of the Application element
  • from $targetentrypoint$
  • to Windows.PartialTrustApplication
• Recompile
• Re-run (causes re-installation)
• Re-create packages

[softworkz]

I did the above for our WinUI3 app and the observations are as follows:

• When running the app, everything is still working (WebView2 and FFmpegInteropX)
• Creating packages succeeds
• When running WACK
  • The list of tests remains unchanged (compared to using runFullTrust)
  • The test results are the same

As usual, always when you try to find an answer that is expected to be simple, it's getting more and more complicated. Here's a conversation with really good information on the details around packaging and trust levels: microsoft/WindowsAppSDK#410 (comment)

[brabebhin]

So basically runfulltrust has no effect whatsoever?

[softworkz]

It does have an effect, for example with regards to which files and other resources can be accessed. Then you can also choose between containerized and not containerized which is independent of "packaged vs. non-packaged", means. A containerized app doesn't need to be packaged and a packaged app can also run without containerization (sighh...).

It's all in the conversation I have referenced above.

[softworkz]

So basically runfulltrust has no effect whatsoever?

Practically speaking: When you have a WinUI3 app (created from the VS template) and you remove the runFullTrust capability, then it's still not at the same restriction level as UWP apps have (by default).

[brabebhin]

So it's more of a runtime permission, you will get exceptions when you try to do stuff that's not allowed. Has no effect on allowed API in wack.

[softworkz]

With the way I tested (partialTrust), it seems to be allowed to use system dlls and non-WinRT APIs, but it's not allowed to access the file system arbitrarily, so you'd either need permission in the manifest (e.g. for Music or Videos library of the user) or the user needs to confirm access via file picker. Those restrictions do not apply when running fullTrust.

[softworkz]

When I add runFullTrust to a UWP app, then I get the same set of tests in WACK like for the WinUI3 app - i.e. it doesn't test for forbidden APIs:

[softworkz]

I also tried to package the WinUI3 app with the same settings like for UWP apps, but this failed (more or less as expected).

To do this, I specified "windowsApp" and the App class as entrypoint:

This caused the app to fail startup due to "not activatable class 'App' could be found".
Makes sense, because were in the UWP world and everything is COM based.
Then I added CsWinRT with the "IsComponent" variable, which tells that my project's classes should be exposed as WinRT objects.

I actually got this working and came one step further.

The App class got loaded indeed but then it errored due to threading (something like the application object must only be accessed from the main thread) - even though it was on the main thread.
The real problem behind is that UWP uses an MtaThread model and WinUI and StaThread model.

By setting the entrypoint to MtaThread, (just for fun) I even got a tiny bit further, but then it errored like this:

Exception thrown at 0x00007FFB31A1565C (KernelBase.dll) in Emby.Client.WinUI.exe: WinRT originate error - 0x80131515 :
'WinUI: Error creating an UWP Window. Creating an UWP window is not allowed.

It's clear that this cannot work. That probably the reason why they dropped UWP support from WindowsAppSDK. (Project Reunion 0.5) still had it. I would guess they had to make a decision towards STA as MTA would have disabled too many use cases.

[softworkz]

Things are getting more clear now.

While the AppCointainer concept exists for both, UWP and WinUI3, it's not the same thing: They are calling the WinUI3 one "AppContainer Light".

In a UWP AppContainer, the app is fully isolated and all API calls are brokered between the container and the OS via an Out-of-Process COM server (one of the oldest Windows concepts btw.). This requires a lot of marshalling of invocations between inside and outside. Since UWP is multi-threaded (not quite, but MTA), the communication can happen on multiple threads simultaneously, which keeps the impact at least moderate.

But with a single-threaded model, like they chose (or had to choose) for WinUI3, all this communication would have had to be marshalled to the main application thread. And doing so, can severely affect application performance and behavior, especially blocking the UI.

That's why they invented "AppContainer Light", which still does virtualization for file system and registry access (and devices, iirc).

The bottom line is that they were essentially forced to do so, and the direct result of that is that there doesn't exist an equally restricted environment for WinUI3 like there is for UWP.

[brabebhin]

TBH, I was among the ones who was very critical about UWP - not necessarily because of the app model, but because of its overall performance, particularly the file system access. I suppose they just remade the app model instead of fixing issues? So I guess UWP app model was just impossible to adapt for desktop? Anyways, whoever was in charge of UWP failed quite spectacularly.

Truth to be told, as long as you had that uber restrictive sandbox and extremely poor performance, no productivity app could ever be done. Whenever we discussed tech stacks for our clients, we wouldn't even consider UWP. Just like we don't even consider MAUI now. Microsoft can't seem to get a break in mobile.

[softworkz]

TBH, I was among the ones who was very critical about UWP - not necessarily because of the app model, but because of its overall performance, particularly the file system access.

I didn't know that it was slow. I never even got that far that I could have noticed ;-) - I've been on the MS/Windows side of things ever since and I've gone through almost everything that MSDN had to offer (had a collection of MSDN deliveries going over almost 20 years, full of CDs and DVDs, the first two years even 3.5" floppy disks).
But then came UWP and it was the first time that I completely skipped a major strategy of theirs. For me it was not only the AppModel but also all the restrictions and limitations and the lack of anything useful you could build and and build with.
At that time I did a lot of enterprise applications (client/server, database, rich UIs for presentation, SyncFusion, DevExpress controls, etc.). I knew almost every trick and technique for doing things which go beyond the ordinary WinForms and WPF framework abilities.
Then came UWP, and it was just laughable to me. Because instead of "all" and "always" and "anything" it was offering "none", "never" and
nothing. Even the component developers largely skipped it.
They were incredibly arrogant at that time in thinking they would be creating the biggest thing ever which would supersede everything else. At the beginning they even wanted to deprecate WinForms, which - I think - is by now still the biggest.

I suppose they just remade the app model instead of fixing issues? So I guess UWP app model was just impossible to adapt for desktop?

With UWP, they have built their own little island which was isolated and didn't need to interact with anything else, That's why they could go by that different - in theory better - but harder to develop - multi-threading model.

The problem is that it's relatively easy to host a MTA UI component in an STA framework (that's why you can have UWP components in a WinUI app. But it doesn't work the other way round. And I think that's what they realized when they were at v0.5 of Project Reunion.
Re-Union was meant to re-unite the not-much-loved "exotic island" UWP with the desktop APIs. But most of the existing UI and COM APIs have always been STA threaded. If they would have made WinUI3 MTA-threaded, then there wouldn't have been a lot to "re-unite". They would have just reated another version of the UWP island. So it had to be STA.
And when you say UWP file access has been slow already, then it would have been much worse with an STA model and doing out-of-process COM for the containerization.

Anyways, whoever was in charge of UWP failed quite spectacularly.

Most definitely. And the successors didn't do much better, since they even got until v0.5 when they realized that there's nothing to re-unite. Because essentially, they didn't re-unite with UWP. It's quite funny that WinUI2 came to UWP even after they had started WinUI3. Since the developers are the same, they must have somehow slowed down WinUI3 to deliver WinUI2.
It's almost a bit like a magic trick they did: after realizing that they cannot re-unite, they brought something (WinUI2) to UWP which is similar to WinUI3, even though WinUI3 existed earlier, but from the naming (2/3), it looked like they would re-unite UWP/WinUI2 with desktop - by delivering WinUI3.
Could it be that they really created this illusion to cover their failure...?

[softworkz]

Microsoft can't seem to get a break in mobile.

Mobile was their biggest mistake from which that series of failure had begun.

MS was ahead of all others with Windows CE and Windows Phone edition. No Apple, no Android in sight.
But instead of building up on this, they let it go sidewards and when they realized that it's getting traction, they still did not build up on it but instead started development of a totally new mobile OS which took three years after which it still was crap. Those are the years by which they lost everything to Google and Apple.

[brabebhin]

What if we statically link all the correct dependencies at build time for the UWP build, and create a self contained build? Shouldn't that remove the problem completely? And replace it with a more manageable one(disk size)?

[softworkz]

What if we statically link all the correct dependencies at build time for the UWP build, and create a self contained build? Shouldn't that remove the problem completely? And replace it with a more manageable one(disk size)?

I have a addressed that idea right in the first message of this conversation:

We're talking about like 12 functions only which are causing the dependency on the one app-local lib.
If we want to be able to use the UWP build for both, uwp and win32 targets, then it would go only by eliminating those dependencies from the ffmpeg dlls. The first and easiest thought would be to statically link these functions into the ffmpeg libs.
But those functions are: memset, memmove, strchr and the likes. If we would (even manage to) statically link them into the ffmpeg libs, then we would end up with a dependency on kernel32.dll in exchange - which is probably the most forbidden dependency you can even have ;-)

[brabebhin]

It's close, but not quite the same. The assumption is that UWP will work on desktop assuming there's no dependency hell situation. This assumption might not hold water.

At the moment, UWP ffmpeg build links correctly against the proper dlls that are allowed in UWP (so no kernel32 which i;m sure will not work on UWP), with the correct functions (no forbidden stuff). I am going to assume those functions will also work fine on desktop. The idea is to statically link against whatever is being used in the UWP build and avoid the dependency problem.

So none of the bellow:

But it might be possible like this:

You make a copy of the vcxproj and name it ffbridge or something
Then remove all midl stuff and code, but keep the configs
Import those 12 functions and create identical exports for them (maybe aliased)
Then build for UWP and WinUI for all platforms, so you get 8 dlls
Remove the project from the solution and Store those dlls. They won't need to be rebuilt (ideally never)
Now it might get a bit tricky and some fiddling will be required: You need to convince ffmpeg to link to those newly created libs (the UWP versions) on build - instead of the MSVCRT140_APP.dll

[softworkz]

I think the misunderstanding is that I was describing multiple options in the OP, it wasn't all about a single one.

[softworkz]

[softworkz]

You missed the important sentence:

If we would (even manage to) statically link them into the ffmpeg libs, then we would end up with a dependency on kernel32.dll in exchange

In case you wonder why would we would end up with a dependency on kernel32 (or kernelbase), that's quite simple:

Because that's what msvcruntime140_app (and friends) are doing eventually. The functions they provide are not all self-contained. Doing something like memset requires calling into the kernel and potentially additional layers in-between. The app-local function are providing such kind of in-between layer to provide isolation and enforce restrictions.
Linking msvcruntime140_app statically means you would end up with a dependency on kernelbase or kernel32 without going through msvcruntime140_app, that's why I'm saying that a dependeny on kernel32 would be "even more forbidden".

Clear now?

[lukasf]

True, for UWP apps, it is forbidden to statically link the runtime. Otherwise, this could have solved our issues.