Skip to content

Commit

Permalink
mac-capture: Add support for improved window capture in macOS 12.3
Browse files Browse the repository at this point in the history
### Description
Add a new capture plugin called General Capture that allows for capture of an entire desktop, a single window, or all windows of an application. Performs similarly to the existing macOS Display Capture and includes the ability to hide the cursor. Outperforms Window Capture with significantly higher framerates, with lower CPU/GPU utilization in both OBS and WindowServer. New application capture capabilities allow users to show an application and all of its windows without displaying the content of their desktop.
### Motivation and Context
The current implementation of Window Capture uses `CGWindowListCreateImage` to generate individual frames for window capture. This process requries a significant amount of CPU and GPU utilization from both OBS and WindowServer. ScreenCaptureKit allows General Capture to use the same video plugin style as Display Capture with better performance and lower resource utilization. ScreenCaptureKit provides `IOSurfaceRef`s instead of having to render `CGImage`s on a timer.
Display Capture uses `CGDisplayStream` which provides efficient access to the screen buffer. Unfortunately, that means that operations like cursor removal are not possible. ScreenCaptureKit allows for the cursor, applications, or windows to be excluded from a display capture. Therefore, General Capture can provide a desktop capture similar to Display Capture with additional features to the existing Display Capture.
Application captures will allow use cases like application tutorials to include only the application in question and the menu bar. This can prevent accidental display of the desktop, notifications, or windows from other applications that may appear on screen. Window Capture allows users to hide other applications and only show the main window of an application, but any additional windows would have to be manually added. Additionally, the menu bar would not be visible in a Window Capture.
<details>
	<summary>Baldur’s Gate 3 Demo</summary>
![Baldur’s Gate 3- Window Capture vs General Capture Comparison]()
</details>
<details>
	<summary>Shadow of the Tomb Raider</summary>
![Shadow of the Tomb Raider - Window Capture vs General Capture Comparison]()
</details>
<details>
	<summary>Shadow of the Tomb Raider (25% Speed Playback)</summary>
![Shadow of the Tomb Raider - Window Capture vs General Capture Comparison - Running at 25% speed]()
</details>
### How Has This Been Tested
Tested on a 2019 16" Macbook Pro with 2.3 GHz 8-Core i9 and AMD Radeon Pro 5500M graphics and an 8GB M1 Mac Mini on macOS Monetery 12.3.
Games tested include: Dota 2, Shadow of the Tomb Raider, Deus Ex: Mankind Divided, Baldur’s Gate 3, and Civilization 6. Games were run through benchmark suites where available, recorded games in the case of Dota 2, or consistent camera movements in the case of BG3. Window Captures were compared side by side with General Captures where the game was running at 1080p, windowed, at at least 60fps. The resulting encoded videos were compared frame-by-frame to analyze framerate differences. Activity Monitor was used to track CPU, GPU, and RAM utilization during the tests.
Demo videos consist of two shots composited in OBS. Each scene shows a game captured using either a Window Capture or General Capture as noted, and a cropped General Capture of the Activity Monitor at time of capture.
### Types of Changes
- Bug fix (non-breaking change which fixes an issue)
- Performance enhancement (non-breaking change which improves efficiency)
- Code cleanup (non-breaking change which makes code smaller or more readable)
#### Summary of Changes
- `libobs-opengl/gl-cocoa.m` - Removed a requirement that frames sent to `gl-cocoa` be the same dimensions as the original frame. This is a requirement to support resizing windows. Without this change, window streams would freeze if the window was ever resized. Testing seems to show that this change does not introduce any issues.
- `plugins/mac-capture/CMakeLists.txt:15,16` - Clean up consistency of tabs/spaces in `include_directories`.
- `plugins/mac-capture/CMakeLists.txt:28` - Inclusion of new capture into compile list.
- `plugins/mac-capture/CMakeLists.txt:42,53` - Inclusion of new libraries needed for General Capture. `CoreVideo` and `CoreMedia` are needed for managing the `CVSampleBuffer` format of source frame buffers. `ScreenCaptureKit` enables the new capture type. Include `ScreenCaptureKit` as `weak_framework` to support compiling on pre-12.3 systems.
- `plugins/mac-capture/mac-display-capture.m:621` - The `__MAC_10_15` define may not exist on devices running < 10.15, resulting in this section not being appropriately avoided during compilation. Swapping to `101500` should work for all versions of macOS.
- `plugins/mac-capture/mac-general-capture.m` - Inclusion of new capture type. A significant amount of code is similar to existing code in `mac-display-capture.m`, intentionally.
- `plugins/mac-capture/mac-general-capture.m:4,6` - Availability check for weak linking.
- `plugins/mac-capture/mac-general-capture.m:8,10` - Blocking compilation on 12.3 and later. Since the compilation block is here, availability is left unchecked for each call. The warning is ignored for this file to reduce noise of warnings. Future additions to this plugin type will need attention paid to potential unguarded checks in availability after 12.3.
- `plugins/mac-capture/mac-general-capture.m:164,211` - Given the different availability tools provided by `ScreenCaptureKit`, matches must be generated from the way OBS stores windows/displays to the format `ScreenCaptureKit` provides.
- `plugins/mac-capture/mac-general-capture.m:296,312` - Initial fetch of available capture targets when plugin created. Fetches are also run whenever the plugin properties are opened to ensure accurate listings in the properties.
### Future Work
- Localization of General Capture not included in PR.
- OBS might want to exclude itself from captures (or provide an option to do so) to avoid the “Hall of Mirrors” problem of bringing OBS within capture bounds to edit.
-“General Capture” UI is simplistic to duplicate whats already present, we’d expect the project to revisit how it exposes this mode
- Areas of code reuse are intentional to ensure familiarity with future maintainers and current implementations. If the project desires abstractions to reduce duplication, that should be discussed.
### Checklist:
- [x] My code has been run through clang-format.
- [x] I have read the contributing document.
- [x] My code is not on the master branch.
- [x] The code has been tested.
- [x] All commit messages are properly formatted and commits squashed where appropriate.
- [x] I have included updates to all appropriate documentation.
  • Loading branch information
Developer-Ecosystem-Engineering committed Jan 27, 2022
1 parent 0524d9d commit 551c54b
Show file tree
Hide file tree
Showing 5 changed files with 702 additions and 4 deletions.
5 changes: 2 additions & 3 deletions libobs-opengl/gl-cocoa.m
Original file line number Diff line number Diff line change
Expand Up @@ -411,9 +411,8 @@ bool gs_texture_rebind_iosurface(gs_texture_t *texture, void *iosurf)
blog(LOG_ERROR, "Unexpected pixel format: %d (%c%c%c%c)", pf,
pf >> 24, pf >> 16, pf >> 8, pf);

if (tex->width != IOSurfaceGetWidth(ref) ||
tex->height != IOSurfaceGetHeight(ref))
return false;
tex->width = IOSurfaceGetWidth(ref);
tex->height = IOSurfaceGetHeight(ref);

if (!gl_bind_texture(tex->base.gl_target, tex->base.texture))
return false;
Expand Down
11 changes: 11 additions & 0 deletions plugins/mac-capture/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ find_library(AUDIOUNIT AudioUnit)
find_library(COREFOUNDATION CoreFoundation)
find_library(IOSURF IOSurface)
find_library(COCOA Cocoa)
find_library(COREVIDEO CoreVideo)
find_library(COREMEDIA CoreMedia)
find_library(SCREENCAPTUREKIT ScreenCaptureKit)

add_library(mac-capture MODULE)
add_library(OBS::capture ALIAS mac-capture)
Expand All @@ -16,13 +19,21 @@ target_sources(
audio-device-enum.h
mac-audio.c
mac-display-capture.m
mac-general-capture.m
mac-window-capture.m
window-utils.m
window-utils.h)

target_link_libraries(mac-capture PRIVATE OBS::libobs ${COREAUDIO} ${AUDIOUNIT}
${COREFOUNDATION} ${IOSURF} ${COCOA})

if(SCREENCAPTUREKIT)
target_link_libraries(mac-capture PRIVATE OBS::libobs ${COREVIDEO} ${COREMEDIA})

target_link_options(mac-capture PRIVATE SHELL:-weak_framework ScreenCaptureKit)
target_link_options(libobs PRIVATE SHELL:-weak_framework ScreenCaptureKit)
endif()

set_target_properties(mac-capture PROPERTIES FOLDER "plugins" PREFIX "")

setup_plugin_target(mac-capture)
2 changes: 1 addition & 1 deletion plugins/mac-capture/mac-display-capture.m
Original file line number Diff line number Diff line change
Expand Up @@ -618,7 +618,7 @@ static bool switch_crop_mode(obs_properties_t *props, obs_property_t *p,
sprintf(dimension_buffer[3], "%d",
(int32_t)[screen frame].origin.y);

#if __MAC_OS_X_VERSION_MAX_ALLOWED >= __MAC_10_15
#if __MAC_OS_X_VERSION_MAX_ALLOWED >= 101500 // __MAC_10_15
if (__builtin_available(macOS 10.15, *)) {
sprintf(name_buffer,
"%.200s: %.12sx%.12s @ %.12s,%.12s",
Expand Down
Loading

0 comments on commit 551c54b

Please sign in to comment.