Allow using images as background in virtual background

[danxuliu]

Fixes #9268

This pull request adds makes possible to use image and video files as virtual backgrounds, instead of only blurring the actual background. The blur strength is also exposed now, so it can be customized if needed. A default blur radius/strength of 10 pixels is used (the same as before), but it can be overriden with a different radius to have stronger or lighter blur effects.

There is no support for animated images, like GIF files (only the first frame would be rendered). Use .webm files instead with type VIDEO instead of IMAGE for animated backgrounds, like a looping animation of sea waves or something like that.

Note that the local video is mirrored for the local user, so the images and videos used for the background of the local video will be mirrored as well. Therefore the images and videos should not contain characters, not only for internationalization reasons, but also because it could be confusing for the local user to see them mirrored.

Finally, internally it is possible to use video streams as a virtual background in addition to video files, but that is not exposed in LocalMediaModel. The reason is that there is no support for setting video streams in a persistent way, as the stream can not be directly stored in the localStorage. It would require storing some special value to signal that a stream needs to be fetched from somewhere (like a camera with certain ID, or from a captured desktop) to then start that stream when the call is started. This would require careful thinking, and it is probably not needed by most users, so... for now only blur, image and video virtual backgrounds are exposed.

How to use from Vue components

• If the local participant is in a call:
  • Similar to how the virtual background is enabled or disabled in the LocalMediaControls or in the TopBarMenu
  • Call LocalMediaModel.setVirtualBackgroundBlur(blurStrength), LocalMediaModel.setVirtualBackgroundImage(imageUrl) or LocalMediaModel.setVirtualBackgroundVideo(videoUrl), depending on the virtual background to set
• If the local participant is not in a call:
  • Similar to how the virtual background is enabled or disabled in the DeviceChecker
  • Call localStorage.setItem('virtualBackgroundType_' + {CONVERSATION_TOKEN}, {VIRTUAL_BACKGROUND_TYPE}) with the desired type, and then depending on the type call localStorage.setItem('virtualBackgroundBlurStrength_' + {CONVERSATION_TOKEN}, {BLUR_STRENGTH}) (for blur type) or localStorage.setItem('virtualBackgroundUrl_' + {CONVERSATION_TOKEN}, {IMAGE_OR_VIDEO_URL}) (for image or video types)
  • It would be recommended, although not strictly needed, to remove the parameters of the other types calling localStorage.removeItem('virtualBackgroundUrl_' + {CONVERSATION_TOKEN}) (for blur type) or localStorage.removeItem('virtualBackgroundBlurStrength_' + {CONVERSATION_TOKEN})` (for image or video types).
  • Be careful not to set a null value for an item, as it will be stored and returned as 'null', rather than as an empty string.

Note that the VirtualBackground node used in the device checker is totally independent from the one used during calls. Therefore, in order to see the virtual background changes in the device checker that node needs to be modified too similar to how it is done in LocalMediaModel:

• Blur:

virtualBackground.setVirtualBackground({
    backgroundType: VIRTUAL_BACKGROUND.BACKGROUND_TYPE.BLUR,
    blurValue: VIRTUAL_BACKGROUND.BLUR_STRENGTH.DEFAULT, // or any other positive integer, depending on the desired strength
})

• Image:

virtualBackground.setVirtualBackground({
    backgroundType: VIRTUAL_BACKGROUND.BACKGROUND_TYPE.IMAGE,
    virtualSource: {THE_URL_TO_THE_IMAGE},
})

• Video:

virtualBackground.setVirtualBackground({
    backgroundType: VIRTUAL_BACKGROUND.BACKGROUND_TYPE.VIDEO,
    virtualSource: {THE_URL_TO_THE_VIDEO},
})

How to test

• Start a call with video
• Enable the virtual background
• Open a browser console
• Execute the following to set an image background (dirty hack to access LocalMediaModel, but it is not exposed in OCA.Talk):
  document.querySelector('.local-media-controls').__vue__.model.setVirtualBackgroundImage('https://{YOUR-NEXTCLOUD-DOMAIN}/apps/theming/img/background/kamil-porembinski-clouds.jpg')
• Leave the call
• Reload the page
• Join the call
• The image background will still be used. To change it again to the blur background execute:
  document.querySelector('.local-media-controls').__vue__.model.setVirtualBackgroundBlur()

[nickvergessen]

Looking at #9251 (comment)
Question: Is it easily possible to apply the filters on the device checker easily or is more effort/restructuring needed there as it's not using call components, views, webrtc or signaling there?

[danxuliu]

Looking at #9251 (comment) Question: Is it easily possible to apply the filters on the device checker easily or is more effort/restructuring needed there as it's not using call components, views, webrtc or signaling there?

I have added an explanation in the pull request description. The device checker is using a VirtualBackground object, so it should be just a matter of calling virtualBackground.setVirtualBackground({ backgroundType: ...}) with the appropriate values.

[nickvergessen]

Code looks good and testing works (when executed in console windows of the correct tab 🤦 )