pass hints to the encoder

[totaam]

Issue migrated from trac ticket # 1699

component: encodings | priority: major | resolution: fixed

2017-11-24 15:32:46: antoine created the issue

---

See #1694#comment:4 and #999#comment:24

We can add some hints to the window-model to try to help the picture encoders make better decisions.
For example:

• "xterm" / "term" or "terminal" window classes should trigger "text" encoding, with higher quality, and avoid video
• a media player (not sure about the best way to detect those), should trigger video
  etc

The webp encoder (#1694) can use this hint, we can also use this hint for selecting video mode.

[totaam]

2017-12-16 07:49:43: antoine commented

---

Added in r17665, see commit message for details.
An xterm shows up as:

$ xpra info | grep content-type
window.1.content-type=text

A Firefox window correctly shows up as:

$ xpra info | grep content-type
window.1.content-type=browser

And the webp encoder now shows details on "preset" and "image hint":

webp.compress(XImageWrapper(BGRX: 0, 1317, 1513, 13), 1513, 13, False, browser) buf=0x7f072c8624f4
webp.compress config: lossless=0    , quality= 40, method=0, alpha=  0,  0,  0, preset=text    , image hint=graph

@maxmylyn: this is not finished yet, but already a good time to hand this over to you:

• to make sure that your application identifiers are added so that we detect it as the correct "content-type" (ie: "browser" for most of your windows)
• for you to run compression tests using webp on a variety of sample window screenshots to see which ones offer the best results (in terms of efficiency, compression, quality, etc..) - so that we can then tune the webp encoder accordingly

The coarse webp tunings using "preset" and "image hint" may not be enough for our needs - in which case we'll need to enhance the webp encoder logic, see libwebp/enc/config.c for what those preset values do.

[totaam]

2018-01-04 19:08:18: maxmylyn commented

---

Okay part one of comment:2 is done - as far as I can tell most major applications I've tried show up correctly (even a couple Electron applications). I'll write a script to test the webp encoder today or tomorrow depending on when I get to it.

[totaam]

2018-03-08 22:37:51: maxmylyn commented

---

Okay, I've finally gotten around to the second part of comment:2 after much reading (mostly learning Bash, to be honest) I cobbled together a not very pretty bash script that will loop through all the presets and quality from 10 to 100 (inclusive) in increments of 10. It will save the outputted file and logs from said compression into subfolders with the name of the preset. If anyone would like to reproduce my metrics, I'll attach the .sh file. You can replace the cwebp......."image.png" lines with whatever image inputs you'd like - just make sure to change the log file name accordingly. It's not pretty nor is it configurable, but it's for a fairly single-use case. Yes it could have been prettier, yes it could have accepted inputs, but at this point I don't care. I'm sorry in advance.

(There's also a second bash script that just does lossless that I'll also attach)

I created three test images - two that are stills from the Documentary "Inside Job" - I chose this one because I watched it again last week on a whim, had the 1080p version available on my Emby Server, and it's also available free on archive.org or Vimeo - one is just a picture of some scenery from Iceland, and the other is the title text. The third image is a Wikipedia page with a picture on the top right. So we got a nice well-rounded three images to work with. They're titled "Video.png", "Image and Text.png", and "Video Large Text.png"

Also of use: [https://developers.google.com/speed/webp/docs/cwebp] and [https://developers.google.com/speed/webp/docs/using]

NOTE: -v adds more verbose logging of the encoding itself - including encoding time!

There's a lot more flags we can use - but I think implementing the presets and quality level is a great first step.

Anyways, I'll throw together a quick spreadsheet with relevant info.

[totaam]

2018-03-09 03:09:51: maxmylyn uploaded file dowebp.sh (0.7 KiB)

webp compression script

[totaam]

2018-03-09 05:07:03: antoine commented

---

r18670 normalizes the webp quality lower.
Please use a wiki table for the data, and maybe also attach the sample pictures so we can reproduce the data later, and again after webp library updates. (the current version in Fedora 27 is 0.6.1)

[totaam]

2018-03-09 22:41:43: maxmylyn commented

---

Okay I managed to comb through the data and put together a nice spreadsheet. Interestingly, the photo preset consistently has a notably higher encoding time than the others.

I'll attach a .csv of the data, and a .xlsx with charts (made it in Google Drive). Yes I know proprietary formats are bad, but the charts make it pretty obvious what is going on at a glance.

I'll write up a Wiki page shortly. I'll put it in WebpEncodingData.

[totaam]

2018-03-09 22:42:34: maxmylyn uploaded file Webp Compression Data no charts.csv (3.0 KiB)

[totaam]

2018-03-09 22:42:42: maxmylyn uploaded file Webp Compression Data.xlsx (14.2 KiB)

[totaam]

2018-03-09 22:46:39: maxmylyn commented

---

Actually, it turns out you can export Google Sheet charts as a .png, so I'll post those instead. Also the original images as you asked.

[totaam]

2018-03-09 22:47:40: maxmylyn uploaded file 1699 Video Chart.png (40.0 KiB)

Chart of data from Video.png

[totaam]

2018-03-09 22:48:46: maxmylyn uploaded file 1699 Video Large Text Chart.png (34.1 KiB)

Chart of data from Video Large Text.png

[totaam]

2018-03-09 22:49:02: maxmylyn uploaded file Image and Text Chart.png (39.2 KiB)

Chart of data from Image and Text.png

[totaam]

2018-03-09 22:49:48: maxmylyn uploaded file Video.png (2063.4 KiB)

The original screenshot of the video

[totaam]

2018-03-09 23:02:22: maxmylyn uploaded file dowebplossless.sh (0.7 KiB)

The same as the other shell script, but lossless and I fixed a typo

[totaam]

2018-03-10 02:31:58: antoine commented

---

The outlier is the "photo" preset, but even then it doesn't take that much more time to encode (~10%).
So, which presets are useful for which types of content? Do they make a noticeable difference?
I also expect the lossless results to look quite different, judging from the API usage.

[totaam]

2018-03-29 18:11:51: maxmylyn commented

---

Okay I finally got around to visually inspecting the images. I'd attach them here, but all together they're ~40mb even compressed with gzip. It'd be better if you recreated them on your own using the bash script I attached.

Before I get into the subjective, I'll point out that the lossless usually performs similar to 50-70% quality for each preset. I'm honestly not sure if that's a good or a bad thing, but it's worth noting that lossless will perform better than higher quality lossy encodings.

On to the subjective:

From what I can see, looking at the worst cases, icon performs the worst in most cases. It makes text readable with no issue even on the lowest quality, so there's that.

For text-only content, text, unsurprisingly, looks the best. However for regular images, text looks subjectively worse than any of the other encodings - it has a hard time with color gradients - in particular trees look rather dreadful on the lower qualities.

Other than that - I can't really tell that much of a difference between the others - at a glance they all kind of look the same to me. I noticed that picture tens to get the colors right most of the time, so there's that - the others can have the lossy color chunks, but picture tends to get the colors more spot on while ignoring the lines.

From my subjective look - I'd say go with text for text only situations, and then with whatever performs the best for anything else.

[totaam]

2018-03-31 15:39:47: antoine changed owner from antoine to maxmylyn

[totaam]

2018-03-31 15:39:47: antoine commented

---

Before I get into the subjective, I'll point out that the lossless usually performs similar to 50-70% quality for each preset.
"performance" - I assume that you mean compression latency only here? Do you have the resulting file size data?
We care about both compression latency and compression ratio.
The lossless mode has completely different performance characteristics: the "quality" setting is then used to drive compression effort, at least when using the API. (more or less what we usually call "speed" in xpra).
A lossless picture is worth spending a bit more time to compress since we can then skip the whole auto-refresh logic and the eventual lossless refresh that this may trigger later. But not if the resulting compressed data is too much bigger.
We currently switch to lossless mode when the compression request has a quality value higher than XPRA_WEBP_LOSSLESS_THRESHOLD which defaults to 75.

icon performs the worst in most cases
We only use the icon as default preset for very small rectangles, I've made this even stricter in r18926.

For text-only content, text, unsurprisingly, looks the best. However for regular images, text looks subjectively worse than any of the other encodings - it has a hard time with color gradients - in particular trees look rather dreadful on the lower qualities.
That's annoying, as I've made "text" the default. It's not as bad as it sounds because the lossless auto refresh should always eventually fix that (and video is handled separately) - should we change it? To what?

I'd say go with text for text only situations, and then with whatever performs the best for anything else.
Most of the time, unless we detect the application or if it supplies us with a hint, we don't know what we're dealing with so "text" seems like the conservative choice: making text unreadable is worse than color changes I would think?

Minor related fix in r18927.

[totaam]

2018-04-03 18:51:51: maxmylyn commented

---

"performance" - I assume that you mean compression latency only here? Do you have the resulting file size data?
We care about both compression latency and compression ratio.

Right I should specify, my bad.

By performance there I meant compression time - lossless compresses fairly quickly.

As for file size, they're all very similar - here's the table for the default encoding of the image "Image and Text.png":

|| Quality || Filesize in Bytes ||
|| original || 851739 ||
|| 10 || 144804 ||
|| 20 || 173878 ||
|| 30 || 198934 ||
|| 40 || 221090 ||
|| 50 || 237386 ||
|| 60 || 255202 ||
|| 70 || 271838 ||
|| 80 || 314068 ||
|| 90 || 397692 ||
|| 100 || 556888 ||
|| lossless || 281438 ||

For all the presets, I see a very similar trend - a notable increase in file-size as quality increases. However, Lossless seems to hover around the 50-70 quality. Which mirrors the encoding times as reported in the spreadsheets I attached.

---

That's annoying, as I've made "text" the default. It's not as bad as it sounds because the lossless auto refresh should always eventually fix that (and video is handled separately) - should we change it? To what?

picture performs similarly and has an acceptable level of quality for text even at the lowest levels of quality. But,

Most of the time, unless we detect the application or if it supplies us with a hint, we don't know what we're dealing with so "text" seems like the conservative choice: making text unreadable is worse than color changes I would think?

I concur with you - leaving text as the default is a good conservative choice, at least as long as we aren't receiving hints as to the content type for a particular application.

[totaam]

2018-04-03 19:21:35: antoine commented

---

OK then I think we're good, we're not using compression levels above 75%.
If I had more time, I would benchmark the lossless compression with different "speed" values, but this will do for now.

[totaam]

2018-09-09 05:35:22: antoine commented

---

Follow up: #1950