Oversampling without smoothing

[HybridDog]

Issue type

• Feature request

Minetest version

Minetest 0.5.0-dev (Linux)
Using Irrlicht 1.8.4
BUILD_TYPE=None
RUN_IN_PLACE=0
USE_GETTEXT=1
USE_SOUND=1
USE_CURL=1
USE_FREETYPE=1
USE_LUAJIT=1
STATIC_SHAREDIR="/usr/share/minetest"

OS / Hardware

Operating system: Lubuntu 17.10
CPU: intel

GPU model: intel graphics chip
OpenGL version: 4.5 (Core Profile) Mesa 17.4.0-devel - padoka PPA

Summary

I think the algorithm I tried for mipmapping (#6917) works great for oversampling.
Please view the screenshots in 1:1 resolution or with nearest neighbour upscaling (integer or big scaling factor).

nearest neighbour downscaled (corresponds most to no oversampling):

ssim based downscaled:

As you can see, oversampling with this downscaling algorithm

• increases details, see the tulip,
• rids Moire-effects (see the bricks (I shouldn't have used mipmapping here…)),
  if mipmapping is used, there're more details instead of Moire effect reduction,
• has an anti-aliasing effect,
• does not smooth the image (or insignificantly), this would happen with linear or bicubic downscaling

It can be implemented with parallelization (shaders), the speed of the algorithm lies between linear and bicubic downscaling.
Also, if implemented on the gpu it maybe doesn't have a noticeable performance impact because minetest is cpu-intensive.
A window resolution threshold setting could be added so that oversampling is only enabled when the resolution is low enough. 2x oversampling means four times as many pixels need to be rendered (in addition to the downscaling overhead) and fortunately oversampling is especially useful when the game is rendering for low resolution output.

Additional informations

Here is a list of some implementations of SSIM-based perceptual downscaling:

• A Shader for mpv: https://gist.github.com/igv/36508af3ffc84410fe39761d6969be10
• My implementation in C: https://gist.github.com/HybridDog/dd95a99d411972f030fb18543280ad60
• My implementation for SSAA in Minetest: https://github.com/HybridDog/minetest/tree/m_oversampling_new

[Calinou]

Remember that even 2× SSAA is very expensive, as it requires rendering four times the original amount of pixels. For example, to output to a 1920×1080 window, you need to render in 3840×2160. Since most people with high-end graphics cards have displays with a resolution higher than 1920×1080, it's going to be even worse for them – keeping above 60 FPS will become difficult on crowded worlds.

[HybridDog]

You're right, I think there're also graphic chips which don't support rendering at very high resolution.
Still, if you have a fast graphics card, the cpu directs the amount of lag in minetest (other than in sauerbraten).

[Calinou]

You're right, I think there're also graphic chips which don't support rendering at very high resolution.

Those graphics cards would be too slow to render high resolutions at playable framerates anyway.

[nerzhul]

please respect the new issue template...

[paramat]

Some of the template sections will be irrelevant for some issues. Version / OS / hardware is mostly for bugs.

[HybridDog]

I thought about adding a configurable threshold resolution, oversampling is automatically en-/disabled when changing the window resolution.

[paramat]

Issue seems useful to keep open for consideration.

[Calinou]

I still like the idea of enabling SSAA manually with a setting like oversampling = 2 in minetest.conf, as recent rendering optimizations make this more viable on high-end machines.

[devshgraphicsprogramming]

Properly configured Anisotropic Filtering is free

[Calinou]

@devshgraphicsprogramming Anisotropic filtering doesn't solve this issue fully when using nearest-neighbor interpolated textures 🙂

Remember that MSAA also only operates on geometry, it doesn't affect the "insides" of textures.

[devshgraphicsprogramming]

OpenGL has two texture filtering modes that can be set separately (which irrlicht doesn't expose):

• Minification (when you go away from the texture)
• Magnification

you could set mag filter to NEARST, and min to LINEAR_MIPMAP_LINEAR

Also Irrlicht SVN maybe has direct mip-map control, what @HybridDog is proposing is simply generating all the mip-maps properly (offline) with nice filters and uploading them explicitly instead of relying on OpenGL mipmap generation hints to do the work for you from the base level.

There's lots of tools (GIMP, Nvidiai Texture Tools, AMD Compressonator) that willl generate mip-maps of textures for you with nice filters like Lanczos or Kalman (or Kelemen, don't remember how its spelt)

MSAA has nothing to do with this, also MSAA is an arcane technique that belongs to the 2000s

[HybridDog]

OpenGL has two texture filtering modes that can be set separately […]

Does Minetest use LINEAR_MIPMAP_NEAREST when mipmapping is enabled (and the magnification filter is NEAREST)?

[…] what @HybridDog is proposing is simply generating all the mip-maps properly (offline) […]

That's another issue: #6917
I've opened this issue separately for oversampling.

[HybridDog]

I have implemented the oversampling with shaders in an opengl example program:
ssim_based_realtime_snippets_20191225.zip

I don't know how to put this into minetest, which uses Irrlicht. The code changes could involve rendering in a higher resolution (minetest already has an undersampling setting, where it renders in lower resolution), and applying the downscaling shaders (this should be similar to adding post-processing).

[grapereader]

Interestingly, your changes to RenderingCorePlain are all that is necessary to add in an FXAA shader. grapereader@7ba37fe

So that's fun 😅

Would be awesome if Minetest supported screen space post-processing more cleanly so it didn't have to be hacked in like this. But I digress.

[Calinou]

@grapereader I agree it would be nice to offer a built-in FXAA option, since it can look quite good at 1440p or 4K. Feel free to open a pull request for this 🙂

[HybridDog]

There's an abandoned Pull Request for post processing: #9895

[grapereader]

I've been playing around some more and honestly, simple SSAA (as implemented here) works great at 1080p and should be an option in the base game. Since Minetest rendering is already so fast, I easily get 60fps despite the double size render. And it just looks beautiful.

On the other hand people with 4K would probably want something faster like FXAA. That's also doable.

I spent all night trying to get SMAA to work hoping that would be the one-size-fits-all solution but it's been a crap shoot. Pipeline works great up to and including edge detection but I can't get the damned thing to calculate weights properly. It's just too complex to grapple with. Oh well.

Tested some other single-pass algorithms like DLAA, DDAA, NFAA. They work fine but FXAA is still king here. DDAA is a close second.

Still, we need a generalized post-processing pipeline before any of this could be merged. I checked out #9895 (thanks for the link). It is on the right path. It adds a new RenderingCore, which is exactly how I would have done it. I don't know if the complaints against the additional 3D mode (which seem to have been the main blocker) are tenable without either:

• a) Adding a new rendering abstraction that executes after a RenderingCore (which seems like overkill just to preserve esoteric modes like anaglyph, and might actually break some of the more advanced post-processing requiring things like depth buffer, which the existing RenderingCores butcher to high heaven)

• b) Somehow re-working these modes to be extensions that can be passed -into- our new RenderingCore. Again, is this really worth it?

• c) ???

At this point I'm not sure if I should be commenting here or on that PR. At least it is linked here, now, because it was mentioned.

[grapereader]

Some screenshots just for fun.

Linear SSAA

FXAA

Phenomenal. MSAA's worst nightmare.

[HybridDog]

I have reimplemented SSAA with SSIM-based perceptual downscaling for the current Minetest version: https://github.com/HybridDog/minetest/tree/m_oversampling_new.
I don't know if it's implemented correctly of if I have made mistakes.
The images should be viewed in 1:1 resolution without scaling.

By the way: I noticed that I've accidentally swapped the EOTF and OETF in the colour conversions before in my old implementation, so I've hidden the screenshots above now.

No anti-aliasing

SSAA currently available in Minetest

This image looks darker because downscaling is done without gamma correction.

SSAA with SSIM-based perceptual downscaling

All YCbCr channels sharpened

Only the Y channel sharpened

No sharpening

This should be the same as SSAA currently available in Minetest except that it has gamma correction.

[HybridDog]

I've experimented with different colour spaces, different values for the numerical precision problem prevention, and tried mpv's shader.
For the experiments I've done small adjustments to my implementation in C.
The full images are here: ssim_based_downscaling_experiments.zip.
Below are my observations regarding colour spaces and the numerical precision problem prevention.

Effect of the chosen colour space

If I use linearised sRGB instead of linear YCbCr, visible colour fringes appear, such as blue pixels in the bricks and red pixels in the text which was originally just white on bright blue background.
If I do the sharpening only on the Y channel, there is less introduced colour contrast and lower memory and computation requirement. I don't know if this looks better or worse than doing it on all YCbCr channels.

Downscaled text; image order: RGB, YCbCr, YCbCr with only Y sharpened

Downscaled bricks; image order: RGB, YCbCr, YCbCr with only Y sharpened

Effect of the threshold to prevent numerical errors

There is a threshold which is required to prevent numerical errors. If the image is stored with 16 bit floats instead of 32 bit floats, I need to increase this threshold to 0.0005 to avoid artifacts, so a lower threshold requires more memory to store images.
In my implementation, if the threshold is not exceeded at a pixel, there is a fixed sharpening (r_all[i] = 2.0f;); I don't know if sharpening in this case is good or bad behaviour.
With a 0.0005 threshold, the i and n melt together in Minetest in the text example and there is a loss of detail in the bricks example. With a 0.05 threshold, the algorithm no longer gives good results, degenerating to image sharpening (in my implementation).
If the downscaling algorithm is used for SSAA, perhaps it's better to use 32 bit floats and sharpen only the Y channel with a lower threshold than to use 16 bit floats and sharpen all YCbCr channels, or perhaps it's not.

Downscaled text with different threshold values; image order (thresholds): 0.000001, 0.0005, 0.05:

Downscaled bricks with different threshold values; image order (thresholds): 0.000001, 0.0005, 0.05:

[HybridDog]

If the downscaling algorithm is used for SSAA, perhaps it's better to use 32 bit floats and sharpen only the Y channel with a lower threshold than to use 16 bit floats and sharpen all YCbCr channels, or perhaps it's not.

I've changed my implementation to use 0.000001 and sharpen only the Y channel and I think it looks better. A backup of the previous implementation which uses the 0.0005 thresholds and sharpens all YCbCr channels is now in my m_oversampling_ycbcr_all_channels branch.