improvements for stereo image generation

[semjon00]

Hello!

Thanks to all the contributors of this fantastic technology and the script.

This code adds two new options for stereo image generation:

1. Alternative algorithm for filling in empty space. For each pixel in a line of empty pixels, it colors them depending not only on the colors of pixels to the left and/or right, but also depending on distance to them. This may make the edges less visible/noticeable. The implementation is CPU heavy (slow), would be great if somebody could do the same, but faster.
2. An option to change how the deviation is distributed between eyes. Currently, the right image is copied directly from the source and the left image is changed. This means that all the (inevitable) distortion from filling up gaps will end up in the left image. For me, this is inconvenient, since my main (leading) eye is the left eye - this means the perceived distortion is much more visible to me than to people, whose main eye is right. This option allows to move the distortion to the right image, or split it in-between two images (to my eye, it makes distortion it less noticeable overall).

Hope I did not break anything in the process.

Now some demos :)
Settings: deviation is 12 (24 would be too extreme), res101 model, 384x384 net, BOOST enabled. The picture is taken from the discussion #45.
Current

With option 1 enabled

Option 1 enabled and the deviation is perfectly balanced

Waiting for improvement suggestions :)

Cheers

[thygate]

Hi, thanks for contributing, it's truly amazing how this community keeps coming up with improvements and new ideas !

I've not had the chance to test it or look at it in detail yet, but that makes a lot of sense, and the results look quite good. I was just experimenting with numba to accelerate the stereo image generation because reading/writing single elements in python is indeed painfully slow.

This slight adjustment (took a few shortcuts to test .. ahem) already gave an insane speed increase. Generation time went from several dozen seconds to less than a second.
I'm just gonna dump that code here for future reference and revert to the last commit so i can merge your PR. We will have to re-apply this on your edited code later.

from numba import njit

@njit
def generate_stereo(left_img, depth, ipd, monitor_w):
	#MONITOR_W = 38.5 #50 #38.5
    h, w, c = left_img.shape

    depth_min = depth.min()
    depth_max = depth.max()
    depth = (depth - depth_min) / (depth_max - depth_min)

    right = np.zeros_like(left_img)

    deviation_cm = ipd * 0.12
    deviation = deviation_cm * monitor_w * (w / 1920)

    print("deviation:", deviation)

    for row in range(h):
        for col in range(w):
            col_r = col - int((1 - depth[row][col] ** 2) * deviation)
            # col_r = col - int((1 - depth[row][col]) * deviation)
            if col_r >= 0:
                right[row][col_r] = left_img[row][col]

    #right_fix = np.array(right)
    right_fix = np.copy(right)

    #gray = cv2.cvtColor(right_fix, cv2.COLOR_BGR2GRAY)
    gray = right_fix #cv2.cvtColor(right_fix, cv2.COLOR_BGR2GRAY)

    #rows, cols = np.where(gray == 0)
    rows, cols, x = np.where(gray == 0)
    for row, col in zip(rows, cols):
        for offset in range(1, int(deviation)):
            r_offset = col + offset
            l_offset = col - offset
            if r_offset < w and not np.all(right_fix[row][r_offset] == 0):
                right_fix[row][col] = right_fix[row][r_offset]
                break
            if l_offset >= 0 and not np.all(right_fix[row][l_offset] == 0):
                right_fix[row][col] = right_fix[row][l_offset]
                break

    return right_fix

np.array -> np.copy
skip making gray map (fix!)
np.where returns 3 elements because of diff gray shape

to check : numba also has options to attempt to automatically parallelize the function..

Ok, so I will check it in detail asap an merge the pr..

[semjon00]

Wow, numba speed bonus is indeed insane! Thanks, now I know this thing exists.

I've got carried away when improving the code, probably more than I would like to. Wish I had noticed (and changed) some of these things before creating the initial MR.. Anyways, new changes are:

• apply_stereo_deviation now uses numba
• correctness fix (I think...) for hard_horizontal, now should actually split the gap in two
• a minor change that I forgot to make in the first part of the function (reversing the sweep direction)
• optimizations for both gap filling techniques
• fix for a potential situation where completely black pixels are regarded as gap
• default gap filling technique is changed to soft_horizontal, IMHO it is fast-enough now

[thygate]

Excellent! I see you added the numba jit too. I was looking into cython first, but that would require an installed c compiler and would complicate being multi platform. So some more googling revealed numba which uses llvm. I was expecting a good speed increase as all the interpreted languages struggle with this, but the increase in speed really is insane.

I'll do a full test to check if nothing broke, and update the version and readme

I was aware of that bug with full lines of black, great if that's solved too.

Wait, did I mess up the merge ?

[semjon00]

I left some debug code inside...: apply_stereo_deviation vs apply_stereo_deviation_core. Not critical, but could be removed...

[semjon00]

Thank you for reviewing and accepting my changes!

[thygate]

I'll remove the _core wrap too, also updated the ui in Script.ui to include the dropdown.

You're very welcome, and thank you for the improvements, I've not fully tested it yet but the results look good, and the original method is still supported, so great.

Still testing ..

[tufeixp]

I can see the uneven left and right image problem fixed here, but code removed at commit 58245f7. Why? @semjon00

[semjon00]

@tufeixp I don't really understand what do you mean. What do you mean by "uneven"? Do you suggest that this commit (58245f7) broke it? This commit (as far as I remember) should not have changed any logic, only simplify the code. The removed code was a duplicate from stereo_generation.py. What fill technique do you use?

[tufeixp]

uneven means unbalanced deviation, no matter what hole fill method I choose, the result is unbalanced.

[semjon00]

@tufeixp Please try putting the stereo balance slider in the middle, try different stereo balance values

[tufeixp]

set balance to -0.5, the right should on the right eye image has no gap.

@semjon00 Sure I have played with the balance slider settings as well. The default 0 should be defined as balanced where the gap fix should perceived on both side, however it's only on the left part of the dual images. Drag it to other values won't change it.

[semjon00]

The eyes were swapped, please see

[semjon00]

@MavenDE Just remembered you notified me of the same/similar problem. Please update the extension and see if it fixed the issue.