Depth Maps, Stereo Image, 3D Mesh and Video generator extension

[thygate]

Create Depth Maps and 3D Stereo Images as side-by-side or anaglyph from generated or existing images with the webui.
Generate an inpainted 3D mesh and render videos.

https://github.com/thygate/stable-diffusion-webui-depthmap-script

[thygate]

Updates

• v0.1.1 bugfixes
  • when not combining, file is now saved as single channel 16 bit
  • fixed overflow issue (see here for details and examples of artifacts)
    
    

[thygate]

Updates

• v0.1.3 bugfix
  • bugfix where some controls where not visible (see issue)
• v0.1.2 new option
  • network size slider. higher resolution depth maps.

[thygate]

Updates

• v0.1.4 update
  • implemented --no-half and precision scope. Should now also work with cards that don't support half precision like GTX 16xx. verified!

edit: verified

[kalkal11]

FYI this is brilliant when used with Photoshop's lens blur, I've used them a few times before but not enough due to the lack of convenience of using cli to do the same thing.

[kalkal11]

https://github.com/vt-vl-lab/3d-photo-inpainting

Perhaps you'd also consider implementing this? As another extension.

Create 3D panning videos from 2D creations. I had a few tests I'd previously done but can't seem to find them sadly.

family_00.mp4

Here's a first party example

[thygate]

I will look into it. The Inpainted 3D mesh could be a nice upgrade for viewing in 3D from the current technique of rendering on a plane with a displacement modifier.

[AugmentedRealityCat]

The Inpainted 3d Mesh it produces is still quite similar to a plane with a displacement. They use a few good tricks to make it better than a simple displacement, but the PLY it produces remains a single polygonal surface. The 4 videos it produces at the end of the process really show the limits of that technique: you can't really move your camera any further than that without breaking the illusion.
But it's still amazing that this is all done from a single picture, and in the right hands it's possible to make something useful out of that PLY model with colors stored at vertex maps.

[cutman100]

I get this error:
WARNING: Could not find the Qt platform plugin "windows" in ""
WARNING: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

[thygate]

I get this error: WARNING: Could not find the Qt platform plugin "windows" in "" WARNING: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Please check the 'Issues' on the project repository: https://github.com/thygate/stable-diffusion-webui-depthmap-script/issues

[thygate]

Updates

• v0.1.5 bugfix
  • Saving as any format other than PNG now always produces an 8 bit, 3 channel RGB image. A single channel 16 bit image is only supported when saving as PNG. (see issue)
• v0.1.6 new option
  • option to invert depthmap (black=near, white=far), as required by some viewers.

[thygate]

Updates

• v0.1.7 bugfixes
  • batch img2img now works (see issue)
  • generation parameters now only saved when enabled in settings
  • model memory freed explicitly at end of script

[thygate]

Updates

• v0.1.8 new options
  • net size can now be set as width and height, option to match input size, sliders now have the same range as generation parameters.
  • better error handling

Added FAQ and Viewing sections to README.md

[thygate]

Updates

• v0.1.9 bugfixes
  • sd model moved to system memory while computing depthmap
  • memory leak/fragmentation issue fixed
  • recover from out of memory error

[thygate]

Updates

• v0.2.0 upgrade
  • the script is now an extension, enabling auto installation.

[thygate]

Updates

• v0.2.1 bugfix
  • Correct seed is now used in filename and pnginfo when running batches. (see issue)

[thygate]

Updates

• v0.2.2 new features
  • added (experimental) support for AdelaiDepth/LeReS (GPU Only!)
  • new option to view depthmap as heatmap
  • optimised ui layout
• v0.2.3 bugfix
  • path error on linux fixed

[thygate]

Big Update

• v0.2.4 HIGH RESOLUTION depthmaps
  • multi-resolution merging is now implemented, significantly improving results!
  • res101 can now also compute on CPU

[thygate]

Updates

• v0.2.5 bugfix
  • error checking on model download (now with progressbar)

[mykeehu]

I was thinking of using it to cut out hair, but it only uses a smooth outline for hair, it can't get the contours down to a fine line. I haven't looked at the 0.2.4-0.2.5 versions yet, but would it be possible to use it for masking and then save it as a transparent PNG?

[thygate]

The newest version implementing multi-resolution merging outputs high resolution depth maps now, and thus has sharper edges. This should already improve your results ..

Cutting the depthmap at some depth level, clipping white and making the rest transparent should not be too hard ..
Is that what you want ?

[mykeehu]

Yes, a simple cutout mask. :) However, maybe @Extraltodeus would be a better target in this case, he made a similar script, I saw him doing absolute black/white masking. I use both of your scripts in parallel for post-processing.

[AugmentedRealityCat]

Yes, a simple cutout mask. :) However, maybe @Extraltodeus would be a better target in this case, he made a similar script, I saw him doing absolute black/white masking. I use both of your scripts in parallel for post-processing.

Same thing over here.
But more often than not I bring the depthmap in photoshop and I adjust it there. That's the step I'd like to be able to do directly in the WebUI if possible.

[AugmentedRealityCat]

The newest version implementing multi-resolution merging outputs high resolution depth maps now, and thus has sharper edges. This should already improve your results ..

Cutting the depthmap at some depth level, clipping white and making the rest transparent should not be too hard .. Is that what you want ?

What is the use of the transparent areas in a mask like this exactly? In my early tests a full white image with a transparency gradient gave me the same results as a second image with the same gradient, but in RGB colors, without any transparency channel. I'd have to test again, but if you have a good usage case, I will test with that instead and learn from your process along the way !

As for myself, what I'd like is a 16 bit color curves control with a GUI and a live histogram, just like in photoshop. Since I suppose this is quite complex, a good first implementation would be a Color Levels effect where you can select the darkest and brightest depth values delimiting the object you want to mask. Then the middle values in between those two parameters would be made white, and both the closest and furthest "borders" of our new mask would be feathered by a user selected value, and that feathering between full-white and full-black would be interpolated in 16 bit using the original depthmap data - this makes a feathering effect that is also 3d in itself, and it can allow for much better compositing in some circumstances.

[Ladypoly]

I just want to say that I love this script. Most of the time I don't even use it for Stable diffusion images but for photos instead.

[AugmentedRealityCat]

If you haven't already, go try the 2.0 Depth model - there is a brand new PR for it over here:
#5542

I can't stop playing with it, it's so much fun.

[AugmentedRealityCat]

@AugmentedRealityCat That's actually a nice idea. I have to look into normal alignment, which shouldn't be a big problem.

Edit: here is the updated verion: https://www.youtube.com/watch?v=AeDngG9kQNI importdepthmap_1.0.3.zip

Thank you so much for this extremely quick response to my feature request ! It's just unbelievable.
That's exactly what I meant, and then some, there is dynamic control of depth and everything - this is better than what I do in C4d, and I thought I was clever !
I hope you'll get more users, but if not know that you have at least one who is very satisfied.

[an303042]

I have to jump in and say how happy I am that I've come across this thread. The depthmap extension is of course amazing, but I also love the blender addon so so much. and the fact that it was written by chatGPT - amazing. Thanks everyone!!

[metzo007]

A tiny automation Addon for Blender. Most of the script was written by chatGPT. I only made the polish and the inputs. https://youtu.be/vfu5yzs_2EU importdepthmap_1.0.2.zip

I get a Error Message using the addon
Report: Error
Python: Traceback (most recent call last): File "CAUserslMetulski AppData1Roaming181ender Foundationeender13.01scripts\addonsNimportdepthmapLinit_.py", line 98, in execute sna_buildsetup_63DA10 File "CAUsergMetulski AppDataNRoamingeender Foundationeender13.0 \scripts1addonsNimportdepthmap1_init_.py", line 126, in sna_buildsetup_63DA1 bpy.ops.object.shade_smooth('INVOKE_DEFAULT, use_auto_smooth=True, auto_smooth_angle=math.radians(60.0)) File "E:1131ende63.01scriptslmoduleslbpylops.py", line 130, in call ret = _op_call(seltidname_pya C_dict, kw, C_exec, C_undo) TypeError: Converting py args to operator properties: : keyword "use_auto_smooth" unrecognized
location: :-1

Can you share your settings like Blender Version; Python Version and Image Size. Thanks.

I will also ask ChatGPT :-)

[Ladypoly]

A tiny automation Addon for Blender. Most of the script was written by chatGPT. I only made the polish and the inputs. https://youtu.be/vfu5yzs_2EU importdepthmap_1.0.2.zip

I get a Error Message using the addon Report: Error Python: Traceback (most recent call last): File "CAUserslMetulski AppData1Roaming181ender Foundationeender13.01scripts\addonsNimportdepthmapLinit_.py", line 98, in execute sna_buildsetup_63DA10 File "CAUsergMetulski AppDataNRoamingeender Foundationeender13.0 \scripts1addonsNimportdepthmap1_init_.py", line 126, in sna_buildsetup_63DA1 bpy.ops.object.shade_smooth('INVOKE_DEFAULT, use_auto_smooth=True, auto_smooth_angle=math.radians(60.0)) File "E:1131ende63.01scriptslmoduleslbpylops.py", line 130, in call ret = _op_call(seltidname_pya C_dict, kw, C_exec, C_undo) TypeError: Converting py args to operator properties: : keyword "use_auto_smooth" unrecognized location: :-1

Can you share your settings like Blender Version; Python Version and Image Size. Thanks.

I will also ask ChatGPT :-)

You have to use Blender 3.3 or up

[AugmentedRealityCat]

website with Depthmap2OBJ !
I found a very useful website where you can upload a depthmap and it accompanying image and get back an OBJ 3d object with you image applied as a UV texture. There is no fancy inpainting or anything like that, but there are some very useful parameters to adjust the 3d object.

Go try this now ! https://depthplayer.ugocapeto.com/
the code repo is over here: https://github.com/spite/depth-player
and there is more info over here: https://3dstereophoto.blogspot.com/2019/01/depthplayerugocapetocom-web-tool-to.html

[thygate]

I will investigate deeper, have not had the time to thoroughly check it out yet. Good find.

[AugmentedRealityCat]

I've just tested the Depthmap loader add-on for Blender written by @Ladypoly and it works very well.

[thygate]

I've added the link to the README too.

[thygate]

Thanks for all the contributions everyone, keep them coming ! =)
I will be looking at getting things running in a separate tab next up.

Updates

• v0.2.6 ui layout and settings
  • added link to repo so more people find their way to the instructions.
  • boost rmax setting

updated README.md with contributions and forks

[AugmentedRealityCat]

I'll try to investigate if there are special uses for your extension when combined with the new PR that allows the use of the 2.0 Depth model for IMG2IMG. I've been playing with it non-stop since I installed it a couple of hours ago - time flies when you have fun !
#5542

[thygate]

Updates

• v0.2.7 separate tab
  • Depth Tab now available for stand-alone (batch) processing

updated README.md with contributions

[thygate]

Updates

• v0.2.8 bugfix
  • boost now also able to run on cpu
  • res101 able to compute on cpu

[thygate]

Updates

• v0.2.9 new feature
  • 3D Stereo (side-by-side) and red/cyan anaglyph image generation.
    (Thanks to @sina-masoud-ansari for the tip! Discussion and examples here)

[thygate]

Updates

• v0.3.0 improved stereo image generation
  • New improved technique for generating stereo images and balancing distortion between eyes by @Semjon00 (See here)
  • Substantial speedup of stereo image generation code using numba JIT

[thygate]

Updates

• v0.3.1 bugfix
  • small speed increase for anaglyph creation
  • clone midas repo before midas 3.1 to fix issue (see here)
• v0.3.2 new feature and bugfixes
  • several bug fixes for apple silicon and other machines without cuda
  • NEW Stereo Image Generation techniques for gap filling by @semjon00 using polylines. (See here) Significant improvement in quality.

[thygate]

midas 3.1 will be implemented asap to bring a permanent fix to the changes that came with it.. meanwhile the install script clones an older commit of the midas repo..

[thygate]

Updates

• v0.3.3 bugfix and new midas models
  • updated to midas 3.1, bringing 2 new depth models (the 512 one eats VRAM for breakfast!)
  • fix Next-ViT dependency issue for new installs
  • extension no longer clones repositories, all dependencies are now contained in the extension

[thygate]

Updates

• v0.3.4 new featues
  • depth clipping option (original idea by @Extraltodeus)
  • by popular demand, 3D-Photo-Inpainting is now implemented
    • generate inpainted 3D mesh (PLY) and videos of said mesh

The long awaited 3D Photo Inpainting is implemented, check the readme#Usage for more info.
More info to follow in readme, on mesh etc ..
Feedback welcome.

You can import this mesh into blender, and use the 'Color Attribute' Node to plug into the Shader 'color' to get the mesh rendering in color, and set the camera to origin and pointing up to align them. Might also need to set the FOV to match.

Who will be first to write a blender import or automation addon with the help of chatgpt ?

[mykeehu]

@thygate I'm just wondering how and with what you made this video. :) I would love to learn.

[ClashSAN]

With depth maps you can create a 3d like experience from an image. This was shown off here, and now I believe @thygate has implemented it in the extension for us.

Example: (Click to expand:)

test.mp4

testcircle.mp4

[thygate]

@mykeehu It was generated using boostinmonoculardepth, res101, and 3dphotoinpainting. All available in the extension as a one click operation.
@ClashSAN Exactly

I have updated the readme#Usage section with the required info.

I've been trying to document as much as possible in the readme, but it has gotten a bit dense indeed. The Changelog could be condensed and moved to a separate file. And I would like to add some additional example images later ..
Suggestions are welcome !

For the description, do you think "Depth Maps, Stereo Image, 3D Mesh and Video generator extension" is adequate ?

[ClashSAN]

yes, I mean on this page: https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Extensions#depth-maps (It's all right though, you don't have to think about it)

[thygate]

ah ok, thanks, I will have to get back to you on that.

[thygate]

Updates

• v0.3.5 bugfix
  • create path to 3dphoto models before download (see issue)

[thygate]

Update

• v0.3.6 new feature
  • implemented binary ply file format for the inpainted 3D mesh, big reduction in filesize and save/load times.
  • added progress indicators to the inpainting process

Blender has no problem importing the binary ply files, and ply ascii file format is still supported for reading.
Currently the output format is always binary, without option, if anyone wants this for compatibility with other software reasons let me know and i'll add the option.

(The large spaceship mesh went from 500MB to 192MB)

[Ladypoly]

That's freaking amazing.
Do you think it is possible to use UVs and textures instead of vertex colors?

And could you separate the generate mesh and video option?

[thygate]

Do you think it is possible to use UVs and textures instead of vertex colors?

That would require UV unwrapping the new mesh and generating a new rgb image that includes the newly inpainted content .. I don't have any plans to do so at the moment, you could do it in blender manually for now ... If there's lasting interest I might consider figuring something out .. (perhaps chatgpt can help make some automation script ;)

And could you separate the generate mesh and video option?

The mesh and video generation are already separate, videos can be rendered from a mesh on the depth tab on the right below the depth output panel.

The mesh generation option does currently also generate 4 demo videos, I will disable these or make it an option to generate them with the mesh .. I'm not sure yet which .. Is that what you meant ?

Suggestions and feedback welcome.

[AugmentedRealityCat]

That's freaking amazing. Do you think it is possible to use UVs and textures instead of vertex colors?

I do that in C4d by creating a new set of UV coordinates and baking the vertex colors into a new texture map based on those UV coordinates.

[thygate]

We could look into doing some sort of uv unwrap and project to the texture.

There are still a lot of 3dphotoinpainting parameters that might be tweakable, i'm still testing ..

Also forgot a setting that should be enabled and repeats some inpainting .. so new version later ..

I think some wiki pages might be in order soon, as the interface gets more and more dense with options, new users might be scared off..

[mykeehu]

I would be interested to know if I can generate the .ply file separately somehow, then create a custom camera movement from it (from where to where and how, e.g. zoom)

[thygate]

The video generation is separate (screenshot: #4252 (reply in thread))

The extension generates 4 demo videos when generating a mesh ... I will likely disable these later.

[AugmentedRealityCat]

There are some simple scripts to define the camera moves for those 4 clips if I remember correctly. I think the amplitude of the movement is defined in one place, and the trajectory (circle, forward, etc.) is defined in another.

[mykeehu]

@AugmentedRealityCat yes, that's what I meant, to make my own pan-and-zoom version in addition to the demo

[thygate]

Yes, the 4 demo video parameters are :

crop border for all: [0.03, 0.03, 0.05, 0.03],

names: ['dolly-zoom-in', 'zoom-in', 'circle', 'swing']
translate x: [0.00, 0.00, -0.015, -0.015],
translate y: [0.00, 0.00, -0.015, -0.00],
translate z: [-0.05, -0.05, -0.05, -0.05],
trajectory: ['double-straight-line', 'double-straight-line', 'circle', 'circle'],

[mykeehu]

@thygate thank you, now I understand better how to set the parameters!

[thygate]

Update

• v0.3.7 new features
  • rembg Remove Background PR by @graemeniedermayer merged
  • setting to flip Left/Right SBS images
  • added missing parameter for 3d inpainting (repeat_inpaint_edge)
  • option to generate demo videos with the mesh

[Ladypoly]

This is no longer an extension. This just an amazing toolkit which found it's way into a stable diffusion web ui.

Can you explain what "pre-depth background removal" does?
I used it with and without and couldn't see any difference. I thought it might skip the depth map process but it was still generated before the background was removed.

[thygate]

Can you explain what "pre-depth background removal" does?

It removes the background on the input image for depth generation, so the whole depth range can be used for the foreground subject.

[thygate]

Update

• v0.3.9
  • use existing/custom depthmaps in output dir for batch mode
  • custom depthmap support for single file
  • wavefront obj output support for inpainted mesh (enabled in settings)
  • option to generate all stereo formats at once
  • bugfix: convert single channel input image to rgb
  • renamed midas imports to fix conflict with deforum
  • ui cleanup

[Ladypoly]

Can we get an option for horizontal tiling to get a seamless depthmap?

[AugmentedRealityCat]

You should look at the asymetric tiling extension - it's old but it's gold !
That's how I manage tiling on the X-axis for the panoramas I'm generating.

https://github.com/tjm35/asymmetric-tiling-sd-webui

[Ladypoly]

This is how i'm doing it currently:
https://imgur.com/a/PMDJF4S

[thygate]

So

• generate depthmap1 of original image
• swap two (horizontal) halves of image to create a second image and create depthmap2
• swap two (horizontal) halves in depthmap2
• blend depthmap1 and depthmap2

any more info on the blending ? Just 50/50 ?

[Ladypoly]

Mostly I'm just using a gradient as mask with 100% opacity in the middle where the seam is and then 0% until the border.
https://imgur.com/a/TXMh2uU

[thygate]

I have set up a discussion with some proof of concept code to follow up on your technique :
thygate/stable-diffusion-webui-depthmap-script#183

[thygate]

Update

• v0.3.10
  • ZoeDepth support (with boost), 3 new models, best results so far
  • better heatmap