[Bug]: Cannot run "512-depth-ema.ckpt" model

[LieDeath]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

I want to run the depth model (512-depth-ema.ckpt (d522022)) of stable diffusion 2.0, but it came to an error.
Though I have run the basic model (768-v-ema.ckpt (2c02b20a)) successfully on previous, the depth one yet not. I emphasize that I put the "512-depth-ema.yaml", which is renamed from "v2-inferance.yaml", correctly in /models/stable-diffusion/, next to the model.
Puzzled, I redownload it, nothing different happen. I stress once again that I have run the basic one smoothly before!

Steps to reproduce the problem

1. Add "512-depth-ema.yaml" and "512-depth-ema.yaml" to /models/stable-diffusion/
2. Go to webui-user.bat and run it
3. select the 512-depth-ema.ckpt (d0522d12) in WEBUI page
4. oops, it crash!!

What should have happened?

I think no error should occur and I can produce depth image by send image in.

Commit where the problem happens

in loading 512-depth-ema.ckpt, the commit hash is ce049c4

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Microsoft Edge

Command Line Arguments

Loading config from: D:\AI6\stable-diffusion-webui\models\Stable-diffusion\512-depth-ema.yaml
LatentDiffusion: Running in v-prediction mode
DiffusionWrapper has 865.91 M params.
Loading weights [d0522d12] from D:\AI6\stable-diffusion-webui\models\Stable-diffusion\512-depth-ema.ckpt
Traceback (most recent call last):
  File "D:\AI6\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 284, in run_predict
    output = await app.blocks.process_api(
  File "D:\AI6\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 982, in process_api
    result = await self.call_function(fn_index, inputs, iterator)
  File "D:\AI6\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 824, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "D:\AI6\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "D:\AI6\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "D:\AI6\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "D:\AI6\stable-diffusion-webui\modules\ui.py", line 1616, in <lambda>
    fn=lambda value, k=k: run_settings_single(value, key=k),
  File "D:\AI6\stable-diffusion-webui\modules\ui.py", line 1457, in run_settings_single
    if not opts.set(key, value):
  File "D:\AI6\stable-diffusion-webui\modules\shared.py", line 478, in set
    self.data_labels[key].onchange()
  File "D:\AI6\stable-diffusion-webui\modules\call_queue.py", line 15, in f
    res = func(*args, **kwargs)
  File "D:\AI6\stable-diffusion-webui\webui.py", line 62, in <lambda>
    shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights()))
  File "D:\AI6\stable-diffusion-webui\modules\sd_models.py", line 292, in reload_model_weights
    load_model(checkpoint_info)
  File "D:\AI6\stable-diffusion-webui\modules\sd_models.py", line 261, in load_model
    load_model_weights(sd_model, checkpoint_info)
  File "D:\AI6\stable-diffusion-webui\modules\sd_models.py", line 192, in load_model_weights
    model.load_state_dict(sd, strict=False)
  File "D:\AI6\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1604, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LatentDiffusion:
        size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 5, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

Additional information, context and logs

The model hash is d522d022, and SHA1: a81e98c0f932a55d8dd08058c1b9ce68525ac6a5
The error seems coming from bad torch size setting.

[smernald]

Version 2.0 works differently from 1.x versions, and right now only the 768-v-ema.ckpt model is supported

[LieDeath]

Version 2.0 works differently from 1.x versions, and right now only the 768-v-ema.ckpt model is supported

Thanks, guy! I have replaced it by extension "depthmap2mask" Extraltodeus.
However, I still want to use it in future...:-(

[LieDeath]

Version 2.0 works differently from 1.x versions, and right now only the 768-v-ema.ckpt model is supported

Now! Boy! I have found a leak of yaml, which can run the 512-basic-ema.ckpt smoothly with webui.
v2-inference-e.zip

[healthyfat]

I have found a leak of yaml, which can run the 512-basic-ema.ckpt smoothly with webui.

Thanks @LieDeath. This yaml config file worked for me.

I was not able to load 512-basic-ema with original instructions.

Where is that file coming from if I may ask?

[AnonymousCervine]

@healthyfat Did you download the right one from the original instructions? They're currently confusingly-phrased, with the location of the 512 file tacked on as an afterthought.

(Amusingly: All of these files get downloaded for us to repositories\stable-diffusion-stability-ai\configs\stable-diffusion anyway when the SD2.0 repo is cloned into a subfolder; it's just a matter of copying them to a place that this repo looks for them)

[healthyfat]

@AnonymousCervine

Did you download the right one from the original instructions?

Yes. I used the config from the link you highlighted. It doesn't work for me.
Only v2-inference-e.yaml does.

[AnonymousCervine]

Interesting. I wonder what the difference between our setups is.

(Probably as much as should be spoken of here; it's off-topic to this issue and probably needs its own?)

[LieDeath]

Interesting. I wonder what the difference between our setups is.

(Probably as much as should be spoken of here; it's off-topic to this issue and probably needs its own?)

I take a review and find the only difference between v2-inference-e.yaml and v2-inference.yaml is a line break at the end of the config file. Perhaps on some devices the missing line break in the v2-inference.yaml could cause problems:-(.

[JaySmithWpg]

I've submitted an improvement that will add support for the 512-depth-ema.ckpt model here: #5542. My fingers are crossed that the merge will go smoothly, because I love this feature. It gives me much more control over the composition.

It looks like you're also using the wrong config. The correct config file for this model is v2-midas-inference.yaml, but it won't work until the change is merged.

[LieDeath]

I've submitted an improvement that will add support for the 512-depth-ema.ckpt model here: #5542. My fingers are crossed that the merge will go smoothly, because I love this feature. It gives me much more control over the composition.

It looks like you're also using the wrong config. The correct config file for this model is v2-midas-inference.yaml, but it won't work until the change is merged.

Big thanks! For your contribution. I am eager to run your solution...

[TWIISTED-STUDIOS]

So I tried the 512 Depth after the Merge, the model loads and the weights attach but then when I try to use it this is given, RuntimeError: Given groups=1, weight of size [320, 5, 3, 3], expected input[2, 9, 64, 64] to have 5 channels, but got 9 channels instead

Anyone have any idea why?

Never mind I'm a clown, I tried to use it in txt2img and it's an img2img model 🤦🏻‍♂️

[DrUmranAli]

So I tried the 512 Depth after the Merge, the model loads and the weights attach but then when I try to use it this is given, RuntimeError: Given groups=1, weight of size [320, 5, 3, 3], expected input[2, 9, 64, 64] to have 5 channels, but got 9 channels instead

Anyone have any idea why?

Never mind I'm a clown, I tried to use it in txt2img and it's an img2img model 🤦🏻‍♂️

I have the same error, however with img2img,....did you do anything else to get it working?

[DobleV55]

I'm having this same issue on a M1 Mac with 512-depth-ema.ckpt)
RuntimeError: Given groups=1, weight of size [320, 5, 3, 3], expected input[2, 9, 64, 64] to have 5 channels, but got 9 channels instead

[chengxy1]

I've encountered a very strange problem. Could you please tell me what the cause is?

*** Error completing request
*** Arguments: ('task(k4w2e2wkjwbegbx)', <gradio.routes.Request object at 0x000001E5EEC54250>, 0, '', '', [], <PIL.Image.Image image mode=RGBA size=1920x1080 at 0x1E56D4B3390>, None, None, None, None, None, None, 4, 0, 1, 1, 1, 7, 1.5, 0.75, 0.0, 512, 512, 1, 0, 0, 32, 0, '', '', '', [], False, [], '', 'upload', None, 0, False, 1, 0.5, 4, 0, 0.5, 2, 20, 'DPM++ 2M', 'Automatic', False, '', 0.8, -1, False, -1, 0, 0, 0, '* CFG Scale should be 2 or lower.', True, True, '', '', True, 50, True, 1, 0, False, 4, 0.5, 'Linear', 'None', '

Recommended settings: Sampling Steps: 80-100, Sampler: Euler a, Denoising strength: 0.8

', 128, 8, ['left', 'right', 'up', 'down'], 1, 0.05, 128, 4, 0, ['left', 'right', 'up', 'down'], False, False, 'positive', 'comma', 0, False, False, 'start', '', '

Will upscale the image by the selected scale factor; use width and height sliders to set tile size

', 64, 0, 2, 1, '', [], 0, '', [], 0, '', [], True, False, False, False, False, False, False, 0, False) {}
Traceback (most recent call last):
File "D:\gitprogram\stable-diffusion-webui\modules\call_queue.py", line 74, in f
res = list(func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^
File "D:\gitprogram\stable-diffusion-webui\modules\call_queue.py", line 53, in f
res = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\gitprogram\stable-diffusion-webui\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\gitprogram\stable-diffusion-webui\modules\img2img.py", line 242, in img2

img
processed = process_images(p)
^^^^^^^^^^^^^^^^^
File "D:\gitprogram\stable-diffusion-webui\modules\processing.py", line 847, in process_images
res = process_images_inner(p)
^^^^^^^^^^^^^^^^^^^^^^^
File "D:\gitprogram\stable-diffusion-webui\modules\processing.py", line 921, in process_images_inner
p.init(p.all_prompts, p.all_seeds, p.all_subseeds)
File "D:\gitprogram\stable-diffusion-webui\modules\processing.py", line 1757, in init
self.image_conditioning = self.img2img_image_conditioning(image * 2 - 1, self.init_latent, image_mask, self.mask_round)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\gitprogram\stable-diffusion-webui\modules\processing.py", line 381, in img2img_image_conditioning
return self.depth2img_image_conditioning(source_image)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\gitprogram\stable-diffusion-webui\modules\processing.py", line 307, in depth2img_image_conditioning
transformed = transformer({"jpg": rearrange(source_image[0], "c h w -> h w c")})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\gitprogram\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\data\util.py", line 22, in call
x = self.transform({"image": x})["image"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\gitprogram\stable-diffusion-webui\venv\Lib\site-packages\torchvision\transforms\transforms.py", line 95, in call
img = t(img)
^^^^^^
File "D:\gitprogram\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\midas\midas\transforms.py", line 168, in call
sample["image"] = cv2.resize(
^^^^^^^^^^^
cv2.error: OpenCV(4.10.0) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\resize.cpp:4030: error: (-215:Assertion failed) func != 0 in function 'cv::hal::resize'