[Bug]: ControlNet Travel script broken? "NameError: name 'self' is not defined"

[marcsyp]

Hi there! Love the prompt travel script, thank you for this!

I am trying to use ControlNet Travel for the first time, and was super excited to get started, but it appears to be broken. Perhaps I am not using it correctly, but I can't find any issue or discussion related to this script, and there's not much in the readme outside of the cool example you've provided, so I'm not sure what to do... I would really love to get something going soon and maybe there's an easy fix!

In any case, here are steps to reproduce:

1. Use txt2img to set up a controlnet that you like, add an image to a controlnet module and play until you're satisfied.
2. Remove the image from the ControlNet module in preparation for batch processing.
3. Select ControlNet Travel script and add the directory path where you are storing your images.
4. Hit Generate, you will get "NameError: name 'self' is not defined"
5. Add your starting image back in to the ControlNet module to see if that's the problem.
6. Hit Generate, you will get "NameError: name 'self' is not defined"

Full error in terminal:

Error completing request
Arguments: ('task(jdnn5h3kpd9x6h8)', 'a portrait of a CEO for a large company, 8k, cinematic, dramatic, late evening light', 'lens flare', [], 20, 0, False, False, 1, 1, 7, 980000670.0, -1.0, 0, 512, 512, True, 512, 512, False, 0.42, 2, 'Latent', 12, 0, 0, [], 9, True, 'keyword prompt', 'keyword1, keyword2', 'None', 'textual inversion first', 'None', 0.7, 'None', True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', False, 'x264', 'mci', 10, 0, False, True, True, True, 'intermediate', 'animation', <scripts.external_code.ControlNetUnit object at 0x00000136AA898700>, <scripts.external_code.ControlNetUnit object at 0x00000136AD6B0D90>, <scripts.external_code.ControlNetUnit object at 0x00000136AD6E18A0>, False, '1:1,1:2,1:2', '0:0,0:0,0:1', '0.2,0.8,0.8', 20, 0.2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0, '', 'None', 30, 4, 0, 0, False, 'None', '
', 'None', 30, 4, 0, 0, 4, 0.4, True, 32, None, False, None, False, None, False, 50, False, 4.0, '', 10.0, 'Linear', 3, False, 30.0, True, False, False, 0, 0.0, 'Lanczos', 1, True, 0, 0, 0.001, 75, 0.0, False, True, 10.0, 30.0, True, 0.0, 'Lanczos', 1, 0, 0, 75, 0.0001, 0.0, False, True, '', False, False, False, False, 'Auto', 0.5, 1, 'linear (weight sum)', '5', 'C:\CEOs', 'Lanczos', 2, 0, 0, 'mp4', 3.0, 0, '', True, False, 'linear', 'lerp', 'token', 'random', '30', 'fixed', 1, '8', None, 'Lanczos', 2, 0, 0, 'mp4', 10.0, 0, '', True, False, False) {}
Traceback (most recent call last):
File "C:\SDW\stable-diffusion-webui\modules\call_queue.py", line 56, in f
res = list(func(*args, **kwargs))
File "C:\SDW\stable-diffusion-webui\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
File "C:\SDW\stable-diffusion-webui\modules\txt2img.py", line 53, in txt2img
processed = modules.scripts.scripts_txt2img.run(p, *args)
File "C:\SDW\stable-diffusion-webui\modules\scripts.py", line 407, in run
processed = script.run(p, *script_args)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 681, in run
gen_image()
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 664, in gen_image
proc = process_images(p)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 240, in process_images
res = process_images_inner(p)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 402, in process_images_inner
samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
File "C:\SDW\stable-diffusion-webui\modules\processing.py", line 869, in sample
samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 358, in sample
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 234, in launch_sampling
return func()
File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 358, in
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 126, in forward
x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict([cond_in], image_cond_in))
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "C:\SDW\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in
setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
File "C:\SDW\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in call
return self.__orig_func(*args, **kwargs)
File "C:\SDW\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
x_recon = self.model(x_noisy, t, **cond)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
out = self.diffusion_model(x, t, context=cc)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 312, in
unet.forward = lambda *args, **kwargs: forward2(unethook, *args, **kwargs)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 215, in forward2
return forward(self, *args, **kwargs)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 195, in forward
h = cfg_based_adder(h, control_in, require_inpaint_hijack)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 48, in cfg_based_adder
if base.shape[0] % 2 == 0 and (self.guess_mode or shared.opts.data.get("control_net_cfg_based_guidance", False)):
NameError: name 'self' is not defined

[marcsyp]

Note: after disabling all other extensions except prompt travel + controlNet, checking boxes and playing around, I got a different error, trying to use HED (but no modules appear to work). Terminal message:

n_stages: 4
n_frames: 19
steps: [5, 5, 5]
seed: 3269063086
subseed: -1
subseed_strength: 0
Loading model from cache: control_sd15_hed [fef5e48e]
Loading preprocessor: hed
0%| | 0/20 [00:00<?, ?it/s]
Error completing request
Arguments: ('task(3gvmgyd5mudw30w)', 'a portrait of a CEO for a large company, 8k, cinematic, dramatic, late evening light', 'lens flare', [], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 512, 512, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, ['Model hash: 9aba26abdf'], 5, <scripts.external_code.ControlNetUnit object at 0x00000136A2C35240>, <scripts.external_code.ControlNetUnit object at 0x00000136AA707070>, <scripts.external_code.ControlNetUnit object at 0x00000136A2C35A20>, False, False, 'positive', 'comma', 0, False, False, '', 1, '', 0, '', 0, '', True, False, False, False, 0, None, False, None, False, None, False, 50, 'linear (weight sum)', '5', 'C:\CEOs', 'Lanczos', 2, 0, 0, 'mp4', 10.0, 0, '', True, False, 'linear', 'lerp', 'token', 'random', '30', 'fixed', 1, '8', None, 'Lanczos', 2, 0, 0, 'mp4', 10.0, 0, '', True, False, False) {}
Traceback (most recent call last):
File "C:\SDW\stable-diffusion-webui\modules\call_queue.py", line 56, in f
res = list(func(*args, **kwargs))
File "C:\SDW\stable-diffusion-webui\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
File "C:\SDW\stable-diffusion-webui\modules\txt2img.py", line 53, in txt2img
processed = modules.scripts.scripts_txt2img.run(p, *args)
File "C:\SDW\stable-diffusion-webui\modules\scripts.py", line 407, in run
processed = script.run(p, *script_args)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 681, in run
gen_image()
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 664, in gen_image
proc = process_images(p)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 240, in process_images
res = process_images_inner(p)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 402, in process_images_inner
samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
File "C:\SDW\stable-diffusion-webui\modules\processing.py", line 869, in sample
samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 358, in sample
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 234, in launch_sampling
return func()
File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 358, in
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 126, in forward
x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict([cond_in], image_cond_in))
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "C:\SDW\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "C:\SDW\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in
setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
File "C:\SDW\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in call
return self.__orig_func(*args, **kwargs)
File "C:\SDW\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
x_recon = self.model(x_noisy, t, **cond)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
out = self.diffusion_model(x, t, context=cc)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 312, in
unet.forward = lambda *args, **kwargs: forward2(unethook, *args, **kwargs)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 215, in forward2
return forward(self, *args, **kwargs)
File "C:\SDW\stable-diffusion-webui\extensions\stable-diffusion-webui-prompt-travel\scripts\controlnet_travel.py", line 147, in forward
control = param.control_model(x=x_in, hint=param.hint_cond, timesteps=timesteps, context=context)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 115, in forward
return self.control_model(*args, **kwargs)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\cldm.py", line 368, in forward
emb = self.time_embed(t_emb)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\container.py", line 204, in forward
input = module(input)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\SDW\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 307, in lora_Linear_forward
return torch.nn.Linear_forward_before_lora(self, input)
File "C:\SDW\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_addmm)

[marcsyp]

automatic1111
commit: 22bcc7be

prompt travel:
d3c8944 (Mon Apr 10 10:37:55 2023)

[marcsyp]

FYI -- I have a hunch that the tensors error above is a result of a bug in ControlNet after turning on and off Low Vram mode. Fresh loads always give the the Self not defined error.

[Kahsolt]

Thanks for your report~ I will check them one by one.

That ControlNet Travel script is so far an experiment, not fully tested, not stable. I wrote it one week ago to quickly verify the idea's viability. From yesterday on I resumed to reformat it, trying to find a more neat way for implementation.

You really feedback with plenty useful debug info, and understand to ask questions in a smart way, nice~ 🤤

[marcsyp]

No worries -- thank you for your contributions to the community! I am very excited about controlnet travel as an idea, I think it could have some really cool use cases that are not possible with other methods. Looking forward to your updates! I'm happy to be a tester. :)

[Kahsolt]

😆 I just got it refactored, add trivial options, and works fine on my machine now.
Give it a try, I'm looking forward to your feedbacks~