Github Actions: automated xformers builds for all platforms!

[pamparamm]

Problem

There are several problems with xformers build process and distribution:

• On Windows, we rely on prebuilt wheels. Though they appear to be safe, the process of building them is not transparent for users.
• On Linux, we rely on a handy guide and manual compilation, which makes installation process a little bit tedious.
• If we update python, torch or CUDA versions, both prebuilt and manually built xformers wheels may become incompatible, meaning manual compilation for any platform.
• Any xformers update leads to same compilation problems.

Solution

I think instead of manually compiling xformers, we can use GitHub actions to make python wheels.
This approach has multiple benefits:

• Transparent process of compilation using GitHub Workers (completely free in public repos): no 🥒pickle🥒 threats in wheels.
• Automated builds for any given platform/python/torch/cuda combination.
• Any xformers update will trigger action to build wheels for all configurations.

Example

Job with compiled xformers artifact (RTX 20xx, Python 3.10, CUDA 11.3.1, torch 1.12.1)

Workflow file

What can be improved or changed

Based on feedback, we can:

• Maintain and improve example workflow, providing access to repo if necessary, and make PR with replaced xformers distribution using our artifacts.
• Make PR to add similar actions to @AUTOMATIC1111 repo and use artifacts from here.
• Make PR to add similar actions to @C43H66N12O12S2 repo and use artifacts from here.

Also instead of artifacts we can make releases tied to commits in original xformers repo, but this IMHO will not fit in main WebUI repo.

[camenduru]

nice build.yml and they started compiling their code https://anaconda.org/xformers/xformers

[kingwingfly]

I wonder if I run conda install -c "xformers/label/dev" xformers, the result equals to I build xformers manurally？

[C43H66N12O12S2]

I'm too busy right now. If nobody else is interested, I'll work on this in a week or two.

[anwac]

anecdotal, n = 1, but in my specific situation (ubuntu, 3090) I spent a week trying to make xformers work, and could not.

A working wheel 'out of the box' (too stupid to do that kind of work myself) would make at least one person very, very happy. Happy to spend some time with tests etc if that is helpful . Am a git(hub) noob though ...

[pamparamm]

Okay, so currently we provide four wheels for different configurations and they seem to work:

• windows py3.10 cuda11.3 torch1.12
• linux py3.10 cuda11.3 torch1.12
• windows py3.8 cuda11.7 torch1.13
• linux py3.8 cuda11.7 torch1.13

All can be downloaded from this job: https://github.com/ninele7/xfromers_builds/actions/runs/3351726935

@anwac, if you want to help, you can try version for linux. Assuming you have python 3.10 and cuda 11.3, all you need to do is to download xformers-wheel-ubuntu-latest-3.10-11.3.1 archive, unpack it somewhere (but don't rename .whl file or it will raise an error during pip install) and do the following:

1. Make sure you are in WebUI's root folder and in correct python environment: ./venv/scripts/activate
2. If you have xformers installed, delete them: pip uninstall xformers
3. Install xformers from whl file: pip install xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl
4. Don't forget to put in webui-user.sh --xformers or --force-enable-xformers flag

[pamparamm]

All wheels are optimized for CUDA arch 7.0-8.6, you can check your compute capability here

Are there any plans to optimize for 8.9? Or will we need to do that ourselves manually?

Will try to, but there are some limits with RAM during compilation, can't guarantee

[pamparamm]

All wheels are optimized for CUDA arch 7.0-8.6, you can check your compute capability here

Are there any plans to optimize for 8.9? Or will we need to do that ourselves manually?

Based on wiki, 8.9 requires minimal version of CUDA 11.8 and I'm not sure this version is currently supported by libraries used in Stable Diffusion. You can test existing artifacts, they might be forward compatible. (Also if you are on Windows using 40xx series GPU, try to disable Hardware-accelerated GPU scheduling for extra performance gain)

[pamparamm]

cool 🎉 please also add linux py3.7 cuda11.3 torch1.12 for colab

I made a build for this configuration, you can access it here. Can you test it, or do you need the wheel to be in releases tab?

[anwac]

sorry for the stupid question, how would I download the whl from the job at https://github.com/ninele7/xfromers_builds/actions/runs/3351726935 ?

I see the four win / py version variations. There are 'colab like' jobs one could run, but I could not figure out how to download from there.

After that your instructions are awesome, and I would love to see how they behave in my Cuda 11.6 situation.
Sorry for troubling you with this kind of basic question.

[pamparamm]

It's no trouble at all, I made a comment below about new releases, right now we don't provide wheels for CUDA 11.6, but tomorrow we probably will.
Assuming your configuration is something like py3.10+cu116+pyt1.12.1, the correct whl for you will be named something like xformers-0.0.14.dev0+cu116-cp310-cp310-linux_amd64.whl. Then you will be able to download it and install following the guide above.

[pamparamm]

Now you can download wheels in releases tab.

It's still WIP, next things to do are:

• Rename wheels according to used cuda versions (so current xformers-0.0.14.dev0-cp310-cp310-win_amd64.whl will become xformers-0.0.14.dev0+cu113-cp310-cp310-win_amd64.whl)
• Add new configurations (such as CUDA 11.6 + pytorch 1.12)
• Make more wheels for python 3.7, 3.8, 3.10
• Make releases after each commit in main branch of xformers repo

All releases will follow this rule for now:

• Wheels for CUDA 11.3-11.6 will require torch==1.12.1
• Wheels for CUDA 11.7 will require torch==1.13.0

[camenduru]

Colab T4
https://github.com/ninele7/xfromers_builds/releases/download/3352937371/xformers-0.0.14.dev0-cp37-cp37m-linux_x86_64.whl

Error completing request
Arguments: ('donald duck', '', 'None', 'None', 50, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 0, 0, 0, False, False, None, '', 1, '', 0, '', True, False, False) {}
Traceback (most recent call last):
  File "/content/stable-diffusion-webui/modules/ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "/content/stable-diffusion-webui/webui.py", line 63, in f
    res = func(*args, **kwargs)
  File "/content/stable-diffusion-webui/modules/txt2img.py", line 48, in txt2img
    processed = process_images(p)
  File "/content/stable-diffusion-webui/modules/processing.py", line 426, in process_images
    res = process_images_inner(p)
  File "/content/stable-diffusion-webui/modules/processing.py", line 519, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength)
  File "/content/stable-diffusion-webui/modules/processing.py", line 646, in sample
    samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
  File "/content/stable-diffusion-webui/modules/sd_samplers.py", line 463, in sample
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "/content/stable-diffusion-webui/modules/sd_samplers.py", line 365, in launch_sampling
    return func()
  File "/content/stable-diffusion-webui/modules/sd_samplers.py", line 468, in <lambda>
    }, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/content/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/sampling.py", line 80, in sample_euler_ancestral
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/stable-diffusion-webui/modules/sd_samplers.py", line 285, in forward
    x_out = self.inner_model(x_in, sigma_in, cond={"c_crossattn": [cond_in], "c_concat": [image_cond_in]})
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/external.py", line 112, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
  File "/content/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/external.py", line 138, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
  File "/content/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddpm.py", line 987, in apply_model
    x_recon = self.model(x_noisy, t, **cond)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddpm.py", line 1410, in forward
    out = self.diffusion_model(x, t, context=cc)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/stable-diffusion-webui/repositories/stable-diffusion/ldm/modules/diffusionmodules/openaimodel.py", line 732, in forward
    h = module(h, emb, context)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/stable-diffusion-webui/repositories/stable-diffusion/ldm/modules/diffusionmodules/openaimodel.py", line 85, in forward
    x = layer(x, context)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/stable-diffusion-webui/repositories/stable-diffusion/ldm/modules/attention.py", line 258, in forward
    x = block(x, context=context)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/stable-diffusion-webui/repositories/stable-diffusion/ldm/modules/attention.py", line 209, in forward
    return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
  File "/content/stable-diffusion-webui/repositories/stable-diffusion/ldm/modules/diffusionmodules/util.py", line 114, in checkpoint
    return CheckpointFunction.apply(func, len(inputs), *args)
  File "/content/stable-diffusion-webui/repositories/stable-diffusion/ldm/modules/diffusionmodules/util.py", line 127, in forward
    output_tensors = ctx.run_function(*ctx.input_tensors)
  File "/content/stable-diffusion-webui/repositories/stable-diffusion/ldm/modules/attention.py", line 212, in _forward
    x = self.attn1(self.norm1(x)) + x
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/stable-diffusion-webui/modules/sd_hijack_optimizations.py", line 227, in xformers_attention_forward
    out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None)
  File "/usr/local/lib/python3.7/dist-packages/xformers/ops.py", line 863, in memory_efficient_attention
    query=query, key=key, value=value, attn_bias=attn_bias, p=p
  File "/usr/local/lib/python3.7/dist-packages/xformers/ops.py", line 313, in forward_no_grad
    causal=isinstance(attn_bias, LowerTriangularMask),
  File "/usr/local/lib/python3.7/dist-packages/xformers/ops.py", line 47, in no_such_operator
    f"No such operator xformers::{name} - did you forget to build xformers with `python setup.py develop`?"
RuntimeError: No such operator xformers::efficient_attention_forward_cutlass - did you forget to build xformers with `python setup.py develop`?

[pamparamm]

Can you please share your ipynb file so I could debug it and try to fix the problem?

[camenduru]

https://github.com/camenduru/stable-diffusion-webui-colab

[camenduru]

https://github.com/camenduru/stable-diffusion-webui-colab/releases

[camenduru]

I build like this !pip wheel git+https://github.com/facebookresearch/xformers#egg=xformers

[anwac]

xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl worked perfectly. Looks like what the result of print(torch.version) would be (1.12.1+cu113) in the fielname is the python version (for me 3.10.6) again.

With 512x512 and a super simple prompt I see 20% better speed and around 10% memory footprint reduction. (3090 ubuntu 22.04)

<ramble_since_happy>
I am particularly delighted: The quest to get xformers going led me into my own valley of tears. Equipped with limited knowledge I got a pretty bloody nose since, as I learned recently, I ran into a bug with nvidia drivers 515.76 and 520.65.01. Those seemingly react to a HDMI cable with simply no longer showing a picture in gfx mode. Crazy that in 2022 switching to a DP cable after a kernel rebuild could have saved me a week of tech ratholes, that all led to a black screen after boot.

This wheel makes me therefor S U P E R happy. Thank you.
</ramble_since_happy>

thanks!!

[pamparamm]

I made repo fork specially for colab builds. @camenduru, can you check if it's working. There was a problem with colab glibc version (blame google for not updating their service). You can download whl from https://github.com/pamparamm/xfromers_builds_colab/releases/download/3355664210/xformers-0.0.14.dev0-cp37-cp37m-linux_x86_64.whl

[camenduru]

working 🎃

[camenduru]

can you add build branch name. maybe first 10 characters like this faa88b1235 this makes devs job easy 🐣
xformers-0.0.14.dev0-faa88b1235-cp37-cp37m-linux_x86_64.whl for https://github.com/facebookresearch/xformers/tree/faa88b123514562dbe8e32ec72a25937b0dd9da5

[C43H66N12O12S2]

I've spent some time on this. A blocker is we need to support Lovelace and PyTorch currently hardblocks anything not in torch.utils.cpp_extensions._get_cuda_arch_flags.supported_arches

I failed to find anyway to modify library files in Github Actions (editing cpp_extension.py would solve this) and I doubt it's possible. Setting a variable alias inside setup.py failed.

PTX is an option, but it would slow down xformers for Lovelace.

[ninele7]

You can store modified cpp_extensions.py in repo, and start action with https://github.com/actions/checkout.

Then you can run build in venv and copy repo version of cpp_extensions over original one.

That should solve the problem, if I understood it correctly.

Other problem might be, that action currently using Cuda-toolkit 11.7 and current script can't install 11.8 which supports Lovelace. We can try to use this fork of original actions script: https://github.com/okazunori2013/cuda-toolkit, which supports newer versions.

[B34STW4RS]

Maybe once more lovelace cards are out in the wild we'll have faster progress on this, because at the moment it is a hassle and a half to get running at full speed. The best I was able to accomplish was dumping the new cuda files into lib and living with 22 it/s single image, or batching jobs at 19it/s (effectively almost doubling the speed but not a workflow compatible with what I'm doing). Spent 4 hours trying to build xformers last night, while completing successfully and installing, I would get the nonsense tensor error pop up and console would nag me to rebuild xformers anytime I tried to generate an image.