Error attempting to use with IP Adapter

[bent1e]

I tried to join IPA as an image control, but the following issue occurred and I am not sure if my usage is correct.

code:
ipa_model_path = f'/ipa_models'
device = "cuda"
face_adapter = f'InstantIDmodel/ip-adapter.bin'
controlnet_path = f'InstantIDmodel/ControlNetModel'
controlnet = ControlNetModel.from_pretrained(controlnet_path, torch_dtype=torch.float16)
pipe = StableDiffusionXLInstantIDPipeline.from_pretrained(
"modelscope/YamerMIX_v8", controlnet=controlnet, torch_dtype=torch.float16
)
pipe.load_ip_adapter_instantid(face_adapter)
pipe.load_ip_adapter(ipa_model_path, subfolder="models", weight_name="ip-adapter-plus_sdxl_vit-h.bin")
pipe.cuda()
pipe.set_ip_adapter_scale(0.8)
ip_image = Image.open("ref.png")
ip_image.resize((512, 512))
images = pipe(
prompt="a woman",
negative_prompt=None,ip_adapter_image=ip_image,
image_embeds=face_emb,
image=face_kps,width=width, height=height,
controlnet_conditioning_scale=0.8,
num_inference_steps=num_inference_steps,
guidance_scale=5,
).images

Error:
ValueError: <class 'diffusers.models.unet_2d_condition.UNet2DConditionModel'> has the config param encoder_hid_dim_type set to 'ip_image_proj' which requires the keyword argument image_embeds to be passed in added_conditions

[ResearcherXman]

For now, our pipeline is not fully compatible with IPAdapterMixin in diffusers. In shorts, we rewrite some functions in pipeline_controlnet_sd_xl.py to load InstantID. We are working on it and planning to integrate into diffusers recently.

[bent1e]

How should I modify the code to make it effective? Or do you have a specific time plan?

[ResearcherXman]

If you just want to make IP-Adapter works, you can refer to pipeline and attention files, and add another K,V for IP-Adapter.

[usamaehsan]

can we use Ip adapter using latest pipeline?

i want to use ip adapter to get reference of body and clothes.