r/StableDiffusion Sep 10 '22

Prompt-to-Prompt Image Editing with Cross Attention Control in Stable Diffusion

217 Upvotes

24

u/bloc97 Sep 10 '22

Github: https://github.com/bloc97/CrossAttentionControl

Last Post: https://www.reddit.com/r/StableDiffusion/comments/x98py5

Original Paper (Not mine, give your thanks to original authors' idea): https://arxiv.org/abs/2208.01626

1

u/Lirezh Oct 06 '22

If those images represent what that method generally produces (not a 1:100 cherry pick) that looks like a huge step forward for AI art.
It's been 1 month, is there any modification of a webui frontend (like the one from AUTOMATIC1111) that incorporates that code ?

1

u/bloc97 Oct 06 '22

The input prompt needs to be cherry picked, because a bad prompt edited is still bad, but the prompt to prompt part was not, I just chose three animals at random.

9

u/Zertofy Sep 10 '22

That's really awesome, but I want to ask some questions?

What is needed for this to work? We have initial prompt, resolution, seed, scale, steps, sampler, and resulting image of course. Then we somehow fixate general composition and change prompt, but leave everything else intact? So the most important elements are prompt and resulting image?

Can we take non-generated picture, write some "original" prompt and associatiate them with each other, then change prompt and expect that it will work? But what with all other parameters...

Or this is what will be achieved in img2img?

Or maybe I completely wrong and it's working in absolutely different ways?

30

u/bloc97 Sep 10 '22

First question: Yes, right now the control mechanisms are really basic, you have a initial prompt (that you can generate to see what the image looks like), then a second prompt that is an edit of the first. The algorithm will generate your second prompt so that it looks as "close" as possible to the first (with the concept of closeness being encoded inside of the network). You can also tweak the weights of each token, such that you can reduce or increase its contribution on the final image (e.g you want less clouds, more trees). Note that tweaking the weights in attention space gives much better results than editing the prompt embeddings, as the prompt embeddings are highly nonlinear and often editing them will break the image.

Second question: Yes, but not right now. What everyone is using as "img2img" is actually a crude approximation of the correct "inverse" process for the network (not to be confused with textual inversion). What we actually want for prompt editing is not to add random noise to an image but find which noise will reconstruct our intended image and use that to modify our prompt or generate variations. I was hoping someone would have already implemented it but I guess I can give it a try when I have more time.

Also, because stable diffusion is slightly different to what I guess was Imagen used in the paper, we have a second self-cross-attention layer, which can be controlled by using an additional mask (that is not yet implemented right now), that means that if image inversion is implemented correctly, we could actually "inpaint" using the cross-attention layers themselves and modify the prompt, this should give us much better results than simply masking out the image and adding random noise...

Exciting times ahead!

7

u/Aqwis Sep 11 '22 edited Sep 11 '22

Regarding point 2 here, is this as simple as running a sampler "backwards"? I made a hacky attempt at modifying the k_euler sampler to run backwards, like so:

s_in = x.new_ones([x.shape[0]])
sigmas = denoiser.get_sigmas(50).flip(0)

for i in range(1, len(sigmas)):
    x_in = torch.cat([x] * 1)
    sigma_in = torch.cat([sigmas[i] * s_in] * 1)
    cond_in = torch.cat([uncond])

    c_out, c_in = [K.utils.append_dims(k, x_in.ndim) for k in denoiser.get_scalings(sigma_in)]
    t = denoiser.sigma_to_t(sigma_in)

    with autocast('cuda'):
        eps = model.apply_model(x_in * c_in, t, cond=cond_in)

    denoised = x_in + eps * c_out
    d = (x_in - denoised) / sigma_in
    dt = sigmas[i] - sigmas[i - 1]

    x = x + d * dt

...and indeed, if I run a txt2img with the output of this as the initial code (i.e. initial latent) I get something that looks a lot like (a somewhat blurry version of) the image I started with (i.e. input into the code above). Not sure if I did this right or if it just happens to "look right" because I added an insufficient amount of noise to the initial image (so that there's still a lot of it "left" in the output of the above code).

6

u/bloc97 Sep 11 '22

This might be how they did inversion in the DDIM paper, but I couldn't find the exact method except a vague description of the inverse process "by running the sampler backwards" just like you described.

Edit to quote the paper: "...can encode from x0 to xT (reverse of Eq. (14)) and reconstruct x0 from the resulting xT (forward of Eq. (14))", page 9 section 5.4

6

u/Aqwis Sep 11 '22

Played around with this a bit more – if I do the noising with 1000 steps (i.e. the number of training steps, instead of 50 above), I get an output which actually "looks like" random noise (and has a standard deviation of 14.6 ~= sigma[0]) but which if used as starting noise for an image generation (without any prompt conditioning and with around 50 sampling steps) actually recreates the original image pretty well (and it's not blurry as when I used 50 steps in the noising)!

Not sure why it's so blurry when I use only 50 steps instead of 1000 to noise it, I'd expect the sampler to be able to approximate the noise using just a few dozen steps roughly as well as it's able to approximate the image when run in the "normal direction". The standard deviation of the noise is only around 12.5 or so when I use 50 steps instead of 1000, so maybe I have an off-by-one error or something somewhere that results in too little noise being added.

4

u/bloc97 Sep 11 '22 edited Sep 11 '22

Great, that's exactly what the authors observed in the DDIM paper! If you don't mind, you are free to setup a quick demo with maybe one or two examples and push it to the github, that would be super cool for everyone to use!

Edit: And for the reason behind why 50 steps doesn't work as well, I guess maybe is that the forward process uses many tricks for acceleration while the inverse process was pretty much neglected and was not optimized (remember the first paper on diffusion models actually needed 1000 sampling steps too for good results), so you actually need to perform the diffusion correctly, for now (eg. 1000 steps).

6

u/Aqwis Sep 11 '22

Yeah, I'm generating a few examples now, and I'll post something in this subreddit and some code on Github later tonight. I didn't actually try your cross attention control code yet, I'll have to do that as well and see how all this fits together. :)

3

u/bloc97 Sep 11 '22

Sounds good, your inversion code can definitively be used standalone but it would be so cool to use it to edit an image!

3

u/ethereal_intellect Sep 11 '22

Wonder if the inversion code could be used to style transfer like in https://github.com/justinpinkney/stable-diffusion . Take clip1 embedding from image1, reconstruct the noise1, take image 2, find clip2, and recreate from noise1 to get a style2 result. Still only just read about it so haven't thought it through, but the reconstruction idea seemed very useful. I will think about it but i'm not sure i'm up to the task of coding it up/trying it out myself

3

u/Zertofy Sep 10 '22

Cool! Also, is it take the same time to generate as usual image? Probably yes, but just to be sure. Some time before I see post here about video editing, and one of the problems was the lack of consistency between frames. I proposed use of the same seed, but it give only partial result. May this technology be the missing element for this?

Anyway, it's really exciting to see how people explore and upgrade SD in real time. Wish you success i quess

5

u/bloc97 Sep 10 '22

It is slightly slower, because instead of 2 u-net calls, we need 3 for the edited prompt. For video, I'm not sure if this can achieve temporal consistency, as the latent space is way too nonlinear, even with cross-attention control you don't always get exactly the same results (eg. backgrounds, trees, rocks might change shape when you are editing the sky). I think hybrid methods (that are not purely end-to-end) will be the way forward for video generation. (eg. augmenting stablediffusion with depth prediction and motion vector generation)

2

u/enspiralart Sep 12 '22

That augmentation, how do you think it should be gone about? For instance, a secondary network that feeds into the U-Net and gives it these depth and motion prediction vectors, which can be used to change the initial latents such that an image is generated from one frame to the next with roughly the same image latent, but motion vectors warping that image? Or yes, how?

2

u/bloc97 Sep 12 '22

I mean, some specific use cases such as animating faces, image fly through and depth map generation for novel view synthesis already exists. To generate video we probably need some kind of new diffusion architecture that can generate temporally coherent images, of which the data can be taken from YouTube, wiki commons, etc. But I don't think our consumer GPUs are powerful enough to run such a model.

2

u/enspiralart Sep 12 '22

There's an amazing conversation going on about it in the LAION discord group video-CLIIP

https://twitter.com/_akhaliq/status/1557154530290290688 this is from that group

Maciek — 08/10/2022 ok so they basically do what we've already done more thoroughly. Architecture is practically the same as well:
"we employ a lightweight Transformer decoder and learn a query token to dynamically collect frame-level spatial features from the CLIP image encoder"
this is just this - https://github.com/LAION-AI/temporal-embedding-aggregation/blob/master/src/aggregation/cross\_attention\_pool.py they
also just do action recognition but they do it on K400 which is easier.
I guess all the more evidence that this approach works.

LAION Discord video-clip group: https://discord.com/channels/823813159592001537/966432607183175730

1

u/TiagoTiagoT Sep 10 '22

Would it be possible to somehow freeze stuff that is not identified by the AI as being what is being changed, sorta like masking but at a deeper level and done automatically?

1

u/Zertofy Sep 10 '22

Hmm yeah, probably that's right

3

u/Daralima Sep 10 '22

This is amazing. I hope this technique is adopted by more user-friendly interfaces that a lot of people seem to be using right now so that a lot of people can work with this to fine-tune their results. Thank you very much for this!

10

u/LetterRip Sep 10 '22

block97,

Doggettx released his variant also that does substitution and weighting via commandline, also interesting is that it allows different aspects to be introduced (and removed) at different steps, including adding keywords later in the diffusion cycle.

https://www.reddit.com/r/StableDiffusion/comments/xas2os/simple_prompt2prompt_implementation_with_prompt/

3

u/thatdude_james Sep 10 '22

This is so cool.

3

u/Ath47 Sep 10 '22

This is awesome. I love that the background in the first pic is blurred according to how big the animal is, because this is accurate to what a camera would do. The mouse is small, close to the camera, so short focus and blurry background. Cat is a bit less blurry, then dog, then no blur for tiger because camera is further back and there's a longer focal plane. Just super interesting that it "knows" to do this.

4

u/bloc97 Sep 10 '22

I like to think that most photos of tigers would be probably taken from very very far away, but you're right! I was first really surprised at this result too. Whether this is a curse or a blessing is still up to debate, as it's one of the reasons of the unpredictability and lack of control for most LLI models.

1

u/TiagoTiagoT Sep 10 '22

Could the depth of field/scale be explicitly defined to avoid the AI just going for the most likely?

3

u/gxcells Sep 10 '22

Thank you so much.

I tried a bit by putting your jupyter notebook on Colab. I had to replace PIL by pillow for installation and also for some reasons it cannot find difflib? But still it seems to work.

I just modified your prompt for a portrait and I kept same seed. It worked very well to change hair color. But if I try to add "a hat" or "sunglasses" or to change eye color, it does not change much the picture. I did not try to change seed yet to see if it is the problem.

I did not try the weight to see if it could help because I did not really understand up to now (but I maybe figured now after reading again the Readme).

Thanks a gain, that's really great work.

1

u/bloc97 Sep 11 '22

When adding or modifying a significant portion of the image, you can try increasing prompt_edit_spatial_start or lowering prompt_edit_spatial_end, this decreases the strictness and allows the network to be a bit less faithful to the original image. The default is maximum strictness as that's what is best for most use cases.

1

u/gxcells Sep 11 '22

Ok, thanks I will try :)

7

u/Incognit0ErgoSum Sep 10 '22

You are an absolute legend.

12

u/bloc97 Sep 10 '22

Thanks, but I'm no legend, we are all here by standing on the shoulders of giants.

5

u/Incognit0ErgoSum Sep 10 '22

Then we're all legends, especially the giants. :)

2

u/1Neokortex1 Sep 10 '22

Very humble of you but your a programming guru👍

Where does one begin learning how to understand what is in the stable diffusion notebooks? I understand its python and will need to learn more about jupyter,colab and machine learning. I want to eventually install locally for animations like the deforum notebook.

2

u/pixelies Sep 11 '22

This is awesome! I hope this feature gets added to the webui :)

1

u/ExponentialCookie Sep 10 '22

Amazing, thanks!

1

u/diffusion_throwaway Sep 11 '22

Oh man. This would be VERY useful. I can't wait until this is implemented into a colab notebook.

1

u/bloc97 Sep 11 '22

It is already available as a jupyter notebook, and there's already a fork with a colab notebook.

1

u/diffusion_throwaway Sep 12 '22

Oh I didn't see the colab workbook. Thanks!

1

u/diffusion_throwaway Sep 12 '22

Sorry, I hate to be this guy, but do you have a link to the colab notebook? I searched around and couldn't find anything.

Thanks!!

1

u/bloc97 Sep 12 '22

It's in the pull requests, it hasn't been merged yet.

1

u/tejank10 Sep 11 '22

Thanks a tonne for creating this! I wanted to know more about the attributes that you've used like last_attn_slice_weights, but I could not find much documentation about it. Can you please point me to what it (and other last_attn_* attributes) are exactly doing? Thanks a lot!

1

u/state2 Sep 12 '22

awsome! does anyone know how i get init_image to work?

2

u/bloc97 Sep 12 '22

Just pass in a PIL image, however it's not yet the full method from the paper so results might be less impressive, I'm still working to make inversion work for the klms scheduler.