r/StableDiffusion Sep 08 '22

Reproducing the method in 'Prompt-to-Prompt Image Editing with Cross Attention Control' with Stable Diffusion Discussion

Post image
274 Upvotes

111

u/bloc97 Sep 08 '22

The most common problem with LLI models is that when you change the prompt slightly, the whole image changes unpredictably (even when using the same seed). The method in the paper "Prompt-to-Prompt Image Editing with Cross Attention Control" allows far more stable generation when editing a prompt by fixing and modifying the Cross-Attention layers during the diffusion process.

I'm still working on the code to make everything work (especially with img2img and inpainting), however, I hope that I will be able to release the code soon on github.

33

u/GBJI Sep 09 '22

Wow, this is one of the most useful developments around Stable Diffusion I've seen so far ! Impressive, really really impressive - maybe even a game changer for the use of Stable Diffusion by digital artists and design studios.

One question: In your paper you wrote "For example, consider an image generated from the prompt “my new bicycle”, and assume that the user wants to edit the color of the bicycle," but I haven't seen any example where you actually changed the color of a synthetic object by specifying the color. You seem to be able to do something from a higher level of abstraction by changing the cake's color according to its "flavor", but can you do it by calling colors directly ? I also saw the cat with the colorful shirt example, and the colorful bedroom as well, but I could not find any precise color being called, like a cat with a red shirt, or a purple bedroom. So can you do that with your system ? If not, what is the difficulty you'd have to overcome to make it work in your opinion ?

Keep us informed, this is very exciting to say the least !

29

u/bloc97 Sep 09 '22

It's not my paper, I've simply implemented the paper as is. You should ask these questions to the original authors and give your thanks! This is indeed a great work! https://amirhertz.github.io/ https://rmokady.github.io/

11

u/GBJI Sep 09 '22

Thanks for turning what is a theoretical paper into something we will likely be able to test for ourselves soon !

Maybe I should just wait for that and try it myself, and if that doesn't answer my questions, I'll go bother the original authors. Thanks for the links, very appreciated.

2

u/Incognit0ErgoSum Sep 09 '22

Is your code available to download anywhere?

3

u/bloc97 Sep 10 '22

2

u/phreakheaven Sep 12 '22 edited Sep 12 '22

I am using the Stable Diffusion WebUI (https://github.com/AUTOMATIC1111/stable-diffusion-webui); can this be implemented and run locally? I really have no idea how most of this works, relating to Colab and Jupyter, etc., as I only run locally and am unsure if that means this requires a non-local setup (because of the Jupyter step in your README).

1

u/phadeb Oct 06 '22

Would like the smae

4

u/Additional-Cap-7110 Sep 09 '22

Whoa man, in that paper it shows how close we are to making a literal animation. It was like showing several frames from a video in some of those examples ie. candels lighting up on a cake or the Teddy’s eyes.

2

u/yugyukfyjdur Sep 09 '22

Thanks--those look like promising results! I could see this being especially interesting for graphic design-style applications (e.g. section headers).

12

u/LordNinjaa1 Sep 09 '22

Thanks to ai I now know what grass fruit looks like!

6

u/Creepy_Dark6025 Sep 09 '22 edited Sep 09 '22

amazing work bloc!, i love these tools around stable difussion that the community makes, that is why open source is always superior to the close models. i can't even image how far will stable difussion go with the support of the community.

5

u/Doggettx Sep 09 '22 edited Sep 09 '22

I'm curious, would it be enough to just swap out the conditioning at a certain step for the prompt variants? Or does the model have to be applied to each prompt and then merged somehow?

Edit: After some hacked in tests, it seems to work when just swapping out prompts. But not sure if that's the right method.

Been playing with it a bit, I'm amazed how early you can insert the new prompt and still keep the composition intact...

I've implemented it in my prompt processing, thanks for the idea!

Banana sushi
Banana icecream

The second one I do now by typing 'Banana [sushi:icecream:0.3]' where the 0.3 is a multiplier for step to insert the swap at, that way you can do multiple swaps in a single prompt and also introduce new concepts with[:new concept:0.3] or remove existing concepts [existing concept::0.3]

6

u/bloc97 Sep 09 '22

Interesting, there are way too many ways to condition a pretrained LLIM, you could use CLIP guidance, you could swap out the prompts like you said, you can edit and control the cross attention layers, you can use inversion on a image, you can use img2img, latent space interpolation, embedding interpolation, and any combination of the above and more that we didn't discover yet... Thanks to stablediffusion we can start experimenting and validating these methods without needing millions of dollars in funding!

1

u/LetterRip Sep 09 '22

nice approach

12

u/Adreitz7 Sep 08 '22

Could you provide a link or something for what you're talking about? I haven't heard of this.

37

u/bloc97 Sep 08 '22

Yes, it's the method from this paper https://arxiv.org/abs/2208.01626. There's no code available yet but I started to implement it from scratch in pytorch, this is the first results that I got and I'm very excited to implement and try out the other methods the paper describes.

12

u/HorrorExpress Sep 09 '22

Well, if these are your first results, I think you've done an incredible job already.

I wasn't aware of this issue until I started hours of iterating prompt text using the same seed. As you say the results can be problematic, or even nonsenical. It really makes the whole endeavour time-consuming, counter intuitive, and, at times, frustrating.

I wasn't aware of this potential solution until reading your post. So thanks for posting it - I'll try and read, and make sense of the article later - and for the work you've done so far. Your example grid is fantastic.

I'm going to keep checking for your future posts on this, and I'm sure I'm not the only one.

Keep up the great work.

4

u/brainscratchings Sep 09 '22

Can you list your github so we can check up on it for updates?

5

u/79dfshsk Sep 10 '22

I had a go myself and got some results: https://imgur.com/a/DwqWB71

I have to say, I haven't a clue what I'm doing, the paper was quite confusing as I don't think the use of attention maps in the paper can be so literally applied in SD (and that I have literally no ML background whatsoever). Also that I really don't have enough RAM/VRAM to hold onto the "attention maps"

3

u/bloc97 Sep 10 '22

3

u/79dfshsk Sep 10 '22 edited Sep 10 '22

Here's my implementation (obviously more hacked-together with global vars): https://pastebin.com/2bvQRGKm

As you can see, I am only picking out the att_slices from certain (inner) blocks of the unet. However, I see you're taking the first down block. I coudn't try this because of my limited memory, but it's interesting to see that mine somehow worked anyway.

One of the things that confused me in the paper - I interpreted it to mean was one attention map per diffusion step, whereas there's actually a load of "slices" - not only for each up/down block, but each up/down block uses a spatial transformer which actually has two cross-attn models. attn1 was just some "hidden" state that I couldn't figure out, but attn2 was 77 (i.e per token, explicitly mentioned in the paper) x 4096 (i.e 256x256). I kept trying with subbing out the attn slices from attn2 without any success before I tried it with attn1 as well.

2

u/bloc97 Sep 10 '22

Hmm, I thought that I had selected all of the Cross attention layers, I'll have to double check the code when I'm back on my project computer...

2

u/79dfshsk Sep 10 '22

Sorry, I misread the code and you are indeed preserving through all layers.

2

u/bloc97 Sep 10 '22

Oh and yea, attn1 is the self-attention layer, which is controlled differently from the prompt, usually we would edit it with a mask but I didn't have time to implement it yet. So right now the edit function for attn1 is simply replacement, while attn2 is implemented as is to the paper

10

u/ExponentialCookie Sep 09 '22

Interesting. Would this help with Textual Inversion's issues by any chance? I would love to test this out.

3

u/Fungunkle Sep 09 '22

This seems deeper than just re-using the same seed.

3

u/m0thercoconut Sep 09 '22

This is amazing! Congrats

2

u/thatdude_james Sep 09 '22

very cool. Just adding my vote to this as a super cool. Can't wait to see it in practice for myself

4

u/dmertl Sep 09 '22

Very impressive results, looking forward to seeing it in action.

1

u/theRIAA Sep 09 '22

Was it trained on the "lemon cake" image specifically?

like, you should be able to use any one of those "w/o" images as a template image, yes?

So how does this compare to results from img2img and/or trained-textual-inversion?

7

u/bloc97 Sep 09 '22

No, there's no pretraining involved and this can be used with img2img and textural inversion. This method helps preserve the image structure when changing a prompt while img2img and textural inversion are tools that allow you to condition your prompt better on one or many images.

1

u/pixelies Sep 11 '22

These are the tools I need make graphic novels. The ability to generate an environment and preserve it while adding characters and props.