r/StableDiffusion Sep 10 '22

Prompt-to-Prompt Image Editing with Cross Attention Control in Stable Diffusion

217 Upvotes

View all comments

9

u/Zertofy Sep 10 '22

That's really awesome, but I want to ask some questions?

What is needed for this to work? We have initial prompt, resolution, seed, scale, steps, sampler, and resulting image of course. Then we somehow fixate general composition and change prompt, but leave everything else intact? So the most important elements are prompt and resulting image?

Can we take non-generated picture, write some "original" prompt and associatiate them with each other, then change prompt and expect that it will work? But what with all other parameters...

Or this is what will be achieved in img2img?

Or maybe I completely wrong and it's working in absolutely different ways?

27

u/bloc97 Sep 10 '22

First question: Yes, right now the control mechanisms are really basic, you have a initial prompt (that you can generate to see what the image looks like), then a second prompt that is an edit of the first. The algorithm will generate your second prompt so that it looks as "close" as possible to the first (with the concept of closeness being encoded inside of the network). You can also tweak the weights of each token, such that you can reduce or increase its contribution on the final image (e.g you want less clouds, more trees). Note that tweaking the weights in attention space gives much better results than editing the prompt embeddings, as the prompt embeddings are highly nonlinear and often editing them will break the image.

Second question: Yes, but not right now. What everyone is using as "img2img" is actually a crude approximation of the correct "inverse" process for the network (not to be confused with textual inversion). What we actually want for prompt editing is not to add random noise to an image but find which noise will reconstruct our intended image and use that to modify our prompt or generate variations. I was hoping someone would have already implemented it but I guess I can give it a try when I have more time.

Also, because stable diffusion is slightly different to what I guess was Imagen used in the paper, we have a second self-cross-attention layer, which can be controlled by using an additional mask (that is not yet implemented right now), that means that if image inversion is implemented correctly, we could actually "inpaint" using the cross-attention layers themselves and modify the prompt, this should give us much better results than simply masking out the image and adding random noise...

Exciting times ahead!

3

u/Zertofy Sep 10 '22

Cool! Also, is it take the same time to generate as usual image? Probably yes, but just to be sure. Some time before I see post here about video editing, and one of the problems was the lack of consistency between frames. I proposed use of the same seed, but it give only partial result. May this technology be the missing element for this?

Anyway, it's really exciting to see how people explore and upgrade SD in real time. Wish you success i quess

5

u/bloc97 Sep 10 '22

It is slightly slower, because instead of 2 u-net calls, we need 3 for the edited prompt. For video, I'm not sure if this can achieve temporal consistency, as the latent space is way too nonlinear, even with cross-attention control you don't always get exactly the same results (eg. backgrounds, trees, rocks might change shape when you are editing the sky). I think hybrid methods (that are not purely end-to-end) will be the way forward for video generation. (eg. augmenting stablediffusion with depth prediction and motion vector generation)

2

u/enspiralart Sep 12 '22

That augmentation, how do you think it should be gone about? For instance, a secondary network that feeds into the U-Net and gives it these depth and motion prediction vectors, which can be used to change the initial latents such that an image is generated from one frame to the next with roughly the same image latent, but motion vectors warping that image? Or yes, how?

2

u/bloc97 Sep 12 '22

I mean, some specific use cases such as animating faces, image fly through and depth map generation for novel view synthesis already exists. To generate video we probably need some kind of new diffusion architecture that can generate temporally coherent images, of which the data can be taken from YouTube, wiki commons, etc. But I don't think our consumer GPUs are powerful enough to run such a model.

2

u/enspiralart Sep 12 '22

There's an amazing conversation going on about it in the LAION discord group video-CLIIP

https://twitter.com/_akhaliq/status/1557154530290290688 this is from that group

Maciek — 08/10/2022 ok so they basically do what we've already done more thoroughly. Architecture is practically the same as well:
"we employ a lightweight Transformer decoder and learn a query token to dynamically collect frame-level spatial features from the CLIP image encoder"
this is just this - https://github.com/LAION-AI/temporal-embedding-aggregation/blob/master/src/aggregation/cross\_attention\_pool.py they
also just do action recognition but they do it on K400 which is easier.
I guess all the more evidence that this approach works.

LAION Discord video-clip group: https://discord.com/channels/823813159592001537/966432607183175730

1

u/TiagoTiagoT Sep 10 '22

Would it be possible to somehow freeze stuff that is not identified by the AI as being what is being changed, sorta like masking but at a deeper level and done automatically?

1

u/Zertofy Sep 10 '22

Hmm yeah, probably that's right