A better (?) way of doing img2img by finding the noise which reconstructs the original image

r/StableDiffusion • u/Aqwis • Sep 11 '22

A better (?) way of doing img2img by finding the noise which reconstructs the original image Img2Img

915 Upvotes

99% Upvoted

u/Aqwis Sep 12 '22 edited Sep 12 '22

Made a few incremental updates to the Gist over the past few hours. Happy to see that a few SD forks/UIs are implementing something like this – they're better situated than me to make something that's useable by non-coders. :)

It seems that the results are quite often best when cond_scale is set to 0.0 – exactly why this is, I don't know. If anyone has an idea, I would love an explanation. With cond_scale at zero, the given prompt has no effect.

In the meantime, I've got to see my share of extremely creepy pictures while experimenting with other cond_scales. Run this on a portrait with cond_scale set to 5.0 and use the resulting noise to generate a picture (also with scale > 2.0) ... or don't. I wouldn't advise doing so personally, especially if you have a superstitious bent. (Or maybe you're going to get completely different results than I got, who knows?)

2

u/gxcells Sep 12 '22

I am using the automatic1111 implementation of your code. It is really difficult to have an effect of a prompt on generating a new image (hair color change or adding a helmet for example). Often it changes the whole face etc

1

u/Limitlez Sep 12 '22

Are you using it through webui? If so, how do you use it? I can't seem to figure it out

2

u/gxcells Sep 12 '22

You use this colab https://colab.research.google.com/drive/1Iy-xW9t1-OQWhb0hNxueGij8phCyluOh Then in img2img tab, at the bottom you can find a dropdown menu for scripts, just use the script "img2imgalternate"

1

u/thedarkzeno Sep 12 '22

https://colab.research.google.com/drive/1Iy-xW9t1-OQWhb0hNxueGij8phCyluOh

got an error:

Loading model [e3b0c442] from /content/stable-diffusion-webui/model.ckpt
---------------------------------------------------------------------------
EOFError Traceback (most recent call last)
<ipython-input-3-75bc94f91c1d> in <module>
2 sys.argv = ['webui.py', "--share", "--opt-split-attention"]
3
----> 4 import webui
5 webui.webui()
3 frames
/usr/local/lib/python3.7/dist-packages/torch/serialization.py in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
918 "functionality.")
919
--> 920 magic_number = pickle_module.load(f, **pickle_load_args)
921 if magic_number != MAGIC_NUMBER:
922 raise RuntimeError("Invalid magic number; corrupt file?")
EOFError: Ran out of input