r/StableDiffusion Sep 04 '22

Stable Diffusion and similar platforms are the best compression models the world currently has!

I think it is important to understand the capabilities of these technologies beyond just making art. What these text to image models are capable of is massive compression of visual data that is orders of magnitude better then what we currently have. Right now if I wanted to share any image I created with stable diffusion, I do not have to share the image with you. I could just give you the metadata for the settings used to generate the said image and save a tremendous amount of bandwidth and hard drive space.

Now you might be thinking that's great and all but what's my point. Well imagine if you are running a massive website that hosts generated images. All of those images are taking up a lot of space on server hard drives which you have to pay to host. Also there is massive amount of bandwidth going through the website which also you have to pay for. Now imagine that you had the idea of embedding the generated images metadata where the jpg image should have been. Now all you need is an browser extension that automatically reads the prompt data, runs the data through your local machine, spit the image back out to the browser for you to view. And now the visitor to the site just viewed the image without you actually hosting the said image. All of the rendering was done on the users end locally. This saves you massive amount of space and bandwidth as you only need to store the images metadata.

Furthermore, you can run censored websites in plain view on the open internet without anyone knowing what's on the website. For example China censors all of the traffic on the internet through the use of automated bots that look out for specific images and phrases. Its a closed off system that is heavily monitored. With the technology I propose above, if you were to host a dissident website in china now, there would be no way for the sensors to pick up the data that is flowing through them. to outside observers all of the websites data would look like gibberish as the prompt metadata could be also encoded and only those with the proper seed information and the text to image model can generate the data on the website.

Now you are also wondering great, but what about images that were not generated with any of the text to image models. Well, those can also be done. As we have already been capable of finding the image representation of your image in the latent space of the model and its appropriate coordinates (prompt). With this we can encode any image in to its similar counterpart that exists in the latent space. This image is not 100% accurate but with large enough model it is indistinguishable to any human eye from the original.

Now apply the data compression tech with every form of visual data we currently have and you save an absolute massive amount of bandwidth and storage space in the process while being ultra secure. And all we have to do is create a browser extension that automatically reads the prompt metadata, runs that info locally, spit it back in to the browser for your viewing in proper format. As the data sets of these models grows and models get better with text, you can do the same thing with text if you wish.

Currently the models are in their infancy, so you cant run it on a non gpu machine or a cellphone efficiently but with very little time we will get there, i suspect within this year actually.

In the mean time you can also have the models run on the cloud if you are a large data center. (temporary hybrid solution). For example someone visits your website, your local servers or cloud servers spin up to generate the images and push those images to the user and dump the generated image from your hard drives. That way you lose out on the bandwidth but win out by getting customers who don't have their own machines and also you don't store massive amounts of data.

Bottom line is there are massive opportunities I see opening up with this tech that few are thinking about. there is massive potential in this tech no doubt about it.

16 Upvotes

9

u/Schyte96 Sep 04 '22

I see one problem with your idea: Running stable diffusion or a similar model is compute heavy. There is a reason people run them on fairly beefy GPUs and such.

You saved on bandwidth and storage, but you massively increase the need for compute (local or cloud) massively.

4

u/digitallyfree Sep 04 '22

I would imagine this would improve over time, especially if this kind of technology become prevalent (it likely will). Hardware acceleration for ML is becoming more and more common in the mobile and datacenter worlds and likely will appear on PCs sooner rather than later. Already Intel has ML x86 extensions and the Apple M-series has an ML coproc.

Think of the evolution of video compression and codecs as the underlying hardware improves - HEVC chokes up on say a C2D, but decodes well on an i5 and can now be encoded in realtime with the ASIC encoder on a modern GPU. We're doing the same now with AV1, which despite the long encoding time places like Youtube are experimenting with since the storage and bandwidth savings are worth it.

For the Youtube case, currently I have AV1 turned off since it consumes a lot of CPU. That will change in the future when my GPU has an AV1 decoder - and the same thing goes with ML inference hardware.

2

u/no_witty_username Sep 04 '22

The computation won't be an issue in the near future. These models are compute heavy only for a brief time as we are only in the infancy of this technology. The resources needed to generate the images will fall as folks find more efficient means and build more efficient models.

1

u/Nms123 Sep 15 '22

I agree with you this likely isn’t gonna be used for day to day image compression barring maybe some breakthrough in quantum computing that massively speeds this process up.

It might, however, have some valid use-cases for satellite communications where data throughput is super expensive.

3

u/AwwwComeOnLOU Sep 04 '22

Wow, you just imagined a new version of the internet with either a large reduction in bandwidth or a massive increase in what we can currently transfer.

It’s likely that the latter will be more attractive to those already rich in bandwidth.

With what you described it’s possible for entire AI generated worlds to be transferred in current limits.

People in the VR space complain about resolution and lag.

Is this a viable business model:

A home VR device that has a stable diffusion image generator in it and using a limited data connection is able to download just the meta data of worlds that the player interfaces with and then their image and changes to the world are converted to meta data and sent out to others quick enough over current bandwidth limits to allow for a VR experience that doesn’t lag and pixilate?

1

u/no_witty_username Sep 04 '22

Yep that will 100% be possible once we get the text to 3d model thing down as well. In fact there are people who are working on that as we speak. Similar goes for audio compression. All media can be compressed with this tech.

2

u/AwwwComeOnLOU Sep 04 '22

So we are on the cusp of a VR revolution where the only remaining pieces are who is going to present the technology (likely Meta and others) and who is going to create worlds compelling enough for people to want to immerse themselves into and stay immersed long enough to make it “the place to be.”

I suspect that large corporations will spend a lot of effort on creating compelling worlds that try to attract the masses, but the cutting edge of where “the cool/hip” people want to be will be created independently.

The system that wins out will be the one that allows creative individuals the most freedom to immerse and create or change sub worlds into their visions.

How do you see this playing out?

3

u/[deleted] Sep 04 '22

The intelligent internet.
Personalised generative search models for everyone that compress knowledge.
Image, audio, text & more.
Distributed & dynamic - a protocol for the next generation.
This is how we build the foundation to activate human potential

https://twitter.com/EMostaque/status/1562762975500374017

2

u/no_witty_username Sep 04 '22

Brilliant, the man himself is already on it and I didn't even know it! I am glad that he realizes the full potential of this tech.

2

u/Bitflip01 Sep 04 '22

What you are describing is similar to the concept of the Babel Image Archives, but with a tradeoff towards more lossy compression and shorter image addresses.

1

u/no_witty_username Sep 04 '22

I was thinking about babel image archive the moment I laid eyes on stable diffusion. Also codecs, I was thinking quite a lot about video codecs and the 4k demo scene.

1

u/[deleted] Sep 04 '22

[deleted]

1

u/no_witty_username Sep 04 '22

With the limited research I did, I believe there are better models out there then diffusion models but non the less its a start. Research in the field is moving at lightning speed, i think we will converge on the most efficient model type soon enough. Regardless what it is though, the doorway has been open.

2

u/[deleted] Sep 04 '22

[deleted]

2

u/no_witty_username Sep 04 '22

When I say better models then diffusion models I mean better models for compressing images and retrieving their likeness from latent space. I am not referring to stable diffusion. For example Gan models would be better at that task then diffusion models. This is regardless if stable diffusion forks to Gan or stays as a diffusion model.

1

u/Tall_Ad426 Sep 04 '22

As we have already been capable of finding the image representation of your image in the latent space of the model and its appropriate coordinates (prompt). With this we can encode any image in to its similar counterpart that exists in the latent space. This image is not 100% accurate but with large enough model it is indistinguishable to any human eye from the original.

any link to any resoure where this kind of image reversing is done?

If i understand this correctly you are proposing going

Real Image -> (some model) -> {prompt, seed, metadata} -> some other model -> Ai generated image, indistinguishable from real image

Any resource where this is usable / demonstrated?

1

u/no_witty_username Sep 04 '22

Here is one video that goes in great depth about the process. There are many more over youtube, I just never saved their links. Search for terms like latent space inversion and similar. https://youtu.be/zyBQ9obuqfQ?t=1091

1

u/Nms123 Sep 15 '22

To be clear, this only works for images that are feasibly describe-able with human language right? There are probably lots of images that won’t be able to be compressed, but they’ll look like noise to us.

1

u/no_witty_username Sep 15 '22

To benefit from the full potential of this compression technology the model needs to be trained with non human language tokens in mind. That means you will be getting subpar results with current models.

1

u/Nms123 Sep 15 '22

Yeah, even then though those tokens have to align with an abstract concept that is larger than an individual pixel, otherwise the model would just be the size of every possible picture. So ultimately there will have to be some possible images not represented

1

u/no_witty_username Sep 15 '22

The images that require great detail and accuracy will not be represented properly in the proposed compression method. But that's not a problem, because the vast majority of images on the internet do not require such precision.