AI Horde

Tazlin has been hard at work with the new worker overhaul and it's looking sexy AF!

Hello everyone!

TL;DR: I want to propose a community-driven effort to research and improve 1-bit LLM models, for use within and outside the Horde. I think having access to such models would be very useful for the overall project, as you do not need a lot of compute to run "bigger" models if they're compressed well. Relevant paper.

We currently live in very interesting times regarding AI development. The big companies in the US seem to still be ahead, but groups from China, working on open-weights models, are making very impressive strides. However, the focus has been and still is on developing really big models. Most of the impressive models that keep coming out are bigger and bigger, leaving most people to pay for API tokens if they want something useful. Probably this is also incentivised, as Nvidia wants to make more money from sales to data centers.

I have been keeping an eye out for the AI Horde project, and I always wanted to help out, but never got around to setting it properly up due to my AMD GPU and running Windows. However, really cool project, and I wanted to congratulate everyone involved!

In my opinion, the goal of the AI Horde community also positions it as one of the very few that are capable of bringing forward some LLMs that can be "smart" on consumer-level hardware. That is, getting models that can handle long-horizon tasks well without paying providers for API access.

Since the first paper on 1.68-bit LLMs, there has been quite a few other suggestions thrown around (recent example). No one really uses these models much, as the quality is seriously degraded. Similarly, no one is really trying to improve these models further, as there is no incentive to do so when you can just pay someone 3$ per million token output from an existing open-weight model. For example, to the best of my knowledge, no one has tried to introduce latent reasoning (example) in this context, or specifically training/fine-tuning models at 1-bit levels.

So, to get to the point, would it make sense to get some community-driven research in this area? I believe that we could all pool together compute, good training data, ideas for fine-tuning / RL-training, etc. If it works out, we could have a method that makes existing larger models (say, up to 200B) available on a single 24GB GPU.

First thing I would try is to expand on the recent NanoQuant paper:

0. Wait for weights to be released.
1. If no weights come out, quantize a Qwen 3 32B, and try a more diverse dataset, with more tokens to see if fidelity can improve. I could get some access to GPUs for this myself. Another option for 1-bit models would be using other existing ones (e.g., Unsloth), but performance degradation is much bigger in those versions, from what I have seen. Furthermore, the compression of these models is not as efficient, and you would not be able to fully run a 70B parameter (as described in the paper), with only 8GB VRAM.
2. Get an LLM (or human volunteers) to determine behaviour on various tasks: find limitations, strengths, etc. Get some human preferences for RLHF, or use a bigger LLM to grade output quality. Preference here on logic tasks.
3. Perform fine-tuning of 1-bit model based on the gathered data, and deploy for use. Return to step 2 after a while.

Fine-tuning the 1-bit model might get a bit hairy, as the binary operation are not differentiable. We also wouldn't be able to up-cast back to F32 for regular training, as this would completely invalidate the consumer-driven access to these models. Simplest idea would be to train a LoRA head, or do some stochastic-driven training (e.g., flipping bits). However, the latter would probably be very unstable, and not work out, and LoRA might be the only option. I'm not a mathematician, so I am open to suggestions here :)

Past the initial prototype, I would consider the following stuff that's already implemented in other quants:

• Try to re-calibrate the LayerNorm / RMSNorm parameters based on the original model's activations.
• Perform some regular KL-divergence distillation
• Other than using an LLM as a judge for RL, perhaps one could also fine-tune using a semantic similarity metric while aligning with the output of the original model. This could ensure that the intent is the same, even if the style differs.
• Depending on token complexity, look into reducing compression just for difficult tokens and compressing further for easy ones (à la FlexQuant)
• Mixture-of-Experts-style quantization, where we increase compression for experts that are not important, and reduce it for higher-frequency ones.

Curious what everyone thinks!

Hey!

Not sure if this is the palce to ask questions, but I have one.

I can't quite figure out how to submit ratings for generated images.

According to the API docs, the /v2/generate/rate/{id} endpoint is for submitting ratings, but I can't figure out how to actually call this enpoint.

For starters, what is {id} in this case?

I've checked the source at github, and to my understanding it's a WaitingPrompt id, which, if I'm correct is the id what /api/v2/generate/async gives back in its response.

Now the API docs tells me I can submit a set of ratings and also a best one from that set. Is this means that if I generate multiple images with one request, I can rate only those images? And should I use the UUID of the images in each rating object, the id I get back from /v2/generate/status/{id} in each generation object?

Also, should I specify the apikey in headers?

I tried to send POST requests to https://aihorde.net/api/v2/generate/rate/%7Bid%7D, but I always get back the following message:

{'message':'The browser (or proxy) sent a request that this server could not understand.'}

In the request URL, I tried to use all the IDs I could think of. Job IDs, image IDs... I also realized in the meantime that ArtBot too generates UUIDs for itself...

My "final" request looked like this:

• in the URL, I used the job id of my image generation request
• In the message, I have put one rating, for which ID I used the id what I got in /v2/generate/status/{id} response. (generations[0].id)

As for why I try to send in ratings manually; the rating feature is broken in ArtBot (CORS errors, in all browser). As well as everywhere else.

I see references to the v1 API in websites that can in theory send in ratings to Horde, as well in the AI-Horde source code.

So, I'm kinda lost and confused with this rating thing. Maybe I'm doing things entirely wrong.

But anyway. Any help or answer is appreciated! :)

Thanks!

Go to this URL and help us make a presence for Haidra and by extension /0 as well!

If you're wondering what canvas 2024 is, check here.

If you're wondering how this fits the instance theme, I invite you to check out this critical reading, which is what inspired the name

The template position is on the bottom right

You will need to login. Once you do, lower the opacity of the template, and place some pixels where there's none

EDIT: Changed position to bottom right as it feels more natural to grow from the bottom up. I also increased its size/fidelity a bit since we had a bit more space there

A long-overdue upgrade. All kudos to rikudou@lemmings.world! Take a gander at it here: https://aihorde.net/

Source is here: https://github.com/RikudouSage/AiHordeFrontpage

If you like using the !aihorde@lemmy.dbzer0.com bot, do join the event and suggest your best options.

it would be nice to extend the competition to Lemmy somehow as most of our community is mostly active on discord. If anyone would be willing to help with that aspect, please let me know.

The Generative AI haters and luddites won the day and bullied Nivenly to remove Haidra, even without clear majority in opposition or a quorum.

I will write a post-mortem about this situation soonish.

Hello there, fellow lemmings!

As the title suggests, I've made an app that changes your wallpaper regularly based on the parameters you set. It uses AI Horde (which uses Stable Diffusion) to generate the images and you can set the interval at which your wallpaper changes (from 1 time per day to 48 times per day).

Other features:

• completely free (in fact, will be open source next week)
• easy to use, but offers also advanced options
• there's a help section with all the parameters explained
• works on both a phone and tablet

There's also an optional Premium subscription which puts you higher in the priority queue, but most of the time it's not needed - unless you're generating during the most busy time, the wait is not really that long even without Premium. Note that if you're a user of the AI Horde, you can put your own api key there and get the same benefits as having a Premium.

The open source version (released next week) will also allow (toggleable on or off) NSFW content, which is not possible in the Google Play version. It also doesn't contain the Premium subscription.

Also please check https://dontkillmyapp.com/ and see steps for your phone vendor, this app needs to work in the background and most vendors kill background apps. It will ask for an exception for battery saver on its own when you set the schedule for the first time, but for some vendors that's not enough.

Do let me know what do you think about the app! And if you want me to tag you when I make a release post for the open source version, let me know. Also huge thanks to @db0@lemmy.dbzer0.com for creating and maintaining the AI Horde, which makes this (and other cool apps) possible!

In addition to that it also supports remembering context from previous messages. You still have to tag it even if you're replying, though.

All for free! As you can see, that comes to around 1.5 PetaPixelsteps.