LocalLLaMA

2841 readers

1 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

founded 2 years ago

MODERATORS

SkySyrup@sh.itjust.works

pax@sh.itjust.works

noneabove1182@sh.itjust.works

Smokeydope@lemmy.world

MonsterBug@sh.itjust.works

Models not loading into RAM (lemmy.ml)

submitted 1 month ago by corvus@lemmy.ml to c/localllama@sh.itjust.works

10 comments fedilink hide all child comments

I didn't expect a 8B-F16 model with 16GB on disk could be run in my laptop with only 16GB of RAM and integrated GPU, It was painfuly slow, like 0.3 t/s, but it ran. Then I learnt that you can effectively run a model from your storage without loading into memory and checked that it was exactly the case, the memory usage kept constant at around 20% with and without running the model. The problem is that gpt4all-chat is running all the models greater than 1.5B in this way, and the difference is huge as the 1.5b model runs at 20 t/s. Even a distilled 6.7B_Q8 model with roughly 7GB on disk that has plenty of room (12GB RAM free) didn't move the memory usage and it was also very slow (3 tokens/sec). I'm pretty new to this field so I'm probably missing something basic, but I just followed the instrucctions for downloading it and compile it.

you are viewing a single comment's thread
view the rest of the comments

[–] gens@programming.dev 1 points 1 month ago* (last edited 1 month ago)

The programs usually mmap the file into memory. That means that parts of it are loaded as used and unloaded if there is no memory left. That's why it does not say it is using memory. Check disc i/o as it is generating the message. For linux that can be seen in htop or iotop, for win idk.

Note that I use lmstudio, that uses llama.cpp to run models. Gpt4all, I think, uses a modified version of same. Doesn't matter they should all be using mmap to load the file.

PS Depending on the model I also get a couple tokens per sec on the cpu.

Edit: Didn't see someone already said the same, I'l leave this here anyway.