this post was submitted on 20 Dec 2024
11 points (92.3% liked)

LocalLLaMA

3306 readers
2 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 2 years ago
MODERATORS
SkySyrup@sh.itjust.works
pax@sh.itjust.works
noneabove1182@sh.itjust.works
Smokeydope@lemmy.world
MonsterBug@sh.itjust.works
11
Can you fine-tune on localized steering of an LLM? (lemmy.dbzer0.com)
submitted 6 months ago* (last edited 6 months ago) by hok@lemmy.dbzer0.com to c/localllama@sh.itjust.works
19 comments fedilink hide all child comments
 

I want to fine tune an LLM to "steer" it in the right direction. I have plenty of training examples in which I stop the generation early and correct the output to go in the right direction, and then resume generation.

Basically, for my dataset doing 100 "steers" on a single task is much cheaper than having to correct 100 full generations completely, and I think each of these "steer" operations has value and could be used for training.

So maybe I'm looking for some kind of localized DPO. Does anyone know if something like this exists?

you are viewing a single comment's thread
view the rest of the comments
[–] hok@lemmy.dbzer0.com 1 points 6 months ago* (last edited 6 months ago)

Thanks for your answer. I think to be clear, what I'm looking for is a kind of masked fine-tuning. You see, I want to "steer" a particular output instead of providing complete examples, which are costly to create.

The steering would be something like this:

  1. I have an LLM generate a sequence.
  2. I find exactly where the LLM goes "off track" and correct it there (for only maybe 10-20 tokens instead of correcting the rest of the generation manually).
  3. The LLM continues "on track" until it goes off track again.

What I would like to do is train the model based on these corrections I give it, where many corrections might be part of the same overall generation. Conceptually I think each correction must have some training value. I don't know much about masking, but what I mean here is that I don't want it to train on a few tens or hundreds of (incomplete) samples but rather thousands of (masked) "steers" that correct the course of the rest of the sample's generated text.