Fine tune of EXPANDED “LFM2-8B-A1B” to “LFM2-12B-A1B” (almost 50%, custom scripting) using Unsloth using custom dataset(s), 128k context in 16 bit precision.
Expands base model from 24 to 32 layers, 256 to 342 tensors to give the model both more knowledge and brainpower.
You must log in or register to comment.
i would argue that the biggest benefit of MoE for low-end users is that it gets done FAST.
but this is not true for A3 / A4 models.
here you have the benefit of 12B …and its done in the blink of an eye. very low battery usage.