requesting an interesting pair(2) models for in-depth comparison benchmark

context: in absence of an established pipeline/workflow ( automated/semi-automated), benchmarking (esp for me), is costly.

i’m thinking that i could focus/concentrate my energy on just 1 crucial pair of models. this would give me the appropiate budget to do a maximally thorough evaluation. and put those resources targeting the pair that truly matters.

i am expecting a promising fine-tuned model. perhaps slightly obscure. any model suggestion is acceptable. lower resource is much better . soft constraint: greater than ~32B is stressing my limit to the point where it risks making my evaluation more shallow.

its most ideal if commenter has deeply vetted the model. but not required. any suggestion is ok.

depending on model sizes(==resources!) i may be tempted to do additional shallow exploration of a larger set of models.

i am trying to test for a variety of my personal uses. ( not hardcore dev . i feel relatively lacking experience for evaluating advanced performance of domains like this).
also, i really really cant promise to cover every usecase. but if you suggest a usecase, and it sounds cool to me, i would love to add tests for it.

i cant really describe my own usecases. very unpredictable. i try to be a productivity/pragmatic person. some topics i like: amateur dev, LLM tech, Tech, evaluation of claims(tech,science,news), society(like politics legal) , interpersonal dilemas, consumer medicine(or eval of latest research), survival( ~bushcraft) , urban ,

hard constraint: must be GGUF that runs on official llama.cpp . that is all i have to work with, currently.

  • e0qdk@reddthat.comEnglish
    1·
    29 days ago

    I’ve found a few models that people made for creative writing / roleplay – they have interesting characteristics as a result. Might be fun for you to evaluate, if you’re curious:

    Those links go to the safetensors. There are existing GGUF quants on HF too, I think, or you can convert them yourself easily with convert_hf_to_gguf.py and llama-quantize from the llama.cpp repo. (That’s what I did.)

    The first one is better at novel-style prose; the Rudo variant is better for dialogue. I don’t know how well either does with non-writing tasks.