I have an rx 6600 and 16gb of ram and an i5 10400f
I am using oobabooga web-ui and I happened to have a gguf file of LLama2-13B-Tiefighter.Q4_K_S .
But it always says that the connection errored out when I load the model.
Anyway, please suggest any good model that I can get started with.
I am using Mistral 7b Instruct for text summary and some light “assistant” type chatting for the last several months. I have been pleased at how accurate it is for my needs, especially given it’s size.
I recall alot of trial and error to find models that were compatible with the version of llama-cpp-python that oobabooga uses (at any given time). GGUF should have made the model format (and therefore model selection) more simple, but i imagine there are still nuances that make it more difficult than it should be to find a working model for a noob.
Best of luck, let us know how it goes