I was experimenting with oobabooga trying to run this model but due to it’s size it wasn’t going to fit in ram, so i tried to quantize it using llama.cpp, and that worked, but due to the gguf format it was only running on the cpu. searching for ways to quantize the model while keeping it in safetensors returned nothing; so is there any way to do that?

I’m sorry if this is a stupid question, i still know almost nothing of this field