Persist everything to disk in real time.
That’s the thing I’m trying to avoid.
Easier to do than to get never-exercised edge-case code to work flawlessly. Are you sure you can’t just throw sqlite at the problem? It’s often overkill but, hey, it’s there on the shelf, might as well use it and I’ve seen it out-perform hand-rolled data structures. Non-persistent ones, written by very confident C coders. And remember crashes are unavoidable, if nothing else then someone can trip over the power cord.
You can’t know that from my issue description, but throwing a database at that problem really is ridiculous overkill.
Still thanks for the suggestion
It’s mostly about throwing ACID at the problem, sqlite just happens to be battle-tested to a ludicrous degree, it’s light enough to not be unconscionable overhead in simple situations (unless you’re on embedded), and performant enough to also deal with nastier situations so I prefer it over some random K/V store with the same guarantees. It’s also a widely-used and stable data format which might come in handy.
That said, if you want to go lightweight do consider good, ole, POSIX filesystem guarantees, in particular that mv is atomic (as long as you stay on the same filesystem but that’s easy to ensure by mv’ing within a directory). Make sure to fsync.
