I have a server running Debian with 24 TB of storage. I would ideally like to back up all of it, though much of it is torrents, so only the ones with low seeders really need backed up. I know about the 321 rule but it sounds like it would be expensive. What do you do for backups? Also if anyone uses tape drives for backups I am kinda curious about that potentially for offsite backups in a safe deposit box or something.
TLDR: title.
Edit: You have mentioned borg and rsync, and while borg looks good, I want to go with rsync as it seems to be more actively maintained. I would like to also have my backups encrypted, but rsync doesn’t seem to have that built in. Does anyone know what to do for encrypted backups?
Well, I’m just starting with serious backups, AFAIK you only need to backup the data which you can’t replicate.
Low seeded torrents are just hard to get, but not impossible. Personal photos, your notes, any other files generated by you are the ones which need backups.
Ideally you want to backup everything that you didn’t explicitly exclude since otherwise there is always something you forgot.
The software borgbackup does some insane compression.
It is more effective if you backup multiple machines tbh (my 3 linux computers with ~600gb used each get compressed down to a single ~350gb backup, because most of the files are the same programs and data over and over again)
But it might do a decent enough job in your case.
So one of the solutions might be getting a NAS and setting up borgbackup.
You could also get a second one and put it in your parents or best friends home for an offsite backup.
That way you don’t have to buy as large of a drive capacity, but will only have fixed costst (+electricity) instead of ongoing costs for some rented server storage.
I guess that would be about 400$ per such a device, if you get a used office pc and buy new drives for it.
Tape seems to be about half the price per TB, but then you need special reader/writer for it, which are usually connected via SAS and are FUCKING EXPENSIVE (over 4000$ as far as I can see).
It only outscales HDDs in price after like ~600TB
How do you handle the cache invalidation issue with Borg when backing up multiple systems to one repo? For me if I access a Borg repository from multiple computers (and write from each) it has to rebuild the cache each time which can take a long time.
I seperate them by archive name prefix and never had the issue you describe.
Edit: it seems I just never noticed it, but the docu suggest you’re right. Now I am confused myself lol.
I’ve been using Borg and Hetzner Storage Box. There are some small VPS hosts that actually beat Hetzner’s pricing but I have been happy with Hetzner so am staying there for now. With 24TB of data you could also look at Hetzner’s SX64 dedicated server. It has a 6 core Ryzen cpu and 4x 16TB HDD’s for 81 euro/month. You could set it up as RAID 10 which would give you around 29 TiB of usable storage, and then you also have a fairly beefy processor that you can use for transcoding and stuff like that. You don’t want to seed from it since Hetzner is sticky about complaints that they might get.
Tape drives are too expensive unless you have 100s of TB of data, I think. Hard drives are too unreliable. If you leave one in a closet for a few years, there’s a good chance it won’t spin back up.
for 81 euro/month.
You can probably find something cheaper from their auction servers.
I’ve got a storage VPS with HostHatch for my backups. It’s one of their Black Friday deals from a few years ago - 10TB storage for $10/month. Not sure they’ll offer that pricing again, but they did have something similar for around double the price during sales last year (still a good deal!)
Tape drives are too expensive unless you have 100s of TB of data, I think
The drives are expensive, and some manufacturers have expensive proprietary software, but the tapes themselves are cheaper per TB than hard drives, and they usually have a 20 or 30 year life guarantee. People seem to think tapes is old technology but modern tapes can fit 18TB uncompressed (they say 45 TB compressed but idk).
The default tier of AWS glacier uses tape, which is why data retrieval takes a few hours from when you submit the request to when you can actually download the data, and costs a lot.
The default tier of AWS glacier uses tape, which is why data retrieval takes a few hours from when you submit the request to when you can actually download the data, and costs a lot.
AFAIK Glacier is unlikely to be tape based. A bunch of offline drives is more realistic scenario. But generally it’s not public knowledge unless you found some trustworthy source for the tape theory?
I have my BD/DVD/CD collection backed up to S3 Glacier. It’s incredibly cheap, offsite, and they worry about the infrastructure. The amount of Hard drive and infrastructure space you’ll need to back up nearly that amount will cost you the about the same give or take. Yes it’ll cost a bit in the event of a catastrophic restore, but if I have something happen at the house, at least I have an offsite backup.
How much does Glacier cost you? Last time I checked, some hosts had warm storage for around the same price, at least during Black Friday or New Year sales.
I can’t recall storage costs (they’re on the website somewhere but are not straightforward).
I was paying maybe $7 a month for a few hundred Gb, although not all of that was glacier.
But retrieval was a pain. There’s no straightforward way to convert back from glacier for a lot of files and there’s a delay. The process creates a non-glacier copy with a limited lifespan to retrieve.
Then the access costs were maybe $50 to move stuff out.
I moved to rsync.net for the convenience and simplicity. It even supported setting up rclone to access s3 directly. So I could do cloud-to-cloud to copy the files over.
Short answer: figure out how much of that is actually irreplaceable and then find a friend or friends who’d be willing to set aside some of their storage space for your backups in exchange for you doing the same.
Tailscale makes the networking logistics incredibly simple and then you can do the actual backups however you see fit.
