Title is TLDR. More info about what I’m trying to do below.
My daily driver computer is Laptop with an SSD. No possibility to expand.
So for storage of lots n lots of files, I have an old, low resource Desktop with a bunch of HDDs plugged in (mostly via USB).
I can access Desktop files via SSH/SFTP on the LAN. But it can be quite slow.
And sometimes (not too often; this isn’t a main requirement) I take Laptop to use elsewhere. I do not plan to make Desktop available outside the network so I need to have a copy of required files on Laptop.
Therefor, sometimes I like to move the remote files from Desktop to Laptop to work on them. To make a sort of local cache. This could be individual files or directory trees.
But then I have a mess of duplication. Sometimes I forget to put the files back.
Seems like Laptop could be a lot more clever than I am and help with this. Like could it always fetch a remote file which is being edited and save it locally?
Is there any way to have Laptop fetch files, information about file trees, etc, located on Desktop when needed and smartly put them back after editing?
Or even keep some stuff around. Like lists of files, attributes, thumbnails etc. Even browsing the directory tree on Desktop can be slow sometimes.
I am not sure what this would be called.
Ideas and tools I am already comfortable with:
-
rsync is the most obvious foundation to work from but I am not sure exactly what would be the best configuration and how to manage it.
-
luckybackup is my favorite rsync GUI front end; it lets you save profiles, jobs etc which is sweet
-
freeFileSync is another GUI front end I’ve used but I am preferring lucky/rsync these days
-
I don’t think git is a viable solution here because there are already git directories included, there are many non-text files, and some of the directory trees are so large that they would cause git to choke looking at all the files.
-
syncthing might work. I’ve been having issues with it lately but I may have gotten these ironed out.
Something a little more transparent than the above would be cool but I am not sure if that exists?
Any help appreciated even just idea on what to web search for because I am stumped even on that.
This sounds like a setup that is broken in a few ways. First, you need to clean up the desktop. Start by replacing the USB drives. It is a bad idea to use them as they are more likely to fail and have limited speeds. I would get either moderately high RPM hard drives or sata SSDs. Both will be more reliable and have greater speed. Just make sure that they are properly mounted. This is especially important for HDDs as they are very sensitive to vibration and need to be kept still by a proper mount with rubber padding.
For the desktop itself, make sure it is fast enough to provide enough throughput. You should be using a CPU with at least 4 cores and DDR4 memory. The more memory you have the more data can be cached. I would go for 16 or 32gb and if you can go even higher. Also make sure you are using gigabit Ethernet with a gigabit switch.
Now for the software. I would install TrueNAS scale as it uses ZFS which is a create filesystem for data storage. TrueNAS can be a little tricky but it is all controllable though a web interface and if you need help guides can be found online.
For accessing the data, I would use either a SMB or NFS share. Smb tends to be simpler and has better cross platform support. NFS can be faster but it is tricky to configure. Make sure that your laptop has a Ethernet connection as doing things over WiFi is going to be slow. You might be able to get away with it if you have a newer wifi revision with a newer WiFi card.
If you are still looking to solve the original problem, you can use Syncthing. Syncthing has a few click install for TrueNAS and you can find guides on how to configure it in a way that works for you.
Maybe Syncthing is the way forward. I use it for years and am reasonably comfortable with it. When it works, it works. Problems is that when it doesn’t work, it’s hard to solve or even to know about. For the present use case it would involve making a lot of shares and manually toggling them on and off all the time. And would need to have some kind of error checking system also to avoid deleting unsynced files.
Others have also suggested NFS but I am having a difficult time finding basic info about what it is and what I can expect. How is it different than using SSHFS mounted? Assuming I continue limping along on my existing hardware, do you think it can do any of the local caching type stuff I was hoping for?
Re the hardware, thanks for the feedback! I am only recently learning about this side of computing. Am not a gamer and usually have had laptops, so never got too much into the hardware.
I have actually 2 desktops, both 10+ years old. 1 is a macmini so there is no chance of getting the storage properly installed. I believe the CPU is better and it has more RAM because it was upgraded when it was my main machine. The other is a “small” tower (about 14") picked up cheaply to learn about PCs. Has not been upgraded at all other than SSD for the system drive. Both running debian now.
In another comment I ran iperf3 Laptop (wifi) —> Desktop (ethernet) which was about 80-90MBits/s. Whereas Desktop —> OtherDesktop was in the 900-950 MBits/s range. So I think I can say the networking is fine enough when it’s all ethernet.
One thing I wasn’t expecting from the tower is that it only supports 2x internal HDDs. I was hoping to get all the loose USB devices inside the box, like you suggest. It didn’t occur to me that I could only get the system drive + one extra. I don’t know if that’s common? Or if there is some way to expand the capacity? There isn’t too much room inside the box but if there was a way to add trays, most of them could fit inside with a bit of air between them.
This is the kind of pitfall I wanted to learn about when I bought this machine so I guess it’s doing its job. :)
Efforts to research what I would like to have instead have led me to be quite overwhelmed. I find a lot of people online who have way more time and resources to devote than I do, who want really high performance. I always just want “good enough”. If I followed the advice I found online I would end up with a PC costing more than everything else I own in the world put together.
As far as I can tell, the solution for the miniPC type device is to buy an external drive holder rack. Do you agree? They are sooo expensive though, like $200-300 for basically a box. I don’t understand why they cost so much.
Simplify it down if you can. Forget NFS, SSHFS and syncthing as those are to complex and overkill at the moment. SMB is dead simple in a lot of ways and is hard to mess up.
I would start by figuring out how to get more drives in your desktop. SSDs can be put anywhere but HDDs need to be mounted. I would get two big disks and put them in raid 1 and then get a small disk for the boot drive. TrueNAS is still going to be the easiest route and TrueNAS scale is based on Debian under the hood although its designed to not be modified for reliability.
Forget the USB drives. I did that for a while and it is bad in so many ways. It will come back to haunt you even if the work fine for now.
For the desktop you always could upgrade the ram or CPU if that becomes the bottleneck. If the ram is 8gb or less you should upgrade it. Make sure you are getting ram that is fast enough to keep up with the CPU. You can find more information about your CPU and its ram speed online.
For your WiFi, it actually isn’t bad. You might be ok using it depending on what you are doing.
For the minipc I wouldn’t use it as a NAS. You can but it is probably going to be limiting.
In case your desktop doesn’t have enough Sata ports you could get a ePCI sata card. They are pretty cheap ($10) and will potentially work better if your hardware is limited.
Forget NFS, SSHFS and syncthing as those are to complex and overkill at the moment. SMB is dead simple in a lot of ways and is hard to mess up.
OTOH, SSHFS and syncthing are already humming along and I’m framiliar with them. Is SMB so easy or having other benefits that would make it better even though I have to start from scratch? It looks like it (and/or NFS) can be administered from cockpit web interface which is cool.
Now that I look around I think I actually have a bit of RAM I could put in the PC. MacMini’s original RAM which is DDR3L; but I read you can put it in a device that wants DDR3. So I will do that next time it’s powered off.
Thanks for letting me know I could use an expansion card. I was wondering about that but the service manual didn’t mention it at all and I had a hard time finding information online.
Is this the sort of thing I am looking for: SATA Card 4 Port with 4 SATA Cables, 6 Gbps SATA 3.0 Controller PCI Express Expression Card with Low Profile Bracket Support 4 SATA 3.0 Devices ($23 USD) I don’t find anything cheaper than that. But there are various higher price points. Assuming none of those would be worthwhile on a crummy old computer like I have. Is there any specific RAID support I should look for?
I have only the most cursory knowledge of RAID but can tell it becomes important at some point.
But am I correct in my understanding that putting storage device in RAID decreases the total capacity? For example if I have 2x6TB in RAID, I have 6 TB of storage right?
Honestly, more than half my data is stuff I don’t care too much about keeping. If I lose all the TV shows I don’t cry over it. Only some of it is stuff I would care enough to buy extra hardware to back up. Those tend to be the smaller files (like documents) whereas the items taking up a lot of space (media files) are more disposable. For these ones “good enough” is “good enough”.
I really appreciate your time already and anything further. But I am still wondering, to what extent is all this helping me solve my original question which is that I want to be able to edit remote files on Desktop as easily as if they were local on Laptop? Assuming i got it all configured correctly, is GIMP going to be just as happy with a giant file lots of layers, undos, etc, on the Desktop as it would be with the same file on Laptop?
2 internal hdds
As I understand the PC case is some sort of small form factor and the motherboard has more SATA ports to plug in disks.
If that’s the case then try to find old pc case in the dumpster or from a neighbour. It should have more space for disks.
Some PCs 15 years ago had incompatible layouts so be a little cautious but gratis is a fair price.
Do you mean take the board out of this case and put it in another, bigger one?
I actually do have a larger, older tower that I fished out of the trash. Came with a 56k modem! But I don’t know if they would fit together. I also don’t notice anywhere particularly suitable to holding a bunch of storage; I guess I would have to buy (or make?) some pieces.
Here is the board configuration for the Small Form Factor:
I did try using #9 and #10 for storage and I seem to recall it kind of worked but didn’t totally work but not sure of the details. But hey, at least I can use a CD drive and a floppy drive at the same time!
git-annex maybe?
- wifi is less responsive than ethernet
- You don’t even need encryption on local connection(no need for ssh) if it is never global.
- if you have that many external HDD, I would rather get a NAS and call it a day, it has everything configured for you as well.
- I don’t know what big data you access on, get a larger SSD to store whatever you might visit often from your HDDs.
Strong disagreement on №2. That kinda thinking is how you get devices on your home network to join a malicious botnet without your knowledge and more identity theft. ALL network communications should assume that a malicious actor may be present and use encryption-in-transit for anything remotely approaching private, identifiable, or sensitive.
Easiest for this might be NextCloud. Import all the files into it, then you can get the NextCloud client to download or cache the files you plan on needing with you.
hmm interesting idea. I do not get the idea that nextcloud is reliably “easy” as it’s kind of a joke how complex it can be.
Someone else suggested WebDAV which I believe is the filesharing Nextcloud uses. Does Nextcloud add anything relevant above what’s available from just WebDAV?
I’d say mostly because the client is fairly good and works about the way people expect it to work.
It sounds very much like a DropBox/Google Drive kind of use case and from a user perspective it does exactly that, and it’s not Linux-specific either. I use mine to share my KeePass database among other things. The app is available on just about any platform as well.
Yeah NextCloud is a joke in how complex it is, but you can hide it all away using their all in one Docker/Podman container. Still much easier than getting into bcachefs over usbip and other things I’ve seen in this thread.
Ultimately I don’t think there are many tools that can handle caching, downloads, going offline, reconcile differences when back online, in a friendly package. I looked and there’s a page on Oracle’s website about a CacheFS but that might be enterprise only, there’s catfs in Rust but it’s alpha, and can’t work without the backing filesystem for metadata.
Why is VPN to your main device not an option?
Nevermind. I read your question way wrong. You have a local network and want better access to things there.
Different filesystems and network service which exposes. SMB will be the worst performance, NFS possibly the best. You’re also going to have lag on however you’re accessing these files from the client, so maybe there’s an issue.
