For context:
I’m copying the same files to the same USB drive for comparison from Windows and from my Fedora 41 Workstation.
Around 10k photos.
Windows PC: Dual Core AMD Athlon from 2009, 4GB RAM, old HDD, takes around 40min to copy the files to USB
Linux PC: 5800X3D, 64GB RAM, NVMe SSD, takes around 3h to copy the same files to the same USB stick
I’ve tried chagning from NTFS to exFAT but the same result. What can I do to improve this? It’s really annoying.
Depends on the distro and desktop environment but some will “transfer” files to a software buffer that doesn’t actually write the data immediately. Works for limiting unnecessary writes on Flash memory but not USB sticks that are designed to be inserted and removed at short notice.
You can force Linux to commit pending writes using the ‘sync’ command. Note it won’t give you any feedback until the operation is finished (multiple minutes for a thumbdrive writing GBs of data) so append & to your command (‘sync &’) to start it as its own process so you don’t lock the terminal.
You can also watch the progress using the command form this Linux Stack Exchange Q;
https://unix.stackexchange.com/questions/48235/can-i-watch-the-progress-of-a-sync-operation#48245
Side question though, it seems that there are faster options. How come we don’t use those in GUI file explorers if they’re faster?
As I’ve already mentioned, sync does absolutely nothing. The copy took so long that the sync command exited 4 times while the files were still transfering and were nowhere near finishing. Regarding the watch -d grep -e Dirty: -e Writeback: /proc/meminfo command, I did not mention it in this thread but I did try it and yes, there was some almost 900k kB of data in the “Dirty” buffer that went up and down constantly even after I’ve disabled the caching.
I haven’t had this problem. It could be the filesystem you’re using? Sometimes Linux gets weird with Windows filesystems. Try formatting it to ext4.
Random peripherals get tested against windows a lot more than Linux, and there are quirks which get worked around.
I would suggest an external SSD for any drive over 32GB. Flash drives are kind of junk in general, and the external SSDs have better controllers and thermals.
Out of curiosity, was the drive reformatted between runs, and was a Linux native FS tried on the flash drive?
The Linux native FS doesn’t help migrate the files between Windows and Linux, but it would be interesting to see exFAT or NTFS vs XFS/ext4/F2FS.
Do you have the same speeds when coping say a single 1gb file? A lot of small files introduces a bit more overhead. Rsync can help somewhat
I’m copying 6 video files that are 40GB total and it’s been over 3h now and still not finished so it’s not just a lot of small files. It’s just slow as hell in general. Yes, the USB is 3.0 connected to 3.0 port verified it’s actually running at 3.0 bus. No, it’s not fault of the USB drive as this takes around 30min from USB 2.0 on Windows 10. Yes, I’ve tried Fat32, exFAT and NTFS… I couldn’t care less about ext4 for this particular use case so it’s not relevant and I haven’t tried it yet because I’m still stuck copying. Not sure what rsync does different, I just use standard CTRL+C/CTRL+V copy/paste that I expect to work flawlessly in 2025. No idea why I would want to use command line for copying files to USB drive. This seems like an ongoing problem for over 10 years from what I’ve been looking at trying to find solution, I found none that worked yet, just the same comments I’m getting here mostly.
Very strange… So it sounds like you’re using whatever the default file manager is for your desktop, there really isn’t any reason the filesystem type would make things that much slower. Something must be very different about your system to be slowing transfers down that much.
I would use iotop to see how much data is being written, where, and what speeds it’s getting but if you prefer a graphical version of that maybe “system monitor” is available to you in gnome or whatever desktop you use. You’ve probably already tested other drives I guess, maybe try just booting a fresh live USB of something and see if the problem persists there too.
Personally, i have never experienced problems while reading from USB sticks, but i have while writing. I have a 15+ years old USB2 stick and a new USB3.x stick. The USB2 stick writes with constant ~20MB/s, while USB3 is all over the place between 200MB/s and ~0.1MB/s. Unusable for me. For a while i used external HDDs and SSDs over USB3, as they somehow run without problems, but they are cumbersome and expensive.
Therefore i have switched to transfer files over the network (for large files i plug in Ethernet) using KDE connect. Unfortunately it can not send folders (yet), so i would .tar them before sending, and untar them after.
LocalSend would also be an option. Maybe that can do folders natively.
I’m aware of these programs but they are just a way around the problem and not a solution. Besides they have their own limitations… I can’t use KDE Connect because it does not work on my network because I run 3 routers in one network and would have to be connected to the same router which is not possible… Because of the reason why I need to run 3 routers
