Grouping large files is abnormally slow, better results by doing incremental runs and increasing max size
I'm using fclones 0.35.0 on macos 26.1 to try to find duplicates on an NVME SSD disk connected via Thunderbolt 4.
I get about 2800MB/s write and 3300MB/s read.
The ssd is formatted with EXFAT for compatibility Windows/Mac
fclones seems to have troubles dealing with large files on initial runs, for example:
fclones group . --name '*.MP4' --cache .cache/fclones --max 700MB
Would for example move quickly a few GB and then slow down to a molasse and takes say 10min.
If I restart from scratch (empty cache) and execute sequentially:
fclones group . --name '*.MP4' --cache .cache/fclones --max 500MB
fclones group . --name '*.MP4' --cache .cache/fclones --max 550MB
fclones group . --name '*.MP4' --cache .cache/fclones --max 600MB
fclones group . --name '*.MP4' --cache .cache/fclones --max 650MB
fclones group . --name '*.MP4' --cache .cache/fclones --max 700MB
then it would take a few seconds to execute each of them.
I initially tried to run on 115K files totalling about 1.3TB which were a mix of jpg, nef, dng, mp4 - After more than 12h it was still grouping content.
Doing the incremental method above, it took a couple minutes to do all the mp4 which were definitely the problem, a few seconds to do the all the other files in one go.
0 条评论