ITADN

Grouping large files is abnormally slow, better results by doing incremental runs and increasing max size

#326Opensbailliez 创建于 2025-12-13
S
sbailliezcommented
I'm using fclones 0.35.0 on macos 26.1 to try to find duplicates on an NVME SSD disk connected via Thunderbolt 4. I get about 2800MB/s write and 3300MB/s read. The ssd is formatted with EXFAT for compatibility Windows/Mac fclones seems to have troubles dealing with large files on initial runs, for example: fclones group . --name '*.MP4' --cache .cache/fclones --max 700MB Would for example move quickly a few GB and then slow down to a molasse and takes say 10min. If I restart from scratch (empty cache) and execute sequentially: fclones group . --name '*.MP4' --cache .cache/fclones --max 500MB fclones group . --name '*.MP4' --cache .cache/fclones --max 550MB fclones group . --name '*.MP4' --cache .cache/fclones --max 600MB fclones group . --name '*.MP4' --cache .cache/fclones --max 650MB fclones group . --name '*.MP4' --cache .cache/fclones --max 700MB then it would take a few seconds to execute each of them. I initially tried to run on 115K files totalling about 1.3TB which were a mix of jpg, nef, dng, mp4 - After more than 12h it was still grouping content. Doing the incremental method above, it took a couple minutes to do all the mp4 which were definitely the problem, a few seconds to do the all the other files in one go.
0 条评论