Page MenuHomePhabricator

Hashing can be very slow for some
Closed, ResolvedPublic

Description

For some people it seems that hashing the local mods can take a very long time for some people. For others it only takes a few minutes.

More information (such as log files) is needed from people suffering from this issue.

Possible fix would be changing the hashing of the files to something quicker, however that would not explain away the discrepancy between users.

Event Timeline

jsm created this task.Apr 16 2016, 11:15 PM
jsm created this object in space S1 Public.
jsm created this object with visibility "Public (No Login Required)".
jsm added a comment.Apr 16 2016, 11:29 PM

Worth checking anti-virus. Should not have any effect (its only reading files) but it might be worth checking.

jsm added a comment.Apr 17 2016, 5:23 PM

Need more information on this but:
SSDs (and the lack of) seem to be the cause. Very easy to actually test.

Tested Hashing on my 7200rpm HDD and it substantially increases the time taken to run through the hashing process.

Started Hashing at 2016-04-19 22:59:58.2315
Completed Hashing at 2016-04-19 23:12:21.0128

Taking 12 minutes 19 seconds.

Compared to my SSD which usually takes under 1 minute.

Whilst running through the hashing process the drive read/write speed stays around 16MB/s until the last 30 seconds where it goes to 112MB/s.

I can only assume the slower speed is because it is creating the hash files and then at the end the last 30 seconds it is reading to check them.

jsm raised the priority of this task from Normal to High.Apr 19 2016, 10:24 PM
Arron17 added a comment.EditedJun 1 2016, 11:06 PM

On SSD:
C:\Games\ARMA MODS - Beowulf
Using new system
Hashed in 00:01:03.1975371
Using MD5
Hashed in 00:00:51.3147188
Using old system
Hashed in 00:01:18.7194159

On HDD:
Folder to hash
D:\ARMA MODS - Beowulf
Using new system
Hashed in 00:03:22.9834684
Using MD5
Hashed in 00:03:22.7837357
Using old system
Hashed in 00:03:21.7188590

jsm added a comment.Jun 2 2016, 12:14 AM

Latest working theory:
The multi-threaded nature of the hashing (ie hashing multiple folders at the same time) is a *horrifically terribly bad idea* on a spinning disk (for obvious reasons..). The program seems to have been forcing the drive to jump all over the place.
This would explain why the problem is worsened on fragmented HDDs.

Thanks to @Arron17 again for spotting this..

jsm moved this task from Backlog to Completed on the BSU.Sync board.Jun 2 2016, 12:33 AM