There are various hashing algorithms in existence like the non-cryptographic 16-character XXH-3 (64-bit) hash (https://github.com/Cyan4973/xxHash) and the cryptographic 64-character BLAKE3 hash (https://github.com/BLAKE3-team/BLAKE3) which hash faster than your drive read speed and as such are significantly faster than the 30-year-old CRC/MD5/SHA-1 combo.
For example, @johnsanc performed some tests hashing a 30 GB file which yielded the following results:
Hashing of a 30 GB File
========================
BLAKE3 = 16.6 sec
CRC32 = 33.6 sec
MD5 = 69.6 sec
SHA1 = 72.1 sec
SHA256 = 143.4 sec
SHA512 = 105.7 sec
^ And XXH3 would be even faster!
If Romvault built in optional, user-customable support for alternative hashing algorithms like XXH3 and BLAKE3, we could theoretically build XXH3 / BLAKE3 etc. variants of existing DATs and significantly improve the scanning speed of multi-TB DATs.
Like for example, I'd love the ability to only scan sets three times faster using XXH3 rather than having to choose from either CRC32 or CRC32/MD5/SHA-1.
While datting projects don't currently offer those hashes, we have to start somewhere. Adding alternative hash support would be a good impetus to get them integrated into No-Intro etc.
rarenight
last year
There are various hashing algorithms in existence like the non-cryptographic 16-character XXH-3 (64-bit) hash (https://github.com/Cyan4973/xxHash) and the cryptographic 64-character BLAKE3 hash (https://github.com/BLAKE3-team/BLAKE3) which hash faster than your drive read speed and as such are significantly faster than the 30-year-old CRC/MD5/SHA-1 combo.
For example, @johnsanc performed some tests hashing a 30 GB file which yielded the following results:
Hashing of a 30 GB File========================BLAKE3 = 16.6 secCRC32 = 33.6 secMD5 = 69.6 secSHA1 = 72.1 secSHA256 = 143.4 secSHA512 = 105.7 sec^ And XXH3 would be even faster!
If Romvault built in optional, user-customable support for alternative hashing algorithms like XXH3 and BLAKE3, we could theoretically build XXH3 / BLAKE3 etc. variants of existing DATs and significantly improve the scanning speed of multi-TB DATs.
Like for example, I'd love the ability to only scan sets three times faster using XXH3 rather than having to choose from either CRC32 or CRC32/MD5/SHA-1.
While datting projects don't currently offer those hashes, we have to start somewhere. Adding alternative hash support would be a good impetus to get them integrated into No-Intro etc.