Git eyes 2.7x faster SHA-1 collision checks via C port

Scott Chacon has proposed a C acceleration path for Git’s SHA-1 collision detection that roughly triples hashing throughput and nearly halves single-threaded index-pack time on common hardware, cutting the safety tax that every untrusted object currently pays.

Unless a build swaps in another SHA-1, Git routes every object through sha1collisiondetection. That is deliberate after the SHAttered attacks, but expensive: on a 4-vCPU Xeon with SHA-NI and AVX2, today’s detector runs near 400-450 MiB/s against OpenSSL’s plain SHA-1 at about 1100 MiB/s; an Apple M5 Max sits near 700 MiB/s. Chacon’s series brings the Xeon path to about 900-1000 MiB/s and the M5 Max to roughly 2 GiB/s (about 2.7× and 2.85× overall). Single-threaded index-pack of git.git falls from 24.3s to 12.7s on the Xeon and from 16.1s to 8.7s on the M5 Max. Collision detection, which once cost 1.5-2.5× on top of hashing, drops to roughly 0.2×.

The work follows Sam Reis’s sha1dc Rust crate, already adopted by gitoxide: hardware SHA-1 compression that spills the message schedule, a vectorized unavoidable-bitconditions filter that discards most blocks before expensive recompression, and hardware recompression of the rest. Git’s optional Rust support is too new and too narrow for every build, and the third-party sha1dc tree is awkward to reshape, so the series adds a parallel accelerator that still uses sha1dc’s context and disturbance-vector tables. Portable, SSE2, AVX2, SHA-NI, and arm64 SHA backends are included.

Chacon disclosed that much of the code is AI-generated and signed off under the DCO. Maintainer Junio C Hamano asked how that squares with Git’s AI policy and license provenance; Chacon said Reis was aware and supportive, and that he had proposed a policy update aligned with Software Freedom Conservancy guidance. Reis confirmed he is “very happy” for Git to take the speedup. Johannes Schindelin posted an overlapping series, argued both have merit, offered a small SHA-NI tweak closing most of a remaining gap to the Rust crate, and supplied a generator so the large filter tables can be regenerated from Reis’s sources rather than hand-maintained. Sebastian Thiel of gitoxide called the win “impactful” at hosting scale, where safe SHA-1 has been a painful clone-time cost.