Slow Hugging Face Downloads? Parallel Range Chunks, Tested
Downloading 16.6 GB of Qwen-Image 2.1 weights, both curl and the official client stalled. Switching to 16 parallel 32 MiB Range chunks reached ~15.8 MiB/s during the main model's chunk phase, and both files finished in 36 minutes with official SHA256 checks passing.
The problem
Two files, 7.26 GB and 9.35 GB:
| Method | Result |
|---|---|
| curl, three files in parallel | Stopped after 7 min; neither large file finished |
| Official HF client | Stopped after 9 min; not finished |
| Single-connection probe (1 MiB) | Took 10.1 s |
Two traps:
- A silent log doesn't mean nothing is downloading. Both curl stderr logs were 0 bytes, while the
.partfiles already held 2.8 GB. - Exit code 0 doesn't mean the download is complete. We once got a 2.4 GB file with only 142 MB downloaded, and curl still returned 0.
The approach: parallel Range chunks
- 16 threads, 32 MiB per chunk, each request with a
Rangeheader. - Every response must be HTTP
206, andContent-Rangestart, end and total must match exactly. Short chunks are thrown away and fetched again. - Up to 4 attempts per chunk, with 1/2/4 s backoff.
- Chunks live in a temp folder. A rerun reuses only chunks whose size, range and chunk hash all match.
- Once every chunk is in, assemble the file, verify it against the official LFS SHA256, and only then move it into the model folder.
Results
| File | Chunks done | Incl. assembly, hashing, copy |
|---|---|---|
| Main model 7.26 GB | 438 s (~15.8 MiB/s) | 693 s |
| Text encoder 9.35 GB | 1217 s | 1504 s |
The whole run took 36 min 38 s, and both files passed the official SHA256. There were 57 failed attempts along the way, 16 on the main model and 41 on the encoder, all recovered by retries.
The encoder reused the first 640 MiB left by the old curl run. Before reusing it, we spot-checked chunks: the old file, the new Range chunk and the chunk metadata had to share the same SHA256. The HF client's .incomplete files can't be reused this way, because they aren't guaranteed to be a contiguous prefix.
Takeaways
- Always verify large files with a full hash. Don't trust exit codes.
- Make downloads resumable and reuse complete chunks, so a network hiccup doesn't send you back to zero.
- Speeds only hold for the network you had at the time. Here the main model's chunk phase ran at 15.8 MiB/s, and the full-run average was 7.2 MiB/s including reuse, retries and verification.
Related
- What we tested with these weights: Qwen-Image 2.1 on 6GB
- LingBot on Windows (uses aria2 multi-connection)
FAQ
How to fix slow Hugging Face downloads? Use resumable multi-connection or parallel Range chunk downloads, then verify the whole file against the official SHA256.