Benchmarks

Test url: https://espn.com Mac M1 64gb 10-core CPU

librariespagesspeed
spider(rust): crawl150,3871m
spider(nodejs): crawl150,387153s
spider(python): crawl150,387186s
scrapy(python): crawl49,5981h
crawlee(nodejs): crawl18,77930m

View the latest runs on github.

-----------------------
Linux
2-core CPU
7 GB of RAM memory
-----------------------

Test url: https://choosealicense.com (small) 32 pages

librariesspeed
spider-rs: crawl 10 samples76ms
scrapy: crawl 10 samples2s

Test url: https://rsseau.fr (medium) 211 pages

librariesspeed
spider-rs: crawl 10 samples3s
scrapy: crawl 10 samples8s
----------------------
mac Apple M1 Max
10-core CPU
64 GB of RAM memory
-----------------------

Test url: https://choosealicense.com (small) 32 pages

librariesspeed
spider-rs: crawl 10 samples286ms
scrapy: crawl 10 samples2.5s

Test url: https://rsseau.fr (medium) 211 pages

librariesspeed
spider-rs: crawl 10 samples2.5s
scrapy: crawl 10 samples10s

Test url: https://a11ywatch.com (medium) 648 pages

librariesspeed
spider-rs: crawl 10 samples2s
scrapy: crawl 10 samples7.7s

Test url: https://espn.com (large) 150,387 pages

librariespagesspeed
spider-rs: crawl 10 samples150,387186s
scrapy: crawl 10 samples49,5981h

Scrapy used too much memory, crawl cancelled after an hour.

Note: The performance scales the larger the website and if throttling is needed. Linux benchmarks are about 10x faster than macOS for spider-rs.