🦞🌯 Lobster Roll

Thread

Crawling a billion web pages in just over 24 hours (andrewkchan.dev)
L 21 pts 14 comments by gerikson Feb 23, 2026 performanceweb thread

Stories related to "Crawling a billion web pages in just over 24 hours" across the full archive.

Crawling a billion web pages in just over 24 hours, in 2025 (andrewkchan.dev)
HN 6 pts 0 comments by emschwartz Jul 16, 2025 thread
Crawling a billion web pages in just over 24 hours, in 2025 (andrewkchan.dev)
HN 3 pts 0 comments by DocFeind Jul 17, 2025 thread
Crawling a billion web pages in just over 24 hours, in 2025 (andrewkchan.dev)
HN 199 pts 58 comments by pseudolus Feb 23, 2026 thread
Crawling a billion web pages in just over 24 hours (andrewkchan.dev)
L 21 pts 14 comments by gerikson Feb 23, 2026 performanceweb Web DevelopmentData / Databases / Infrastructure thread
How to crawl a quarter billion webpages in 40 hours (michaelnielsen.org)
L 4 pts 0 comments by jcs Aug 10, 2012 networkingreleasescaling Data / Databases / InfrastructureProductivity / Career / Business thread
We rendered a million web pages to find out what makes the web slow (catchjs.com)
L 59 pts 14 comments by friendlysock Dec 23, 2020 performanceweb Web DevelopmentData / Databases / Infrastructure thread
Why Web Pages Suck (stratechery.com)
L 17 pts 3 comments by tjvantoll Jul 21, 2015 web Web Development thread
Accelerated Mobile Pages – A new approach to web performance (ampproject.org)
L 6 pts 0 comments by asthasr Oct 26, 2015 mobileweb Web Development thread
How We Moved 34,000 WIRED Pages to One Site in 9 Hours (wired.com)
L 1 pts 0 comments by flyingfisch Nov 2, 2015 javascriptprogrammingweb Web DevelopmentProgramming (General) thread
Why Do Websites Publish AMP Pages? (daringfireball.net)
L 26 pts 23 comments by flyingfisch Oct 26, 2016 practicesweb Web DevelopmentProductivity / Career / Business thread
Social Fixer malware campaign uses NPM to inject code into webpages (numin.it)
L 4 pts 12 comments by friendlysock Jul 3, 2017 nodejssecurityweb Web DevelopmentProgramming (General)Security / Privacy thread
Is WEBrick Webscale? (schneems.com)
L 15 pts 4 comments by schneems Aug 1, 2017 performancerubyweb Web DevelopmentProgramming (General)Data / Databases / Infrastructure thread
Optimizing web servers for high throughput and low latency (blogs.dropbox.com)
L 2 pts 0 comments by hwayne Sep 6, 2017 performanceweb Web DevelopmentData / Databases / Infrastructure thread
How 3rd Party Scripts can be performant citizens on the web (twnsnd.com)
L 2 pts 5 comments by ryantownsend Sep 14, 2017 javascriptperformanceweb Web DevelopmentData / Databases / Infrastructure thread
A Guide to Faster Web App I/O and Data Operations with Streams (sitepen.com)
L 1 pts 0 comments by pushcx Oct 26, 2017 performanceweb Web DevelopmentData / Databases / Infrastructure thread
A Pinterest Progressive Web App Performance Case Study (medium.com)
L 1 pts 0 comments by soulcutter Nov 30, 2017 javascriptperformanceweb Web DevelopmentData / Databases / Infrastructure thread
Just because your site isn't for emerging markets, doesn't excuse you from web performance optimisation (twnsnd.com)
L 17 pts 10 comments by ryantownsend Jan 18, 2018 performanceweb Web DevelopmentData / Databases / Infrastructure thread
The Unbearable Lightness of Web Pages (thelocalyarn.com)
L 24 pts 13 comments by calvin Jan 24, 2018 web Web Development thread
nuster- v1.7.9.4 is released (A web caching proxy server) (github.com)
L 2 pts 0 comments by NusterCacheServer Feb 5, 2018 devopslinuxperformancereleaseshowweb Web DevelopmentSystems / Low-Level / OSProgramming (General)Data / Databases / Infrastructure thread
Web cache server performance benchmark: nuster vs squid (github.com)
L 7 pts 0 comments by NusterCacheServer Feb 7, 2018 devopslinuxperformanceweb Web DevelopmentSystems / Low-Level / OSData / Databases / Infrastructure thread
Smaller Lodash bundles with Webpack and Babel (nolanlawson.com)
L 2 pts 0 comments by nolan Mar 20, 2018 javascriptperformanceweb Web DevelopmentData / Databases / Infrastructure thread
Yesquel: Scalable SQL Storage for Web Applications (2015) (cs.nyu.edu)
L 1 pts 0 comments by nickpsecurity May 7, 2018 databasespdfperformanceweb Web DevelopmentData / Databases / Infrastructure thread
Abstract: "Web applications have been shifting their storage systems from SQL to NOSQL systems. NOSQL systems scale well but drop many convenient SQL features, such as joins, secondary indexes, and/or transactions. We design, develop, and evaluate Yesquel, a system that provides performance and sca...
Web Framework Performance Comparison Round 16 (techempower.com)
L 6 pts 5 comments by yumaikas Jun 7, 2018 performanceweb Web DevelopmentProgramming (General)Data / Databases / Infrastructure thread
Web cache server HTTP/2 performance benchmark: nuster vs nginx (github.com)
L 4 pts 3 comments by NusterCacheServer Jun 13, 2018 performanceweb Web DevelopmentData / Databases / Infrastructure thread
How to Hurricane-Proof a Web Server (2017) (arstechnica.com)
L 4 pts 3 comments by nickpsecurity Jun 22, 2018 linuxnetworkingperformanceweb Web DevelopmentSystems / Low-Level / OSData / Databases / Infrastructure thread
Fairytale about performance in web application (itnext.io)
L 4 pts 0 comments by bliashenko Aug 15, 2018 debuggingjavascriptperformanceweb Web DevelopmentProgramming (General)Data / Databases / Infrastructure thread
Story describes how I fixed JS execution time from 26 to 1 sec
Conservative web development (drewdevault.com)
L 49 pts 22 comments by mrr Sep 4, 2018 performanceweb Web DevelopmentData / Databases / Infrastructure thread
Staticman --- add comments to GitHub Pages without JavaScript (staticman.net)
L 13 pts 8 comments by inactive-user Sep 27, 2018 performanceweb Web DevelopmentProgramming (General)Data / Databases / Infrastructure thread
Installing Hugo and publishing Hugo web-pages on OpenBSD server (bsdboy.ml)
L 4 pts 3 comments by bsdb0y Oct 1, 2018 openbsdweb Web DevelopmentSystems / Low-Level / OS thread
Noria: dynamic, partially-stateful data-flow for high-performance web applications (usenix.org)
L 35 pts 9 comments by jonhoo Oct 8, 2018 compscidatabasesdistributedperformancerustweb Web DevelopmentSystems / Low-Level / OSProgramming Languages / CS TheoryData / Databases / Infrastructure thread
A while ago, I [asked](https://lobste.rs/s/cqnzl5/lobste_rs_access_pattern_statistics_for) about traffic statistics for lobste.rs for a research project. Finally, the result of that work has now been published as the linked system, which speeds up lobste.rs by ~5x over that provided by MySQL. The so...