Ask HN: Scaling a targeted web crawler beyond 500M pages/day

1 pointby honungsburk2 hours ago