Scraping Zillow for the Little Guys(caleb.software)
5 pointsby karagenita day ago1 comment
  • bob1029a day ago
    My go-to for scraping "like a human" is to use headless browser automation. Dump the DOM after a few seconds to allow react jank to settle, and then parse the HTML document for desired properties.

    If you do things correctly this approach should be mostly indistinguishable from an actual user. The automated browser instance is just a normal build of chrome. The biggest trick seems to be patience. If you only make a request once every 60 seconds or so, it's very unlikely that someone or something is going to care about it.

    • karagenita day ago
      Yeah, I'd probably go that route if I wanted to scrape more than the handful of pages I needed for this project. I wonder if it would work on Zillow or not. Even my simple workflow of "click the next button, save the request in devtools, repeat" was suspicious enough to trigger a captcha-type "are you a bot?" challenge. Maybe it was just too many requests quickly like you mentioned, or maybe they're doing something more advanced like mouse movement tracking.