I understand wanting to get money, but honestly, there is no way I would give money to this website in it's current state, you are giving me far too little info before asking me to hand over a credit card.
Then, if someone gives you £19, a crazy amount of money honestly, the last page of the report is an advert to give them 4 times more!
I don't know if I would pay £19 for a general state-of-the-area report. I would almost certainly have paid £100-300 for a service that took my planning application, critically reviewed it and told me which aspects were and were not likely to pass, with references to specific examples within my local area.
1. Brilliant! Governments (and corps) treat public data like it’s theirs not ours. Information yearns to be free.
2. Having said that, you are likely violating T&Cs by scraping at all.
3. It is a lot easier to defend your position if you are making it free and public yourself.
4. But paying for food is nice
5. I suggest the business model here is providing architects and lawyers with strong evidence of prior planning decisions nationally
Most people applying for (difficult) planning have experience locally. But the planning system is a mess because it is not coherent nationally or regionally. The win here is not providing a copy of your data (that has legal issues) but providing pointers to decisions that support the case of the person paying you.
So I want to turn an old pub into tasteful housing and a cafe for the local village. The local planning team don’t like it, I could spend money bribing them and the councillors (see how much I understand British democracy) or I could get from you the fifteen pub to housing conversion decisions from around the country and use that to help my bribed councillors defend their u-turn
Everyone wins :-)
But it’s a big mindset chnage (one that will benefit the whole country) but it’s slow.
I think the “push for public policy improvements” angle if genuine will get you a lot more respect and kudos when things get sticky. Good luck
No-one has figured out how to make money off open source (while sticking to the basic principles. Jeff Bezos makes a fortune off of it)
Most people who open source their code that I have known and still wanted to be paid / recognised for their effort have always been disappointed
Can I suggest you mentally put the work you have done to date into a box marked “the past”, open the data, start yourself as part of the community trying to make government code and data open, and sell your skills - the old “consultancy paying the bills” approach
Trying to make cash off public data will just confuse the message, and start to build resentment. Make a clean and clear statement, Sell your services on top. Expand to other forms of data scraping in government.
It’s a tough road - good luck
See also the open addresses project by Data Adaptive [1] which is using Freedom of Information requests to publish public council tax address data. The problem they have run into there is that their address datasets are derived from proprietary Ordnance Survey data.
It looks like data.gov.uk is in the process of standardising the planning application process, and publishing them under OGL [2].
[1]: https://www.owenboswarva.com/blog/post-addr44.htm [2]: https://www.planning.data.gov.uk/dataset/planning-applicatio...
It would be good to add appeal data in (also a public gateway) to show which councils are just being unreasonable.
I personally think the planning regulations in this country are the cause of many ills, including the housing shortage. It just costs so much to get through planning these days, it is often just not worth it. Data like this could help us get that changed.
You guys have all kinds of pro-individualistic, borderline nonsensical residental housing laws like "right to light" and "right to view". It's completely incompatible with "build more". Most British people view their privacy (or perceived privacy) as a higher priority than fixing the housing market. "It's so overlooked" is such common comment and it's almost bizarre to someone used to living in a higher density environment (like the UK very much is).
For the more challenging scrapes, would highly recommend using the Chrome Devtools MCP to be able to attach the network requests, being made by the browser to the site, as context for your agent/LLM chat - this approach really helped me to write a solid API-based scraper (also using curl_cffi) and bypassed the old tedious playwright-based approach I used to rely on.
If you really want the data, just FOI it for goodness' sake.
I get the distinct impression that many of these outfits aren't really advocating for impoved transparency but are simply trying to exploit and monetise illicitly obtained government data to make a quick buck.
I'm not implying that anything would get deliberately redacted, but it seems likely that information released through other channels would not match the web. A request might also reveal information that was not on the web.
What other choices are there?
I appreciate that won't necessarily capture live / recent data. But it might be quicker than waiting for rate-limits to reset.
There's a Royal Institute of Town Planners, they probably have a magazine you could advertise in (but equally that might get you blocked, idk).
RICS people could probably use the data too? I guess it's useful house-buyer info; houses in the vicinity had successful loft conversions, say.
On the data side - it's something of a moat for you now, but I could see you being successful with FOI requests. An MP might be interested in championing open data access.
> UK planning data is technically public.
it's public, but still copyrighted by those who submitted it
the councils also have database rights over their database, unless you've obtained explicit permission from them directly
https://en.wikipedia.org/wiki/Database_right#United_Kingdom
> I ended up writing several scrapers: a standard requests-based one, a Playwright-based one for councils that block anything that doesn't look like a real browser, and a curl_cffi one for TLS fingerprinting.
so they're explicitly trying to stop you doing this, and ... you're openly admitting to bypassing their technical measures to try and stop you?
have you heard of the Computer Misuse Act?
I doubt the 240 councils are going to be happy once they find out you've done this, especially if you're selling it on for profit
I did a search for my postcode and got given results for a different area and council miles away
The script was extracting prefixes from individual application addresses, and Ceredigion's database happened to have a chunk of records with HR* postcodes (data entry errors at source. The addresses are Welsh places like Aberystwyth and Borth, but with Hereford postcodes attached). Those errors polluted the lookup table, so HR* ended up pointing at Ceredigion.
Fixed by trusting only the manual postcode-to-council mappings for councils that have them, rather than supplementing from address extraction. The postcode you sent now correctly shows "not yet in coverage" rather than results from rural Wales. Cheers for flagging. That was a zinger of a bug.
On the grind, why not get an agent to help you build the long tail of deterministic scrapers? Claude etc is really shockingly good at this kind of moderate-complexity iterative work, it will just keep going around the fetch/parse/understand loop until it has what you're looking for.
Careful not to expose the councils too publicly before they shut you off
> You may not use automated tools to scrape, copy, or bulk-download data from our service.
Pot kettle, huh.
The open source angle is something I'm increasingly considering, especially after a local government IT person made a fair point on this thread about the strain it causes. It won't fix the scraping load directly, but might frame the project as public-interest rather than attempting to make a bit of extra money. Tbh, the real value is probably in serving property developers and consultants, not emailing £19 PDFs to homeowners. Got a lot to think about.