What started as a Flask + Scrapy job board has turned into a small data engineering / analytics pipeline behind the scenes:
scheduled scraping + post-processing daily dataset snapshots data quality reports trend metrics over time DuckDB warehouse outputs internal dashboard with Plotly visualizations Stack is mostly:
Python Flask Scrapy SQLite DuckDB Plotly
I’m especially interested in feedback on:
job board usefulness / UX the filtering around junior and visa discovery whether the analytics / data pipeline side is interesting enough to expose more publicly.
Would love feedback from people who’ve built scraping, analytics, or job-search products.