Ask HN: Is our data warehouse setup normal or over-complicated?
4 pointsby ealready_value3 hours ago1 comment
  • icedchai2 hours ago
    Without understanding differences between the "source" and "canonical" forms, it is tough to say. Also how much data are we actually talking about? The pipeline you describe may be entirely reasonable, or it may be an over engineered, convoluted contraption that could be replaced with a single DB replica and a few views to simplify queries.

    My experience with QuickSight has been pretty negative. The overall UI/UX is pretty meh. If you're embedding it in your product you may be better off generating your own reports, in app.

    • ealready_value2 hours ago
      The source form is the production database, which is what the current reports pull from. The canonical form is the form that in theory all of the verticals get rolled into, but many of the nuances that our customers are used to having end up getting replaced with similar, but are not quite the same. Right now that's my biggest concern that customers are not going to get the data they need because of this canonical form.

      We're talking about a few-hundred megabytes of data for all of the customers that these reports pull, but that's also for the past 15 years. We do have like 25k customers, which shrinks how much a customer can pull in even further. One last point is that we already de-normalize the report data into its own table specifically for these reports, so that's not something the data warehouse is doing for us.

      I agree with your experience with QuickSight, it is exactly my experience. My preference is to continue using the reports we generate in the app, but I'm trying to wrap my head around cases where this ends up being the better direction.