What the author is saying in practice is: it's nice to have Meta level funding for effectively infinite distributed storage and compute. In other areas of the industry where you want to stay profitable, metrics stay metrics, because you're submitting 2 integers for an aggregate, rather than tens of megabytes of JSON logs per minute. Then you want to retrieve this in seconds on demand... so that's either preprocessing (ongoing compute) or some serious parallel IO.

I'm glad the author likes wide events for everything. I do too. But once you try to justify the cost, you understand why it's metrics, logs and (often sampled) traces.

  > I’m going to blame Open Telemetry. Yes, it’s powering the modern observability stack and yet I blame it for the mass confusion

Of course. It’s the classic Internet dev thing of “make a blog post about how using tool X well was a boon to what you were trying to do” and a mass of people walk away with “tool X magically solves things in its default configuration without me learning anything”.

The nice bit is this post will then lead to a bunch of companies’ log storage prices spiking and then a post about how you don’t need wide events trending.

Ludicrous take. I'm tired of people who aren't experts on observability having such strong opinions.

Having used scuba at meta, I can authoritatively say it's bottom-tier quality. Especially in the era of having competent LLMS able to write sophisticated queries.

Genuine expert level observability is being able to query things like "Of all the users who updated versions of the APP in the last week, what % of them had fewer errors after the update (per hour of usage) than before"

That will never happen in a "wide event."

A metric is a measure that can be aggregated over time. You can derive a metric from logs (count) but that doesn't make them the same thing. Spans have start and end times and parent pointer. Logs are arbitrary data, but can be correlated, can have application/org defined structure that's beyond the scope of the observability stack.

You could draw a class diagram with "Wide" - or whatever - as the base class, metric, span, log subclasses. Metric class requires an additional metric field, span requires start, end, parent, log is just open.

undefined

(2024)

prev. submission - https://news.ycombinator.com/item?id=39529775

As someone who has been on that side of things for 10 years, can we stop building really complicated stuff that requires this pile of shit and dedicated staff to run it.

The whole otel, logging and metrics thing is based around "record everything" and dig through the pile of shit looking for things that might be helpful. Fine until you have 50TB of logs/traces online which is our problem. Oh that and the 50 gig prometheus instance. 99.99999% of all information we record is entirely and utterly useless and costs us money

Make simpler things, log only exceptional things, use the rule of silence.

wide events feel like structured logs, great for high cardinality data but not so much for time series

Very ignorant take. If all of us had infinite infrastructure there would indeed be no reason to separate the three at generation, but for those that have to actually count the costs, paying for servers to store that amount of duplication is real issue, and even more (for some reason, by order of magnitude or more) if you're using one of big observability providers.

It is divided mostly because database to store metrics, and database to store essentially freeform documents are very different products and need different features from query language, and having "just metrics" allows to store them with compression reaching 1byte per metric data point or less.

Logs and traces I admit, split here is unnecessary, and it should be just presentation difference. But metrics is often just unrelated to any single event summary of the state of the system. Sure you can just log every single log entry with request metadata and get request rate, percentiles etc. by calculating the events, but that is just too wasteful for most.

With AWS this compares to EMF with (additional) properties?

You get logs with full content + metrics you can query.

Or am I missing something?

Potential PII nightmare. You're now potentially circumventing access controls from sensitive data.

Wide events are great if you don't face an error that can only be exposed by a more detailed metric, log or trace. And that in my experience is a big if.

undefined

Meh. This is just structured logging with some post processing. You can definitely derive metrics and traces based on them, but doing so is more expensive in many ways.

Why is 77rushi77's comment flagged + dead? Seemed like a good question.

Edit: Looks like it's been restored.

Not this bullshit again

The Scuba example is convincing for debugging. What the post doesn’t cover is cost. If you keep raw wide events instead of pre-aggregating, how do you handle storage and query cost at scale? Is sampling the main lever, or do teams still keep cheap metrics for alerting and use events only for investigation?

[flagged]