- Consider decoupling your collector from whatever is consuming your traces with something like kafka. Traces can be pretty heavy and it can be tricky to scale collectors. If something goes down, it's probably a good idea to continue writing the traces to queue or topic.
- https://www.otelbin.io is a nice little tool to help with collector configuration
My ideal setup would be to just write SQL on telemetry data and plot dashboards / set alerts.
Also, thoughts on Vector vs otel agent?
This isn't a lot to go on.
The important thing is what you're trying to instrument - hosts, applications, network, microservices, all of the above? (And then whether you want a few weeks retention, or keeping years worth.)
Grafana in front of Prometheus with node-exporter or telegraf (it can expose in prometheus mode) on the clients -- will tick a lot of boxes and is fast to get going.
Grafana in front of InfluxDB + telegraf is similar, but personally I find PromQL easier than InfluxQL.
> ... write SQL on telemetry data and plot dashboards / set alerts.
Read up about the design of TSDBs and log / tracing datastores - their design & intent heavily influences their query languages.
IMO, with the current tech, it entirely depends on what data you're talking about.
For metrics and traces, I would use the OTel collector personally. You will have much more flexibility and it's pretty easy to write custom processors in Go. Support for traces is quite mature and metrics isn't far off. We've been running collectors for production scale of metric and trace ingest for the past couple of years, on the order of 1m events/sec (metric datapoints or spans). You mentioned low volume so that's less important, but I just wanted to mention in case others find this comment.
Logs are a bit different. We looked in to this in the past year. Vector has emerging support for OTLP but it's pretty early. Still, I bet it's pretty straightforward if your backend can ingest via OTLP. Our main concern with running the otel-collector as the log ingest agent was around throughput/performance. Vector is battle-tested, otel is still a bit early in this space. I imagine over time the gap will be closed but I would probably still reach for Vector for this use-case for higher scale. That said, YMMV and as with any technical decision, empirical data and benchmarking on your workloads will be the best way to determine the tradeoffs.
For your scale you could probably get away with an OTel collector daemonset and maybe a deployment with the Target Allocator (to allocate Prometheus scrapes) and call it a day :)
Don’t use vector or otel-agent. Add a materialized view in clickhouse to transform data and swap HyperDX to load from your view (in the UI.)
It's been solid, but the UI is kind of clunky and a little buggy here and there. Dashboards are tricky to setup too. But it has no dependencies, and was easy to setup, and I couldn't find anything else that handled logs too.
PS: I am one of the maintainers
The UI is predictably an annoying mess, but that's the case with EVERY tracing solution I've tried. Very much including SigNoz.
HyperDX is just a lot better, sure a few papercuts but they got all the important stuff right imo.
Can you share which version of SigNoz did you try or what time frame? We recently made a lot of improvement in how you can host SigNoz including support for Postgres and better docs fro self hosting corretcly - https://signoz.io/docs/collection-agents/get-started/
It's the shame the docs on it are still quite bad. The example config in the article here does look almost identical to the one we use everywhere, just without the redact, and should probably be pasted somewhere into the official docs.
Every provider seems to produce their own soft fork of the collector for branding (eg Alloy, ADOT, etc) and slightly changes the configuration, which doesn't help.
I've dabbled in building a project that collects metrics from the logs for smaller projects. Everyone tells me it's a bad idea, but it seems to work well for me.
Eventually it'll have successors that are better in some way, more efficient, or whatever, but right now there are no alternatives at all. Open Telemetry is the first common standard that multiple vendors have signed up to.
If it were to give more fine-grained control over write-only access -- would probably just write directly and let it handle the load.
We agree that fine-grained access control is important. A read-only user role will be available in the next major release.
Having stats is nice but i am not choosing your product because of stats. I actually think greptimedb is exactly what I am looking for, I.e. a humio / falcon logscale alternative. But I had to do some digging to actually infer that.
Your material doesn't highlight what sets you apart from the competition. If you want to target developers which you might not. I dont know.
I want to debug issues using freetext search, i want to be able to aggregate stats i care about on demand.
And while all the tracing providers speak the OTEL protocol, the way your do auth is not the same. Sometimes you need to specify it in a header, sometimes it's a part of the URL.