I've found OTel to still have rough edges and it's not really the one stop shop for telemetry we want it to be at $work yet. In particular, no good (sentry-style) exception capturing yet. They also recently changed a lot of metric names(for good reason it seems), which breaks most dashboards you find on the internet.
I have been warned by people that OTel isn't mature yet and I find it to still be true, but it seems the maintainers are trying to do something about this nowadays
I think that the "issue" around otel is that instrumentation is easy and free (as both in beer and freedom) but then for the dashboarding part is where there are literally tens of different SaaS solutions, all more or less involved with the OTel development itself, that want you to use their solution (and pay $$$ for it). And even if you can go a loooong way with a self-hosted Grafana + Tempo, even Grafana Labs are putting more and more new features behind the Grafana Cloud subscription model.
This layout allows us to quickly merge hundreds of millions of samples into a single profile. The only practical limit is protobuf's 2GB message size cap.
Relatedly: Has anyone profiled the performance and reliability characteristics of rsyslogd (Linux and FreeBSD distributed syslogger, maybe other platforms too) in its mode where it’s shipping logs to a central node? I’ve configured and used it with relatively small (high single digit nodes, bursts of activity to a million or two requests per minute or so) set-ups but have wondered if there’s a reason it’s not a more common solution for distributed logging and tracing (yes it doesn’t solve the UI problem for those, but it does solve collecting your logs)
Like… has anyone done a Jepsen-like stress test on rsyslogd and shared the results? I’ve half-assedly looked before and not been able to find anything.
30 comments
I've found OTel to still have rough edges and it's not really the one stop shop for telemetry we want it to be at $work yet. In particular, no good (sentry-style) exception capturing yet. They also recently changed a lot of metric names(for good reason it seems), which breaks most dashboards you find on the internet.
I have been warned by people that OTel isn't mature yet and I find it to still be true, but it seems the maintainers are trying to do something about this nowadays
This layout allows us to quickly merge hundreds of millions of samples into a single profile. The only practical limit is protobuf's 2GB message size cap.
Like… has anyone done a Jepsen-like stress test on rsyslogd and shared the results? I’ve half-assedly looked before and not been able to find anything.
https://grafana.com/oss/pyroscope/
https://github.com/grafana/pyroscope
> Continuously capturing low-overhead performance profiles in production
It suprises me that anything designed by the OTel community could ever meet 'low-overhead' expectations.