Hacker News new | past | comments | ask | show | jobs | submit login

Someone from F5 worked on this with OpenTelemetry [0] for Arrow, another effort was done for Parquet but was dropped [1]

[0]: https://github.com/open-telemetry/oteps/pull/171

[1]: https://github.com/open-telemetry/opentelemetry-proto/pull/3...




Oh nice, thank you (and also solumos) for the links! It looks like oteps/pull/171 (merged June 2023) expanded and superseded the opentelemetry-proto/pull/346 PR (closed Jul 2022) [0]. The former resulted in merging OpenTelemetry Enhancement Proposal 156 [1], with some interesting results especially for 'Phase 2' where they implemented columnar storage end-to-end (see the Validation section [2]):

* For univariate time series, OTel Arrow is 2 to 2.5 better in terms of bandwidth reduction ... and the end-to-end speed is 3.1 to 11.2 times faster

* For multivariate time series, OTel Arrow is 3 to 7 times better in terms of bandwidth reduction ... Phase 2 has [not yet] been .. estimated but similar results are expected.

* For logs, OTel Arrow is 1.6 to 2 times better in terms of bandwidth reduction ... and the end-to-end speed is 2.3 to 4.86 times faster

* For traces, OTel Arrow is 1.7 to 2.8 times better in terms of bandwidth reduction ... and the end-to-end speed is 3.37 to 6.16 times faster

Pretty exciting results! The OTEL-Arrow adapter has subsequently been donated to the otel community; here's a comment that does a good job of summarizing the results and the recommendations that came out of the test [3].

[0]: https://github.com/open-telemetry/opentelemetry-proto/pull/3...

[1]: https://github.com/open-telemetry/oteps/blob/main/text/0156-...

[2]: https://github.com/open-telemetry/oteps/blob/main/text/0156-...

[3]: https://github.com/open-telemetry/community/issues/1332#issu...




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: