Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New metric system (Prometheus data format) #4325

Merged
merged 30 commits into from Feb 28, 2023
Merged

New metric system (Prometheus data format) #4325

merged 30 commits into from Feb 28, 2023

Conversation

MrAnno
Copy link
Collaborator

@MrAnno MrAnno commented Feb 13, 2023 •
edited

This PR adds a new metric naming scheme (labeling capability) to syslog-ng and a querying tool to produce Prometheus-like metric exports.

$ sbin/syslog-ng-ctl stats prometheus

syslogng_events_allocated_bytes 18446744073709550560
syslogng_filtered_events_total{id="#anon-filter0",result="matched"} 0
syslogng_filtered_events_total{id="#anon-filter0",result="not_matched"} 2
syslogng_filtered_events_total{id="ff",result="matched"} 0
syslogng_filtered_events_total{id="ff",result="not_matched"} 2
syslogng_input_events_total{id="#anon-source0#0",driver_instance="-",result="processed"} 0
syslogng_input_events_total{id="s_network#1",result="processed"} 2
syslogng_internal_source{result="dropped"} 0
syslogng_internal_source{result="queued"} 0
syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="delivered"} 0
syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="dropped"} 0
syslogng_output_events_total{id="d_dest#0",driver_instance="tcp,127.0.0.1:5555",result="queued"} 0
syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="delivered"} 0
syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="dropped"} 0
syslogng_output_events_total{id="d_dest#1",driver_instance="http,https://localhost:8080",result="queued"} 0
syslogng_parsed_events_total{id="#anon-parser0",result="discarded"} 0
syslogng_parsed_events_total{id="#anon-parser0",result="processed"} 0
syslogng_scratch_buffers_bytes 0
syslogng_scratch_buffers_total 3
syslogng_tagged_events_total{id=".source.#anon-source0",result="processed"} 0
syslogng_tagged_events_total{id=".source.s_network",result="processed"} 2

$ sbin/syslog-ng-ctl stats prometheus --with-legacy-metrics

syslogng_src_facility_processed{stat_instance="18"} 0
syslogng_dst_network_memory_usage{id="d_dest#0",stat_instance="tcp,127.0.0.1:5555"} 0
syslogng_src_host_processed{id="s_network#1",stat_instance="AnnoW"} 2
syslogng_global_sdata_updates_processed 0
syslogng_global_msg_clones_processed 0
syslogng_global_internal_queue_length_processed 0
syslogng_global_payload_reallocs_processed 4
...

syslog-ng stats labels are similar to the labels in the Prometheus data model:
https://prometheus.io/docs/concepts/data_model/

The (id, labels) pair identifies a new-style key. The order of the labels
should not matter, and a label with an empty value should be equivalent to
a label non-existing, but the current implementation is rudimentary and
does not take these considerations into account.

The old (component, instance , id) keying is marked as legacy; any of the legacy functions should not be used when introducing new metrics.

Questions:

  • when should we remove stats-log?

Related PR: #4318

@github-actions
Copy link
Contributor

No news file has been detected. Please write one, if applicable.

alltilla
alltilla reviewed Feb 14, 2023
View reviewed changes
Copy link
Collaborator

@alltilla alltilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed until stats: add labeling capability, inclusive. It looks good, I only had UT related comments.

I will check the last 2 commits soon.

lib/stats/tests/test_stats_cluster.c Show resolved Hide resolved
lib/stats/tests/test_stats_cluster.c Outdated Show resolved Hide resolved
@kira-syslogng
Copy link
Contributor

Build FAILURE

@MrAnno MrAnno force-pushed the stats-labeling branch 2 times, most recently from e24141b to 9c11d46 Compare February 15, 2023 00:18
alltilla
alltilla reviewed Feb 15, 2023
View reviewed changes
lib/stats/stats-cluster.c Outdated Show resolved Hide resolved
lib/stats/stats-cluster.c Outdated Show resolved Hide resolved
lib/stats/stats-cluster.c Outdated Show resolved Hide resolved
@alltilla alltilla mentioned this pull request Feb 15, 2023
Merged
@MrAnno MrAnno force-pushed the stats-labeling branch 2 times, most recently from 0ee8a78 to 95f2e77 Compare February 16, 2023 12:19
@MrAnno MrAnno marked this pull request as ready for review February 16, 2023 12:20
@kira-syslogng
Copy link
Contributor

Build FAILURE

@kira-syslogng
Copy link
Contributor

Build FAILURE

@MrAnno MrAnno force-pushed the stats-labeling branch 2 times, most recently from 187437d to 49891a2 Compare February 16, 2023 15:24
@MrAnno MrAnno changed the title New metric system WIP: New metric system Feb 16, 2023
@MrAnno MrAnno force-pushed the stats-labeling branch 5 times, most recently from a89d3a3 to c5f98a4 Compare February 18, 2023 12:40
@MrAnno
Copy link
Collaborator Author

MrAnno commented Feb 18, 2023

@kira-syslogng do stresstest

@MrAnno MrAnno changed the title WIP: New metric system New metric system (Prometheus data format) Feb 18, 2023
@kira-syslogng
Copy link
Contributor

Kira-stress-test: Build FAILURE

@MrAnno
Copy link
Collaborator Author

MrAnno commented Feb 18, 2023

@kira-syslogng do stresstest

@MrAnno
Copy link
Collaborator Author

MrAnno commented Feb 18, 2023 •
edited

@HofiOne Can you help us fix the stress test?

15:53:49 # An exception occured:
15:53:49 # Type: IOError
15:53:49 # Message: [Errno 28] No space left on device

https://ci.syslog-ng.com/jenkins/job/kira-performance-test-disk-buffer/645/console

@kira-syslogng
Copy link
Contributor

Kira-stress-test: Build FAILURE

@MrAnno
lib: rename stats_cluster_*_key_set() to legacy_set()
c93bec0 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
stats: only show legacy counters in stats-{csv, log}
fd89611 
The CSV and log output are now considered legacy, stats-query can be
used to display new "labeled" metrics.

Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
stats: skip NULL external counters (or aliases)
2a8bbe7 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
stats: add labeling capability
d09af90 
This commit adds labels to stats cluster keys.

These labels are similar to the labels in the Prometheus data model:
https://prometheus.io/docs/concepts/data_model/

The (id, labels) pair identifies a new-style key. The order of the labels
should not matter, and a label with an empty value should be equivalent to
a label non-existing, but the current implementation is rudimentary and
does not take these considerations into account.

Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
stats: add stats_cluster_get_type_label()
b8d69b1 
To provide labels for individual counters within a cluster.

Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
stats: rename written to delivered
848b5f7 
NOTE: Currently, it is not guaranteed that the message has reached its
destination (the old name is misleading as well).

Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
stats: allow setting both new and legacy cluster keys
9cc2190 
This helps creating legacy aliases for new metrics without adding a lot of
boilerplate to the user side.

Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
stats: remove memory_usage from logpipe cluster
02bd7d3 
The new metric system requires metrics within the same cluster to have the
same unit of measure.

Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
logqueue: remove unused abstract stats counter registration function
e2136e6 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
lib: add new metric "output_events_total"
78ad676 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
lib: add new metric "input_events_total"
d528b26 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
lib: add new metric "filtered_events_total"
b152497 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
lib: add new metric "parsed_events_total"
1e1a959 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
lib: add new scratch_buffers metrics
8c3d919 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
lib: add new metric "tagged_events_total"
b864017 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
lib: add new metric "internal_source"
72858b2 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
lib: add new metric "events_allocated_bytes"
f4a7e6b 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
lib: exclude destination "processed" from new metrics
a41d773 
sum(syslogng_output_events_total) wouldn't not be meaningful otherwise.

Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
control: enable scratch-buffers usage
2784dab 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
stats: add syslog-ng-ctl stats prometheus
f063ba1 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
stats: extract stats_prometheus_format_counter()
1ba6d4f 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
stats: add unit test for stats_prometheus
e649dbc 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
news: add feature syslog-ng#4325
d33a548 
Signed-off-by: László Várady <laszlo.varady@anno.io>
@MrAnno
Copy link
Collaborator Author

MrAnno commented Feb 26, 2023

@kira-syslogng do stresstest

@kira-syslogng
Copy link
Contributor

Kira-stress-test: Build SUCCESS

@alltilla alltilla merged commit 3bb57f6 into syslog-ng:master Feb 28, 2023
15 checks passed
Genfood pushed a commit to Genfood/syslog-ng that referenced this pull request Jun 14, 2023
@MrAnno @Genfood
news: add feature syslog-ng#4325
6cf61f4 
Signed-off-by: László Várady <laszlo.varady@anno.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alltilla alltilla alltilla approved these changes

@bazsi bazsi Awaiting requested review from bazsi

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
syslog-ng 4.1
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@MrAnno @kira-syslogng @folti @alltilla @HofiOne