Skip to content

Log level for unsuccessful probes #1401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
winadm87 opened this issue Apr 4, 2025 · 3 comments
Open

Log level for unsuccessful probes #1401

winadm87 opened this issue Apr 4, 2025 · 3 comments

Comments

@winadm87
Copy link

winadm87 commented Apr 4, 2025 •
edited
Loading

We have update to v.0.26 and now observing the situation with probe logs. And these observations show some logic we are not sure about.
If we are probing some target that the exporter can not resolve - we get an ERROR level log, e.g.

Mar 30 10:44:31 prometheus-test docker[658698]: time=2025-03-30T08:44:31.841Z level=ERROR source=utils.go:86 msg="Resolution with IP protocol failed" module=icmp target=fake_target.example.net target=fake_target.example.net err="lookup fake_target.example.net on 172.0.0.44:53: no such host"

Ok. And now we probing some existing target, but get not expected status code and probe is failing. But exporter logging the reason for the failed probe with INFO level, e.g.

2025-03-30 11:08:01.634 
time=2025-03-30T08:08:01.633Z level=INFO source=http.go:488 msg="Invalid HTTP response status code" module=http_2xx_no_verify target=https://existing.target.com/ status_code=403 valid_status_codes=[200]

So we can not start the exporter with --probe.level=warn to get only failed probes, and we have to start it with --probe.level=info which produces millions of logs for successful probes.
So the question is if the reason of probe failing could be logged with ERROR/WARN level?
@winadm87
Copy link
Author

winadm87 commented Apr 4, 2025

cc @tjhop

@Abhinand-Gokul
Copy link

We would be interested in this as well, getting HTTP status codes for failures on warn/error probe.level.

@tjhop
Copy link
Contributor

tjhop commented Apr 10, 2025

There's a related conversation to the log prober logging in #1377, I've asked for some clarification there and would like to make a determination on how to proceed there before attempting to tune leveling further

@Jakob3xD Jakob3xD mentioned this issue Apr 24, 2025
Open
tjhop added a commit to tjhop/blackbox_exporter that referenced this issue May 8, 2025
@tjhop
fix: stop scrape logger spam, refactor scrape logger design
7df3031 
Based on discussion in prometheus#1377, slack, etc, it seems unclear what the
intended functionality/behavior of the probe scrape logger was supposed to be.
This is an attempt to align behavior with current user expectations. The
improved defaults and updated configs here should address the log spam
as well.

User facing changes:
- the `--log.prober` flag now sets the _level at which prober scrape
  logs will be emitted at_.
- the `--log.prober` flag now defaults to `debug` level. Because the
  default log level is info, this means that in order to see probe
scrape logs, a user must either:
    - increase the level of the logger via `--log.level` to be >= that
      of `--log.prober` (ie, set both to `debug`)
    - decrease the level of the probe logger via `--log.prober` to be <=
      that of `--log.level` (ie, set both to `info`)

Note:
This approach (emitting all logs from the probe logger at the level
specified) involves hijacking calls to the logger and overriding the
level. This comes with the very obvious side effect that the developers
working on code using the probe scrape logger no longer have control
over the control over the level of the log that gets emitted. Regardless
of whether they use `l.Info()` or `l.Debug()`, it'll still get logged at
the configured log prober level. This approach likely would not be
compatible with some of the other discussions around how probe scrape
logs work, like prometheus#1401.

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
@tjhop tjhop mentioned this issue May 8, 2025
Merged
electron0zero pushed a commit that referenced this issue May 21, 2025
@tjhop
fix: stop scrape logger spam, refactor scrape logger design (#1381)
a72fbd7 
* fix: stop scrape logger spam, refactor scrape logger design

Based on discussion in #1377, slack, etc, it seems unclear what the
intended functionality/behavior of the probe scrape logger was supposed to be.
This is an attempt to align behavior with current user expectations. The
improved defaults and updated configs here should address the log spam
as well.

User facing changes:
- the `--log.prober` flag now sets the _level at which prober scrape
  logs will be emitted at_.
- the `--log.prober` flag now defaults to `debug` level. Because the
  default log level is info, this means that in order to see probe
scrape logs, a user must either:
    - increase the level of the logger via `--log.level` to be >= that
      of `--log.prober` (ie, set both to `debug`)
    - decrease the level of the probe logger via `--log.prober` to be <=
      that of `--log.level` (ie, set both to `info`)

Note:
This approach (emitting all logs from the probe logger at the level
specified) involves hijacking calls to the logger and overriding the
level. This comes with the very obvious side effect that the developers
working on code using the probe scrape logger no longer have control
over the control over the level of the log that gets emitted. Regardless
of whether they use `l.Info()` or `l.Debug()`, it'll still get logged at
the configured log prober level. This approach likely would not be
compatible with some of the other discussions around how probe scrape
logs work, like #1401.

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>

* docs(README): add section on --log.prober behavior with sample output

Address PR feedback

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>

---------

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Abhinand-Gokul @tjhop @winadm87