clue needed! prometheus/node_exporter + grafana

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

clue needed! prometheus/node_exporter + grafana

Stuart Henderson
I'm trying to get graphs out of prometheus+node_exporter displaying
in grafana (all using our packages).

I have node_exporter graphs showing in Prometheus /graph page if I manually
enter a query, and the built-in prometheus graphs in grafana ("Prometheus 2.0
Stats" etc) were very straightforward and are working fine.

I'm trying to add any of the common dashboards for node_exporter from
grafana.com ("Node Dashboard", "Node Exporter 0.16 for Prometheus
Monitoring display board", "Node Exporter Full", "Node exporter single
server"), they're linked to the Prometheus data source, but I can't
get them to display any stats. e.g. "Node Dashboard" just show a blank
screen, "Node Exporter Full") shows various empty graphs/dials showing
"N/A" or "No data points".

If anyone has got them working, can you throw me a clue please?

This looks like it should all be much less faff than symon+syweb but I
seem to be missing something ..

Reply | Threaded
Open this post in threaded view
|

Re: clue needed! prometheus/node_exporter + grafana

Ralf Horstmann-2
Hi Stuart,

the most likely explanation is that some grafana.com dashboards are
primarily targeting Linux or an older node_exporter version, or they are
just poor quality.

For some metrics, the metric names are different between Linux and
OpenBSD (or other BSDs). The memory metrics on Linux for example use the
metric names derived from /proc/meminfo, while the BSDs have different
metric names.

In my Grafana dashboards I have worked around that by adding
expressions like this:

        (node_memory_SwapTotal_bytes{instance=~'$node'} - node_memory_SwapFree{instance=~'$node'})
         or (node_memory_SwapTotal{instance=~'$node'} - node_memory_SwapFree{instance=~'$node'})
         or (node_memory_swap_used_bytes{instance=~'$node'})

This one has three expressions: Linux with node_exporter >= 0.16.0,
Linux with node_exporter < 0.16.0, BSD with node_exporter >= 0.16.0.

In node_exporter 0.16.0 many metrics were renamed to a more standardized
format. Some grafana.com dashboards have not been updated for that.

I'm currently running prometheus-2.0.0p0 and grafana-5.4.3 from packages
on OpenBSD 6.5. The node_exporters are either the one I sent to ports@
in February (which was not imported) or 0.18.0 from snapshots. Works
fine for me.

As a starting point I'd suggest this dashboard:
    https://grafana.com/grafana/dashboards/405

It should show at least some graphs out of the box. The memory graph
needs some work though.

Cheers,
Ralf

* Stuart Henderson <[hidden email]> [2019-08-04 22:29]:

> I'm trying to get graphs out of prometheus+node_exporter displaying
> in grafana (all using our packages).
>
> I have node_exporter graphs showing in Prometheus /graph page if I manually
> enter a query, and the built-in prometheus graphs in grafana ("Prometheus 2.0
> Stats" etc) were very straightforward and are working fine.
>
> I'm trying to add any of the common dashboards for node_exporter from
> grafana.com ("Node Dashboard", "Node Exporter 0.16 for Prometheus
> Monitoring display board", "Node Exporter Full", "Node exporter single
> server"), they're linked to the Prometheus data source, but I can't
> get them to display any stats. e.g. "Node Dashboard" just show a blank
> screen, "Node Exporter Full") shows various empty graphs/dials showing
> "N/A" or "No data points".
>
> If anyone has got them working, can you throw me a clue please?
>
> This looks like it should all be much less faff than symon+syweb but I
> seem to be missing something ..
>

Reply | Threaded
Open this post in threaded view
|

Re: clue needed! prometheus/node_exporter + grafana

Stuart Henderson
On 2019/08/28 19:29, Ralf Horstmann wrote:

> Hi Stuart,
>
> the most likely explanation is that some grafana.com dashboards are
> primarily targeting Linux or an older node_exporter version, or they are
> just poor quality.
>
> For some metrics, the metric names are different between Linux and
> OpenBSD (or other BSDs). The memory metrics on Linux for example use the
> metric names derived from /proc/meminfo, while the BSDs have different
> metric names.
(snip)

Many thanks (and to Paco Esteban who also replied off-list) - I'll pick
this up again soon, your suggestions are likely to save me a bunch of time
figuring it out.