Skip to content

mountstats/linux: fix multiple transports statistics labeling#3358

Merged
SuperQ merged 2 commits into
prometheus:masterfrom
sagigrimberg:sagi/fix-nfs-multi-xprt
Feb 17, 2026
Merged

mountstats/linux: fix multiple transports statistics labeling#3358
SuperQ merged 2 commits into
prometheus:masterfrom
sagigrimberg:sagi/fix-nfs-multi-xprt

Conversation

@sagigrimberg
Copy link
Copy Markdown
Contributor

@sagigrimberg sagigrimberg commented Jul 7, 2025

When mounting nfs with nconnect, mountstats will show multiple transports. Currently the mountstats exporter fails to create metrics from this information due to repeated entries.

Fix this by adding transport metrics a "transport" label enumerating their order in the mountstats output.

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Jul 7, 2025

@discordianfish @SuperQ FYI

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Jul 22, 2025

@discordianfish @SuperQ ping again..... must be a sad joke...

@discordianfish
Copy link
Copy Markdown
Member

discordianfish commented Sep 17, 2025

@sagigrimberg Whats the intention? Why do you need individual metrics per transport? Not familiar with nconnect

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Nov 26, 2025 •
edited
Loading

@discordianfish Sorry for the super long delay - I stopped checking this PR frequently.

The intention is to support nconnect mount option in NFS. Which creates multiple connections (transports). Currently any nconnect>1 breaks the exporter.

See: https://man7.org/linux/man-pages/man5/nfs.5.html

@sagigrimberg
mountstats/linux: fix multiple transports statistics labeling
53503f8 
When mounting nfs with nconnect, mountstats will show multiple transports.
Currently the mountstats exporter fails to create metrics from this information
due to repeated entries.

Fix this by adding transport metrics a "transport" label enumerating their
order in the mountstats output.

Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
@sagigrimberg sagigrimberg force-pushed the sagi/fix-nfs-multi-xprt branch from 566270c to 53503f8 Compare November 30, 2025 07:16
@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Nov 30, 2025

@discordianfish @SuperQ ping

4 similar comments
@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Dec 9, 2025

@discordianfish @SuperQ ping

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Dec 22, 2025

@discordianfish @SuperQ ping

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Dec 24, 2025

@discordianfish @SuperQ ping

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Dec 30, 2025

@discordianfish @SuperQ ping

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Jan 1, 2026

@discordianfish @SuperQ ping

1 similar comment
@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Jan 11, 2026

@discordianfish @SuperQ ping

@discordianfish discordianfish requested a review from SuperQ January 16, 2026 10:40
discordianfish
discordianfish approved these changes Jan 16, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@discordianfish discordianfish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makese sense then!

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Jan 17, 2026

Thank you. So you will merge this branch?

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Jan 19, 2026

@discordianfish @SuperQ can this PR be merged please?

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Feb 1, 2026

@discordianfish @SuperQ ping

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Feb 5, 2026

@discordianfish @SuperQ ping can we get this merged?

1 similar comment
@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Feb 8, 2026

@discordianfish @SuperQ ping can we get this merged?

@sagigrimberg
Copy link
Copy Markdown
Contributor Author

sagigrimberg commented Feb 15, 2026

@discordianfish @SuperQ ping is there anything I can do to get this merged?

@jbaiter
Copy link
Copy Markdown

jbaiter commented Feb 17, 2026 •
edited
Loading

We need this as well, the bug is currently breaking our monitoring setup in production. We rely on the mountstats metrics to track NFS performance indicators. We're using nconnect as well, so I can confirm the cause of this bug.

The bug causes excessive log volume from the node-exporter, see #3258.

Is there any way to get this merged @discordianfish @SuperQ? In our case the bug was triggered when upgrading from Debian 12 (v1.5.0) to Debian 13 (v1.9.0), I can imagine we're not the only ones experiencing this.

@SuperQ SuperQ merged commit 29f6ec0 into prometheus:master Feb 17, 2026
13 checks passed
@SuperQ SuperQ mentioned this pull request Feb 17, 2026
Closed
ShashwatHiregoudar pushed a commit to ShashwatHiregoudar/node_exporter that referenced this pull request Mar 3, 2026
@sagigrimberg @ShashwatHiregoudar
mountstats/linux: fix multiple transports statistics labeling (promet…
92c0a7c 
…heus#3358)

When mounting nfs with nconnect, mountstats will show multiple transports.
Currently the mountstats exporter fails to create metrics from this information
due to repeated entries.

Fix this by adding transport metrics a "transport" label enumerating their
order in the mountstats output.

Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Shashwat Hiregoudar <shashwathiregoudar@gmail.com>
SuperQ added a commit that referenced this pull request Apr 4, 2026
@SuperQ
Release v1.11.0
22b5aec 
* [CHANGE] Adds distroless container image. #3592
* [FEATURE] infiniband: Add `hw_counters` metrics #2827
* [FEATURE] kernel_hung: Add collector for kernel hung tasks #3470
* [FEATURE] mixin: Add "All" option to cluster variable in dashboards #3509
* [FEATURE] netlink: Add node_network_altnames_info metric #3541
* [FEATURE] thermal: Add support for CPU temperature metrics on Apple Silicon #3547
* [FEATURE] bcachefs: Add new collector for bcachefs #3523
* [ENHANCEMENT] mixin: Make devices in network graphs configurable #3525
* [ENHANCEMENT] nvme: Add namespace parameters #3360
* [BUGFIX] netdev: Respect deviceFilter for `node_network_address_info` metric #3340
* [BUGFIX] mountstats: Fix multiple transports statistics labeling #3358
* [BUGFIX] diskstats: Fix panic when using device-include flag only #3603

Signed-off-by: Ben Kochie <superq@gmail.com>
@SuperQ SuperQ mentioned this pull request Apr 4, 2026
Merged
SuperQ added a commit that referenced this pull request Apr 4, 2026
@SuperQ
Release v1.11.0
a63a2b6 
* [CHANGE] Adds distroless container image. #3592
* [FEATURE] infiniband: Add `hw_counters` metrics #2827
* [FEATURE] kernel_hung: Add collector for kernel hung tasks #3470
* [FEATURE] mixin: Add "All" option to cluster variable in dashboards #3509
* [FEATURE] netlink: Add node_network_altnames_info metric #3541
* [FEATURE] thermal: Add support for CPU temperature metrics on Apple Silicon #3547
* [FEATURE] bcachefs: Add new collector for bcachefs #3523
* [ENHANCEMENT] mixin: Make devices in network graphs configurable #3525
* [ENHANCEMENT] nvme: Add namespace parameters #3360
* [BUGFIX] netdev: Respect deviceFilter for `node_network_address_info` metric #3340
* [BUGFIX] mountstats: Fix multiple transports statistics labeling #3358
* [BUGFIX] diskstats: Fix panic when using device-include flag only #3603

Signed-off-by: Ben Kochie <superq@gmail.com>
SuperQ added a commit that referenced this pull request Apr 4, 2026
@SuperQ
Release v1.11.0 (#3608)
14fddfc 
* [CHANGE] Adds distroless container image. #3592
* [FEATURE] infiniband: Add `hw_counters` metrics #2827
* [FEATURE] kernel_hung: Add collector for kernel hung tasks #3470
* [FEATURE] mixin: Add "All" option to cluster variable in dashboards #3509
* [FEATURE] netlink: Add node_network_altnames_info metric #3541
* [FEATURE] thermal: Add support for CPU temperature metrics on Apple Silicon #3547
* [FEATURE] bcachefs: Add new collector for bcachefs #3523
* [ENHANCEMENT] mixin: Make devices in network graphs configurable #3525
* [ENHANCEMENT] nvme: Add namespace parameters #3360
* [BUGFIX] netdev: Respect deviceFilter for `node_network_address_info` metric #3340
* [BUGFIX] mountstats: Fix multiple transports statistics labeling #3358
* [BUGFIX] diskstats: Fix panic when using device-include flag only #3603

Signed-off-by: Ben Kochie <superq@gmail.com>
renovate Bot added a commit to sdwilsh/sOS that referenced this pull request Apr 7, 2026
@renovate
chore(deps): update quay.io/prometheus/node-exporter docker tag to v1…
ad531bd 
….11.1

##### [\`v1.11.1\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.1)

- \[BUGFIX] Fix kernel\_hung for no data [#3613](prometheus/node_exporter#3613)

---
##### [\`v1.11.0\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.0)

- \[CHANGE] Adds distroless container image. [#3592](prometheus/node_exporter#3592)
- \[FEATURE] infiniband: Add `hw_counters` metrics [#2827](prometheus/node_exporter#2827)
- \[FEATURE] kernel\_hung: Add collector for kernel hung tasks [#3470](prometheus/node_exporter#3470)
- \[FEATURE] mixin: Add "All" option to cluster variable in dashboards [#3509](prometheus/node_exporter#3509)
- \[FEATURE] netlink: Add node\_network\_altnames\_info metric [#3541](prometheus/node_exporter#3541)
- \[FEATURE] thermal: Add support for CPU temperature metrics on Apple Silicon [#3547](prometheus/node_exporter#3547)
- \[FEATURE] bcachefs: Add new collector for bcachefs [#3523](prometheus/node_exporter#3523)
- \[ENHANCEMENT] mixin: Make devices in network graphs configurable [#3525](prometheus/node_exporter#3525)
- \[ENHANCEMENT] nvme: Add namespace parameters [#3360](prometheus/node_exporter#3360)
- \[BUGFIX] netdev: Respect deviceFilter for `node_network_address_info` metric [#3340](prometheus/node_exporter#3340)
- \[BUGFIX] mountstats: Fix multiple transports statistics labeling [#3358](prometheus/node_exporter#3358)
- \[BUGFIX] diskstats: Fix panic when using device-include flag only [#3603](prometheus/node_exporter#3603)
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Apr 8, 2026
@renovate
chore(deps): update quay.io/prometheus/node-exporter docker tag to v1…
54d5e87 
….11.1

##### [\`v1.11.1\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.1)

- \[BUGFIX] Fix kernel\_hung for no data [#3613](prometheus/node_exporter#3613)

---
##### [\`v1.11.0\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.0)

- \[CHANGE] Adds distroless container image. [#3592](prometheus/node_exporter#3592)
- \[FEATURE] infiniband: Add `hw_counters` metrics [#2827](prometheus/node_exporter#2827)
- \[FEATURE] kernel\_hung: Add collector for kernel hung tasks [#3470](prometheus/node_exporter#3470)
- \[FEATURE] mixin: Add "All" option to cluster variable in dashboards [#3509](prometheus/node_exporter#3509)
- \[FEATURE] netlink: Add node\_network\_altnames\_info metric [#3541](prometheus/node_exporter#3541)
- \[FEATURE] thermal: Add support for CPU temperature metrics on Apple Silicon [#3547](prometheus/node_exporter#3547)
- \[FEATURE] bcachefs: Add new collector for bcachefs [#3523](prometheus/node_exporter#3523)
- \[ENHANCEMENT] mixin: Make devices in network graphs configurable [#3525](prometheus/node_exporter#3525)
- \[ENHANCEMENT] nvme: Add namespace parameters [#3360](prometheus/node_exporter#3360)
- \[BUGFIX] netdev: Respect deviceFilter for `node_network_address_info` metric [#3340](prometheus/node_exporter#3340)
- \[BUGFIX] mountstats: Fix multiple transports statistics labeling [#3358](prometheus/node_exporter#3358)
- \[BUGFIX] diskstats: Fix panic when using device-include flag only [#3603](prometheus/node_exporter#3603)
sdwilsh pushed a commit to sdwilsh/ansible-playbooks that referenced this pull request Apr 8, 2026
@renovate @sdwilsh
chore(deps): update quay.io/prometheus/node-exporter docker tag to v1…
3db740e 
….11.1

##### [\`v1.11.1\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.1)

- \[BUGFIX] Fix kernel\_hung for no data [#3613](prometheus/node_exporter#3613)

---
##### [\`v1.11.0\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.0)

- \[CHANGE] Adds distroless container image. [#3592](prometheus/node_exporter#3592)
- \[FEATURE] infiniband: Add `hw_counters` metrics [#2827](prometheus/node_exporter#2827)
- \[FEATURE] kernel\_hung: Add collector for kernel hung tasks [#3470](prometheus/node_exporter#3470)
- \[FEATURE] mixin: Add "All" option to cluster variable in dashboards [#3509](prometheus/node_exporter#3509)
- \[FEATURE] netlink: Add node\_network\_altnames\_info metric [#3541](prometheus/node_exporter#3541)
- \[FEATURE] thermal: Add support for CPU temperature metrics on Apple Silicon [#3547](prometheus/node_exporter#3547)
- \[FEATURE] bcachefs: Add new collector for bcachefs [#3523](prometheus/node_exporter#3523)
- \[ENHANCEMENT] mixin: Make devices in network graphs configurable [#3525](prometheus/node_exporter#3525)
- \[ENHANCEMENT] nvme: Add namespace parameters [#3360](prometheus/node_exporter#3360)
- \[BUGFIX] netdev: Respect deviceFilter for `node_network_address_info` metric [#3340](prometheus/node_exporter#3340)
- \[BUGFIX] mountstats: Fix multiple transports statistics labeling [#3358](prometheus/node_exporter#3358)
- \[BUGFIX] diskstats: Fix panic when using device-include flag only [#3603](prometheus/node_exporter#3603)
renovate Bot added a commit to sdwilsh/sOS that referenced this pull request Apr 10, 2026
@renovate
chore(deps): update quay.io/prometheus/node-exporter docker tag to v1…
3a4600f 
….11.1

##### [\`v1.11.1\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.1)

- \[BUGFIX] Fix kernel\_hung for no data [#3613](prometheus/node_exporter#3613)

---
##### [\`v1.11.0\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.0)

- \[CHANGE] Adds distroless container image. [#3592](prometheus/node_exporter#3592)
- \[FEATURE] infiniband: Add `hw_counters` metrics [#2827](prometheus/node_exporter#2827)
- \[FEATURE] kernel\_hung: Add collector for kernel hung tasks [#3470](prometheus/node_exporter#3470)
- \[FEATURE] mixin: Add "All" option to cluster variable in dashboards [#3509](prometheus/node_exporter#3509)
- \[FEATURE] netlink: Add node\_network\_altnames\_info metric [#3541](prometheus/node_exporter#3541)
- \[FEATURE] thermal: Add support for CPU temperature metrics on Apple Silicon [#3547](prometheus/node_exporter#3547)
- \[FEATURE] bcachefs: Add new collector for bcachefs [#3523](prometheus/node_exporter#3523)
- \[ENHANCEMENT] mixin: Make devices in network graphs configurable [#3525](prometheus/node_exporter#3525)
- \[ENHANCEMENT] nvme: Add namespace parameters [#3360](prometheus/node_exporter#3360)
- \[BUGFIX] netdev: Respect deviceFilter for `node_network_address_info` metric [#3340](prometheus/node_exporter#3340)
- \[BUGFIX] mountstats: Fix multiple transports statistics labeling [#3358](prometheus/node_exporter#3358)
- \[BUGFIX] diskstats: Fix panic when using device-include flag only [#3603](prometheus/node_exporter#3603)
renovate Bot added a commit to sdwilsh/sOS that referenced this pull request Apr 17, 2026
@renovate
chore(deps): update quay.io/prometheus/node-exporter docker tag to v1…
0c2da88 
….11.1

##### [\`v1.11.1\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.1)

- \[BUGFIX] Fix kernel\_hung for no data [#3613](prometheus/node_exporter#3613)

---
##### [\`v1.11.0\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.0)

- \[CHANGE] Adds distroless container image. [#3592](prometheus/node_exporter#3592)
- \[FEATURE] infiniband: Add `hw_counters` metrics [#2827](prometheus/node_exporter#2827)
- \[FEATURE] kernel\_hung: Add collector for kernel hung tasks [#3470](prometheus/node_exporter#3470)
- \[FEATURE] mixin: Add "All" option to cluster variable in dashboards [#3509](prometheus/node_exporter#3509)
- \[FEATURE] netlink: Add node\_network\_altnames\_info metric [#3541](prometheus/node_exporter#3541)
- \[FEATURE] thermal: Add support for CPU temperature metrics on Apple Silicon [#3547](prometheus/node_exporter#3547)
- \[FEATURE] bcachefs: Add new collector for bcachefs [#3523](prometheus/node_exporter#3523)
- \[ENHANCEMENT] mixin: Make devices in network graphs configurable [#3525](prometheus/node_exporter#3525)
- \[ENHANCEMENT] nvme: Add namespace parameters [#3360](prometheus/node_exporter#3360)
- \[BUGFIX] netdev: Respect deviceFilter for `node_network_address_info` metric [#3340](prometheus/node_exporter#3340)
- \[BUGFIX] mountstats: Fix multiple transports statistics labeling [#3358](prometheus/node_exporter#3358)
- \[BUGFIX] diskstats: Fix panic when using device-include flag only [#3603](prometheus/node_exporter#3603)
@github-actions github-actions Bot mentioned this pull request Apr 20, 2026
Closed
1 task
renovate Bot added a commit to sdwilsh/sOS that referenced this pull request Apr 24, 2026
@renovate
chore(deps): update quay.io/prometheus/node-exporter docker tag to v1…
50fa724 
….11.1

##### [\`v1.11.1\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.1)

- \[BUGFIX] Fix kernel\_hung for no data [#3613](prometheus/node_exporter#3613)

---
##### [\`v1.11.0\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.0)

- \[CHANGE] Adds distroless container image. [#3592](prometheus/node_exporter#3592)
- \[FEATURE] infiniband: Add `hw_counters` metrics [#2827](prometheus/node_exporter#2827)
- \[FEATURE] kernel\_hung: Add collector for kernel hung tasks [#3470](prometheus/node_exporter#3470)
- \[FEATURE] mixin: Add "All" option to cluster variable in dashboards [#3509](prometheus/node_exporter#3509)
- \[FEATURE] netlink: Add node\_network\_altnames\_info metric [#3541](prometheus/node_exporter#3541)
- \[FEATURE] thermal: Add support for CPU temperature metrics on Apple Silicon [#3547](prometheus/node_exporter#3547)
- \[FEATURE] bcachefs: Add new collector for bcachefs [#3523](prometheus/node_exporter#3523)
- \[ENHANCEMENT] mixin: Make devices in network graphs configurable [#3525](prometheus/node_exporter#3525)
- \[ENHANCEMENT] nvme: Add namespace parameters [#3360](prometheus/node_exporter#3360)
- \[BUGFIX] netdev: Respect deviceFilter for `node_network_address_info` metric [#3340](prometheus/node_exporter#3340)
- \[BUGFIX] mountstats: Fix multiple transports statistics labeling [#3358](prometheus/node_exporter#3358)
- \[BUGFIX] diskstats: Fix panic when using device-include flag only [#3603](prometheus/node_exporter#3603)
@github-actions github-actions Bot mentioned this pull request Apr 27, 2026
Open
1 task
renovate Bot added a commit to sdwilsh/sOS that referenced this pull request May 8, 2026
@renovate
chore(deps): update quay.io/prometheus/node-exporter docker tag to v1…
2c65804 
….11.1

##### [\`v1.11.1\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.1)

- \[BUGFIX] Fix kernel\_hung for no data [#3613](prometheus/node_exporter#3613)

---
##### [\`v1.11.0\`](https://github.com/prometheus/node_exporter/releases/tag/v1.11.0)

- \[CHANGE] Adds distroless container image. [#3592](prometheus/node_exporter#3592)
- \[FEATURE] infiniband: Add `hw_counters` metrics [#2827](prometheus/node_exporter#2827)
- \[FEATURE] kernel\_hung: Add collector for kernel hung tasks [#3470](prometheus/node_exporter#3470)
- \[FEATURE] mixin: Add "All" option to cluster variable in dashboards [#3509](prometheus/node_exporter#3509)
- \[FEATURE] netlink: Add node\_network\_altnames\_info metric [#3541](prometheus/node_exporter#3541)
- \[FEATURE] thermal: Add support for CPU temperature metrics on Apple Silicon [#3547](prometheus/node_exporter#3547)
- \[FEATURE] bcachefs: Add new collector for bcachefs [#3523](prometheus/node_exporter#3523)
- \[ENHANCEMENT] mixin: Make devices in network graphs configurable [#3525](prometheus/node_exporter#3525)
- \[ENHANCEMENT] nvme: Add namespace parameters [#3360](prometheus/node_exporter#3360)
- \[BUGFIX] netdev: Respect deviceFilter for `node_network_address_info` metric [#3340](prometheus/node_exporter#3340)
- \[BUGFIX] mountstats: Fix multiple transports statistics labeling [#3358](prometheus/node_exporter#3358)
- \[BUGFIX] diskstats: Fix panic when using device-include flag only [#3603](prometheus/node_exporter#3603)
@evkuzin
Copy link
Copy Markdown

evkuzin commented May 9, 2026

This change introduced another issue - a big spike in cardinality.
I'm trying to solve it in

#3650

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SuperQ SuperQ SuperQ approved these changes

@discordianfish discordianfish Awaiting requested review from discordianfish

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sagigrimberg @discordianfish @jbaiter @evkuzin @SuperQ