Problem disabling monitors or removing hosts

[salderma]

I am running Foreman+Puppet+PuppetDB on the same host.

There are strange cases where services or hosts are left dangling in Nagios after being disabled or removed from target node's classification:

Scenario 1:
Node has a service monitored like OpenSSH. Puppet creates the nagios service monitor for the Node. Suppose due to an infrastructure firewall, Nagios can no longer reach node:tcp_22, and I wish to set openssh_monitor = false. Puppet run on the nagios server produces the following output:

 notice: /File[/etc/nagios/auto.d/services/buildtest.example.edu-openssh_process.cfg]/owner: owner changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest.example.edu-openssh_process.cfg]/group: group changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest.example.edu-openssh_process.cfg]/mode: mode changed '644' to '755'
 notice: /File[/etc/nagios/auto.d/services/buildtest.example.edu-openssh_tcp_22.cfg]/owner: owner changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest.example.edu-openssh_tcp_22.cfg]/group: group changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest.example.edu-openssh_tcp_22.cfg]/mode: mode changed '644' to '755'

After this action the puppet run completes, with no service restart/reload of the nagios service. Even if there was, the service files are still readable by the nagios user and still get parsed upon service restart.

Scenario2:
Node is removed from Foreman+Puppet+PuppetDB via Foreman Delete + puppet node deactivate . Nagios host and service monitors are left hanging on the nagios server. Puppet run on the nagios server looks similar to the above, but also changes host configs:

 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-00-baseservices.cfg]/owner: owner changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-00-baseservices.cfg]/group: group changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-00-baseservices.cfg]/mode: mode changed '644' to '755'
 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-nrpe_process.cfg]/owner: owner changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-nrpe_process.cfg]/group: group changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-nrpe_process.cfg]/mode: mode changed '644' to '755'
 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-openssh_process.cfg]/owner: owner changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-openssh_process.cfg]/group: group changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-openssh_process.cfg]/mode: mode changed '644' to '755'
 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-openssh_tcp_22.cfg]/owner: owner changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-openssh_tcp_22.cfg]/group: group changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/services/buildtest2.example.edu-openssh_tcp_22.cfg]/mode: mode changed '644' to '755'
 notice: /File[/etc/nagios/auto.d/hosts/buildtest2.example.edu.cfg]/owner: owner changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/hosts/buildtest2.example.edu.cfg]/group: group changed 'root' to 'nagios'
 notice: /File[/etc/nagios/auto.d/hosts/buildtest2.example.edu.cfg]/mode: mode changed '644' to '755'
 notice: Finished catalog run in 3.60 seconds

Again, no service nagios restart/reload within the puppet run.

The only resolution is to manually delete the files modified and reload nagios.

[alvagante]

For case 1 the problem is due to the fact that in most modules the monitor defines are declared only if $monitor is true, otherwise they are not evaluated. Since monitoring is an optional feature I don't want to have the relevant resource declared (with an ensure => absent) if $monitor is false.
A possible approach could be to declare them whenever $monitor_tool is set, then they are enabled or disabled explicitly according to the value of $monitor .
Would it make sense for you? This is a change that has to be made on ALL the modules :-I so I'd test it first on a specific one (openssh?)

[salderma]

If you can mock up an example of how you'd like it to work, I'd be happy to help push it into the modules.

I would think a less popular module might be a good one to start with, but I will defer to your expertise.

TFTP or Splunk come to mind for me. As I was about to dig into these to adjust how the application of monitoring and firewall are applied specifically to the method of deployment - e.g. avoid xinetd port monitoring from the tftp module, since it seems to create a malformed Nagios Service definition (perhaps that's another bug).

[alvagante]

Please, give a look to this:
example42/puppet-tftp@67eb4e5
and let me know if it works (for tftp).
If yes we can extend the solution to other modules.

[salderma]

Ok, some notes on my testing.
test host:
puppet 2.6.18-3.el6
puppet master:
foreman-1.2.2-1.rl6
puppet-3.2.2-1.el6
puppet-server-3.2.2-1.el6
puppetdb-1.3.2-1.el6
nagios host:
puppet-2.6.18-3.el6
nagios-3.5.0-1.el6

1. Setting tftp_monitor = true, produces no tftp specific monitors on the Nagios Server, or in Puppi on the localhost.
2. Monitor checks for Nagios and Puppi:
   a. puppi checks for xinetd_process and xinetd_tcp_ are created.
   b. nagios checks for xinetd_process and xinetd_tcp_ are created.
3. Setting tftp_monitor = false, produces no significant change in monitors or status (output in paste: http://pastebin.com/Lj2ydvyV)
4. Setting tftp_startup_mode = standalone, tftp_monitor = true, results in a non-functional TFTP daemon.
   a. It appears there is no init script, so the service start fails.
   b. Output of puppet run is at pasted here: http://pastebin.com/urCLR9xA
5. Monitor checks for Nagios and Puppi
   a. puppi checks for xinetd have been removed from the local host.
   b. puppi check for tftp_process has been created on the local host.
   c. nagios check for xinetd remain on the nagios host.
   d. nagios check for tftp_process has been created, but remains in a critical state.
6. Changing tftp_startup_mode from xinetd to standalone after xinetd has been configured does not disable the xinetd config for tftp.

I guess I picked a bad module to start with, it would seem we've got some other problems in the way of the issue with module_monitor = false between module and nagios server.

I believe I can cleanup the xinetd monitoring by 1) removing xinetd port monitoring from the xinetd module and 2) take an ensure absent approach when switching modes in the tftp module.

[alvagante]

Yes, definitively tftp is not the proper module to test this.
Would you try to apply the changes I did to the tftp module to another module you use, which doesn't use xinetd, and let me know how it works?

[salderma]

Sure, not a problem. I do have a ? on the tftp module - do you have any idea how that xinetd_tcp_ check originates? I need to figure out how to get rid of it, I'm guessing it's an artifact in the puppetdb from including the puppet-xinetd module but I'm not sure how to clean it up.

[alvagante]

Yes, that's due to the inclusion of xinetd module in the tftp one, when $startup_mode = 'xinetd' (as by default).
So in order to remove its checks, apply to the xinetd module a patch similar to the one done on tftp and pass monitor => false, (or disable => true in you don't need the xinetd service (since it has been installed by tftp...).
Incidentally, the tdtp module should support the standalone service for Debian derivatives, on RedHta no default init script is provided (you can pass it via $file_init_template but it should be better to have a working one embedded in the module).

[salderma]

Ok, thanks, the mystery for me is that xinetd module doesn't have a port monitor at all in init.pp, oddly there is a firewall stanza which, given xinetd's purpose, I'm not at all sure how you'd handle that.

I've moved your code in to my fork of the splunk module. I'll run a quick test and update the issue.

[alvagante]

Actually is a mystery also for me, there should not be any trace of xinetd_tcp_ monitor.
And the firewall stanza is actually pointless, there. (Since no port is defined in params.pp it's not used in any case).

[salderma]

Couple notes on pre-testing tasks to undo previous configuration...

1. had to set tftp_disable = true
2. had to explicitly include xinetd module and set xinetd_disable = true

See commit: https://github.com/salderma/puppet-splunk/commit/3ea8cc9b96aba161ee0f4fb3c3ce649788672b21

Going from splunk_monitor = true to splunk_monitor = false, Success on the nagios end:
http://pastebin.com/wZztgRKt

Going back to splunk_monitor = true works as expected to re-establish monitoring.

Going to splunk_disable = true provides the same results as splunk_monitor = false.

Looks good!

[salderma]

related commits:
https://github.com/salderma/puppet-apache/commit/da1b013a05340f34fd349c7f84bb165a7865c56b
https://github.com/salderma/puppet-puppet/commit/ed9e05a74003f843e9512e51e1dcfa8413faecdf
https://github.com/salderma/puppet-vsftpd/commit/0e2015c3dacf4d7b4ee366820984c6da6ae92295

[alvagante]

They look good, when you are confident about them (I am :-) would you please submit a pull request?

[lieutdan13]

To help with scenario 2, I this can be (partially) solved by adding tags to exported resources.

For example, in nagios::host:

tag     => "nagios_check_${nagios::target::magic_tag}",

becomes

tag     => ["nagios_check_${nagios::target::magic_tag}", "nagios_host_${name}"],

When the node no longer exists and you are unable to run puppet on it, add the following to the nagios server's node:

File <| (tag == "nagios_check_${nagios::target::magic_tag}" and tag == "nagios_host_[FQDN of deceased host]") |> {
    ensure => absent
}

Note: Exact syntax may be different.