Barth W. - Nagios. System and Network Monitoring [2005, PDF, ENG]

Contents

Contents
Introduction 15
From Source Code to a Running Installation 23
1 Installation 25
1.1 Compiling the Source Code . . . . . . . . . . . . . . . . . . . . . . 26
1.2 Installing and Testing Plugins . . . . . . . . . . . . . . . . . . . . . 30
1.2.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . 30
1.2.2 Plugin test . . . . . . . . . . . . . . . . . . . . . . . . . . 32
1.3 Configuration of the Web Interface . . . . . . . . . . . . . . . . . 33
1.3.1 Setting Up Apache . . . . . . . . . . . . . . . . . . . . . . 33
1.3.2 User Authentication . . . . . . . . . . . . . . . . . . . . . 34
2 Nagios Configuration 37
2.1 The Main Configuration File nagios.cfg . . . . . . . . . . . . . . . 38
2.2 Objects—an Overview . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.3 Defining the Machines to Be Monitored, with host . . . . . . . . . 44
2.4 Grouping Computers Together with hostgroup . . . . . . . . . . . 46
2.5 Defining Services to Be Monitored with service . . . . . . . . . . . 47
2.6 Grouping Services Together with servicegroup . . . . . . . . . . . 50
2.7 Defining Addressees for Error Messages: contact . . . . . . . . . . 50
2.8 The Message Recipient: contactgroup . . . . . . . . . . . . . . . . 52
2.9 When Nagios Needs to Do Something: the command Object . . . 53
2.10 Defining a Time Period with timeperiod . . . . . . . . . . . . . . . 54
2.11 Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.12 Configuration Aids for Those Too Lazy to Type . . . . . . . . . . . . 56
2.12.1 Defining services for several computers . . . . . . . . . . . 56
2.12.2 One host group for all computers . . . . . . . . . . . . . . 57
2.12.3 Other configuration aids . . . . . . . . . . . . . . . . . . . 57
2.13 CGI Configuration in cgi.cfg . . . . . . . . . . . . . . . . . . . . . 57
2.14 The Resources File resource.cfg . . . . . . . . . . . . . . . . . . . . 59
3 Startup 61
3.1 Checking the Configuration . . . . . . . . . . . . . . . . . . . . . . 61
3.2 Getting Monitoring Started . . . . . . . . . . . . . . . . . . . . . . 63
3.2.1 Manual start . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.2.2 Automatic start . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2.3 Making configuration changes come into effect . . . . . . 64
3.3 Overview of the Web Interface . . . . . . . . . . . . . . . . . . . . 64
In More Detail . . . 69
4 Nagios Basics 71
4.1 Taking into Account the Network Topology . . . . . . . . . . . . . 72
4.2 Forced Host Checks vs. Periodic Reachability Tests . . . . . . . . . . 75
4.3 States of Hosts and Services . . . . . . . . . . . . . . . . . . . . . 75
5 Service Checks and How They Are Performed 79
5.1 Testing Network Services Directly . . . . . . . . . . . . . . . . . . . 81
5.2 Running Plugins via Secure Shell on the Remote Computer . . . . 82
5.3 The Nagios Remote Plugin Executor . . . . . . . . . . . . . . . . . 82
5.4 Monitoring via SNMP . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.5 The Nagios Service Check Acceptor . . . . . . . . . . . . . . . . . . 84
6 Plugins for Network Services 85
6.1 Standard Options . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
6.2 Reachability Test with Ping . . . . . . . . . . . . . . . . . . . . . . 88
6.2.1 check_icmp as a service check . . . . . . . . . . . . . . . 90
6.2.2 check_icmp as a host check . . . . . . . . . . . . . . . . . 91
6.3 Monitoring Mail Servers . . . . . . . . . . . . . . . . . . . . . . . . 92
6.3.1 Monitoring SMTP with check_smtp . . . . . . . . . . . . . 92
6.3.2 POP and IMAP . . . . . . . . . . . . . . . . . . . . . . . . 95
6.4 Monitoring FTP and Web Servers . . . . . . . . . . . . . . . . . . . 97
6.4.1 FTP services . . . . . . . . . . . . . . . . . . . . . . . . . . 97
6.4.2 Web server control via HTTP . . . . . . . . . . . . . . . . . 98
6.4.3 Monitoring Web proxies . . . . . . . . . . . . . . . . . . . 101
6.5 Domain Name Server under Control . . . . . . . . . . . . . . . . . 105
6.5.1 DNS check with nslookup . . . . . . . . . . . . . . . . . . 106
6.5.2 Monitoring the name server with dig . . . . . . . . . . . . 107
6.6 Querying the Secure Shell Server . . . . . . . . . . . . . . . . . . . 108
6.7 Generic Network Plugins . . . . . . . . . . . . . . . . . . . . . . . 110
6.7.1 Testing TCP ports . . . . . . . . . . . . . . . . . . . . . . . 110
6.7.2 Monitoring UDP ports . . . . . . . . . . . . . . . . . . . . 112
6.8 Monitoring Databases . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.8.1 PostgreSQL . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.8.2 MySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.9 Monitoring LDAP Directory Services . . . . . . . . . . . . . . . . . 121
6.10 Checking a DHCP Server . . . . . . . . . . . . . . . . . . . . . . . . 124
6.11 Monitoring UPS with the Network UPS Tools . . . . . . . . . . . . 126
7 Testing Local Resources 133
7.1 Free Hard Drive Capacity . . . . . . . . . . . . . . . . . . . . . . . 134
7.2 Utilization of the Swap Space . . . . . . . . . . . . . . . . . . . . . 136
7.3 Testing the System Load . . . . . . . . . . . . . . . . . . . . . . . . 137
7.4 Monitoring Processes . . . . . . . . . . . . . . . . . . . . . . . . . 138
7.5 Checking Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.5.1 The standard plugin check_log . . . . . . . . . . . . . . . 142
7.5.2 The modern variation: check_logs.pl . . . . . . . . . . . . 143
7.6 Keeping Tabs on the Number of Logged-in Users . . . . . . . . . . 144
7.7 Checking the System Time . . . . . . . . . . . . . . . . . . . . . . 145
7.7.1 Checking the system time via NTP . . . . . . . . . . . . . . 145
7.7.2 Checking system time with the time protocol . . . . . . . 146
7.8 Regularly Checking the Status of the Mail Queue . . . . . . . . . . 147
7.9 Keeping an Eye on the Modification Date of a File . . . . . . . . . 148
7.10 Monitoring UPSs with apcupsd . . . . . . . . . . . . . . . . . . . . 149
7.11 Nagios Monitors Itself . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.11.1 Running the plugin manually with a script . . . . . . . . . 151
7.11.2 check_nagios as a tool for CGI programs . . . . . . . . . . 152
7.12 Hardware Checks with LM Sensors . . . . . . . . . . . . . . . . . . 152
7.13 The Dummy Plugin for Tests . . . . . . . . . . . . . . . . . . . . . 154
8 Manipulating Plugin Output 155
8.1 Negating Plugin Results . . . . . . . . . . . . . . . . . . . . . . . . 155
8.2 Inserting Hyperlinks with urlize . . . . . . . . . . . . . . . . . . . 156
9 Executing Plugins via SSH 157
9.1 The check_by_ssh Plugin . . . . . . . . . . . . . . . . . . . . . . . 158
9.2 Configuring SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
9.2.1 Generating SSH key pairs on the Nagios server . . . . . . . 160
9.2.2 Setting up the user nagios on the target host . . . . . . . 161
9.2.3 Checking the SSH connection and check_by_ssh . . . . . 161
9.3 Nagios Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 162
10 The Nagios Remote Plugin Executor (NRPE) 165
10.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
10.1.1 Distribution-specific packages . . . . . . . . . . . . . . . . 166
10.1.2 Installation from the source code . . . . . . . . . . . . . . 167
10.2 Starting via the inet Daemon . . . . . . . . . . . . . . . . . . . . . 168
10.2.1 xinetd configuration . . . . . . . . . . . . . . . . . . . . . 168
10.2.2 inetd configuration . . . . . . . . . . . . . . . . . . . . . 169
10.3 NRPE Configuration on the Computer to Be Monitored . . . . . . . 170
10.3.1 Passing parameters on to local plugins . . . . . . . . . . . 171
10.4 Nagios Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 172
10.4.1 NRPE without passing parameters on . . . . . . . . . . . . 172
10.4.2 Passing parameters on in NRPE . . . . . . . . . . . . . . . 173
10.4.3 Optimizing the configuration . . . . . . . . . . . . . . . . 173
10.5 Indirect Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
11 Collecting Information Relevant for Monitoring with SNMP 177
11.1 Introduction to SNMP . . . . . . . . . . . . . . . . . . . . . . . . . 178
11.1.1 The Management Information Base . . . . . . . . . . . . . 179
11.1.2 SNMP protocol versions . . . . . . . . . . . . . . . . . . . 183
11.2 NET-SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
11.2.1 Tools for SNMP requests . . . . . . . . . . . . . . . . . . . 184
11.2.2 The NET-SNMP daemon . . . . . . . . . . . . . . . . . . . 187
11.3 Nagios’s Own SNMP Plugins . . . . . . . . . . . . . . . . . . . . . 196
11.3.1 The generic SNMP plugin check_snmp . . . . . . . . . . . 196
11.3.2 Checking several interfaces simultaneously . . . . . . . . . 201
11.3.3 Testing the operating status of individual interfaces . . . . 203
11.4 Other SNMP-based Plugins . . . . . . . . . . . . . . . . . . . . . . 205
11.4.1 Monitoring hard drive space and processes with nagiossnmp-
plugins . . . . . . . . . . . . . . . . . . . . . . . . 205
11.4.2 Observing the load on network interfaces with checkiftraffic
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
11.4.3 The manubulon.com plugins for special application purposes
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
12 The Nagios Notification System 215
12.1 Who Should be Informed of What, When? . . . . . . . . . . . . . . 216
12.2 When Does a Message Occur? . . . . . . . . . . . . . . . . . . . . 217
12.3 The Message Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
12.3.1 Switching messages on and off systemwide . . . . . . . . 218
12.3.2 Enabling and suppressing computer and service-related
messages . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
12.3.3 Person-related filter options . . . . . . . . . . . . . . . . . 221
12.3.4 Case examples . . . . . . . . . . . . . . . . . . . . . . . . 222
12.4 External Notification Programs . . . . . . . . . . . . . . . . . . . . 224
12.4.1 Notification via e-mail . . . . . . . . . . . . . . . . . . . . 225
12.4.2 Notification via SMS . . . . . . . . . . . . . . . . . . . . . 227
12.5 Escalation Management . . . . . . . . . . . . . . . . . . . . . . . . 231
12.6 Dependences between Hosts and Services as a Filter Criterion . . . 234
12.6.1 The standard case: service dependencies . . . . . . . . . . 234
12.6.2 Only in exceptional cases: host dependencies . . . . . . . 238
13 Passive Tests with the External Command File 239
13.1 The Interface for External Commands . . . . . . . . . . . . . . . . 240
13.2 Passive Service Checks . . . . . . . . . . . . . . . . . . . . . . . . . 241
13.3 Passive Host Checks . . . . . . . . . . . . . . . . . . . . . . . . . . 242
13.4 Reacting to Out-of-Date Information of Passive Checks . . . . . . 243
14 The Nagios Service Check Acceptor (NSCA) 247
14.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
14.2 Configuring the Nagios Server . . . . . . . . . . . . . . . . . . . . 249
14.2.1 The configuration file nsca.cfg . . . . . . . . . . . . . . . 249
14.2.2 Configurung the inet daemon . . . . . . . . . . . . . . . . 251
14.3 Client-side Configuration . . . . . . . . . . . . . . . . . . . . . . . 252
14.4 Sending Test Results to the Server . . . . . . . . . . . . . . . . . . 253
14.5 Application Example I: Integrating syslog and Nagios . . . . . . . . 254
14.5.1 Preparing syslog-ng for use with Nagios . . . . . . . . . . 255
14.5.2 Nagios configuration: volatile services . . . . . . . . . . . 257
14.5.3 Resetting error states manually . . . . . . . . . . . . . . . 258
14.6 Application Example II: Processing SNMP Traps . . . . . . . . . . . 260
14.6.1 Receiving traps with snmptrapd . . . . . . . . . . . . . . 260
14.6.2 Passing on traps to NSCA . . . . . . . . . . . . . . . . . . 261
14.6.3 The matching service definition . . . . . . . . . . . . . . . 263
15 Distributed Monitoring 265
15.1 Switching On the OCSP/OCHP Mechanism . . . . . . . . . . . . . . 266
15.2 Defining OCSP/OCHP Commands . . . . . . . . . . . . . . . . . . . 267
15.3 Practical Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
15.3.1 Avoiding redundancy in configuration files . . . . . . . . . 269
15.3.2 Defining templates . . . . . . . . . . . . . . . . . . . . . . 270
16 The Web Interface 273
16.1 Recognizing and Acting On Problems . . . . . . . . . . . . . . . . 275
16.1.1 Comments on problematic hosts . . . . . . . . . . . . . . 276
16.1.2 Taking responsibility for problems: acknowledgements . . 278
16.2 An Overview of the Individual CGI Programs . . . . . . . . . . . . . 279
16.2.1 Variations in status display: status.cgi . . . . . . . . . . . 279
16.2.2 Additional information and control center: extinfo.cgi . . 284
16.2.3 Interface for external commands: cmd.cgi . . . . . . . . . 288
16.2.4 The most important things at a glance: tac.cgi . . . . . . 290
16.2.5 Network plan: the topological map of the network (statusmap.
cgi) . . . . . . . . . . . . . . . . . . . . . . . . . . 291
16.2.6 Navigation in 3D: statuswrl.cgi . . . . . . . . . . . . . . . 293
16.2.7 Querying the status with a cell phone: statuswml.cgi . . . 295
16.2.8 Analyzing disrupted partial networks: outages.cgi . . . . . 295
16.2.9 Querying the object definition with config.cgi . . . . . . . 295
16.2.10 Availability statistics: avail.cgi . . . . . . . . . . . . . . . 296
16.2.11 What events occur, how often? histogram.cgi . . . . . . . 298
16.2.12 Filtering log entries after specific states: history.cgi . . . . 299
16.2.13 Who was told what, when? notifications.cgi . . . . . . . 300
16.2.14 Showing all logfile entries: showlog.cgi . . . . . . . . . . 301
16.2.15 Evaluating whatever you want: summary.cgi . . . . . . . 301
16.2.16 Following states graphically over time: trends.cgi . . . . . 303
16.3 Planning Downtimes . . . . . . . . . . . . . . . . . . . . . . . . . 304
16.3.1 Maintenance periods for hosts . . . . . . . . . . . . . . . 305
16.3.2 Downtime for services . . . . . . . . . . . . . . . . . . . . 306
16.4 Additional Information on Hosts and Services . . . . . . . . . . . . 307
16.4.1 Extended host information . . . . . . . . . . . . . . . . . 307
16.4.2 Extended service information . . . . . . . . . . . . . . . . 310
16.5 Configuration Changes through the Web Interfaces: the Restart Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
17 Graphic Display of Performance Data 313
17.1 Processing Plugin Performance Data with Nagios . . . . . . . . . . 314
17.1.1 The template mechanism . . . . . . . . . . . . . . . . . . 314
17.1.2 Using external commands to process performance data . . 317
17.2 Graphs for the Web with Nagiosgraph . . . . . . . . . . . . . . . . 317
17.2.1 Basic installation . . . . . . . . . . . . . . . . . . . . . . . 318
17.2.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 319
17.3 Preparing Performance Data for Evaluation with Perf2rrd . . . . . 325
17.3.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . 326
17.3.2 Nagios configuration . . . . . . . . . . . . . . . . . . . . . 326
17.3.3 Perf2rrd in practice . . . . . . . . . . . . . . . . . . . . . . 327
17.4 The Graphics Specialist drraw . . . . . . . . . . . . . . . . . . . . . 330
17.4.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . 330
17.4.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 331
17.4.3 Practical application . . . . . . . . . . . . . . . . . . . . . 332
17.5 Automated to a Large Extent: NagiosGrapher . . . . . . . . . . . . 336
17.5.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . 336
17.5.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 338
17.6 Other tools and the limits of graphic evaluation . . . . . . . . . . . 349
Special Applications 351
18 Monitoring Windows Servers 353
18.1 NSClient and NC Net . . . . . . . . . . . . . . . . . . . . . . . . . 354
18.1.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . 354
18.1.2 The check_nt plugin . . . . . . . . . . . . . . . . . . . . . 355
18.1.3 Commands which can be run with NSClient and NC Net . 356
18.1.4 Advanced functions of NC Net . . . . . . . . . . . . . . . 363
18.2 NRPE for Windows: NRPE NT . . . . . . . . . . . . . . . . . . . . . 371
18.2.1 Installation and configuration . . . . . . . . . . . . . . . . 372
18.2.2 Function test . . . . . . . . . . . . . . . . . . . . . . . . . 373
18.2.3 The Cygwin plugins . . . . . . . . . . . . . . . . . . . . . . 373
18.2.4 Perl plugins in Windows . . . . . . . . . . . . . . . . . . . 374
19 Monitoring Room Temperature and Humidity 377
19.1 Sensors and Software . . . . . . . . . . . . . . . . . . . . . . . . . 378
19.1.1 The PCMeasure software for Linux . . . . . . . . . . . . . 378
19.1.2 The query protocol . . . . . . . . . . . . . . . . . . . . . . 379
19.2 The Nagios Plugin check_pcmeasure . . . . . . . . . . . . . . . . 379
20 Monitoring SAP Systems 383
20.1 Checking without a Login: sapinfo . . . . . . . . . . . . . . . . . . 384
20.1.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . 384
20.1.2 First test . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
20.1.3 The plugin check_sap.sh . . . . . . . . . . . . . . . . . . . 386
20.2 Monitoring with SAP’s Own Monitoring System (CCMS) . . . . . . 388
20.2.1 CCMS—a short overview . . . . . . . . . . . . . . . . . . . 388
20.2.2 Obtaining the necessary SAP usage permissions for Nagios 390
20.2.3 Monitors and templates . . . . . . . . . . . . . . . . . . . 392
20.2.4 The CCMS plugins . . . . . . . . . . . . . . . . . . . . . . 394
20.2.5 Performance optimization . . . . . . . . . . . . . . . . . . 398
Appendixes 399
A Rapidly Alternating States: Flapping 401
A.1 Flap Detection with Services . . . . . . . . . . . . . . . . . . . . . 402
A.1.1 Nagios configuration . . . . . . . . . . . . . . . . . . . . . 403
A.1.2 The history memory and the chronological progression of
the changes in state . . . . . . . . . . . . . . . . . . . . . 404
A.1.3 Representation in the Web interface . . . . . . . . . . . . 404
A.2 Flap Detection for Hosts . . . . . . . . . . . . . . . . . . . . . . . . 406
B Event Handlers 409
B.1 Execution Times for the Event Handler . . . . . . . . . . . . . . . . 410
B.2 Defining the Event Handler in the Service Definition . . . . . . . . 411
B.3 The Handler Script . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
B.4 Things to Note When Using Event Handlers . . . . . . . . . . . . . 413
C Writing Your Own Plugins: Monitoring Oracle with the
Instant Client 415
C.1 Installing the Oracle Instant Client . . . . . . . . . . . . . . . . . . 416
C.2 Establishing a Connection to the Oracle Database . . . . . . . . . . 417
C.3 A Wrapper Plugin for sqlplus . . . . . . . . . . . . . . . . . . . . . 417
C.3.1 How the wrapper works . . . . . . . . . . . . . . . . . . . 418
C.3.2 The Perl plugin in detail . . . . . . . . . . . . . . . . . . . 419
D An Overview of the Nagios Configuration Parameters 423
D.1 The Main Configuration File nagios.cfg . . . . . . . . . . . . . . . 424
D.2 CGI Configuration in cgi.cfg . . . . . . . . . . . . . . . . . . . . . 443
D.2.1 Authentication parameters . . . . . . . . . . . . . . . . . 443
D.2.2 Other Parameters . . . . . . . . . . . . . . . . . . . . . . . 444
Index 447