December 4, 2008

Day 4 - Extending snmpd

Do you monitor your hosts with snmp? Ever wanted to add additional data sources to your snmp agent? Net-SNMP's snmpd lets you do this.

There are a few different options available to extend snmpd. The first is the most primitive, simply running a program and reporting the first line of output and the exit status. This is done with the 'exec' statement in snmpd.conf.

# Format is
# exec <name> <command> [args]
exec googleping /bin/ping -c 1 -w 1 -q
You need to specify the full path for 'exec' commands. If you want to run your command in /bin/sh, swap in 'sh' for 'exec' and you get to avoid the full path requirement. The 'exec' and 'sh' extensions command show results through the UCD-SNMP-MIB::extTable table:
% snmpwalk -v2c -c secret localhost UCD-SNMP-MIB::extTable
UCD-SNMP-MIB::extIndex.1 = INTEGER: 1
UCD-SNMP-MIB::extNames.1 = STRING: googleping
UCD-SNMP-MIB::extCommand.1 = STRING: /bin/ping
UCD-SNMP-MIB::extResult.1 = INTEGER: 0
UCD-SNMP-MIB::extOutput.1 = STRING: PING ( 56(84) bytes of data.
UCD-SNMP-MIB::extErrFix.1 = INTEGER: noError(0)
UCD-SNMP-MIB::extErrFixCmd.1 = STRING: 
You can see that the first line of output is available in extOutput. This is nice, but the order of commands depends entirely on the order in snmpd.conf, so if you put another 'exec' above the googleping one, the googleping check becomes .2 instead of .1, which is not so stable with respect to adding new exec statements or moving them around. Boo.

The second option available is called 'extend,' and it works similarly to 'exec,' but better. The 'extend' configuration accepts multiline output from your command and is indexed on the name (ie; "googleping") instead of an index number (ie; 1, 2, etc). Just change 'exec' to 'extend':

extend googleping /bin/ping -c 1 -w 1 -qn
extend mysqlstatus /usr/bin/mysqladmin status
These 'extend' commands show up in NET-SNMP-AGENT-MIB::nsExtensions. If you only want the output, you can walk NET-SNMP-EXTEND-MIB::nsExtendOutput1Table (or nsExtendOutput2Table). If you want only the exit code, you can walk nsExtendResult. If you want to view the output of walking nsExtensions (it's too long to post here), click here.

Remember the benefit of 'extend' over 'exec' was that the indexing was on the name, so let's query for only the googleping result:

% snmpget -v2c -c secret localhost 'NET-SNMP-EXTEND-MIB::nsExtendResult."googleping"' 
NET-SNMP-EXTEND-MIB::nsExtendResult."googleping" = INTEGER: 0

# If I null route all IPs, and requery:
% snmpget -v2c -c secret localhost 'NET-SNMP-EXTEND-MIB::nsExtendResult."googleping"' 
NET-SNMP-EXTEND-MIB::nsExtendResult."googleping" = INTEGER: 2
Take note above that the OID is in single quotes and "googleping" still needs to be sent as quoted to snmpget, this is so snmpget understands that this is really an octet-string OID. (See what "googleping" becomes with snmpget -On)

The output and exit code of your 'extend' and 'exec' statements are cached for a short period of time. The exact time saved in cache is determined by the nsExtendCacheTime OID. If you have write access configured in snmp, you can issue a SET command to change the cache time.

# Cache the googleping results for 15 seconds
% snmpset -v2c -c secret localhost 'NET-SNMP-EXTEND-MIB::nsExtendCacheTime."googleping"' i 15
NET-SNMP-EXTEND-MIB::nsExtendCacheTime."googleping" = INTEGER: 15

% snmpwalk -v2c -c secret localhost NET-SNMP-EXTEND-MIB::nsExtendCacheTime
NET-SNMP-EXTEND-MIB::nsExtendCacheTime."googleping" = INTEGER: 15
NET-SNMP-EXTEND-MIB::nsExtendCacheTime."mysqlstatus" = INTEGER: 5
Lastly, you can tell snmpd to 'pass' (that's the name of the config statement) handling of an entire OID subtree to an external program, which seems like a nice feature. This lets you write a subtree handler in your language of choice rather than being required (while still an option) to write your more complex handlers using snmpd's perl support or C module support. For brevity, I'll skip coverage of that, but it works similar to 'extend' and 'exec,' but has it's own (simple) text protocol for telling your subprocess what OID it wants data on (See further reading).

Extending SNMP to support your own data sources is a good way to allow your existing monitoring tools (nagios, etc) to monitor remotely without having to have local access such as with ssh or nrpe.

Further reading:

Net-SNMP snmpd extension configuration and documentation


Thomas said...

Thanks a lot for this so quick and clear tutorial !

Dieter_be said...

I falsely assumed that if i wanted to access all lines of output separately, I could just use for example 'NET-SNMP-EXTEND-MIB::nsExtendOutput2Line."mystring"' for the second line but this gave unrecognized OID's.

Turns out I needed to do:

Also, for monitoring systems such as zenoss, you need the numeric OID, not the textual representation. by doing `snmpwalk -On` you can get the numeric oid from the string one.

Thanks for this great article.

Sebastián Alvarez said...

I know, your post is old... but you save my day.

Thanks from Argentina :D.

Aaron said...

Hi! First off, thank you for this post, really useful. My question however is some of my servers are giving a "Permission denied" when I do an SNMPWalk on NET-SNMP-AGENT-MIB:nsExtensions from my Nagios server. Any thoughts?

Anonymous said...

Great tutorial and easy to follow, but one question I made an entry the snmpd.conf as such:

exec power_isurge /usr/bin/perl /tmp/

it works fine i get my scripts output, but when I run an snmp walk on UCD-SNMP-MIB::extNames i get this about 50 times. do you know why?

UCD-SNMP-MIB::extNames.1 = STRING: power_isurge
UCD-SNMP-MIB::extNames.2 = STRING: power_isurge
UCD-SNMP-MIB::extNames.3 = STRING: power_isurge
UCD-SNMP-MIB::extNames.4 = STRING: power_isurge
UCD-SNMP-MIB::extNames.5 = STRING: power_isurge
UCD-SNMP-MIB::extNames.6 = STRING: power_isurge
UCD-SNMP-MIB::extNames.7 = STRING: power_isurge
UCD-SNMP-MIB::extNames.8 = STRING: power_isurge
UCD-SNMP-MIB::extNames.9 = STRING: power_isurge
... same all the way to 50.

Jay said...

Thanks for this! It's helped me gather info from some very customised Linux boxes.

Also a big thanks to Dieter_be for pointing out potential problems when doing this sort of thing via Zenoss.

Ian said...

This insists on the values being strings. How do I get them to treat it as integers or gauges?

. = STRING: 1
. = STRING: 0
. = STRING: 0
. = STRING: 1
. = STRING: 0