PerfSONAR PS Perl SNMP MA Installation

From GEANT2-JRA1 Wiki

Contents 1 Information 2 Introduction 3 Caveats 4 Installation 4.1 Automatic 4.1.1 configure.PL Script 4.1.1.1 Post Install 4.1.1.2 Known Issues 4.1.1.2.1 Perl Modules 4.1.1.2.2 RRDp.pm Patch 4.2 Manual 4.2.1 Prerequisites 4.2.1.1 RRDTool 4.2.1.1.1 Known Issues 4.2.1.2 RRDp.pm Patch 4.2.1.3 Perl Modules 4.2.1.3.1 Known Issues 5 SNMP MA 5.1 Configuration 6 Testing 7 Running 8 References

Information

Version(s)	RC1 - RC6
Date	November 5, 2007
Author	Jason Zurawski

Introduction

This document describes the installation procedures of the perfSONAR-PS SNMP based Measurement Archive (MA). perfSONAR-PS is an implementation of the perfSONAR standard written in the Perl programming language. 12 perfSONAR-PS aims for complete compatibility with all other implementations of the perfSONAR standard.

This software offers the ability to publish data collected via Simple Network Management Protocol ('SNMP') software. 3 This particular release focuses on data that offers certain characteristics of the network, namely network utilization, errors, and discards. These three metrics are collected via (external) means and are normally stored in Round Robin Databases (RRD). 4 The collection of these metrics is beyond the scope if this software release, but many tools exist such as MRTG, SNAPP, and CACTI that are capable of collecting and storing network data. 567

The publishing of data is accomplished by exposing the data located in RRD files by describing the metadata (i.e. static information about the data such as how/where it was collected, etc.) A Web Services interface is offered that allows client applications such as perfSONAR-UI to contact this service, query for specific resources, and finally present graphs of the results. 8

Caveats

This software represents the first widespread release of perfSONAR-PS. As such we are still experimenting with packaging and delivery methods that will best fit the user community.

This release is being presented as a stand-alone application. Other perfSONAR-PS services will be presented the same way, and do not share the underlying libraries. Future releases WILL share the core code, and will be available through the more accessible and standardized Comprehensive Perl Archive Network (CPAN) tool.

We thank the user community in advance for bearing with this flux in installation procedure and still look forward to hearing comments that will make the process of obtaining and installing perfSONAR code much faster and easier.

Installation

Before installing the SNMP based MA, be sure to check if currently have the software listed in the prerequisite sub-section. Software versions may change from prior releases of perfSONAR-PS without explicit notice.

Automatic

An automatic installation script has been developed that will take care of obtaining libraries and software needed for this application.

configure.PL Script

To execute the script:

     
(sudo) perl configure.PL

The script will check for and attempt to install RRDTool for you. This process may fail if not all of the supporting libraries for RRDTool are found. Please see the remainder of this document for more information. The script can be run repeatedly to find issues with no harm to the system.

The script will also check for required perl libraries and use CPAN to install any that are missing or old. The script will report any errors that CPAN encountered due to missing components.

Post Install

Set necessary paths for RRDTool (this can be done from /etc/profile, or from other recognized locations):

      
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/rrdtool/lib
 export PATH=$PATH:/usr/local/rrdtool/bin 

Known Issues

In addition to the issues regarding the installation of RRDTool, these issues may arise:

Perl Modules

Some modules require software libraries that may not be installed on most systems. Although it is normally the job of the module designer to take care of these situations, this is not always the case. Specifically the XML modules (XML::XPath, XML::LibXML) require the the XML::Parser module that depends on several internal libraries.

If you system is equipped with a software management system (such as up2date, yum, or apt) try and locate the xml-parser-perl package via the search interface. This is the easiest way to avoid trouble with CPAN. Additionally, ensure that libxml2-dev, the LibXML version 2 development libraries, are also available.

RRDp.pm Patch

RRDTool will enter into an infinite loop due to an error condition in the RRDp.pm Perl module. A patch is available in util/ of this distribution.

To apply this patch:

Find your copy of RRDp.pm:

 
find /usr | grep RRDp.pm

Copy the patchfile to that location:

 
sudo cp patchfile /PATH/TO/FILE && cd /PATH/TO/FILE 

Patch:

  
sudo patch -p0 -u < patchfile

Remove the patchfile:

 
sudo rm -f patchfile

Manual

The software can be installed without the use of the configure script. If this script also fails it is possible to follow these steps.

Prerequisites

To ensure proper installation and functioning, the SNMP MA requires two key areas of prerequisite software:

RRDTool

RRDTool is the OpenSource industry standard, high performance data logging and graphing system for time series data. 4

Download and unpack RRDTool:

wget -c http://oss.oetiker.ch/rrdtool/pub/rrdtool.tar.gz
tar -zxvf rrdtool.tar.gz

From inside of the install directory, configure RRDTool:

./configure --prefix=/usr/local/rrdtool --enable-perl-site-install

Finalize the installation of RRDTool:

make
sudo make install

Set necessary paths for RRDTool (this can be done from /etc/profile, or from other recognized locations):

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/rrdtool/lib
export PATH=$PATH:/usr/local/rrdtool/bin 

Known Issues

RRDTool requires four development libraries that may not be found on most systems:

• libart 2 dev
• zlib dev
• libpng 1.2 dev
• freetype 2 dev

It is recommended that a package updating system (such as apt, yum, or up2date) be used to find the correct versions of these libraries and install them automatically.

If a software management system is unavailable, the libraries can be downloaded and installed from this location:

http://oss.oetiker.ch/rrdtool/pub/libs/

Ensure that all install locations are accessible through path variables.

RRDp.pm Patch

RRDTool will enter into an infinite loop due to an error condition in the RRDp.pm Perl module. A patch is available in util/ of this distribution.

To apply this patch:

Find your copy of RRDp.pm:

 
find /usr | grep RRDp.pm

Copy the patchfile to that location:

 
sudo cp patchfile /PATH/TO/FILE && cd /PATH/TO/FILE 

Patch:

  
sudo patch -p0 -u < patchfile

Remove the patchfile:

 
sudo rm -f patchfile

Perl Modules

There are several Perl modules that are required for the SNMP MA to function properly. Some of these are included in the base Perl installation on many machines, although it is important to check all to be sure upgrades are not available.

The most straightforward way to install Perl modules is through the CPAN administrative interface available on most machines:

sudo perl -MCPAN -e shell

If this is your first time using CPAN, there will be several questions to answer as you prepare the interface. You will eventually be presented with a shell:

cpan[1]>

Modules may be installed by typing:

      
cpan[1]>install MODULE::NAME

Instructions may appear as the installation proceeds, errors may also occur that will need to be handled. If a module is installed, a message may appear:

  
cpan[1]> install MODULE::NAME
MODULE::NAME is up to date (1.9707).   

This is a list of the necessary modules for the LS:

• strict
• warnings
• Exporter
• Cwd
• POSIX
• Fcntl
• Getopt::Long
• Log::Log4perl
• Params::Validate
• Log::Dispatch
• Log::Dispatch::Screen
• Log::Dispatch::FileRotate
• IO::File
• Time::Local
• HTTP::Daemon
• LWP::UserAgent
• XML::XPath
• XML::LibXML
• File::Temp
• File::Basename
• Time::HiRes

Known Issues

Some modules require software libraries that may not be installed on most systems. Although it is normally the job of the module designer to take care of these situations, this is not always the case. Specifically the XML modules (XML::XPath, XML::LibXML) require the the XML::Parser module that depends on several internal libraries.

If you system is equipped with a software management system (such as apt, yum, or up2date) try and locate the xml-parser-perl package via the search interface. This is the easiest way to avoid trouble with CPAN. Additionally, ensure that libxml2-dev, the LibXML version 2 development libraries, are also available.

SNMP MA

After installing the prerequisite software, the remaining steps to getting the SNMP MA working are few. The software directory should be placed in a common location such as /usr/local or /opt. This is not a requirement.

There are three major configuration files that may be adjusted, although the software is configured to run out of the box. The configurations files can be adjusted to personalize settings, and to add the metrics you are collecting of course.

Configuration

• logger.conf - This file contains information regarding logging in the perfSONAR-PS architecture. There is a choice between three methods:
  • log4perl.appender.A1=Log::Dispatch::Screen - sends log files to the output screen
  • log4perl.appender.A1=Log::Dispatch::Syslog - append the log messages directly to the Syslog system
  • log4perl.appender.A1=Log::Dispatch::FileRotate - appends log messages to a specified log file (features rotation of old files)
• ms.conf - This file contains detailed information and settings that help operate the SNMP MA. Here is a listing of the major directives, and their purpose. Defaults are included in the initial file, and for most instances will work fine:
  • METADATA_DB_TYPE - Type of the metadata database, this should be file (and only file) for this particular version.
  • METADATA_DB_NAME - N/A
  • METADATA_DB_FILE - Location of the 'store.xml' file, the location of all metadata and key descriptions.
  • RRDTOOL - Path to the BINARY for rrdtool.
  • PORT - Listen port for the application. This should be something that can be accessed from the outside world.
  • ENDPOINT - An endPoint is a contact string that a service is listening on. This is used in conjunction with the port and hostname: http://HOST:PORT/end/Point/String
  • DEFAULT_RESOLUTION - This is a value that pertains to the resolution used in the RRAs in an RRD. It can be left blank if unsure.
  • LS_REGISTRATION_INTERVAL - The interval of time (in seconds) that this service will contact the LS_INSTANCE to register itself.
  • LS_INSTANCE - The LS that will be contacted to register this service.
  • SERVICE_NAME - String describing the 'name' of this MA.
  • SERVICE_ACCESSPOINT - String describing the accessPoint of this MA. This string is how the service can be reached: http://HOST:PORT/end/Point/String
  • SERVICE_TYPE - String describing the type of this MA, it can be just MA or SNMP MA, etc.
  • SERVICE_DESCRIPTION - String describing this MA, it can include location info, etc.
  • MAX_WORKER_PROCESSES - Maximum number of child processes that can be spawned at a given time.
  • MAX_WORKER_LIFETIME - Maximum lifetime a child has to do work (eliminates hung children)
• store.xml - This file contains the metadata descriptions of each of the interfaces that will be offering data to the service. Additionally, there are also keys that list information used to access the data. See the current store file for an example of the format, it is important to keep the format very similar as much of the service's usefulness depends on being able to search/locate information via this description.

A helper utility named configure.pl (located in util/) can help create th-is file by providing questions regarding each value.

Testing

As stated previously, the MA can run essentially out of the box (locally at least). You will need two terminal sessions to test out the MA, one for the server and the other for a client application (included).

In the first terminal:

Enter the directory where the software is installed, run this command:

      
./ma.pl --verbose

You may see output like this:

Can't write pidfile to /var/run

This is a warning letting you know that you don't have permission to use the default pid file location (/var/run). Normally only the root user will have the ability to do this, so a new pid directory must be tried. Try this:

./ma.pl --verbose --piddir=/tmp

This will write the pid file to /tmp. You will now see output similar to this:

    
Use of uninitialized value in scalar chomp at ./ma.pl line 188.
Use of uninitialized value in string ne at ./ma.pl line 189.
2007/10/31 13:02:13 (30564) DEBUG> ma.pl:119 main:: - Starting '30564'
2007/10/31 13:02:13 (30564) DEBUG> Transport.pm:166 perfSONAR_PS::Transport::startDaemon - Starting daemon.
2007/10/31 13:02:13 (30564) DEBUG> ma.pl:136 main:: - Not registering with an LS instance
2007/10/31 13:02:13 (30565) DEBUG> ma.pl:226 main::measurementArchive - Starting '30565' as the MA.
2007/10/31 13:02:13 (30565) DEBUG> Transport.pm:187 perfSONAR_PS::Transport::acceptCall - Accepting calls.

The server is started, and it has stared its major functions: registering data with an LS and listening for calls. The LS registration functionality is disabled by default.

In the second terminal:

Enter the directory where the software is installed, change into the client directory, and run this command:

./client.pl --server=localhost --port=8081 --endpoint=perfSONAR_PS/services/snmpMA requests/EchoRequest.xml

This will tell the client to contact the MA which is running on:

    
http://localhost:8081/perfSONAR_PS/services/snmpMA

And will send a simple Echo message (to see if the server is active). The client will see this output if everything is running smoothly:

2007/10/31 13:22:01 (31823) DEBUG> Transport.pm:145 perfSONAR_PS::Transport::getHttpURI - Created URI: http://localhost:8081//perfSONAR_PS/services/snmpMA
2007/10/31 13:22:01 (31823) DEBUG> Transport.pm:346 perfSONAR_PS::Transport::sendReceive - Sending information to "http://localhost:8081//perfSONAR_PS/services/snmpMA".
2007/10/31 13:22:01 (31823) DEBUG> Transport.pm:364 perfSONAR_PS::Transport::sendReceive - Response returned.
<?xml version="1.0"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap-.org/soap/envelope/">
  <SOAP-ENV:Header/>
  <SOAP-ENV:Body>
    <nmwg:message xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" id="message.8731091" messageIdRef="echoRequest1" type="EchoResponse">
      <nmwg:metadata xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" id="metadata.15782028" metadataIdRef="meta1">
        <nmwg:eventType>success.echo</nmwg:eventType>
      </nmwg:metadata>
      <nmwg:data xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" id="data.14442771" metadataIdRef="metadata.15782028">
        <nmwgr:datum xmlns:nmwgr="http://ggf.org/ns/nmwg/result/2.0/">The echo request has passed.</nmwgr:datum>
      </nmwg:data>
    </nmwg:message>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

If you are not seeing a similar message, you may see something such as this:

2007/10/31 13:32:29 (32494) DEBUG> Transport.pm:145 perfSONAR_PS::Transport::getHttpURI - Created URI: http://localhost:8081//perfSONAR_PS/services/snmpMA
2007/10/31 13:32:29 (32494) DEBUG> Transport.pm:346 perfSONAR_PS::Transport::sendReceive - Sending information to "http://localhost:8081//perfSONAR_PS/services/snmpMA".
2007/10/31 13:32:29 (32494) DEBUG> Transport.pm:356 perfSONAR_PS::Transport::sendReceive - Send to "http://localhost:8081//perfSONAR_PS/services/snmpMA" failed.
Empty String at ./client.pl line 106

This message indicates that the server is not properly listening on the port you are contacting, perhaps it is listening on another, the port is firewalled, or it may not be started at all. Please check the above steps, and try again.

In the first terminal:

This information will appear when the call comes in:

2007/10/31 13:21:58 (31815) DEBUG> ma.pl:119 main:: - Starting '31815'
2007/10/31 13:21:58 (31815) DEBUG> Transport.pm:166 perfSONAR_PS::Transport::startDaemon - Starting daemon.
2007/10/31 13:21:58 (31815) DEBUG> ma.pl:136 main:: - Not registering with an LS instance
2007/10/31 13:21:58 (31817) DEBUG> ma.pl:226 main::measurementArchive - Starting '31817' as the MA.
2007/10/31 13:21:58 (31817) DEBUG> Transport.pm:187 perfSONAR_PS::Transport::acceptCall - Accepting calls.
2007/10/31 13:22:01 (31817) INFO> Transport.pm:198 perfSONAR_PS::Transport::acceptCall - Received call from:    127.0.0.1
2007/10/31 13:22:01 (31817) DEBUG> Common.pm:281 perfSONAR_PS::Common::reMap - Setting namespace "http://schemas.xmlsoap.org/soap/envelope/" with prefix "SOAP-ENV".
2007/10/31 13:22:01 (31817) DEBUG> Common.pm:281 perfSONAR_PS::Common::reMap - Setting namespace "http://ggf.org/ns/nmwg/base/2.0/" with prefix "nmwg".
2007/10/31 13:22:01 (31817) DEBUG> Request.pm:108 perfSONAR_PS::Request::parse - 
<?xml version="1.0"?>
<nmwg:message xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" type="EchoRequest" id="echoRequest1">
  <nmwg:metadata id="meta1">
    <nmwg:eventType>http://schemas.perfsonar.net/tools/admin/echo/2.0</nmwg:eventType>
  </nmwg:metadata>
  <nmwg:data id="data1" metadataIdRef="meta1"/>
</nmwg:message>
2007/10/31 13:22:01 (31817) DEBUG> Request.pm:183 perfSONAR_PS::Request::setResponse - 
<?xml version="1.0"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP-ENV:Header/>
  <SOAP-ENV:Body>
    <nmwg:message xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" id="message.8731091" messageIdRef="echoRequest1" type="EchoResponse">
      <nmwg:metadata xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" id="metadata.15782028" metadataIdRef="meta1">
        <nmwg:eventType>success.echo</nmwg:eventType>
      </nmwg:metadata>
      <nmwg:data xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" id="data.14442771" metadataIdRef="metadata.15782028">
        <nmwgr:datum xmlns:nmwgr="http://ggf.org/ns/nmwg/result/2.0/">The echo request has passed.</nmwgr:datum>
      </nmwg:data>
    </nmwg:message>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>
2007/10/31 13:22:01 (31817) DEBUG> SNMP.pm:147 perfSONAR_PS::MA::SNMP::receive - Received 'shadow' request from below; no action required.
2007/10/31 13:22:01 (31817) INFO> Request.pm:271 perfSONAR_PS::Request::finish - Seconds required to handle request from 127.0.0.1: 0
2007/10/31 13:22:01 (31817) DEBUG> Request.pm:275 perfSONAR_PS::Request::finish - Closing call.
2007/10/31 13:22:01 (31817) DEBUG> Transport.pm:187 perfSONAR_PS::Transport::acceptCall - Accepting calls.

The other messages in the client/requests may also be tried, with similar results.

Running

The software can be run as:

  
./ma.pl

This will background the process automatically. Be sure to adjust your logging settings in logger.conf.

References

• 1 perfSONAR
• 2 The Perl Directory
• 3 SNMP
• 4 RRDTool
• 5 MRTG
• 6 SNAPP
• 7 CACTI
• 8 perfSONAR-UI