Hades MP Service
From GEANT2-JRA1 Wiki
Contents |
Objectives
The Hades framework is a measurement system for active one-way delay measurements. Hades means Hades Active Delay Evaluation System and
was formerly known as DFN-IPPM. Hades consists of three distinct parts:
The measurements are done in the GÉANT2 network with dedicated measurement boxes.
The results of these measurements are collected and stored by a central server, thus providing all the Hades data of the GÉANT2 network
at one access point.
On this server, the Measurement Archive (MA) for the Hades data is running, implementing the perfSONAR interface for retrieving
measurement data. The implementation of this MA is done in Perl.
Achievements
Functionality on the Hades measurement boxes
A sender process is sending groups of UDP packets from one measurement point to another, residing on another measurement box. The groups
are sent at a fixed interval rate, usually each 30 or 60 seconds. The receiving box logs the timestamps of each received packet in a text
file. For each day and each connection, a new file is created. The time gap between the packets is a fixed value as small as possible.
For every packet the sequence number and a send and a receive timestamp are logged. The timestamps have a maximal precision of
nanoseconds and are divided into one integer, representing the UNIX epoch seconds, and one integer, representing the nanoseconds
fraction. Each log file also has a header containing information about the measurement data, such as packet size, group size, and group
sending interval.
Collecting and analysing the data
The log files are collected by the Hades server from each measurement box and stored in dedicated directories, one for each day. An
analysing process parses the log files, line by line, where each line represents the data for one packet. In the first step of the
analysing process, the data is "sanitised", or cleaned up. The following cleanups are made:
Fix the sequence numbers, which can be reset due to a sequence number overflow or if the sending process on the measurement box was
stopped and then restarted.
Sort packets into the groups that were originally sent together by the sending process. This is done by the sequence numbers of the
packets.
Detect groups that are incomplete due to a restart of the sending or the receiving process on the measurement box. These groups should
not be used for further analysis as the non-existing packets could be mistaken for a packet loss.
Valid groups having less then the right number of packets are marked in the group with a counter for lost packets.
If packets were duplicated, the first received packet is used for analysis. Further duplicates are marked in the group with a counter.
Packet reordering is marked in the group, too. All information is stored in a fast and memory-optimised data structure useful for the further aggregation processes.
Data aggregation
Another aggregation process used in Hades tries to extract the most important information from the sanitised data and combines them in
one structure per group (for a detailed description of the group structure, see section 4.1.2.2) One-Way Delay (OWD) and One-Way Delay
Variation (OWDV) are calculated from the absolute send and receive timestamps. For both the groups, minimum, maximum, and median are
computed. Numbers of lost and duplicate packets, as well as the information about reordered packets in this particular group, are also
added (see section 4.1.1.2). The send time for the first packet is stored as the identifier for the whole group. The group structures are
then archived in files again, corresponding to the raw data files.
Implementation status
A Hades prototype is currently running in test mode, providing the data from measurements taken in the GÉANT2 network at this time. The
service is implemented as daemon process, handling the communication with clients. The Perl implementation is a modular design, using
CPAN modules [CPAN] and own modules. Additionally, a client application is provided, which is compliant with all perfSONAR services. It
is able to send XML request files to the service and displays the XML result.
Modules
The following modules from CPAN were used:
Log::Dispatch: Flexible log dispatcher to make logging easy for daemons.
Date::Manip: Powerful time and date handling, used for internal time handling.
Data::Dumper: Used for debugging purposes.
XML::LibXML: Perl module to use functionality of libxml, the open source library most often used for handling XML.
XML::Writer: Simple Perl module for manipulating XML.
SOAP::Lite: The Perl module most often used for SOAP. Very flexible and powerful. This module needed to be patched, to be useful for the
perfSONAR implementation. SOAP::Lite only has limited support for document/literal SOAP interaction. Therefore, a patch was created and
added.
The data handling was done by Hades-specific modules.
Hades::Data: A basic, abstract class from which all other data type classes are derived. It also contains code for managing all derived
data classes.
Hades::Data::IPPM, Hades::Data::IPPM_Raw, Hades::Data::IPPM_Sanitized, and Hades::Data::IPPM_Aggregated: Derived data classes, describing the data formats (see above).
Hades::Data::Finder: Interface to the measurement archive itself.
NMWG and NMWG::Message: These modules provide basic functionalities for handling NMWG/perfSONAR XML messages. NMWG::Message provides a
service and message type independent representation of an NMWG/perfSONAR message.
MS.pm: This module provides the functions doing the service-specific work using NMWG::Message. They are called via the SOAP::Lite server
deamon, called ms.pl.
Next steps
At the moment, it is possible to request the whole Measurement archive of Hades measurements residing on gandalf.rrze.uni-erlangen.de (at
port 8090). Provided data formats are raw and aggregated (see above). Moreover, the data itself can be viewed on http://www.win-labor.dfn.de, visualised as plots per connection and data set. Also, traceroute
information for the measurement paths are displayed. The deployment status of measurement boxes can be seen at
http://www.perfsonar.net/jra1-wiki/index.php/Equipment_installation.
References
www.win-labor.dfn.de
Contact
* Dipl.-Inf. Roland Karch * Dipl.-Phys. Ralf Kleineisel * Dipl.-Inf. Birgit König * Dr.-Ing. Stephan Kraft * Dipl.-Inf. Jochen Reinwand * Dipl.-Inf. Verena Venus
WiN Labor Regionales Rechenzentrum Erlangen Martensstraße 1 91058 Erlangen
Phone: +49 9131 85-28800 Fax: +49 9131 302941 E-Mail: win-labor@dfn.de
Work in progress
The next step is to implement key requests, thereby enabling simpler requests using a key, rather than a whole parameter set. Moreover, the storing format of the Hades data will change, from text files to a fast database backend for the MA. This has the advantage
of providing a fast and efficient search mechanism and a much more efficient data storing mechanism than storing data in files. In addition, an authentication mechanism, lookup service registration, on-demand capability, and resource protection will be implemented.
