Hades MA Service
From GEANT2-JRA1 Wiki
Contents |
Objectives
The Hades framework is a measurement system for active one-way delay measurements. Hades means Hades Active Delay Evaluation System and
was formerly known as DFN-IPPM. Hades consists of three distinct parts:
The measurements are done in the GÉANT2 network with dedicated measurement boxes.
The results of these measurements are collected and stored by a central server, thus providing all the Hades data of the GÉANT2 network
at one access point.
On this server, the Measurement Archive (MA) for the Hades data is running, implementing the perfSONAR interface for retrieving
measurement data. The implementation of this MA is done in Perl.
Achievements
Measurement archive
The MA itself is a process enabling access to the data files on the Hades server. A search mechanism retrieves the data files needed to
answer a certain request. Then, the data for the requested time span is extracted from the files and concatenated for the response.
Specification
This section provides details on the input, output, and errors for the IPPM MA.
Input
Requesting measurement data is done via XML request files conforming to the Global Grid Forum’s Network Measurement Working Group (NMWG)
schema used by perfSONAR. The data can be requested in two different forms, namely raw or aggregated, specified by the eventType element
in the XML message (the data formats are described in detail in section 4.1.2.2). To make a valid request, some parameters must be
specified to query the data on the Hades server. Mandatory declarations are the type of data format, a time span over which the data are
requested, and the two end-points. This minimum set of information, however, may not be enough to specify a single measurement. If there
are more than one matching data sets, the measurement identification number (ID) or other parameters are necessary to identify one
measurement. The measurement ID is a unique integer number, representing one set of parameters. The data will only be returned if a measurement exists for the given parameter set. In the case of an incomplete set of requesting
parameters, the measurement may not be identified uniquely. The response is then an error message containing a list of possible matching
data sets, stating the parameter information and measurement IDs. Example for requesting raw data:
<?xml version="1.0" encoding="UTF-8"?>
<nmwg:message type="SetupDataRequest"
id="datarq2-1"
xmlns="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/"
xmlns:perfsonar="http://ggf.org/ns/nmwg/tools/org/perfsonar/1.0/"
xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/"
xmlns:nmtm="http://ggf.org/ns/nmwg/time/2.0/"
xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/">
<nmwg:metadata id="meta1">
<perfsonar:subject id="subj1">
<nmwgt:endPointPair>
<nmwgt:src type="IFname" value="Budapest_GEANT"/>
<nmwgt:dst type="IFname" value="Milano_GEANT"/>
</nmwgt:endPointPair>
</perfsonar:subject>
<nmwg:parameters>
<nmwg:parameter name="packetsize" value="41"/>
<nmwg:parameter name="precedence" value="0x0"/>
</nmwg:parameters>
<nmwg:eventType>ippm_raw</nmwg:eventType>
</nmwg:metadata>
<nmwg:metadata id="meta2">
<perfsonar:subject id="subj2" metadataIdRef="meta1"/>
<nmwg:eventType>select</nmwg:eventType>
<nmwg:parameters id="param2">
<nmwg:parameter name="startTime">1138025449</nmwg:parameter>
<nmwg:parameter name="endTime">1138026469</nmwg:parameter>
</nmwg:parameters>
</nmwg:metadata>
<nmwg:data id="1" metadataIdRef="meta2"/> </nmwg:message>
The example shows a request of measurement data from Budapest_GEANT to Milano_GEANT in raw data format. The select eventType is used to
specify the time span. As parameters, the packet size with value 41 and precedence with value 0x0 are given. The parameters, which can be specified directly if necessary to identify measurements, include:
Precedence: the QoS bits, may be e.g. 0x0 or 0xb8
Packet size: size of the UDP packet, e.g. 41 (minimum) or 1427 (maximum) byte
Group size: number of packets sent together by the sender, e.g. 5
Packet interval: inter-packet gap in a group, e.g. 20 ms
Group interval: sending interval between groups, e.g. 30 s
Output
Dependent on the eventType, the returned data is either in raw or aggregated format. One datum of raw data format (see description above) representing a single measurement packet contains these elements:
'seqnr': sequence number of packet,
'senttime_sec': sent time of packet in epoch seconds,
'senttime_nsec': additional nano seconds of sent time,
'recvtime_sec': receive time of packet in epoch seconds,
'recvtime_nsec': additional nano seconds of receive time.
The returned datum lines in the XML response look like this:
<nmwg:datum recvtime_nsec="182970000" recvtime_sec="1138026363" senttime_sec="1138026363" seqnr="189574" senttime_nsec="167321000"/>
Whereas the raw data consists of a set of information for each single packet, the aggregated data format provides a whole group as one
datum element. The information derived from sanitisation and aggregation for each group are returned:
'time': sent time of earliest sent, received packet of group,
'max_owd': maximum OWD of all received packets of group,
'med_owd': median OWD of all received packets of group,
'min_owd': minimum OWD of all received packets of group,
'max_ipdv': maximum OWDV of all received packets of group,
'med_ipdv': median OWDV of all received packets of group,
'min_ipdv': minimum OWDV of all received packets of group,
'lost_packets': number of lost packets,
'duplicate_packets': number of duplicate packets,
'reordered_packets': number of reordered packets.
It is possible to implement other data formats, if the need arises. The raw data format must be pre-processed for further use in any
case, but the aggregated format may not be suitable for all purposes. As there are currently no other applications using the Hades data,
these two formats are sufficient.
Errors
In some cases, the system is not able to deliver the desired data. In this case, an error message is returned with the XML response. If
the requested data does not exist (because there were no measurements made with the specified parameters for the given measurement path
and the selected time span), the message "No data found!" is returned. If more than one measurement is available, the message "More than
one measurement available" and a list of all possible measurements, with measurement IDs and the corresponding parameter sets, is
returned. If requesting a measurement that does not cover the whole requested time span, the error message "Incomplete data" is returned. Internal errors are not delivered unless they influence the content of the response. For example, a read timeout in the data should be
reported to the requesting client.
Implementation status
A Hades prototype is currently running in test mode, providing the data from measurements taken in the GÉANT2 network at this time. The
service is implemented as daemon process, handling the communication with clients. The Perl implementation is a modular design, using
CPAN modules [CPAN] and own modules. Additionally, a client application is provided, which is compliant with all perfSONAR services. It
is able to send XML request files to the service and displays the XML result.
Modules
The following modules from CPAN were used:
Log::Dispatch: Flexible log dispatcher to make logging easy for daemons.
Date::Manip: Powerful time and date handling, used for internal time handling.
Data::Dumper: Used for debugging purposes.
XML::LibXML: Perl module to use functionality of libxml, the open source library most often used for handling XML.
XML::Writer: Simple Perl module for manipulating XML.
SOAP::Lite: The Perl module most often used for SOAP. Very flexible and powerful. This module needed to be patched, to be useful for the
perfSONAR implementation. SOAP::Lite only has limited support for document/literal SOAP interaction. Therefore, a patch was created and
added.
The data handling was done by Hades-specific modules.
Hades::Data: A basic, abstract class from which all other data type classes are derived. It also contains code for managing all derived
data classes.
Hades::Data::IPPM, Hades::Data::IPPM_Raw, Hades::Data::IPPM_Sanitized, and Hades::Data::IPPM_Aggregated: Derived data classes, describing the data formats (see above).
Hades::Data::Finder: Interface to the measurement archive itself.
NMWG and NMWG::Message: These modules provide basic functionalities for handling NMWG/perfSONAR XML messages. NMWG::Message provides a
service and message type independent representation of an NMWG/perfSONAR message.
MS.pm: This module provides the functions doing the service-specific work using NMWG::Message. They are called via the SOAP::Lite server
deamon, called ms.pl.
Next steps
At the moment, it is possible to request the whole Measurement archive of Hades measurements residing on gandalf.rrze.uni-erlangen.de (at
port 8090). Provided data formats are raw and aggregated (see above). Moreover, the data itself can be viewed on http://www.win-labor.dfn.de, visualised as plots per connection and data set. Also, traceroute
information for the measurement paths are displayed. The deployment status of measurement boxes can be seen at
http://www.perfsonar.net/jra1-wiki/index.php/Equipment_installation.
References
www.win-labor.dfn.de
Contact
* Dipl.-Inf. Roland Karch * Dipl.-Phys. Ralf Kleineisel * Dipl.-Inf. Birgit König * Dr.-Ing. Stephan Kraft * Dipl.-Inf. Jochen Reinwand * Dipl.-Inf. Verena Venus
WiN Labor Regionales Rechenzentrum Erlangen Martensstraße 1 91058 Erlangen
Phone: +49 9131 85-28800 Fax: +49 9131 302941 E-Mail: win-labor@dfn.de
Work in progress
The next step is to implement key requests, thereby enabling simpler requests using a key, rather than a whole parameter set. Moreover, the storing format of the Hades data will change, from text files to a fast database backend for the MA. This has the advantage
of providing a fast and efficient search mechanism and a much more efficient data storing mechanism than storing data in files. In addition, an authentication mechanism, lookup service registration, on-demand capability, and resource protection will be implemented.
