JRA1 D1.4 Data Extraction
From GEANT2-JRA1 Wiki
Contents |
Data extraction techniques
As part of D1.4 information about data extraction techniques to retrieve the data from measurement points is collected.
SNMP
SNMP can be used to retrieve data collected by the network equipment, i.e. routers and switches. Metrics for the links that can be retrieved in this way include the status, link bandwidth, and average link utilisation/throughput which is derived by the subtraction of packet counters. Status, counters for forwarded packets, not forwarded packets, and received packets as well as CPU load are relevant metrics for the network nodes. Some variables are part of standard MIBs e.g. the status, while others like the CPU load are non standard variables. In the latter case it needs to be considered whether the variable is available or not.
SNMP uses object identifiers (OIDs) which identifiy a specific type of information. The OIDs have two parts: one related to the type of information to be retrieved and a second one related to the location from where this information is retrieved.
examples:
.1.3.6.1.2.1.1.3 = .iso.org.dod.internet.directory.mgmt.mib-2.system.sysUpTime
consisting of .1.3.6.1.2.1 = .iso.org.dod.internet.directory.mgmt.mib-2: Standard Internet II MIB and .1.3 = system.sysUpTime: relative address of system uptime.
.1.3.6.1.4.1.9.9.23.1.1.1.1:
consisting of .1.3.6.1.4.1.9 = .iso.org.dod.internet.private.enterprises.cisco: CISCO MIB and .9.23.1.1.1.1 = relative address of CISCO CDP Table
SNMP is unreliable because it sits ontop of UDP. So if a message is lost, it must be retransmitted by the sender (e.g. using timeout mechanism waiting for the reply).
A protection mechanisms is needed for the network equipment to block direct information retrieval. Scripts, e.g. written in Perl, can be used to retrieve the data and store the data in regular time intervals. In huge networks and tight time intervals the data retrieval has to be performed in a parallel manner. This includes to allow for out-of-order answers and time-outs.
XML
For Juniper routers there is a possibility to retrieve router data via XML. A graphical interface called JUNOScope exists to fetch and show/configure the data. The data contains the status of the interface cards, outstanding alarms, statistics for the interface cards, routing table information (MPLS, BGP, OSPF). A guide for using JUNOScript to retrieve the information is also available.
The Netopeer project at Cesnet has developed infrastructure to access different routers (Cisco, Juniper, etc.) using a common XML interface. There are several backends translating XML <--> router interface and several frondends translating XML <--> user interface.
IPPM
The DFN-IPPM measurement boxes deployed within a network perform tests of one-way delay, jitter, and packet loss.
Hardware Requirements
To permit this measurements it is necessary to have at least two measurement devices whose time is exactly synchronized. For the IPPM measurements this is achieved by connecting two PCs in different locations to one GPS or PZF receiver (using the DCF77 signal) each. Other similar receivers can be used, too. An NTP server on the machines adjusts its time according to the external time signals.
Architecture
The simplest IPPM system could be built with two measurement boxes where each of those is running the analysis of raw measurement data itself. A third computer is used to analyse the data provided by the two measurement boxes. The measurement boxes store data in a configurable directory from where the analyser collects them.
While the measurement boxes just store send and receive times in files, the analyser calculates the delays and checks whether packets were lost on the way from sender to receiver. That information is stored in binary format on the analyser machine where it can be accessed by a visualisation script.
Format of measurement data
Once the measurement boxes are running the senders send the configured amount of measurement packets (packetgroupsize) regularly to the receivers.
The receivers receive those packets and store the measurement data sent by the sender plus the time of receiving into files in a previously configured directory.
The header consists of several comment lines that contain the following information:
qos_mi_instance data file: /data/2004/11/10/Wuerzburg_DFN.Marburg_Uni.429.dat
data file number : 49
qos_mi_sender address : 188.1.36.198 (Wuerzburg_DFN)
qos_mi_receiver address : 137.248.1.20:50459 (Marburg_Uni)
packet size : 429
precedence : 0x0 (routine)
interval length : 30
packet group size : 5
date & time : Wed Nov 10 00:00:00 2004
table columns:
sequence_number:sent_sec:sent_usec:received_sec:received_usec:packet_size
700631:1100041212:508473:1100041212:519469:429
700632:1100041212:528470:1100041212:539448:429
700633:1100041212:548472:1100041212:559450:429
700634:1100041212:568474:1100041212:579449:429
700635:1100041212:588476:1100041212:599445:429
Description:
• qos_mi_instance data file
Complete path and filename of the current data file
• data file number
Sequence number of the current data file. The sequence number is incremented each time a new file is created by the currently running process. That normally happens once a day.
• qos_mi_sender address
IP address and name of sender
• qos_mi_receiver address
IP address and name of receiver
• packet size
Reflecting the size of a measurement packet.
• precedence
This line contains the value set in the ToS field in the measurement packets.
• interval length
Describes the time in seconds the sender had to wait between sending one measurement packet group and the next one.
• packet group size
Number of packets the sender sends in each packet group. The time between these sending events is described by the interval length.
• date & time
Creation time of data file.
Below these header lines and a short comment describing the data format the actual data follow. Each line contains information about one measurement packet. Different fields are separated by colons.
The first field contains a sequence number which allows the detection of packet loss. In the next two fields the sending time seconds and microsecond are saved respectively. Field four and five contain the corresponding data for the receiving side. The sixth and last field contains the size of the measurement packet.
Analyser
The measurement data files that are created on the measurement boxes are copied to a dedicated analyser host. This is done by calling the /proj/gwin/ipqos/bin/data2www.pl script via a cron job. First of all this script copies the data files of all configured measurement boxes to /data/ipqos/data/YYYY/MM/DD. In addition for each data file it calls /proj/gwin/ipqos/bin/analyzer.pl.
The analyzer.pl script calculates delays and checks whether packets were lost on the way from sender to receiver. The resulting information is stored in /data/ipqos/www which is exported via NFS and can be mounted by a visualisation engine that can possibly run on another host.
OWAMP
There are two tools in owamp that can be used to request measurements.
1) owping is a command-line tool that will generate a binary file or output ascii data in both human readable and machine friendly formats. The binary data files can be parsed using owstats.
2) powstream is a client-side daemon program that can be used to create a perpetual stream of owamp test packets toward the client (i.e. the client is the receiver). It generates sub-session data files including summary statistics. The summary statistics are ascii formated, the data files can be parsed using owstats.
The owampd process also buffers session data for one-way delay streams that are received at the server. The OWAMP protocol itself can be used to fetch the data results from these files from 3rd parties given the session identifier and liberal data retention policy settings in owampd. The owfetch command-line application can be used to fetch the data - it can also generate the same ascii statistical reports on the data that owstats and owping can do.
To integrate this into a database the -R (raw) output flag is probably the most appropriate. Internet2 uses custom perl scripts to collect data from measurement hosts to a central database host.
RIPE TTM
RIPE TTM can provide raw data in two ways
1) Request the RIPE NCC for past data along with the parameters and they will provide you the data
2) Copy files directly from the RIPE TTM box. Data is available on the box for last 14 days.
The one of interest to us is option 2. For each day in the last 14 days,files are present on the measurement box and they contain the measurement data for that day. The files can be identified depending on the name (ddmmyy format) and they also carry a timestamp tag at the end. The filenames also specify what kind of data is contained in them. These can be of four types
- Information about sent packets (filenames start with SNDP)
- Information about recieved packets (filenames start with RCDP)
- Information about GPS sychronization (filenames start with GENE)
- Information about Route Changes - Routing Vector (filenames start with RVEC)
RCDP files contain one way delay measurements. There can be one or more of these files per day. If there is one file, the file will contain the RCDP prefix along with the date and also a timestamp equal to 0000. If more files are present for a particular day then the timestamp will differ and will be equal to the starting time value that the file contains.
Measurement information is contained in each line. The format is proprietary and is described in http://www.ripe.net/projects/ttm/General/access.html
Files can only be copied by 'SCP' (Secure CoPy)ing them. Permission (in the form of an a/c on the box) has to be got from the RIPE NCC explicitly in order to gain access to these files.
Multicast Beacon
Perl scripts to measure packet delay, jitter, loss.
It stores the aggregated data in several text files that can be accessible through the web server (e.g. Apache). The history file contains date, IP, loss, RTT and Jitter but the data is removed after 60 seconds. The history file format is CSV (comma separated values).
MAPI
MAPI (Monitoring API) is a programming interface used by passive monitoring applications in SCAMPI platform. It is build around a concept of a flow.
A flow in the SCAMPI architecture means all packets on a monitored line on which a sequence of monitoring functions is applied. Monitoring functions applied to a flow typically select only some packets (using header filtering, sampling or payload searching) and compute various statistics on them (number of packets, average delay between succeeding packets, etc.).
MAPI is a set of functions in C language that allow monitoring applications to open as many flows as they wish and apply an arbitrary sequence of monitoring functions on each flow. A different sequence of functions can be applied to different flows. Thus a monitoring application can use one flow to count the number of packets exchanged between two subnets, it can use another flow to detect string "virus" in traffic produced by another subnet, etc.
To illustrate the environment that MAPI provides to applications, the following code fragment counts the number of packet sent from network 192.168.1.0/24 that include "virus" in their payload:
fd = mapi_create_flow("/dev/scampi/0");
fid1 = mapi_apply_function(fd, "BPF_FILTER", "src net 192.168.1.0/24");
fid2 = mapi_apply_function(fd, "STR_SEARCH", "virus");
fid3 = mapi_apply_function(fd, "PKT_COUNTER");
mapi_connect(fd);
while(1) {
sleep(5);
cnt = mapi_get_result(fd, fid3, MAPI_REF);
printf("Packets with virus: %d\n", *cnt);
}
The mapi_get_result() function was used to retrieve the measured metric.
There are several possibilities how JRA1 can interact with a passive monitoring application written on top of MAPI. The two most straightfoward options are that the monitoring executor of the monitoring point service talks a) using MAPI interface, b) using user interface of application written on top of MAPI. We will discuss it in more detail in a separate document "Integration of passive monitoring into JRA1 architecture".
Flow-based tools
Todo (GARR): provide some information about data retrieval from flow-based measurements.
CoralReef
Passive monitoring tool for flow analysis.
Metrics measured
The tool is capable of capturing and analyzing a lot of packet types, like ATM cells, IP packets and so on. It stores information about the packet count, the size of the packets and the header information. Additionally it can detect and analyze network flows. This is only possible for IPv4. Here an overview of supported protocols: ATM, Packet Over SONET, Ethernet (including VLAN), HDLC, PPP (PPPoHDLC, PPPoED, PPPoES), IPv4, IPv6, ARP, IPX, AppleTalk, ICMP(6), SNMP, DNS.
Database
CoralReef uses text or binary files to store its data. It also works on these files for aggregation and converts them to different output formats. As input traces from a few other tools are usable.
Measurement results storage method
The command line tools work with tables. They can be created from the different sources: directly from the monitoring device or from different types of trace files. They can also be written to different formats.
Data aggregation method
The command line tools can be connected with pipes to allow different aggregation types. Aggregation is done by working on the tables mentioned above. For example it is possible to monitor network flows with crl_flow and then use tools like t2_convert and t2_top to get a list of the five machines in the net creating the most traffic:
crl_flow -I -b -Ci=10 if:fxp0 |
t2_convert src_IP_Table |
t2_top -Sb -n5
Data concatenation
Most CoralReef tools are capturing packets continuously and give summaries at a specified interval. Other tools then aggregate these summaries further. This aggregation is possible with data from different machines.
flowscan
Passive flow monitoring tool. FlowScan is a network analysis and reporting tool. It processes IP flows recorded in cflowd-format raw flow files and reports on what it finds. It can be configured to work with flow-tools as well. FlowScan depends on a netflow collector subsystem (cflowd or flow-tools) to provide data collected from network devices, it is only data analyzing tool. When graphical representation of a data is needed, rrd-like tool RRGrapher is provided.
Author Dave Plonka, home page of a tool http://net.doit.wisc.edu/~plonka/FlowScan/.
FlowScan is licenced under GNU General Public license (Copyright (c) 2000-2001 Dave Plonka <plonka@doit.wisc.edu>. All rights reserved.)
BWCTL
bwctl is a command-line tool that will output iperf results either to stdout or to a file.
The tool itself does not provide any mechanism for distributing the data. Internet2 uses perl scripts to collect the data from measurement hosts to a central database host.
SSH
SSH can be used for secure communication in the data retrieval.
