From GEANT2-JRA1 Wiki

Contents 1 Servers 1.1 Monitoring stations in CESNET network 1.2 Monitoring stations for GN2 - CESNET link 1.3 Operating system 1.4 DAG cards 2 Discussion 2.1 Optical splitters 2.2 Router/switch port setup to mirror packets 2.3 Remote access

Servers

Monitoring stations in CESNET network

We have very good experience with SuperMicro servers. So far their mainboards worked flawlessly with any monitoring cards we tried and their cases are well cooled and well mechanically built.

As of January 2007, if you want to use DAG cards, we recommend the [X7DB8 mainboard] with two dual-core Xeon Woodcrest CPUs and a 4U case. You can buy the whole server, such as [7045B-8R] (which has a slightly different and more expensive mainboard) or you can request the server custom-built from components. We use such servers for internal monitoring in CESNET network.

Monitoring stations for GN2 - CESNET link

As part of JRA1 two passive monitoring stations were deployed in CESNET and two more in ARNES. We do not have good experience with these servers and we do not recommend buying more servers of this type (applies to the server itself, not to the monitoring cards, which are good). The following is just to give account of what has been installed.

• Two 5U-high servers (we requested 4U to save rack space, but they were not available) with the Intel SE7520AF2 mainboard from the Bee company. The first server includes a DAG6.2 card (10 Gb/s) and the second server includes a DAG4.3 card (1 Gb/s) with coprocessor. Each server is used for monitoring of one direction of the GN2-CESNET link.

• The mainbord was mounted in the case such that holes for cards in the back of the case did not align with PCI connectors and there was no slack that would allow to shift the mainboard slightly. I carefully bent the metal plate on the DAG cards so that they could be inserted without bending their boards. The mainboard takes very long to boot up even though we tried various BIOS options to speed it up.

• The DAG cards ran at 66 MHz instead of 133 MHz, which limited throughput. After long investigations and discussions with Bee and Endace, it was located as compatibility problem between DAG cards and the particular mainboard. Endace modified firmware for us and the cards now run at 133 MHz.

Operating system

• We installed SuSe 10.0 linux from DVD-ROM. It booted flawlessly, whereas the lastest Fedora CD-ROM downloaded at that time did not recognize SCSI adapter in the server. We selected 32-bit (instead of default 64-bit) installation on the initial installation screen of SuSe 10.0, because compilation of some packages was unsucessful when we first tried 64-bit installation (I later installed a 64-bit SuSe 10.1 and 10.2 version on another server without problems).

• We started yast utility, selected "online update" (for security updates), set "installation source" to ftp://suse.inode.at/opensuse/distribution/SL-10.0-OSS/inst-source and selected "Software management" for update of RPM packages to current versions.

• Alternative installation souces can be found at [1], for example ftp://ftp.gwdg.de/pub/opensuse/distribution/SL-10.0-OSS/inst-source

• We installed new kernel 2.6.14.2 from sources

• We installed gcc-3.4.4 from sources to /usr/local/gcc-3.4.4, because compilation of some packages failed with gcc 4.0.2 distributed by default

DAG cards

• We installed DAG 2.5.5r2 software downloaded from www.endace.com. To load the necessary drivers and to initialize the card after PC reboot, we use dag.sh script. The script requires DAG 2.5.5.r2 software (or hopefully also later versions), but works with both 2.4 and 2.6 kernels. DAG4.3 or DAG 6.2card is selected by argument.

• DAG cards are connected to mirroring ports on Cisco 7600 router. Since 10GE DAG card does not have a transmitter, the mirroring port must be faked to stay up even when its receiver is not connected. One way to do it is to short-connect its transmitter to receiver and branch off the signal by a splitter to DAG card (another possible way could be to reprogram the transceiver to stay up, in theory should be possible with Xenpak, we are investigating this option)

Discussion

Optical splitters vs. mirroring ports. Minus signs should be taken as considerations rather than purely disadvantages.

Optical splitters

   + do not require spare router/switch port
   + it is certain that we get the monitored packets with the original
     traffic dynamics
   - it requires to disconnect the line for installation (and possibly
     causes flops in routing), network operation people certainly
     do not like this
   - it requires two monitoring adapters or one monitoring adapter
     with two ports (on 10 Gbps only 1-port monitoring adapters currently
     exist)
   - we get packets exactly as they are on the line, that is if they include
     MPLS headers, we must consider them when filtering/classifying packets
   - to monitor a DWDM line, we need to separate selected wavelength first
     by an optical filter
   - a splitter must be designed for the waveband of the monitored line,
     there are broadband (1310nm-1550nm) splitters available, but 10BASE-SR
     (850nm) reguires different splitters
   - some network devices used for monitoring (such as regular NICs) may 
     have problems when we only connect signal to their receiver (from the
     splitter) and leave their transmitter open (because there is no point
     to connect it)

Router/switch port setup to mirror packets

   + we do not need to disconnect the line
   + we can change the link that we monitor remotely by software 
     configuration only
   + we can choose to mirror traffic from the port where MPLS headers
     are already stripped
   + we can stop/start mirroring remotely and read the exact number of
     mirrored packets from the router/switch counters, which is convenient
     for debugging of monitoring devices/applications
   - we need a spare router/switch port with mirroring capability
     and a transceiver for this port (all middle- and high-end Cisco
     switched and routers do, but optical transport platforms such
     as Cisco 15454 do not)
   - routers/switches may not support mirroring from certain port
     types, such as GE WAN or PoS (due to limitations in router
     capabilities)
   - routers/switches support a very limited number of mirroring
     sessions, usually only one or two and you need two sessions
     to monitor both directions of one link, which does not leave
     any spare monitoring session for operational use (debugging, etc.)
   - if a monitoring device has only a receiver and not a transmitter
     (such as some DAG cards), the mirroring port on a router/switch
     may have difficulty to keep its transmitter up when its receiver
     is not connected anywhere (in which case we can use a splitter
     to fork off signal from its own transmitter to light up its receiver)

Remote access

Passive monitoring cards are expensive. One approach to limit their number could be to take advantage of a DWDM network. We should be able to capture traffic by an optical splitter in one place and send it using two additional lighpaths (one for each direction of the monitored line) through a DWDM system to a remote monitoring station. In this way a pair of monitoring stations could be sufficient to monitor traffic in any selected point. However, the cost of extra wavelength (transponders and transceivers) is currently even higher than the cost of passive monitoring cards. Moreover, this approach would add extra delay and possibly even packet loss to the monitored traffic, which would distort measurements. Also we could only monitor traffic in one place at a time, rather than obtaining a comprehensive view on what happens in our network.

Return to Passive monitoring