SQL MA Stitching Guide

From GEANT2-JRA1 Wiki

This is the guide to stitch (configure) your SQL MA to provide metadata information about one or both of the following

  • Network interfaces and interface utilisation data
  • Network links and status of these links (a.k.a Lightpath Monitoring or e2eMon)

Contents

[edit]

Stitching for perfSONAR Bundle 2.x

[edit]

Path Status Metric

Objective

  • To create a metadata configuration file for your network which will expose information about the status of network links/circuits which are usually part of an end-to-end link

Requirements before using this guide

  1. Sample file for you to follow - available here
  2. The metadata configuration file guide - available here
  3. The suggested (default) database schema is available for MySQL database here and for postgresql here. The database schema is explained in detailed here and here
    1. Look for the sections on l2 status
    2. It is possible to use a different database schema than the one provided. Please contact the support team if you are interested in doing this.
  4. There are two types of metadata - link metadata and node metadata. Link metadata provides meta information about a link while a node metadata provides meta information about a node.
  5. A link metadata contains references to two nodes. In a configuraiton file, there can only be one metadata per link or per node. However, two link metadata can refer to the same nodes.
  6. For each link that you want to make available via the MA, you will need the following information
    1. Two Nodes/end points that are present at the end points of this link. For each node, you will need to know
      1. A node identifier unique across this configuration file - assigned and maintained by yourself
      2. The logical name assigned to the node (any naming convention is ok)
      3. The location information (country, city, lat, long) - leave blank or provide dummy values if you don't want to provide any
      4. The name of the institution responsible/associated to this node
    2. Link related information such as
      1. Local name of the link
      2. Assigned Global name for the link
      3. Type of link (Domain_link or ID_link)
      4. References to the two nodes associated with this link and the role of each node (EndPoint or DemarcPoint)
    3. Other information such as
      1. The Domain name(network domain or organisation)
      2. A Static 'parameter' indicating that path.status network metric is supported. See example for illustration
[edit]

Step 1 - Understanding the configuration file structure

  • Take a look at the metadata configuration file guide and the sample file. Try to understand the general logic of the file
    • Remember that each metadata-data chain represents one link
    • The metadata in this metadata chain has references to two other metadata elements which describe the node which are part of the link
    • The data in this metadata chain contains 'key' information related to how the data can be accessed from the database. Take a look at the example file which illustrates this
    • You will have to create as many metadata-data chains as the number of links that you want to expose
    • You can have as many chains as you want in your metadata configuration file. There are currently no limits
    • The global name assigned to links should be agreed/assigned by other parties invovled in the link/project
[edit]

Step 2 - Creating your own metadata configuration file

  • Either manually or with the help of your own programs or scripts, you will need to generate the metadata configuration file.
  • The metadata for the link and the key in the linked data contains a name (not the global name) of the link which is local to your domain. The database will be expected to store status of these links.
    • You can make use of programs available to help you in automating this process.
  • Remember that whenever something changes in your network, the metadata configuration file will need to be updated. It is hence worth automating the process.
[edit]

Step 3 - Applying your metadata configuration file to your installation

  1. Generate and Verify the xml configuration file
  2. Open the exist xml database web interface
    1. If you installed exist database on your tomcat as a web application you should be able to find the web interface by simply typing http://server_name:port/exist (for example: mu.dante.org.uk:8080/exist) where server_name is the dns/ip address of the server where you installed tomcat, exist and the service and port is the port on which tomcat is running. Scroll down the page and on the left menu bar, you can find the Admin button. Clicking on this button will give you a login screen. Use the username and password that you provided during the installation (NOT THE ADMIN USERNAME AND PASSWORD) and login to the database. Remember to log out after you are done. If not, exist won't be happy and will create problems when you try to log in again.
    2. If you installed exist database as a standalone application, you can use the exist xml database client application (from any machine. For example, your desktop) to access this database. You will need to provide your server_name, port, username and password
  3. Once you have logged into the exist database, you should be able to locate the collection for sqlmaconfig. Double click on this collection.
  4. If you find previous xml files in the collection, remove them (maybe you want to back it up somewhere?) and add your new file with the help of the menus available on the screen.
  5. Confirm that your new xml file has been loaded and then log out
  6. This completes this step on applying your metadata configuration
  7. Other options
    1. With the help of the installer
      1. Copy your metadata configuration file to the $perfSONAR-bundle/perfSONAR-SQL-MA-1.0/perfsonar/conf directory
      2. Start the perfSONAR bundle installer and choose the RRD MA service installation that you wish to modify
      3. Go through the 'configure' step
      4. During the configure step, you will be asked to specify the complete path of the metadata configuration file.
      5. Instead of choosing the default value for the metadata configuration, provide the complete path of your configuration file
      6. Complete the configure step and then follow the 'deploy' step via the 'modify your service' menu
      7. In the deploy step, make sure you ask the installer to update the xml database.
      8. Confirm that the xml file has been uploaded into the exist xml database by accessing its contents from the exist xml UI. See notes in the previous option (With the help of the exist xml database web based UI) described above.
    2. With the help of the service installation scripts (ant targets)
      1. Copy your metadata configuration file to the $perfSONAR-bundle/perfSONAR-SQL-MA-1.0/perfsonar/conf directory
      2. In the directory in $perfSONAR-bundle/perfSONAR-SQL-MA-1.0/perfsonar/conf, look for the service.properties file
      3. Open the file for editing and look for the property 'service.ma.conf_file'
      4. Modify its value - provide the complete path to your metadata configuration file (instead of the default configuration file)
      5. In the ant directory ($perfSONAR-bundle/perfSONAR-SQL-MA-1.0/perfsonar/ant), execute the following command
        1. ant xmldb-init-sqlma
      6. Confirm that the xml file has been uploaded into the xml database by following the first option above (With the help of the exist xml database web based UI). Instead of viewing the files in the xml database collection, you can simply view the files to see if your file has been added to the database
[edit]

Link Utilisation metric

Objective

  • To create a metadata configuration file for your network which will expose information about all network interfacees data (stored in the config file) that you wish to make available to the measurement archive for export

Requirements before using this guide

  1. Sample file for you to follow - available here
  2. The metadata configuration file guide - available here
  3. You will need to modify the table(s) in the database that you wish to export utilisation data from.
    1. Add new collumn to each table. This collumn should have a name 'metadataId' (VARCHAR)
    2. The suggested (default) database schema is available for MySQL database here and for postgresql here
  4. For each network interface data that you want to expose,
    1. For the configuration you will need
      1. The database information (hostname, tables & collumn names), and the usernames and passwords for accessing these entries.
      2. The list of interfaces for which utilisation data is stored in each table. For each direction of the interface (in/out), you need to know
      3. The unit in which data is stored (bps or Bps)
      4. Name of the interface for which data is stored (Example: so-1/0/0 which could be an IP interface on a router)
      5. Description of the interface - usually configured on the ifConfig (ex: Link to GÉANT2)
      6. IPv4 address of the interface
      7. The traffic direction (in/out) being monitoring on each interface and being stored in each data source
      8. Capacity/Interface speed (units = bits per second or the units which are using for storing data in the rrd file)
      9. DNS entry of the Router/network equipment which contains this interface
      10. Authentication Realm or authRealm - Usually the domain name or NREN name for this router (ex: GÉANT2, RENATER)
    2. In the database tables storing measurement data, you will need
      1. One row of data in the table pointing to a specific direction (in/out) of that interface
      2. The 'metadataId' collumn value for each row pointing to the related parameter in the configuration file (explained later as well as in the document)
[edit]

Step 1 - Understanding the configuraiton file structure

  • Take a look at the metadata configuration file guide and the sample file. Try to understand the general logic of the file
    • Remember that each metadata-data chain represents one counter (in or out) value associated with an interface
    • You will have to create as many metadata-data chains as the number of counters (one per direction) that you want to expose
    • You can have as many chains as you want in your metadata configuration file. There are currently no limits
    • Each metadata in the metadata-data chain will have a metadata Identifier (part of the metadata element)
[edit]

Step 2 - Creating your own metadata configuration file

  • Either manually or with the help of your own programs or scripts, you will need to generate the metadata configuration file.
    • In the database, whenever you add a new row to the table storing utilisation data, along with the information about the utilisation value and time, you will also need to add the metadataId field pointing to the appropriate metadata identifier available in the configuration file
  • Remember that whenever something changes in your network, the metadata configuration file will need to be updated. It is hence worth automating the process which will periodically contact the network elements/routers and automatically generate the file and the database.
  • Manually creating these files for a large number of interfaces is not recommended.
[edit]

Step 3 - Applying your metadata configuration file to your installation

  1. Generate and Verify the xml configuration file
  2. Open the exist xml database web interface
    1. If you installed exist database on your tomcat as a web application you should be able to find the web interface by simply typing http://server_name:port/exist (for example: mu.dante.org.uk:8080/exist) where server_name is the dns/ip address of the server where you installed tomcat, exist and the service and port is the port on which tomcat is running. Scroll down the page and on the left menu bar, you can find the Admin button. Clicking on this button will give you a login screen. Use the username and password that you provided during the installation (NOT THE ADMIN USERNAME AND PASSWORD) and login to the database. Remember to log out after you are done. If not, exist won't be happy and will create problems when you try to log in again.
    2. If you installed exist database as a standalone application, you can use the exist xml database client application (from any machine. For example, your desktop) to access this database. You will need to provide your server_name, port, username and password
  3. Once you have logged into the exist database, you should be able to locate the collection for sqlmaconfig. Double click on this collection.
  4. If you find previous xml files in the collection, remove them (maybe you want to back it up somewhere?) and add your new file with the help of the menus available on the screen.
  5. Confirm that your new xml file has been loaded and then log out
  6. This completes this step on applying your metadata configuration
  7. Other options
    1. With the help of the installer
      1. Copy your metadata configuration file to the $perfSONAR-bundle/perfSONAR-SQL-MA-1.0/perfsonar/conf directory
      2. Start the perfSONAR bundle installer and choose the RRD MA service installation that you wish to modify
      3. Go through the 'configure' step
      4. During the configure step, you will be asked to specify the complete path of the metadata configuration file.
      5. Instead of choosing the default value for the metadata configuration, provide the complete path of your configuration file
      6. Complete the configure step and then follow the 'deploy' step via the 'modify your service' menu
      7. In the deploy step, make sure you ask the installer to update the xml database.
      8. Confirm that the xml file has been uploaded into the exist xml database by accessing its contents from the exist xml UI. See notes in the previous option (With the help of the exist xml database web based UI) described above.
    2. With the help of the service installation scripts (ant targets)
      1. Copy your metadata configuration file to the $perfSONAR-bundle/perfSONAR-SQL-MA-1.0/perfsonar/conf directory
      2. In the directory in $perfSONAR-bundle/perfSONAR-SQL-MA-1.0/perfsonar/conf, look for the service.properties file
      3. Open the file for editing and look for the property 'service.ma.conf_file'
      4. Modify its value - provide the complete path to your metadata configuration file (instead of the default configuration file)
      5. In the ant directory ($perfSONAR-bundle/perfSONAR-SQL-MA-1.0/perfsonar/ant), execute the following command
        1. ant xmldb-init-sqlma
      6. Confirm that the xml file has been uploaded into the xml database by following the first option above (With the help of the exist xml database web based UI). Instead of viewing the files in the xml database collection, you can simply view the files to see if your file has been added to the database


Retrieved from "http://wiki.perfsonar.net/jra1-wiki/index.php/SQL_MA_Stitching_Guide"
Personal tools