Java SQL MA Service Installation Guide

From GEANT2-JRA1 Wiki

Contents

[edit]

Installation

[edit]

Pre-requisities

Use these instructions to download and install additional software.
Hypothesis, the installation guide is to install the service over a datase with a predefined schema (see step 8). Should you use anoter database schema, then you need to modify the a configuration file (contact Roman Laplacz on how to use it).

[edit]

Step 1 - Java Development Kit (JDK) 2 Platform

For a successful install we recommend JDK version 1.5, which has been tested. Current code will not compile on version 1.4. Download and extract Java 2 Platform Standard Edition from http://java.sun.com/j2se/1.5.0/download.jsp.


[edit]

Step 2 - Apache Ant

Apache Ant is a Java-based build tool compulsory to proceed with other steps that are using Ant scripts. Download Ant (ver. 1.6.x) from http://ant.apache.org/bindownload.cgi and install it.


[edit]

Step 3 - eXist XML database

[edit]
Step 3.1 - Getting the application

eXist database needs to be installed. Download eXist (ver. 1.0rc) from http://exist-db.org/index.html#download.

[edit]
Step 3.2 - Installing eXist

To run graphical installer of eXist enter the following command and select the path for installation when asked: 

java -jar eXist-{version}.jar

To run command line installation of eXist enter the following command and specify installation path as a parameter (you should replace $EXIST_HOME with your own selected path): 

java -jar eXist-{version}.jar -p $EXIST_HOME

The rest of this document assumes that eXist directory name is under $EXIST_HOME e.g /usr/local/exist.

[edit]

Acquiring service code

[edit]

Step 4 - Getting service sources

In order to get sources one can use SVN or prepared snapshot file.

[edit]
Step 4.1 - SVN

The latest version of MA is available from perfSONAR SVN:

You can download Measurement Archive code to any directory in your server. By default, SVN will create and download files into ./perfsonar. The rest of this document assumes that the perfSONAR full directory name is under $PERFSONAR e.g./usr/local/jra1-project/sqlma/perfsonar.

[edit]
Step 4.2 - Snapshots

A snapshot tar file can be downloaded from http://monstera.man.poznan.pl/jra1-wiki/index.php/Java_SQLMAService

[edit]

Step 5 - Getting Tomcat and Axis

While using SVN you also need Tomcat (ver. 5.5.17) and Axis (ver. 1.4) downloaded separately. To download and install them use Apache homepage or run the following command in $PERFSONAR/ant directory: 

ant -f build-sqlma.xml webservice-container-download

If you have another service (e.g. MA) running on the same machine you will have to run another instance of Tomcat using another port. Follow appropriate instructions.

[edit]

Step 6 - Download required java libraries

  • In order to download all required libraries (jars) to compile and run the service execute the following command in $PERFSONAR/ant directory: 
ant -f build-sqlma.xml libs-sqlma

Warning: If downloading the libraries failed run the script again (the connection might get broken).


Add the path $PERFSONAR/lib to the env variable $CLASSPATH 

export CLASSPATH=.:$PERFSONAR/lib
[edit]

Relational database startup

[edit]

Step 7 - Installing relational database

Install and run SQL database (mySQL (v.5) or postreSQL) or use already available one. You may want to use some installation tips of perfSONAR developers for mySQL and postgreSQL.

[edit]

Step 8 - Database configuration

In order to setup the SQL database you have to download and run or follow one of the SQL scripts. They will create appropriate user account, tables and fill it with some test data. 

mysql -u root -p < mysql-sqlma-dbsetup.sql

psql -f psql-sqlma-dbsetup.conf

Scripts configure the database to store utilization and L2 link status data.

To remove the configuration use $PERFSONAR/conf/mysql-sqlma-dbremove.sql or $PERFSONAR/conf/psql-sqlma-dbremove.sql.


Warning: If postgreSQL database is used for L2 path status data then SQL statements prepared for this database must be uncommented in $PERFSONAR/conf/ibatis-L2-status.xml file (default SQL statements are valid for mySQL; this is beacuse the difference between time conversion functions of both databases).

FAQ:
- What if L2 path status measurement data are stored in already existing database table with different name and different columns? See answer.

[edit]

eXist startup

You can skip steps 8-10 of eXist startup provided you have already installed it for another service e.g. Measurement Archive. The only addition is that you will have to configure create additional user name and password used fro SQL MA.

[edit]

Step 9 - Changing default port

To change the eXist default server port, open $EXIST_HOME/bin/startup.sh file and find OPTIONS line which should look like: 

 OPTIONS="-Dexist.home=$EXIST_HOME"

Then alter that line adding new port number at the end: 

 OPTIONS="-Dexist.home=$EXIST_HOME -Djetty.port=8680"

8680 is an example of new port number (default is 8080). This 8680 port number will be used in next examples. You must change it into your own one where necessary.

[edit]

Step 10 - Running an eXist server

After the installation and configuration has completed, you should be ready to launch eXist. To run eXist server enter the following commands: 

cd $EXIST_HOME/bin
./startup.sh

It starts the included Jetty webserver at previously configured port i.e. eXist and Cocoon are installed as a web application, located at http://localhost:8680/exist/

[edit]

Step 11 - Running the eXist client tool

eXist client can be launched in two ways: first, you may use eXist's GUI, second, you may use the shell mode. Without an action, the client switches to interactive mode and displays the graphical user interface. It is also possible that the client switches to shell-mode and prompts for user input.

[edit]
GUI

Simply run client script without any parameters: 

 cd $EXIST_HOME/bin
 ./client.sh

and enter URL in database login window pointing to your eXist location e.g. xmldb:exist://localhost:8680/exist/xmlrpc Then connect to the server. See example screenshot.

[edit]
Shell mode

The shell-mode is invoked using the following command (it provides access to a server running inside the Jetty webserver at port 8680 on a local host): 

cd /$EXIST_HOME/bin
./client.sh -s -ouri=xmldb:exist://localhost:8680/exist/xmlrpc

When connected successfully you should see eXist database prompt: 

exist:/db>

You will find more information about eXist integrated command-line client pointing towards eXist home page.

[edit]

Step 12 - Creating user

The following examples will guide you through shell-mode of eXist client. If you use GUI you will find all relevant comands in menus. For more information refer to eXist quick start pages.

  • First create user account for SQL Measurement Archive service entering the following commands in eXist shell: 
exist:/db>adduser sqlmaservice
password: sqlmaservice
re-enter password: sqlmaservice
home collection [none]: 
enter groups: dba
user <...> created

These are just examples and you can use other login/password of your own choice.

[edit]

Metadata configuration file

[edit]

Step 13 - Creating metadata configuration file

You can use one of these templates or you can also take the configuration file used for another MA adding additional data in it based on these examples. In order to create metadata configuration file containing metric utilization use the following file as a template:

In order to create metadata configuration file containing L2 link status use the following file as a template:

Also one can create metadat configuration file with data representing more than one metric:

For testing purposes you don't have to create new configuration files.

[edit]

Configuring service

[edit]

Step 14 - Service properties setup

Here we are going to set up some properties necessary to run the service. It's better that you call Ant script which will interact with you to do it automaticaly. To generate config files and setup properties run the following command: 

cd $PERFSONAR/ant
ant -f build-sqlma.xml configure-sqlma

You can always edit above files manually to change settings or modify supported functionality. But remember, all these files are required for running the MA. See here for detailed description of configuration properties.

[edit]

Step 15 - Sending metadata configuration file to XML database

First init eXist database with the following command: 

cd $PERFSONAR/ant
ant -f build-sqlma.xml xmldb-init-sqlma

This command results in sending metadata configuration file (see 'service.ma.conf_file' property in $PERFSONAR/conf/service.properties file) to XML database. If such file already exists in the database then it will be removed but before that the copy will be created in $PERFSONAR/conf directory. The name of this copy will get the prefix "dump".

XML database should have only one resource i.e. the current metadata configuration file. It's important to remove other resources which could have been used before current configuration. To print a list of resources existing in XML database run the following command: 

cd $PERFSONAR/ant
ant -f build-sqlma.xml xmldb-list-sqlma

There should be only current metadata configuration file. If there are other resources run the following command: 

cd $PERFSONAR/ant
ant -f build-sqlma.xml xmldb-resource-remove-sqlma

The script will ask for a name of resource to remove.

Warning: If for some reasons Ant script can not init the xml database successfully then one can use eXist client tool to create the new collection 'sqlmaconfig' (Owner: sqlmaservice) and store there the metadata configuration file (see step 13).


[edit]

Step 16 - Configuration of SQL database connection

In $PERFSONAR/conf/ directory you will find ibatis-SqlMapConfig-L2-status-domain.xml, ibatis-SqlMapConfig-L2-status-interdomain.xml and ibatis-SqlMapConfig-utilization.xml files containing SQL database configuration parameters. Put valid values for properties depending on the database of you choice.

  • For L2 status in ibatis-SqlMapConfig-L2-status.xml
    • JDBC.Driver. Example value = com.mysql.jdbc.Driver
    • JDBC.ConnectionURL. Example value = jdbc:mysql://127.0.0.1/perfsonar_ma
    • JDBC.Username. Example value = perfsonar_ma
    • JDBC.Password. Example value = testpass
  • For utilization in ibatis-SqlMapConfig-utilization.xml
    • Properties are the same as above
[edit]

Compiling service

[edit]

Step 17 - Measurement Archive compilation

If you chose using SVN or source distribution you will now have to compile your sources and create jar files. Simply enter the following commands: 

cd $PERFSONAR/ant
ant -f build-sqlma.xml build-sqlma


[edit]

Starting Web applications

[edit]

Step 18 - Running Tomcat

  • Execute the following command in Tomcat home directory in order to launch Tomcat 
$TOMCAT/bin>./startup.sh
  • Or restart Tomcat if it has been running previously 
$TOMCAT/bin>./shutdown.sh
$TOMCAT/bin>./startup.sh
[edit]

Step 19 - Deploying your web service

  • Undeploy application from Tomcat (if necessary) 
cd $PERFSONAR/ant
ant -f build-sqlma.xml undeploy-sqlma
  • Deploy the SQL Measurement Archive in Tomcat 
ant -f build-sqlma.xml deploy-sqlma
  • Restart your Tomcat 
$TOMCAT/bin>./shutdown.sh
$TOMCAT/bin>./startup.sh
  • If you want to validate your service installation follow those simple checks.
  • In case of any changes in the code of the web service or update using SVN you have to compile the service again and redeploy it in Tomcat.
  • In case of any changes only in the configuration files restart Tomcat 
$TOMCAT/bin>./shutdown.sh
$TOMCAT/bin>./startup.sh
[edit]

Testing

It is presumed that you have completed all steps before. Now you can check whether SQL Measurement Archive deployment has been successful.

[edit]

Testing database connection

  • mySQL

Warning: Put correct values for url, userid and password attributes in $PERFSONAR/ant/sqlma/sqlma-test-run-targets.xml and run the following command 

$PERFSONAR/ant> ant -f build-sqlma.xml run-mysql-sqlma-test1
  • postgreSQL

Warning: Put correct values for url, userid and password attributes in $PERFSONAR/ant/sqlma/sqlma-test-run-targets.xml and run the following command 

$PERFSONAR/ant> ant -f build-sqlma.xml run-psql-sqlma-test1
[edit]

Running sample requests

In order to test requests and replies enter the following commands which use sample XML request files as specified.

[edit]

L2 status

[edit]
Requests for a key
$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-key-L2status-1

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-key-L2status-2

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-key-L2status-3
[edit]
Request for data
$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-data-L2status-1

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-data-L2status-2

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-data-L2status-3

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-data-L2status-4

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-data-L2status-5

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-data-L2status-6
[edit]
Request to store data

In order to enable writing data in SQL MA the property 'service.ma.xmldb.db_store' in $PERFSONAR/conf/service.properties file must have set value 'on' (after this change Tomcat has to be restarted). 

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-store-L2status-1

This request uses the following example $PERFSONAR/schema/example-instances/sonar/MeasurementArchiveStore-L2status-Req-1.xml. NOTE Only data which are described by metadata in XML database (eXist) can be stored.

[edit]

Utilization

[edit]
Request for key

Request file: $PERFSONAR/schema/example-instances/sonar/MetadataKey-Req.xml 

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-key-util
[edit]
Request for data

Request file: $PERFSONAR/schema/example-instances/sonar/SetupDataRequest-FilterRRDSelect-2.xml 

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-data-util-1

Request file: $PERFSONAR/schema/example-instances/sonar/SetupDataRequest-FilterRRDSelect-4.xml 

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-data-util-2

Request file: $PERFSONAR/schema/example-instances/sonar/SetupDataRequest-FilterRRDSelect-5.xml 

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-data-util-3
[edit]
Request to store data

In order to enable writing data in SQL MA the property 'service.ma.xmldb.db_store' in $PERFSONAR/conf/service.properties file must have set value 'on' (after this change Tomcat has to be restarted). 

$PERFSONAR/ant> ant -f build-sqlma.xml run-sqlma-client-store-util

This request uses the following example $PERFSONAR/schema/example-instances/sonar/MeasurementArchiveStoreRequest-1.xml


[edit]

Ping request

This request can be used to monitor the availability of service. 

$PERFSONAR/ant> ant -f build-sqlma.xml run-echo

Request file: $PERFSONAR/schema/example-instances/sonar/Echo-Req.xml
Response file: $PERFSONAR/schema/example-instances/sonar/Echo-Resp.xml


[edit]

LS Registration

Some properties of the following files are required in order to register SQL MA in LS.

After the configuration of LS registration SQL MA should be redeployed (step 18) !!!

[edit]

$PERFSONAR/conf/service.properties

  • service.r.access_point - URL of SQL MA service (presented value below is only an example and should be changed)
  • service.r.service_name - symbolic name of SQL MA service (presented value below is only an example and may be changed)
  • service.r.service_type - service type (presented value below should not be changed)
  • service.r.service_description - Description of service (presented value below is only an example and may be changed)
  • service.r.ls_url - RL of LS service which will accept registration requests (presented value below is only an example and should be changed)
  • component.registrator.interval - Interval time (milliseconds) between registration messages (presented value below could be used in real life configuration)
  • component.registrator.scheduler_component - The name of a scheduler used for registration (presented value below should not be changed) 


service.r.access_point=http://xxx.yyy.zz:8080/axis/services/MeasurementArchive
service.r.service_name=Test MA
service.r.service_type=MA
service.r.service_description=Test MA, perfSONAR project
service.r.ls_url=http://aaa.bbb.cc:8080/axis/services/LookupService
component.registrator.interval=30000
component.registrator.scheduler_component=ma_scheduler
[edit]

$PERFSONAR/conf/components.properties

  • ma_scheduler - name of scheduler application component (presented value below should not be changed)
  • registrator - name of LS registration application component (presented value below should not be changed) 


ma_scheduler=org.perfsonar.commons.auxiliary.components.simplescheduler.Scheduler
registrator=org.perfsonar.service.measurementArchive.register.LSRegistrationComponent


[edit]

Storing the data with metadata which do not exist in the metadata configuration (xml database) yet

If the request of type MeasurementArchiveStoreRequest contain metadata element which does not exist in the metadata configuration (xml database) yet then this metadata should also contain parameters which are required by the key in the metadata configuration.

For example, if metadata/data pair in the metadata configuration looks like this: 

<nmwg:metadata id="md-link-LRZ-SARA-DEISA-002">
        <nmwg:subject id="sub1">
            <nmtl2:link>
                ...
            </nmtl2:link>
        </nmwg:subject>
        <nmwg:parameters>
            <nmwg:parameter name="supportedEventType">Path.Status</nmwg:parameter>
        </nmwg:parameters>
    </nmwg:metadata>
 
    <nmwg:data id="d2" metadataIdRef="md-link-LRZ-SARA-DEISA-002">
        <nmwg:key>
            <nmwg:parameters>
                <nmwg:parameter name="pathId">DFN-Surfnet-Link-5678</nmwg:parameter>
                <nmwg:parameter name="ibatisConfig">ibatis-SqlMapConfig-L2-status-domain.xml</nmwg:parameter>
            </nmwg:parameters>
        </nmwg:key>
    </nmwg:data>


then the request with new metadata should have the same key parameters (in this case pathId and ibatisConfig): 

<nmwg:message id="msg4"
              type="MeasurementArchiveStoreRequest"
              xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/"
              xmlns:nmtl2="http://ggf.org/ns/nmwg/topology/l2/3.0/"
              xmlns:select="http://ggf.org/ns/nmwg/ops/select/2.0/"
              xmlns:ifevt="http://ggf.org/ns/nmwg/event/status/base/2.0/">
 
    <nmwg:metadata id="meta1">
        <nmwg:subject id="sub1">
            <nmtl2:link>
                ...
            </nmtl2:link>
        </nmwg:subject>
        <nmwg:eventType>Path.Status</nmwg:eventType>
        <nmwg:parameters>
            <nmwg:parameter name="pathId">XYZ</nmwg:parameter>
            <nmwg:parameter name="ibatisConfig">ibatis-SqlMapConfig-L2-status-domain.xml</nmwg:parameter>
        </nmwg:parameters>
    </nmwg:metadata>
 
    <nmwg:data id="data1" metadataIdRef="meta1">
      <ifevt:datum timeType="unix" timeValue="1157107685">
        <ifevt:stateAdmin>unknown</ifevt:stateAdmin>
        <ifevt:stateOper>up</ifevt:stateOper>
      </ifevt:datum>
    </nmwg:data>
 
  </nmwg:message>


Very important is not to use chaining of metadata elements for such requests. In case of the request for L2 path status metric node elements should be declared in the link element (example).


If the metadaId is used to tie the data stored in the relational database with metadata elements kept in the metadata configuration (xml database), like it is for the utilization metric, then there is no need to use metadataId parameter in the request because it will be added automatically by the service.

Example of metadata/data pair in the metadata configuration (xml database) for the utilization metric: 

<nmwg:metadata id="meta1-test">
        <netutil:subject id="subj1">
            <nmwgt:interface>
                ...
            </nmwgt:interface>
        </netutil:subject>
        <nmwg:parameters>
            <nmwg:parameter name="supportedEventType">utilization</nmwg:parameter>
        </nmwg:parameters>
    </nmwg:metadata>
 
    <nmwg:data id="data1-test" metadataIdRef="meta1-test">
        <nmwg:key>
            <nmwg:parameters>
                <nmwg:parameter name="metadataId">meta1-test</nmwg:parameter>
                <nmwg:parameter name="ibatisConfig">ibatis-SqlMapConfig-utilization.xml</nmwg:parameter>
            </nmwg:parameters>
        </nmwg:key>
    </nmwg:data>

Example of the request for the utilization metric: 

<nmwg:message id="msg1"
                type="MeasurementArchiveStoreRequest"
                xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/"
                xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/"
                xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/">
 
    <nmwg:metadata id="meta1">
        <netutil:subject id="subj1">
            <nmwgt:interface>
                ...
            </nmwgt:interface>
        </netutil:subject>
        <nmwg:eventType>utilization</nmwg:eventType>
        <nmwg:parameters>
            <nmwg:parameter name="ibatisConfig">ibatis-SqlMapConfig-utilization.xml</nmwg:parameter>
        </nmwg:parameters>
    </nmwg:metadata>
 
    <nmwg:data id="data1" metadataIdRef="meta1">
        <nmwg:datum value="12345" timeValue="1139270800" timeType="unix" valueUnits="bps" />
    </nmwg:data>
 
  </nmwg:message>


If the SQL MA service supports only one metric then the name of ibatis configuration file (ibatisConfig parameter) can be put in the service configuration file ($PERFSONAR/conf/service.properties) and the request does not have to contain this parameter. The property which should be used to keep the name of that file is service.ma.sql.default_ibatis_config.

Retrieved from "http://wiki.perfsonar.net/jra1-wiki/index.php/Java_SQL_MA_Service_Installation_Guide"
Personal tools