Build ARCo

For building ARCo some additional third party tools are needed. The easiest way to get access to these tools currently is to download the N1 Grid Engine portion of  the Solaris Enterprise System.

Check the N1 Grid Engine check mark and press the orange 'Get Downloads & Media' button.  After registration you will have access to the necessary files. 

From the n1ge6u9-arco.tar.gz.zip file you can extract the third party libraries that are required for building. Alternatively you can download them from the links delivered below. Additional build preliminaries like the JavaHelp system and the JWSDP package are listed as well and are required for building.

The next step is to checkout the correct repository and to adjust some configuration files to your local setup. See the following example for a Grid Engine maintrunk system (for Grid Engine 6.0u8 or higher the same procedure applies with the difference that you have to use cvs co -r V60_ARCO_BRANCH arco for the checkout and the references for sge.root and sge.srcdir should point to a Grid Engine 6.0u8 or higher)

Extraction of the packages above

% cvs co arco
% cd arco/source
% cp build.properties build_private.properties

edit build_private.properties
8<-----------8<-----------8<-----------8<-----------8<-----------
product.version=Maintrunc
product.name=GE
product.longname=Grid Engine

#
# Commmon site directories
#
copyright.dir=/home/aa114085/ARCHIVE/n1ge6-copyright
junit.jar=<somedir>/junit.jar
jwsdp.home=<somedir>/jwsdp-1.6
#
# These files are in the swc distribution usually under
# /opt/sun/private/share/lib
# /usr/share/webconsole/lib
#
servlet.jar=<somedir>/servlet.jar
swc.jato.jar=<somedir>/jato.jar
swc.cc.jar=<somedir>/cc.jar
swc.services_impl.jar=<somedir>/serviceimpl.jar
swc.services.jar=<somedir>/serviceapi.jar
swc.webcommon.jar=<somedir>/SMIWebCommon.jar
oracle.jdbcdriver=<somedir>/classes12.jar
postgres.jdbcdriver=<somedir>/postgresql-7.4.2.jar
Webapps.home=/var/opt/webconsole/webapps
jfreechart.jar=<somedir>/jfreechart-1.0.3/lib/jfreechart-1.0.3.jar
jcommon.jar=<somedir>/jfreechart-1.0.3/lib/jcommon-1.0.6.jar
avalon.jar=<somedir>/avalon-framework-4.2.0.jar
fop.jar=<somedir>/fop.jar
batik.jar=<somedir>/batik.jar
jimi.zip=<somedir>/JimiProClasses.zip
jh.home=/usr/java/packages/javax.help-2.0/javahelp
emma.dir=<somedir>/emma-2.0.5312/lib

#
# Path to SGE_ROOT 
#
sge.root=<SGE_ROOT dir path>
#
#  Path to the SGE source tree
#
sge.srcdir=<path to the sge source directory  or just a dummy string>

#
#  Compile options
#
compile.debug=true

8<-----------8<-----------8<-----------8<-----------8<-----------

Please check if all jar files are available at the listed places. Otherwise your compilation will fail.

edit build.site
ANT_HOME=<somewhere>/apache-ant-1.6.5
WSDP_HOME=<somewhere>/jwsdp-1.6
JAVA_HOME=/usr/java/jdk1.5.0_06

Compilation:
% build.sh clean
% build.sh dist

The result are reporting/reporting.tar.gz and dbwriter/dbwriter.tar.gz.

Install ARCo

Install PostgreSQL

Configure PostgreSQL

Follow the PostgreSQL instructions, set up a user, create a data directory, and init the database:

bash-2.05# useradd -c "PostgreSQL admin" -d /export/home/postgres -m -s /bin/bash postgres
bash-2.05# passwd postgres
bash-2.05# mkdir /usr/local/pgsql/data
bash-2.05# chown postgres /usr/local/pgsql/data
bash-2.05$ initdb -D /usr/local/pgsql/data 
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale C.

fixing permissions on existing directory /usr/local/pgsql/data... ok
creating directory /usr/local/pgsql/data/base... ok
creating directory /usr/local/pgsql/data/global... ok
creating directory /usr/local/pgsql/data/pg_xlog... ok
creating directory /usr/local/pgsql/data/pg_clog... ok
selecting default max_connections... 40
selecting default shared_buffers... 800
creating configuration files... ok
creating template1 database in /usr/local/pgsql/data/base/1... ok
initializing pg_shadow... ok
enabling unlimited row size for system tables... ok
initializing pg_depend... ok
creating system views... ok
loading pg_description... ok
creating conversions... ok
setting privileges on built-in objects... ok
creating information schema... ok
vacuuming database template1... ok
copying template1 to template0... ok

Success. You can now start the database server using:

    /usr/local/pgsql/bin/postmaster -D /usr/local/pgsql/data
or
    /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start

To start the database I run pg_ctl -D /usr/local/pgsql/data -l /var/log/postgres start -o "-i". The -o "-i" option is suposed to allow TCP/IP connections to the database.

To get pg_ctl to work without having to specify where the data is set the PGDATA:

PGDATA=/usr/local/pgsql/data/
export PGDATA

Consult the PostgreSQL documentation for further details.

Configuring PostgreSQL for ARCo

Follow these instructions on docs.sun.com. The following 

• Adjustment of pg_hba.conf file as follows:

# TYPE  DATABASE    USER        IP-ADDRESS        IP-MASK           METHOD

local   all         postgres                                        trust
local   all         all                                             md5
# IPv4-style local connections:
host    all         all         192.168.100.0     255.255.255.0     trust

• Modified my postgresql.conf as per the instructions to enable IP connections, simply change the tcpip_socket line to true and uncomment it.

Setting up ARCo propper

Work through the instructions.

• Enable monitorting

bash-2.05$ qconf -mconf

Change the following:

...
reporting_params    accounting=true  reporting=true flush_time=00:00:15 joblog=true  sharelog=00:00:00
...

Also:

bash-2.05$ qconf -me global

Change the following:

hostname              global
...
report_variables      cpu,np_load_avg,mem_free,virtual_free
...

• Install the dbwriter software. Make sure you do this as root (or more properly, make sure your sgeadmin user actually owns the files!)

bash-2.05$ ./inst_dbwriter 

Welcome to the Grid Engine installation
---------------------------------------

Grid Engine dbWriter installation
---------------------------------
The dbWriter installation will take approximately 5 minutes

Hit  to continue >> 


Generic Parameters
------------------

Please enter your SGE_ROOT [/usr/N1GE6] >> 

Please enter your SGE_CELL [default] >> 
Please enter the path to your java 1.4 installation [] >> /usr/java

Setup your database connection parameters
-----------------------------------------

Enter your database type ( o = Oracle, p = PostgreSQL ) [p] >>    


Please enter the name of your postgresql db host [] >> b100x-1
Please enter the port of your postgresql db [5432] >> 

Please enter the name of your postgresql database [arco] >> 

Please enter the name of the database user [arco_write] >> 

Please enter the password of the database user >> 
Please retype the password >> 

Please enter the name of the database schema [public] >> 

Search for the jdbc driver org.postgresql.Driver 
in directory /usr/N1GE6/dbwriter/lib .........
found in /usr/N1GE6/dbwriter/lib/pg73jdbc2.jar

Enter your database type ( o = Oracle, p = PostgreSQL ) [p] >> 


Please enter the name of your postgresql db host [b100x-1] >> 

Please enter the port of your postgresql db [5432] >> 

Please enter the name of your postgresql database [arco] >> 

Please enter the name of the database user [arco_write] >> 

Please enter the password of the database user >> 
Please retype the password >> 

Please enter the name of the database schema [public] >> 

Search for the jdbc driver org.postgresql.Driver 
in directory /usr/N1GE6/dbwriter/lib .........
found in /usr/N1GE6/dbwriter/lib/pg73jdbc2.jar

Should the connection to the database be tested? (y/n) [y] >> 

Test db connection to 'jdbc:postgresql://b100x-1:5432/arco' ... OK


Please enter the interval between two dbwriter runs in seconds [60] >> 

Please enter the file with the derived value rules [$SGE_ROOT/dbwriter/database/postgres/dbwriter.xml] >> 

The dbWriter can run with different debug levels
Possible values: WARNING INFO CONFIG FINE FINER FINEST
Please enter the debug level of the dbwriter [INFO] >> 

All parameters are now collected
--------------------------------

      SGE_ROOT=/usr/N1GE6
      SGE_CELL=default
     JAVA_HOME=/usr/java (java version "1.4.1_03")
        DB_URL=jdbc:postgresql://b100x-1:5432/arco
       DB_USER=arco_write
      INTERVAL=60
REPORTING_FILE=$SGE_ROOT/$SGE_CELL/common/reporting
  DERIVED_FILE=$SGE_ROOT/dbwriter/database/postgres/dbwriter.xml
   DEBUG_LEVEL=INFO

Are this settings correct? (y/n) [y] >> 

Create start script sgedbwriter in /usr/N1GE6/dbwriter/bin

Create configuration file for dbWriter in /usr/N1GE6/default/common
Installation of dbWriter completed
Start the dbWriter with /usr/N1GE6/dbwriter/bin/sgedbwriter start

• Set it to run on every boot

bash-2.05# pwd
/usr/N1GE6/dbwriter
bash-2.05# cp bin/sgedbwriter /etc/init.d
bash-2.05# ln -s /etc/init.d/sgedbwriter /etc/rc2.d/S98sgedbwriter
bash-2.05# /etc/init.d/sgedbwriter start
starting dbWriter
Spool directory /usr/N1GE6/default/spool/b100x-1 does not exist.
Should it be created? (y/n) >> y

Install Sun Web Console if not present (solaris, linux only)

Install reporting

Go with the defaults . . .

bash-2.05# pwd
/usr/N1GE6/reporting
bash-2.05# ./inst_reporting 

Welcome to the  N1 SGE reporting module installation
----------------------------------------------------
The installation will take approximately 5 minutes

Hit  to continue >> 

Please enter the path to your java 1.4 installation [] >> /usr/java
Spool directory
---------------

In the spool directory the N1 SGE reporting module will
store all queries and results

Please enter the path to the spool directory [/var/spool/arco] >> 


Database Setup
--------------

Enter your database type ( o = Oracle, p = PostgreSQL ) [p] >> 


Please enter the name of your postgresql db host [] >> b100x-1

Please enter the port of your postgresql db [5432] >> 

Please enter the name of your postgresql database [arco] >> 

Please enter the name of the database user [arco_read] >> 

Please enter the password of the database user >> 
Please retype the password >> 

Please enter the name of the database schema [public] >> 

Search for the jdbc driver org.postgresql.Driver 
in directory /usr/N1GE6/reporting/WEB-INF/lib ..............
found in /usr/N1GE6/reporting/WEB-INF/lib/postgresql-7.4.2.jar

Should the connection to the database be tested? (y/n) [y] >> 

Test db connection to 'jdbc:postgresql://b100x-1:5432/arco' ... OK

Configure users with write access
---------------------------------
Enter a login name of a user (Press enter to finish) >> pj113073 

Users: pj113073
Enter a login name of a user (Press enter to finish) >> user2

Users: pj113073 user2
Enter a login name of a user (Press enter to finish) >> 

All parameters are now collected
--------------------------------
       SPOOL_DIR=/var/spool/arco
          DB_URL=jdbc:postgresql://b100x-1:5432/arco
         DB_USER=arco_read
ARCO_WRITE_USERS=pj113073 user2

Are this settings correct? (y/n) [y] >> y

/usr/sadm/bin/smcwebserver: server not running - /var/run/smcwebserver.pid does not exist

Directory /var/spool/arco does not exist, create it? (y/n) [y] >> 

Create directory /var/spool/arco
Create directory /var/spool/arco/queries
Copy predefined queries into /var/spool/arco/queries
Create directory /var/spool/arco/results
Create directory /usr/N1GE6/reporting/charts
Create directory /usr/N1GE6/reporting/export
Register the N1 SGE reporting module in the webconsole

    Registering com.sun.grid.arco_6.

Starting Sun(TM) Web Console Version 2.0.3...
See /var/log/webconsole/console_debug_log for server logging information