|
|
ARCo FAQs
Can the database connection parameters for the dbwriter be changed?The database connection parameters for the dbwriter are stored in $SGE_ROOT/$SGE_CELL/default/common/dbwriter.conf: ... Can the database connection parameters for the ARCo web application be changed?The connection parameter for the ARCO web application has a configuration file. Use the smreg commands of the java webconsole to determine the location of this configuration file: # smreg list -p The password is encrypted with the dbpwd util. How can I get metrics from the dbwriter?If the dbwriter is running in debug level INFO it prints after the import of a reporting file the number of processed lines and how long it takes. 02/02/2006 07:23:57|foo.bar|ile.ReportFileReader.parseFile|I|Processed 63 lines in 2.22s (28.34 lines/s) You can find the log file add $SGE_ROOT/$SGE_CELL/spool/dbwriter/dbwriter.log How do I change the debug level of the dbwriter?The debug level of the dbwriter is defined in the dbwriter configuration file ($SGE_ROOT/$SGE_CELL/common/dbwriter.conf). ... After changing the dbwriter configuration file you have to restart the dbwriter. How to import an accounting file into the ARCo database?With the acct2reporting.sh conversion script it is possible to convert an accouting file into a reporting file. Output of the script can be imported by the dbwriter. Warning: The import of the accouting file is only recommened if an empty database is used. Importing the accouting file into existing data will result in database conflicts. However it is a good possiblity to import historical data installation into the ARCo modul.
# $SGE_ROOT/$SGE_CELL/common/sgedbwriter stop
# acct2reporting.sh $SGE_ROOT/$SGE_CELL/common/accounting > $SGE_ROOT/$SGE_CELL/common/reporting_from_accouting
# vi $SGE_ROOT/$SGE_CELL/common/dbwriter.conf
# $SGE_ROOT/$SGE_CELL/common/sgedbwriter start
# qconf -mconf
# $SGE_ROOT/$SGE_CELL/common/sgedbwriter stop Can not save any query in ARCo
During the installation of the ARCo reporting module the users is asked # ./inst_reporting
Only those users are allowed save modifications on ARCo. % cat /var/opt/webconsole/webapps/reporting/config.xml <configuration> Edit this file to add or delete users. After editing this file you have to restart the sun webconsole # smcwebserver restart How to get the version of the installed ARCo database model?With N1GE6u1 the table sge_version was introduced. This table containsthe installed versions of the ARCo database model.
Use a sql util the show the content of this table: * oracle % sqlplus arco_write@arco.dbhost
% psql -h dbhost arco arco_write How do I query the average cpu usage of all jobs requesting resource 'foo'The following query returns the job number of all job requesting resource 'foo' select distinct j_job_number from sge_job, sge_job_request The cpu usage of a job is stored in the table sge_job_usage. The average cpu usage of all jobs requesting resource 'foo' can be calculated by joining the table sge_job_usage and sge_job_request: select AVG(ju_cpu) from sge_job_usage, sge_job_request Installation issues
Installation description of Duncan Hardie and Peter Jenkinshttp://dino.uk.sun.com/bin/view/PSEMEA/N1GridEngine6#ARCo Installation with Oracle
http://n1wiki.sfbay.sun.com/bin/view/GE/InstallationOracle DatabaseInstallation from http://www.Blastwave.org
# pkgadd -d <download_dir>/pkg_get.pkg 3. add # setenv PATH ${PATH}:/opt/csw/bin
5. Install the postgresql package # pkg-get -i postgresql The postgres binaries will be installed in /opt/csw/bin. If not yet installed, the packages CSWcommon, CSWisaexec, CSWossl, CSWreadline and CSWzlib will be installed as well. The postgresql installation will create a user postgres. 6. Activate the postgres user account # mkdir /opt/csw/var/pgdata 7. Initialize the postgresql database # su postgres 8. Setup access permissions > vi /opt/csw/var/pgdata/pg_hba.conf 9. Startup postmaster daemon (in background, allow socket connections) > postmaster -S -i -D /opt/csw/var/pgdata 10. Create users and database for ARCo > createuser -P arco_write
Sample queriesThe following is a gzip'ed tar file which contains a number of queries for ARCO. They include some of the built-in, simple queries that come with a standard ARCO installation, as well as some more advanced queries that illustrate some more sophisticated reports. Please study the individual queries for further information.
|
| Column | Type | Description |
| job_number | integer | Unique record identifier |
| task_number | integer | Array task id |
| pe_taskid | text | ID of a task of a tightly integrated parallel task |
| name | text | job name (script name or value set with the submit option -N) |
| groupname | text | UNIX group name of the promary group the job was executed in. References the group table. |
| username | text | UNIX user account the job was running in. References the user table. |
| account | text | Account string set with the submit option -A |
| project | text | Project, References the project table |
| department | text | Department, References the department table |
| submission_time | timestamp | Time of the job submission |
| start_time | timestamp | Time when the job was started |
| end_time | timestemp | Time when the job finished |
| wallclock_time | integer | end_time - start_time |
| cpu | double | The CPU time usage in seconds |
| io | double | The amount of data transferred in input/output operations |
| iow | double | The io wait time in seconds |
| maxvmem | double | The maximum vmem size in bytes |
| wait_time | integer | start_time - submission_time |
| turnaround_time | integer | end_time - submission_time |
same as view_accounting
| Column | Type | Description |
| completed | integer | Completed jobs |
| time | timestamp | Time when the jobs finished |
| Column | Type | Description |
| job_number | integer | JOB_ID |
| task_number | integer | Array task id |
| pe_taskid | text | ID of a task of a tightly integrated parallel task |
| name | text | job name (script name or value set with the submit option -N) |
| username | text | UNIX group name of the primary group the job was executed in. References the group table |
| account | text | UNIX user account the job was running in. References the user table |
| project | text | Project. References the project table |
| department | Department. References the department table | |
| time | Unix timestamp | Time when the job logging entry was generated |
| event | text | Event being recorded |
| state | text | Job state after the reported event |
| initiator | text | User who initiated action for the event |
| host | text | Host on with the event action was initiated |
| message | text | A message explaining what happened |
| Column | Type | Description |
| department | text | Name of the department |
| time_start | timestamp | Start time for the validity of a value |
| time_end | timestamp | End time for the validity of a value |
| variable | text | Variable name, e.g. h_sum_jobs |
| str_value | text | Varaible value, e.g. 5 |
| num_value | double precision | Variable value as number |
| num_config | double precision | In case of consumables: Consumable maximum available value (configured value) |
| Column | Type | Description |
| groupname | text | Name of the group |
| time_start | timestamp | Start time for the validity of a value |
| time_end | timestemp | End time for the validity of a value |
| variable | text | Variable name, e.g. h_sum_jobs |
| str_value | text | Variable value, e.g. 53 |
| num_value | double precision | Variable value as number |
| num_config | double precision | In case of consumables: Consumable maximum available value (configured value) |
| Column | Type | Description |
| hostname | text | The hostname |
| time_start | timestamp | Start time for the validity of a value |
| time_end | timestamp | End time for the validity of a value |
| variable | text | Variable name, e.g. load_avg |
| str_value | text | Variable value, e.g. 0.34 |
| num_value | double precision | Variable value as number |
| num_config | double precision | In case of consumables: Consumable maximum available value (configured value) |
| Column | Type | Description |
| project | text | Name of the project |
| time_start | timestamp | Start time for the validity of a value |
| time_end | timestamp | End time for the validity of a value |
| variable | text | Variable name, e.g. h_avg_cpu |
| str_value | text | Variable value, e.g. 345.5 |
| num_value | double precision | Variable value as number |
| num_config | double precision | In case of consumables: Consumable maximum available value (configured value) |
| Column | Type | Description |
| qname | text | Name of the queue |
| hostname | text | Name of host |
| time_start | timestamp | Start time for the validity of a value |
| time_end | timestamp | End time for the validity of a value |
| variable | text | Variable name, e.g. slots |
| str_value | text | Variable value, e.g. 5 |
| num_value | double precision | Variable value as number |
| num_config | double precision | In case of consumables: Consumable maximum available value (configured value) |
| Column | Type | Description |
| username | text | Name of the user |
| time_start | timestamp | Start time for the validity of a value |
| time_end | timestamp | End time for the validity of a value |
| variable | text | Variable name, e.g. h_sum_cpu |
| str_value | text | Variable value, e.g. 23.2 |
| num_value | double precision | Variable value as number |
| num_config | double precision | In case of consumables: Consumable maximum available value (configured value) |
/opt/sge
default
/usr/j2sdk1.4.2
swchost
oraclehost
1521
arco
arco_write exists,
password is known, schema of this user is arco_write
arco_read
The ARCo modules uses two database users (arco_write and arco_read). Due to the versatile configuration possibilities of oracle we can give here only a example how to setup the database accounts. Please the oracle documenation for futher information. Use your favorite sql util the execute the following sql statements.
CREATE USER "ARCO_WRITE" PROFILE "DEFAULT" IDENTIFIED BY "<secret password>"
DEFAULT TABLESPACE "USERS" TEMPORARY TABLESPACE "TEMP" QUOTA 100 M ON "USERS" ACCOUNT UNLOCK;
CREATE USER "ARCO_READ_XX" PROFILE "DEFAULT" IDENTIFIED BY "arco_read"
DEFAULT TABLESPACE "USERS" TEMPORARY TABLESPACE "TEMP" QUOTA 100 M ON "USERS" ACCOUNT UNLOCK;
The arco_write users needs the priviliges to create the sge_* tables and views in it's own database schema (the tables will be created during the dbwriter setup).
GRANT CREATE TABLE TO "ARCO_WRITE";
GRANT CREATE VIEW TO "ARCO_WRITE";
The arco_read user is used by the ARCo webapplication. For security issues this user should have a minimum set of privileges. All other privileges will be granted during the installation of the dbwriter.
GRANT CREATE SESSION TO "ARCO_WRITE;
In oracle each database user uses it's own database schema. To make it possible that the arco_read users has access to the sge_* tables and views during the installation of the dbwriter some synonmys are created in the schema of the arco_read users (A synonmy can seen as link from one table/view to a another). The arco_write user needs the privilege to setup these synonms.
GRANT CREATE ANY SYNONYM TO "ARCO_WRITE";
# cd /opt/sge
# tar xf /tmp/sun_sge6_dist/n1ge-6_0u1-arco.tar
# cd dbwriter
# ./inst_dbwriter
Welcome to the Grid Engine installation
---------------------------------------
Grid Engine dbWriter installation
---------------------------------
The dbWriter installation will take approximately 5 minutes
Hit <RETURN> to continue >>
Generic Parameters
------------------
Please enter your SGE_ROOT [/opt/sge] >>
Please enter your SGE_CELL [default] >>
Please enter the path to your java 1.4 installation [/user/j2sdk1.4.2] >>
Setup your database connection parameters
-----------------------------------------
Enter your database type ( o = Oracle, p = PostgreSQL ) [p] >> o
Please enter the name of your oracle db host [] >> oraclehost
Please enter the port of your oracle db [1521] >>
Please enter the name of your oracle database [arco] >>
Please enter the name of the database user [arco_write] >>
Please enter the password of the database user >>
Please retype the password >>
Please enter the name of the database schema [arco_write] >>
Search for the jdbc driver oracle.jdbc.driver.OracleDriver
in directory /opt/sge/dbwriter/lib ...........
jdbc driver not found in any jar file of directory /sopt/sge/dbwriter/lib
Please copy your driver jar into the directory /opt/sge/dbwriter/lib
Open another terminal and copy the oracle jdbc driver into /opt/sge/dbwriter/lib
# cp <OraHome>/jdbc/lib/classes12.jar /opt/sge/dbwriter/lib
Press enter in the shell with the dbwriter installation
Search for the jdbc driver oracle.jdbc.driver.OracleDriver in directory /scratch3/rh150277/31002/sge/dbwriter/lib .
found in /scratch3/rh150277/31002/sge/dbwriter/lib/classes12.jar
Should the connection to the database be tested? (y/n) [y] >> y
Test db connection to 'jdbc:oracle:thin:@oraclehost:1521:arco' ... OK
In the next step the dbwriter installation determines the version of the ARCo database model. For N1GE6u1 the version of the database model is 1.
Query db version ... no sge tables found
New version of the database model is needed
Should the database model be upgraded to version 1? (y/n) [y] >>
In my case the schema of the user arco_write contains no
tables. The installation will create all necessary tables. The arco_read
has it's own database schema, some grants and synonms will be created
for this user.
The ARCo web application connects to the database
with a user which has restricted access.
The name of this database user is needed to grant
him access to the sge tables.
Please enter the name of this database user [arco_read] >>
Upgrade to database model version 1 ... Install version 6.0 (id=0) -------
Create table sge_job
Create index sge_job_idx0
Create index sge_job_idx1
create table sge_job_usage
Create table sge_job_log
Create table sge_job_request
Create table sge_queue
Create index sge_queue_idx0
Create table sge_queue_values
Create index sge_queue_values_idx0
Create table sge_host
Create index sge_host_idx0
Create table sge_host_values
Create index sge_host_values_idx0
Create table sge_department
Create index sge_department_idx0
Create table sge_department_values
Create index sge_department_values_idx0
Create table sge_project
Create index sge_project_idx0
Create table sge_project_values
Create index sge_project_values_idx0
Create table sge_user
Create table sge_user_values
Create index sge_user_values_idx0
Create table sge_group
Create index sge_group_idx0
Creat table sge_group_values
Create index sge_group_values_idx0
Create table sge_share_log
Create view view_accounting
Create view view_job_times
Create view view_jobs_completed
Create view view_job_log
Create view view_department_values
Create view view_group_values
Create view_host_values
Create view view_project_values
Create view view_queue_values
Create view view_user_values
grant privileges on sge_department to arco_read
grant privileges on sge_department to arco_read
grant privileges on sge_group to arco_read
grant privileges on sge_group_values to arco_read
grant privileges on sge_host to arco_read
grant privileges on sge_host_values to arco_read
grant privileges on sge_job to arco_read
grant privileges on sge_job_log to arco_read
grant privileges on sge_job_request to arco_read
grant privileges on sge_job_usage to arco_read
grant privileges on sge_project to arco_read
grant privileges on sge_project_values to arco_read
grant privileges on sge_queue to arco_read
grant privileges on sge_queue_values to arco_read
grant privileges on sge_share_log to arco_read
grant privileges on sge_user to arco_read
grant privileges on sge_user_values to arco_read
grant privileges on view_accounting to arco_read
grant privileges on view_department_values to arco_read
grant privileges on view_group_values to arco_read
grant privileges on view_host_values to arco_read
grant privileges on view_job_log to arco_read
grant privileges on view_job_times to arco_read
grant privileges on view_jobs_completed to arco_read
grant privileges on view_project_values to arco_read
grant privileges on view_queue_values to arco_read
grant privileges on view_user_values to arco_read
create synonym sge_department for user arco_read
create synonym sge_department_values for user arco_read
create synonym sge_group for user arco_read
create synonym sge_group_values for user arco_read
create synonym sge_host for user arco_read
create synonym sge_host_values for user arco_read
create synonym sge_job for user arco_read
create synonym sge_job_log for user arco_read
create synonym sge_job_request for user arco_read
create synonym sge_job_usage for user arco_read
create synonym sge_project for user arco_read
create synonym sge_project_values for user arco_read
create synonym sge_queue for user arco_read
create synonym sge_queue_values for user arco_read
create synonym sge_share_log for user arco_read
create synonym sge_user for user arco_read
create synonym sge_user_values for user arco_read
create synonym view_accounting for user arco_read
create synonym view_department_values for user arco_read
create synonym view_group_values for user arco_read
create synonym view_host_values for user arco_read
create synonym view_job_log for user arco_read
create synonym view_job_times for user arco_read
create synonym view_jobs_completed for user arco_read
create synonym view_project_values for user arco_read
create synonym view_queue_values for user arco_read
create synonym view_user_values for user arco_read
commiting changes
version 6.0 (id=0) successfully installed
Install version 6.0u1 (id=1) -------
Create table sge_version
Update view view_job_times
Update version table
commiting changes
version 6.0u1 (id=1) successfully installed
OK
Please enter the interval between two dbwriter runs in seconds [60] >>
Please enter the file with the derived value rules [$SGE_ROOT/dbwriter/database/oracle/dbwriter.xml] >>
The dbWriter can run with different debug levels
Possible values: WARNING INFO CONFIG FINE FINER FINEST
Please enter the debug level of the dbwriter [INFO] >>
All parameters are now collected
--------------------------------
SGE_ROOT=/opt/sge
SGE_CELL=default
JAVA_HOME=/usr/j2sdk1.4.2 (java version "1.4.2")
DB_URL=jdbc:oracle:thin:@oraclehost:1521:arco
DB_USER=arco_write
INTERVAL=60
REPORTING_FILE=$SGE_ROOT/$SGE_CELL/common/reporting
DERIVED_FILE=$SGE_ROOT/dbwriter/database/oracle/dbwriter.xml
DEBUG_LEVEL=INFO
Are this settings correct? (y/n) [y] >> y
Create start script sgedbwriter in /opt/sge/dbwriter/bin
Create configuration file for dbWriter in /opt/sge/default/common
Installation of dbWriter completed
Start the dbWriter with /opt/sge/dbwriter/bin/sgedbwriter start
The installation of dbwriter is now finished. Start the dbwriter
# /opt/sge/dbwriter/bin/sgedbwriter start
The log file of the dbwriter is in $SGE_ROOT/$SGE_CELL/spool/dbwriter/dbwriter.log
# tail -f $SGE_ROOT/$SGE_CELL/spool/dbwriter/dbwriter.log
22/09/2004 08:47:46|balrog|.ReportingDBWriter.initLogging|I|Starting up ReportingDBWriter
22/09/2004 08:47:50|balrog|e.ReportFileReader.processFile|I|Renaming reporting to reporting.processing
22/09/2004 08:48:04|balrog|BWriter$DerivedValueThread.run|I|Next regular task (derived values and delete) will be done at 9/22/04 9:11 AM
22/09/2004 08:48:22|balrog|ile.ReportFileReader.parseFile|I|Deleting file reporting.processing
22/09/2004 08:48:23|balrog|ile.ReportFileReader.parseFile|I|Processed 327 lines in 32.54s (10.05 lines/s)
...
# su
Password:
#
# cd /opt/sge/reporting
Welcome to the N1 SGE reporting module installation
----------------------------------------------------
The installation will take approximately 5 minutes
Hit <RETURN> to continue >>
Please enter the path to your java 1.4 installation [/usr/j2sdk1.4.2] >>
Spool directory
---------------
In the spool directory the N1 SGE reporting module will
store all queries and results
Please enter the path to the spool directory [/var/spool/arco] >>
Database Setup
--------------
Enter your database type ( o = Oracle, p = PostgreSQL ) [p] >> o
Please enter the name of your oracle db host [] >> oraclehost
Please enter the port of your oracle db [1521] >>
Please enter the name of your oracle database [arco] >>
Please enter the name of the database user [arco_read] >>
Please enter the password of the database user >>
Please retype the password >>
Search for the jdbc driver oracle.jdbc.driver.OracleDriver
in directory /opt/sge/reporting/WEB-INF/lib .................
jdbc driver ... not found in any jar file of directory /opt/sge/reporting/WEB-INF/lib
Please copy your driver jar into the directory /opt/sge/reporting/WEB-INF/lib
Press enter to continue >>
Search for the jdbc driver oracle.jdbc.driver.OracleDriver
in directory /scratch3/rh150277/31002/sge/reporting/WEB-INF/lib ...
found in /scratch3/rh150277/31002/sge/reporting/WEB-INF/lib/classes12.jar
Should the connection to the database be tested? (y/n) [y] >> y
Test db connection to 'jdbc:oracle:thin:@bilbo:1521:arco' ... OK
In the next step you have to define users with write permission in the reporting module:
Configure users with write access
---------------------------------
Enter a login name of a user (Press enter to finish) >> user1
Users: user1
Enter a login name of a user (Press enter to finish) >> user2
Users: user2 user2
Enter a login name of a user (Press enter to finish) >>
All parameters are now collected
--------------------------------
SPOOL_DIR=/var/spool/arco
DB_URL=jdbc:oracle:thin:@oraclehost:1521:arco
DB_USER=arco_read
ARCO_WRITE_USERS=user1 user2
Are this settings correct? (y/n) [y] >> y
Shutting down Sun(TM) Web Console Version 2.2.1...
Found old versions of the N1 SGE reporting module
-------------------------------------------------
Should the application com.sun.grid.arco_6 be deleted? (y/n) [n] >> y
Unregistering com.sun.grid.arco_6.
Directory /var/spool/arco does not exist, create it? (y/n) [y] >>
reate directory /var/spool/arco
Create directory /var/spool/arco/queries
Copy predefined queries into /var/spool/arco/queries
Create directory /var/spool/arco/results
Create directory /scratch3/rh150277/31002/sge/reporting/charts
Create directory /scratch3/rh150277/31002/sge/reporting/export
Register the N1 SGE reporting module in the webconsole
Registering com.sun.grid.arco_6u1.
Starting Sun(TM) Web Console Version 2.2.1...
See /var/log/webconsole/console_debug_log for server logging information
Open your browser and go to the url https://reporting_host:6789
|
![]() |
By any use of this Website, you agree to be bound by these Policies and Terms of Use. |