CPF

Log

Log Files & Diagnostics Overview

The CPF creates additional log files in addition to those created for the Apache web server and tomcat servlet container. These log files can be accessed on the machine that the master and worker are running on. These maybe exposed for download via the Apache web server if configured.

Log File Format

Each log file has lines with the following format, separated by tabs (\t).

{Date YYYY-MM-DD HH:MI:SS,SSS}\t{Level DEBUG, INFO,WARN, or ERROR}\t{Category}\t{Message}

For example:

2014-07-30 13:56:16,432 INFO  CPF-TEST  Start Module Start  moduleName=CPF-TEST

Log entries may also have a [java stack trace](http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html#printStackTrace()), shown on multiple lines. For example.

2014-07-17 17:08:25,305  ERROR ca.bc.gov.open.cpf.api.scheduler.BatchJobPreProcess Could not open JDBC Connection for transaction; nested exception is java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
  at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)
  at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
  at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
  at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
  at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
  at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:280)
  at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:207)
  at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:139)
  at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:88)
  at oracle.jdbc.pool.OracleImplicitConnectionCache.makeCacheConnection(OracleImplicitConnectionCache.java:1598)
  at oracle.jdbc.pool.OracleImplicitConnectionCache.makeOneConnection(OracleImplicitConnectionCache.java:515)
  at oracle.jdbc.pool.OracleImplicitConnectionCache.getCacheConnection(OracleImplicitConnectionCache.java:475)
  at oracle.jdbc.pool.OracleImplicitConnectionCache.getConnection(OracleImplicitConnectionCache.java:357)
  at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:395)
  at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:179)
  at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:157)
  ... 13 more
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
  at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:392)
  at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:434)
  at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:687)
  at oracle.net.ns.NSProtocol.connect(NSProtocol.java:247)
  at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102)
  at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320)
  ... 29 more
Caused by: java.net.SocketTimeoutException: connect timed out
  at java.net.PlainSocketImpl.socketConnect(Native Method)
  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:579)
  at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:150)
  at oracle.net.nt.ConnOption.connect(ConnOption.java:133)
  at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:370)
  ... 34 more

Log Files

The following table summarizes the log files created by the CPF. The log files by default are stored in /apps/logs/cpf/. Each module and business application has separate directories for their log files. Log files will be rolled over once they reach 10MB. The last 7 log files will be retained with extensions .{i}.log (e.g. .2.log).

File Description
catalina.out If running on tomcat the tomcat server log file. Exact file name and location will vary based on the servlet container used and it's configuration. If the CPF does not start check this log file for any errors.
cpf-app.log Log messages for components on the master not related to a module or a business application. New log file created on module restart or if > 10MB. Previous 7 log files retained with extensions .{1-7}.log (e.g. .1.log).
worker_{environmentName}_{hostName}_{port}_{contextPath}.log Log messages for components on the worker not related to a module or a business application. New log file created on module restart or if > 10MB. Previous 7 log files retained with extensions .{1-7}.log (e.g. .1.log). 
{moduleName}/
{moduleName}_master.log
 Log messages for a module's components on the master not related to a business application. New log file created on module restart or if > 10MB. Previous 7 log files retained with extensions .{1-7}.log (e.g. .1.log). 
{moduleName}/
{moduleName}_worker_{environmentName}_{hostName}_{port}_{contextPath}.log
 Log messages for a module's components on the worker not related to a business application. New log file created on module restart or if > 10MB. Previous 7 log files retained with extensions .{1-7}.log (e.g. .1.log). 
{moduleName}/
{appName}/
  {moduleName}_{appName}_master.log
 Log messages for a business application's components on the master. New log file created on module restart or if > 10MB. Previous 7 log files retained with extensions .{1-7}.log (e.g. .1.log). 
{moduleName}/
{appName}/
  {moduleName}_{appName}_worker_{environmentName}_{hostName}_{port}_{contextPath}.log
 Log messages for a business application's components on the worker. New log file created on module restart or if > 10MB. Previous 7 log files retained with extensions .{1-7}.log (e.g. .1.log).

Diagnosing Issues

This section describes how to diagnose issues with the CPF system or plug-ins running within the system.

The first step in diagnosing an issue is to detect if it is a plug-in specific issue that must be reported to the plug-in developer or a general CPF issue that must be reported to the CPF developer. The general rule is check to see if the problem is isolated to this plug-in by submitting jobs to other plug-ins (e.g. MapTileByTileId). If the error is specific to a single plug-in then contact the plug-in developer for resolution first. Only contact the CPF developer if the plug-in developer cannot resolve the issue.

Server Errors

Sometimes the server will return an error page instead of a valid page. The CPF returns a formatted and styled error page similar to as shown below. This kind of error is generated by the CPF application. If the server returns an unformatted page (black text on white background) then this is an error generated by the web server. For example the application server was down, or the CPF was not deployed. Those errors must be reported to the server administrator.

400 Bad Data

If the input data to the web service was incorrect the CPF may return a 400 error with a message about the invalid parameter. Check the parameters are correct using the Client REST API or the Overview tab for the business application. If the issue can’t be resolved by reading the documentation. contact the plug-in developer for business application specific parameters or the CPF developer for general CPF parameters.

404 Not Found

This error indicates that the resource could not be found. Verify that the plug-in is loaded, the resource exists (e.g. correct job id and hasn’t been deleted), or that the correct URL is used.

500 Server Error

This error indicates an unknown error has occurred on the server. Verify that the error can be repeated. Check the cpf-app.log or catalina.out log files for more details. The contents of these files should be sent to the CPF developer with a detailed description of the steps (pages) and parameters used to generate the error.

It may be possible to diagnose and resolve the following types of error without contacting the CPF developer.

  • java.io.IOException subclasses
    • If the error is a broken pipe within a tomcat/catalina stack trace then it’s probably that the connection from the web browser to the CPF process was disconnected. Not much can be done except increasing the timeouts for communications between the apache and tomcat servers.
    • If the error is a file not found, permission denied, out of disk space error then verify the server configuration and check available disk space.
  • java.sql.SQLException subclasses
    • Check the error code/message to verify if it is an invalid username or password or for password expiry. Update the passwords or configuration as required.
    • Check the error code/message to verify that the database server is available. It might be down for a cold backup, daily restart or maintenance. Verify the JDBC URL to check it’s pointing to the correct server
    • Check the error code/message to verify if the database server was out of tablespace storage. Delete old jobs or increase the storage as required.
    • Check the error code/message to verify if it’s an invalid SQL statement. If it is contact the CPF developer if it has CPF has a table schema name in the SQL, otherwise contact the plug-in developer.
  • java.lang.OutOfMemoryError This could occur if the request data is to large to fit into memory or if the plug-ins or CPF has been re-dployed too many times and is not being garbage collected (perm gen errors). Typically this error will show up in the catalina.out file. Rebooting the application server is the best way to resolve these issues. If the error occurs repeatedly then verify that the -Xmx option for tomcat is large enough for the business applications and for perm gen errors check the -XX:MaxPermSize option. The CPF and plug-ins should be developed so that they can be garbage collected when stopped or restarted. Details on this are in the client developers guide.
  • org.springframework.security.authentication.*Exception This indicates that the user is not valid or does not have permission to access the resource. Contact the CPF developer as these should not be logged in the log file and just displayed to the user.
Verify Plug-in Started

Use the following steps to verify that the plug-in is started on the master or worker.

  • Check that the plug-in is enabled and started on the master using the View Module page.
    • Enable the module if it is not enabled.
    • Start the module if not started.
    • If there is a module error:
      • Check the master module log file {moduleName}/{moduleName}_master.log for more details on the error.
      • Check the configuration properties using the plug-ins documentation to verify they are correct. Especially check database URL, username and password.
      • Check that the Maven Module Id is correct. Fix and start module if incorrect.
      • If there is a Maven error downloading files verify that the Maven repository has all the required modules for the plug-in. Clear the CPF cache if necessary to reset the system. Verify using a web browser that the Maven modules exist in the Maven repository and can be downloaded. If not contact the Maven repository administrator.
  • Check there is a worker connected using the Worker List page.
  • Check the module is started using the Worker Module List page.
  • If not started check for module errors using the Worker Module View page.
  • Check the worker module log file {moduleName}/{moduleName}_worker_{environmentName}_{hostName}_{port}_{contextPath}.log for more details on the error.

If the plug-in loaded successfully the master module log file {moduleName}/{moduleName}_master.log and worker module log file {moduleName}/{moduleName}_worker_{environmentName}_{hostName}_{port}_{contextPath}.log will be rolled over and the following entries will appear at the top of the log file.

2014-07-30 16:28:24,183 INFO  [Module Name]  Start Module Start    moduleName=[Module Name]
2014-07-30 16:28:24,200 INFO  [Module Name]  Start Loading plugin  class=[businessApplicationClass]
2014-07-30 16:28:24,212 INFO  [Module Name]  End Loading plugin    class=[businessApplicationClass]  businessApplicationName=[businessApplicationName]
2014-07-30 16:28:24,238 INFO  [Module Name]  End Module Start      moduleName=[Module Name] time=0.06
Job Submission

Use the following steps if experiencing difficulty accessing the business application pages and in submitting instant, single or multiple request jobs.

  • Using above verify the plug-in for the business application is started.
  • Verify the business application is available on the 'Business Application' page in the client application. Refresh if necessary. If it does not appear in the list then the plug-in does not exist (contact plug-in developer) or is not loaded.
  • Click on the name of the business application and verify that the Overview, Create Single Request Job, Create Multi-Request Job and Batch Job Tabs are available. If the plug-in supports instant mode check the ‘Instant’ tab is visible. Click on each tab to verify it is displayed correctly.
  • Use the Create Single Request Job and Create Multi-Request Job tabs to verify that a job can be submitted.
    • If the job status page was returned then the job was submitted correctly.
    • If you get a red error message at the top of the form or next to a field then the parameters were incorrect. Verify the parameters used. Consult the Overview tab for more details or the plug-in developer.
    • If the size of a mult-request job file and other form parameters exceed 20MB then a 400 error will be returned. Use the URL upload instead.
    • If a 400 error is returned check the input parameters to see if they are valid. Contact the plug-in developer for application specific parameters or the CPF developer for general parameters.
    • If a 500 error is returned check the cpf-app.log for errors and send the log file to the CPF developer.
Job Processing

If the job was submitted but is not completed in the expected time check the following.

  • Check the module is loaded on the worker using the Worker Module View page. Some modules may take 30 minutes to initialize after restart on the worker
  • Submit jobs to another business application so see if the system generally works.
  • If the job is stuck in the creatingRequests or creatingResults states check the master module and app log file for any errors. If the error was due to out of memory or database space allocate more space or cancel the job. Otherwise send the errors to the CPF developer for review, don’t cancel delete the job until it has been investigated.
  • If the job is stuck in the processing stage check the master and worker module and app log files for any errors. Send to the CPF developer for review.
  • Additional information about the scheduling and execution of execution groups and requests can be enabled by Edit Business Application to turn on Debug level logging for the business application. NOTE: Change back to ERROR logging after the problem is diagnosed as the logs can get large and slow down the system.

Back to top

Copyright ©2008-2015 Province of British Columbia. All Rights Reserved.

Version: 4.1.4-SNAPSHOT. Last Published: 2015-Mar-12.