JQR

decouple yourself

Documentation

From JQR

Jump to: navigation, search

Contents


[edit] Introduction

Many application actions depends on unreliable network resources or just take long to execute. In such environment it is good practice to decouple application front and back end. JQR is a software application component written in Java that provides easy job queuing through different popular web protocols and supports jobs to be written in almost any language compatible with Java environment (Python, Ruby, etc.)

[edit] Getting Started

[edit] Unix Binary Installation

  • Download the appropriate binary distribution of JQR (sensatic-jqr-x.x.x.tar.gz)
  • Extract the archive to the desired folder 
$ tar zxvf sensatic-jqr-x.x.x.tar.gz
  • Go to the folder and start the application 
$ cd sensatic-jqr-x.x.x
$ bin/jqr start

[edit] System Configuration

All configuration files for JQR are located in etc/ directory. The following sections explain different aspects of JQR configuration in detail.

[edit] Quick configuration

In most cases you will want just to change just basic properties, such as port numbers, administration email, etc. This can be done through etc/jqr.properties file, which looks like this 

# internal broker URL
broker.url=tcp://localhost:61618
# expose JMX
broker.useJmx=true
# JMX port
broker.jmxPort=1099
# JQR server port
jqr.server.port=25256
# JQR client redelivery count
jqr.client.retryCount = 1
# send notification emails of failed jobs and other warnings to
jqr.admin=admin@example.com
# SMTP host for email subsystem
email.smtpHost=localhost
# send emails as
email.from=application@example.com
# customize subject of emails
email.subject=[JQR] Application message

[edit] Broker

JMS Broker (AcitveMQ implementation) is the central part of the JQR application, used to queue jobs. By default, JQR integrates a broker that it is used in the application. This integrated broker is configured in the etc/broker.xml file. The full specification of the ActiveMQ broker configuration can be found in the appropriate documentation. In most cases, you will only want to adapt original etc/broker.xml file to your needs. Here's a list of common tasks:

  • To turn off JMX monitoring set useJmx property of the <amq:broker> tag to false
  • To change JMX port, set connectorPort of the <amq:managementContext> to the appropriate value

[edit] Jobs

Jobs are the central part of the JQR system as they represent tasks that should be executed. This section covers all aspects of job development, deployment and configuration.

[edit] Development

All jobs in JQR are currently implemented in Java. Every class that implements org.sensatic.jqr.jobs.Job interface could be seen as a suitable JQR job. The Job interface is simple 

public interface Job {

	public void execute(JobDetails details) throws JobException;

	public void onQueue(JobDetails details);

	public void onSuccess(JobDetails details);

	public void onFailure(JobDetails details);

}

The interface defines four methods that should enclose actions to be executed when certain event occurs:

  • onQueue - defines action to be triggered when the job arrives in the queue.
  • execute - defines the actual main logic of the job
  • onSuccess - defines action to be triggered on job's successful execution
  • onFailure - defines action to be triggered on job's execution failure

These four methods defines a basic life-cycle of a job execution. The job is first queued in the system. When its time comes, it is executed. In case that execution fails for any reason, it could be re-executed number of times depending on the system configuration. After all execution attempts fail, job will be delivered to the dead job que where it can be examined by the application administrator.

[edit] Job Details

As you could notice in the previous section, every method of the Job interface method takes JobDetails class as an argument. This class is used to set up the context of the job execution. It holds parameters such as:

  • Job name
  • Job group
  • Job parameters

all these properties are defined in a client and passed to the jobs defined in JQR for the execution, as we will see in the moment.

[edit] Deployment

In order to be able to use your jobs, you have to provide them to the JQR's classpath. The easies (and recommended) way to do it is to place your jobs in a JAR archive and place the archive in the lib/jobs. After the application restart your jobs should be successfully deployed. All that is left to be done is to configure JQR to use them.

[edit] Configuration

Jobs are configured by the etc/jobs.xml file. It's a common Spring bean configuration file that initializes QueueFactory object, responsible for managing queues and jobs. Let's walk through the default configuration provided with the JQR distribution: 

        <bean id="queueFactory" class="org.sensatic.jqr.jobs.QueueFactory">
                <property name="queues">
                        <map>
                                <entry>
                                        <key>
                                                <value>testGroup</value>
                                        </key>
                                        <map>
                                                <entry key="testJob">
                                                        <bean id="testJob"
                                                                class="org.sensatic.jqr.jobs.TestJob"/>
                                                </entry>
                                        </map>
                                </entry>
                        </map>
                </property>
        </bean>

As you can see, jobs are grouped into groups. Each group basically defines a queue in which job will be handled. Process of making new queues and jobs is as simple as creating new map entries and beans (of course, beans have to implement Job interface). In the future, there will be more mechanisms avaliale for job configuration.

[edit] Off-The-Shell jobs

One of the main design goals of JQR is to be easy to use. Therefore, JQR comes with a set of jobs that could be used off-the-shelf and which represents some of the most common tasks. Also, its job package contains a few abstract classes which could help you easily develop more complex jobs.

[edit] AbstractJob

The org.sensatic.jqr.jobs.AbstractJob class implements Job interface and adds some basic logging functionality. If you want to use logging configuration described in the "Logging" section below. This class is probably the starting place of your future jobs.

[edit] HttpNotificationJob

As described earlier, the Job interface defines methods that should be executed on certain events related to the job. These methods are commonly used to send notifications back to client when certain job event occurs (such as successful job execution or execution failure). One of the methods you an use to send your notifications is by calling appropriate URLs back. The org.sensatic.jqr.jobs.HttpNotificationJob job allows you to define URLs that will be called when certain event occurs. It is done through the following properties:

  • onQueueURL
  • onSuccessURL
  • onFailureURL
  • executeURL

There are two things worth mentioning about this job. If a property is not configured (or is null) for any reason, no HTTP call will be made. So, you are free to extend this class instead of AbstractJob class, even if you are not planning to made HTTP calls at the time. As a matter of fact, HttpNotificationJob extends AbstractJob, so it will log all activities properly.

Another important thing is that all HTTP calls will be made using POST and all parameters passed to the job will be passed back to the application as POST call parameters. In future there will be support for more configuration options, such as HTTP method to be used and such.

[edit] SSL

If you need to make callbacks to HTTPS sites, you can use Java Secure Socket Extension (JSSE). In most cases all you have to do is:

  • use this utility tool to download the certificate from the desired host
  • copy generated jssecacerts file to $JAVA_HOME/jre/lib/security/

[edit] ShellJob

org.sensatic.jqr.jobs.ShellJob simply executes a shell command. It contains two configuration parameters:

  • script - defines the shell script to be executed asynchronously
  • backupScript - defines the shell script to be executed if the original script execution fails

Additionally, this job extends HttpNotificationJob class, so it could be configured to trigger responses through the HTTP POST method calls.

Here's the example of simple ShellJob configuration: 

<entry key="createHtgroup">
    <bean id="createHtgroup" class="org.sensatic.jqr.jobs.ShellJob">
        <property name="script"><value>/usr/local/test.sh</value></property>
        <property name="onSuccessURL"><value>http://example.com/success</value></property>
        <property name="onFailureURL"><value>http://example.com/fail</value></property>
    </bean>
</entry>

[edit] Logging

JQR uses Log4J as its logging subsystem. All logs are located in the data/ folder. After the first start of the application there will be the following files:

  • debug.log - this is a central debug log of the application. You should check it if you have problems in application behavior.
  • stdout.log - captures all text application tries to write to standard output
  • stderr.log - captures all text application tries to write to standard error. You should check this file in case application don't want to start properly

In case there is no additional configuration all jobs will write their messages to debug.log. In most cases, however, you will want your jobs to log in separate files. It ease the administration and help you find problems in job execution easily. Most of the jobs log messages by their group name. That means that they use specified log name for all jobs in one group and that name is the same as the name of the group. So in order to configure JQR to write all that messages to a separate file, you have to put the following configuration snippet in your data/log4j.xml file. 

<appender name="testGroupLog" class="org.apache.log4j.RollingFileAppender">
	<param name="File" value="data/test.log"/>
	<param name="MaxFileSize" value="10240KB"/>
	<param name="MaxBackupIndex" value="9"/>   
	<layout class="org.apache.log4j.PatternLayout">
	<param name="ConversionPattern" value="%d %p [%c] - <%m>%n"/>
	</layout>
</appender>		
	
<logger name="testGroup" additivity="false">
	<level value="info"/>
	<appender-ref ref="myJobGroupLog"/>
	<appender-ref ref="emailLog"/>
</logger>

This configuration snippet instructs JQR to put all log messages from jobs in data/test.log file.

[edit] Clients

Once jobs have been implemented, configured and deployed, we can use client code to trigger their asynchronous execution. Below is a list of clients implemented in different programming languages and examples of their usage.

[edit] PHP

[edit] Installation

Once you have downloaded PHP client distributed with the JQR release (sensatic-jqr-client-php-x.x.x.tar.gz), extract the archive somewhere in your web application directory. Note that distribution contains two folder: main and test. The main folder contains classes that implement PHP client and that folder must be in the include path of your application. The test folder contains a few scripts which tests and demonstrates this client. You can look at executor.php script as an example and use it to start your development.

[edit] Usage

There are two main classes of this library:

  • JobExecutor - is a client class that communicates with the server. It is base on Zend REST library and needs to be configured with the appropriate server URL.
  • Job - holds details, such as name, group and parameters of a job that has to be executed.

Here's the code sample of their usage: 

require_once('Job.php');
require_once('JobExecutor.php');

$exec = new JobExecutor("http://localhost:25256/jqr/");

$job = new Job();
$job->group = "testGroup";
$job->name = "testJob";
$job->params->firstname = "Dejan";
$job->params->lastname = "Bosanac";

When executed, this code sample executes testJob which is configured in default configuration.

[edit] Management

[edit] Web Console

ActiveMQ web console is integrated into JQR and could be used by default at the following URL: http://localhost:25256/admin/

[edit] JMX

You can use any JMX console, such as jconsole, to monitor and control JQR. Default configuration registers JMX to port 1099, so you can use the following URL, service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi, to access it.

[edit] JMS

Because JQR uses JMS architecture as its infrastructure, ActiveMQ implementation in particular, you can use browser tools such as Hermes JMS to browse queues.

Retrieved from "http://www.sensatic.org/jqr/Documentation"