Abstract
This paper demonstrates creating a flexible, extensible, plug-in-based .NET Application (JOM - Smart Job Manager). JOM is an asynchronous job processing engine built using the MS Provider design pattern and the .NET technology available today. Technologies, libraries and patterns used here are .NET Framework 1.1, C#, MSMQ, Enterprise Library, MS Provider Model design pattern, .NET Windows Service and .NET Windows Application.
Introduction
Asynchronous processing is desirable in many situations, for example, in a hospital the discharge summary needs to be printed for the patient along with email, FAX and SMS being sent to the Doctors, Clinicians notifying the discharge of the patient. All these tasks require processing time but you do not want the patient to be waiting for all these tasks to be completed. In this situation, you would implement an asynchronous process for FAX, EMAIL etc. so that you can concentrate on discharging the patient as quickly as possible. Smart Job Manager Framework is designed with this asynchronous scenario in mind. Smart Job Manager (JOM) runs as a Windows Service and the client applications will queue the jobs to Microsoft Message Queuing (MSMQ). JOM is developed using the MS Provider Design Pattern which gives the framework flexibility, extensibility and plug ability on the fly. JOM receives the job from the queue and allocates a proper Provider to accomplish it asynchronously.
Introducing the patterns and technologies used in Smart Job Manager
If you are already familiar with the MS Provider Design Pattern, MSMQ and Enterprise Library you can skip this section.
MS Provider design pattern
MS Provider design pattern helps to abstract the application from specific provider dependency and makes the application extensible and flexible. An abstract base class holds properties and methods required to be implemented in the concrete provider API. Also the description of the provider is defined in the configuration file, which allows the plug-in implementation of the API. Have a look at the ASP.NET Provider Toolkit where Rob Howard describes about the provider model and its specifications.
MSMQ
JOM uses MSMQ to receive a task and adds it to the queue. Message queuing provides a kind of store-and-forward messaging in a pre-built infrastructure that helps to address the asynchronous scenarios. Message queuing provides guaranteed message delivery, efficient routing, security, and priority-based messaging. The .NET Framework provides access to all message queuing operation out of the box via the System.Messaging.dll.
Enterprise library
The current buzz in the .NET community surrounds the Enterprise Library from Patterns and Practices Group, which ships with seven reusable application blocks caching, configuration, cryptography, data access, exception handling, logging and instrumentation, and security. The aim of the application blocks is to reduce development cost and increase confidence. It took me only couple of hours to add functionalities like caching, exception handling and logging features into JOM and for this the June 2005 release of the application blocks was used. These application blocks are great tools that enable us to maintain consistency and efficiency in our applications. Using the application blocks makes our programs easier to write as it follows similar patterns throughout and the common tasks are encapsulated into easy to use classes and static methods.
Smart Job Manager
JOM facilitates communication between two or more dissimilar systems that are not in sync. JOM runs as a Windows Service, receiving jobs from MSMQ and dynamically assigning providers to accomplish the jobs.
Fig. Smart Job Manager workflow
To understand the JOM Framework we need to understand the role of the Job
object and the providers. Job
is a business object responsible for passing provider specific data with the provider name from the client to the queue. JOM Windows Service seeks the message containing the Job
object in the queue every 5 second. On the arrival of a job, JOM reads it from the queue and assigns it the correct provider to do the job. Providers normally implement all the codes that they want to accomplish on a given job for example, a provider can send a FAX. Remember a provider also specs out what it expects from the Job
object to correctly perform the job. Client application should be aware of the Job
object's specification before creating them and putting them in the queue. An example may be, to send a FAX provider it may need to know the faxNo and the Job
object should contain the faxNo.
Job object in the JOM Framework
Job
object is simply a business object that can be used commonly across different systems. The Job
object contains the data that the provider expects for a specific task and the Job.JobDetailCollection
property contains the job specific data. The Job
object should also specify which provider to use in the JobManagerProviderName
property. For example, to send a FAX the Job
object will contain the faxNo
in the JobDetailCollection
. The JobDetail
object acts a name/value pair and JobDetailCollection
acts as a name/ value pair collection.
Fig. Business object: Job
, JobDetail
, JobDetailCollection
A client application simply needs to create a Job
object and send it to the specified message queue:
Job job = new Job(0,0,DateTime.Now,
DateTime.Now,"SimpleJobManagerProvider");
JobDetailCollection jobDetails = new JobDetailCollection();
JobDetail jobDetail = new JobDetail(0,0,
"Copyright","Use at your own risk.");
jobDetails.Add(jobDetail);
jobDetail = new JobDetail(0,0,"PersonName","Shahed Khan");
jobDetails.Add(jobDetail);
jobDetail = new JobDetail(1,0,"Email","shahed.khan@gmail.com");
jobDetails.Add(jobDetail);
job.JobDetailCollection = jobDetails;
QueueJob(job, QueuePath);
In the above code, we simply created a Job
object specifying the provider and in this case we used SimpleJobManagerProvider
. We also assigned Job.JobDetailCollection
with the data such as PersonName, Email etc. and queued the job in the message queue (MSMQ) by calling the QueueJob
method. A client application only needs to create a Job
object with the proper JobDetailCollection
and the correct ProviderName and pass it to the message queue (MSMQ). The JOM Framework will take care of the rest. Job
and JobDetail
has a 1:N relation and you will notice both the Job
and the JobDetail
have an ID
property. The JobDetail
also has a JobID
property. This is done to facilitate easy incorporation of the Job
and the JobDetail
object in a DB environment.
An implementation of the provider pattern in JOM
As JOM implements the provider pattern it is extensible, flexible and provides a plug-in based implementation of the provider APIs. The provider pattern is used here to overcome the limitations of a predetermined provider API and predetermined functionalities. Instead, it is designed to adopt any provider API that implements the JobManagerProvider
.
Let's walk through a simple example demonstrating how this pattern has been implemented in JOM.
Fig Smart Job Manager and the provider pattern
public abstract class JobManagerProvider : ProviderBase {
public abstract bool DoJob(Job job);
}
The abstract
base class JobManagerProvider
has the method DoJob(Job job)
which is required to be implemented on concrete providers. For example, in the demo application we have implemented a couple of concrete providers such as TextToFileJobManagerProvider
, TextToImageJobManagerProvider
, and SimpleJobManagerProvider
:
public class SimpleJobManagerProvider : JobManagerProvider {
public override bool DoJob(Job job){
StreamWriter sw = new StreamWriter(string.Format("{0}{1}{2}{3}",
OutputFilePath, "SimpleJob",
DateTime.Now.Ticks.ToString(), ".txt"));
sw.AutoFlush = true;
sw.WriteLine(job.JobManagerProviderName);
foreach(JobDetail jobDetail in job.JobDetailCollection){
sw.WriteLine(string.Format("{0}: {1}",
jobDetail.ElementDesc, jobDetail.ElementData));
}
sw.Close();
return false;
}
}
In the above code, the SimpleJobManagerProvider
inherits from JobManagerProvider
and implements the DoJob( Job job)
. In the above code, this provider is writing the contents of the Job
into a text file.
JOM decides on dynamically loading the provider API based on the provider name passed to it via the Job
object and runs the DoJob
method:
private void OnMessageArrival(IAsyncResult ar) {
MessageQueue mq = (MessageQueue)ar.AsyncState;
Job job = new Job();
try{
Message msg=mq.EndReceive(ar);
job = (Job) msg.Body;
JobManager.DoJob(job);
}
catch(Exception e){
EntLibHelper.JobException("OnMessageArrival",job,e);
}
finally{
mq.BeginReceive(TimeSpan.FromSeconds(5),mq,
new AsyncCallback(OnMessageArrival));
}
}
The above code shows the Job
object is derived from the queue on the arrival of a message and the JobManager.DoJob(job)
is called. Exceptions are logged via Enterprise Library Logging App Block:
public class JobManager {
public static bool DoJob (Job job){
JobManagerProvider provider =
JobManagerProvider.Instance(job.JobManagerProviderName);
return provider.DoJob(job);
}
}
In the above code we see that the JobManager
acts as a factory to decide on which provider to load dynamically. Job
object should pass the JobManagerProviderName
to be instantiated. The methods in the JobManager
class are static
, so we can simply call JobManager.DoJob(Job job)
instead of having to instantiate JobManager
before use.
Configuration information: As described in the MS Provider specification after the concrete provider is implemented, it must be described in the configuration section. The beauty of the provider pattern is that the appropriate provider is instantiated at run time from the information that is contained in the configuration file and you can define an unlimited number of providers.
<smartJobManager.providers>
<jobManager defaultProvider="SimpleJobManagerProvider">
<providers>
<add name = "SimpleJobManagerProvider"
type = "SmartJobManager.SimpleJobManagerProvider,
SimpleJobManagerProvider"/>
</providers>
</jobManager>
</smartJobManager.providers>
In the above section, we have added the SimpleJobManagerProvider
to our providers list. You can add as many as you wish. For the complete specification of the configuration file see ASP.NET Provider Toolkit.
For a new provider to be implemented, we simply need to write a new provider extending from JobManagerProvider
and add the provider specific information to the configuration file and that’s it.
Use of Enterprise Library in the JOM Framework
JOM uses Enterprise Library June 2005 for caching, logging and exception handling. This facilitates the JOM to easily transform these features according to your needs. The following code block demonstrates how caching, logging and exception handling is implemented in JOM.
Caching of providers in JOM
Providers are added to the cache as the reflection used later is expensive:
Type type = null;
string cacheKey = null;
JobManagerConfiguration config =
JobManagerConfiguration.GetConfig();
Provider jobManagerProvider =
(Provider) config.Providers[providerName];
cacheKey = "Job::" + providerName;
if (EntLibHelper.GetCachedObject(cacheKey) == null){
try {
type = Type.GetType( jobManagerProvider.Type );
Type[] paramTypes = System.Type.EmptyTypes;
EntLibHelper.StoreInCache( cacheKey,
type.GetConstructor(paramTypes) );
}
catch (Exception e) {
throw new Exception("Unable to load provider", e);
}
}
object[] paramArray = new object[0];
return (JobManagerProvider)(((ConstructorInfo)
EntLibHelper.GetCachedObject(cacheKey)).Invoke(paramArray) );
Logging in JOM
For logging, simple flat file sink that ships with Enterprise Library is used. You can transfer easily to a DB sink or any of your customized sink:
try{
Message msg=mq.EndReceive(ar);
job = (Job) msg.Body;
if (job != null){
EntLibHelper.JobInfo("Recieved New Job",job);
if (!JobManager.DoJob(job)){
EntLibHelper.JobError("JobError Occured",job);
}
}
}
catch(MessageQueueException e) {
if (e.MessageQueueErrorCode !=
MessageQueueErrorCode.IOTimeout){
EntLibHelper.Exception("OnMessageArrival",e);
}
}
Exception handling in JOM
All exceptions are logged using a flat file sink. You can modify according to your need:
catch(Exception e){
EntLibHelper.JobException("OnMessageArrival",job,e);
}
Enterprise Library configuration
Fig Enterprise Library configuration screen
How to use Smart Job Manager in your own application
JOM steps
Implement a concrete provider inherited from JobManagerProvider
similar to the SimpleJobManagerProvider
used earlier. You will find a couple of providers in the demo application. Some example providers can be written to print a file, send a FAX, send an email, log into database, transfer a file etc.
Decide what you will be passing via the Job
and the JobDetailCollection
objects. For example, to send an email via the provider you might need to send the email address using the Job
and the JobDetailCollection
objects.
In your provider, include a DoJob
method and write the codes. This method should do the task you wish. Compile and copy your newprovider.dll to the bin folder of JOM. Modify the configuration file and add a new key for your provider as it is demonstrated in the configuration section above.
Client application steps
Prepare a Job
object by specifying the provider name, create JobDetailCollection
with the data specified by the provider and send it to the message queue from your client application. JOM should pick it up and process it asynchronously. Now, modify the Enterprise Library specific configuration.
Comparing with "Distributed system design with command pattern"
Brad King in his article Simplify Distributed System Design Using the Command Pattern, MSMQ, and .NET takes a different approach by passing a command object to the queue:
Fig Distributed system with command pattern workflow
The good thing about this approach is it’s strongly typed and it can apply validations on business objects. But the limitation is both the client and the invoker need to know the original definition of every command object. So all the command objects should be residing in both the client app and the invoker app. However in JOM we used a common business object Job
which is common for all applications. The client is not aware of the provider as the client only knows the name of the provider and what to pass via the Job
object. JOM hides the providers (business logic) from the client completely.
Fig. Smart Job Manager workflow
Future enhancements
Here in the demo application simple recoverable messages are sent to MSMQ. However you might want to implement reliable messages, create transactional queues, and implement acknowledgement, timeout and journaling on the messages. Also this application can be designed so that you can manage the configuration from the Enterprise Library UI.
The code snippet demonstrating how the Job
is queued in MSMQ in the demo application:
private static void QueueJob(Job job, string destinationQueue){
try{
MessageQueue mq = new MessageQueue(destinationQueue);
mq.DefaultPropertiesToSend.Recoverable = true;
mq.Formatter = new BinaryMessageFormatter();
mq.Send(job, "Job Message Test");
mq.Close();
}
catch (Exception e){
Console.WriteLine(e.ToString());
}
}
This code reads messages from the queue:
try{
Message msg=mq.EndReceive(ar);
job = (Job) msg.Body;
JobManager.DoJob(job);
}
Demo app
A full demonstration of the JOM workflow is provided in the demo application. Demo includes a number of concrete providers implemented from JobManagerProvider
, JOM Windows service and the client application.
Providers
TextToFileJobManagerProvider
: Creates a text file with end user specified text.
TextToImageJobManagerProvider
: Creates a JPEG image file with end user specified text.
SimpleJobManagerProvider
: Creates a text file with some predefined text.
ScreencaptureJobManagerProvider
: Captures the screen where the client application is running and creates an image file.
JOM Windows service
JOM Windows service hosts all the providers mentioned above. If you want to host your own providers with JOM simply follow the instructions discussed earlier. JOM seeks new jobs in the MSMQ every 5 sec.
Fig. View and start the .NET Job Manager Service with computer management tool
Fig. View queued jobs with the computer management tool
Client application
The client application has the interface to send jobs to the queue. This can create jobs for all the providers mentioned above:
Fig. Smart Job Manager workflow
Checklist to run the demo application
View the Readme.txt file provided with demo.
Conclusion
Smart Job Manager is architected using provider pattern which makes it flexible, pluggable and extensible. Also the use of Job
object from all the clients introduces common communication platform among all the applications. This also hides all the business logic implemented in the providers. JOM fits in any scenario where asynchronous task is desirable. Use of MSMQ in the JOM Framework makes it reliable and recoverable. And the implementation of Enterprise Library based logging, caching and exception handling allows easy accommodation of this (JOM) with other applications in the enterprise environment.
Special Thanks to Christopher Heale for proof reading.
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.