Connecting to Tibco EMS in Mule using JMS Connector

In this article, I will brief about messaging interfaces using JMS connector in Mule. We will be connecting to Tibco EMS Server.

Queues and Topics:

• Point-to-Point(Queues): Message delivered to one recipient.
• Publish Subscribe(topics): Message delivered to multiple recipients. 

Let create a new mule project and add the following references to the projects as shown below.

Lets not  create a simple flow to read a message from the queue and write it to file system. Drag and drop the JMS connector and then file activity as shown and then configure the JMS connector as shown below,

Configure the connector configurations as shown below,

Configure the JMS to poll message from queue and then file activity as shown; Run the mule app to verify the behavior. Make sure Tibco EMS Server is up and running.

Using GEMS tools, we would connect to the EMS server and then push a message to tibcodj.queue as shown.

Once the message is available in the queue, as the mule flow is polling to the same queue, this message would be picked up and sent to file system.

This flow would pick up all the message that arrives at this queue. Lets add a JMS filter to select a type of message based on the specific property of the message. From GEMS tool, lets a custom property to the message as shown below, we add a ServiceType Property and provide as Cab.

In the message flow, go to the configuration tab and add JMS Selector filter as shown and run the project to verify the behavior.

Now lets connect to a Topics using JMS connector, as earlier we would configure another JMS connector to connect to Topic.

Here is the simple, which polls on a file directory and then push the text available in the file to a EMS topic. Flow configurations are shown below,

Run the project and push a message from GEMS to a queue,

1st flow would create a dat file in the directory and as 2nd flows polls this same directory the information in file is picked up and processed to the Tibco EMS topic, as  shown below.

Now lets try to add a subscriber flow, as shown

Now in the 2nd flow i.e topic publisher lets add a message property to message using message properties and sends to the EMS topic and the subscriber i.e 3 flow, subscribes to message based on the property value as shown below,

Performance Tuning In Mule ESB

In this article, I would brief on tuning mule application through threading profile. Fine tuning can be performed at various stages.

1. Design Phase Tuning
1. Tune Mule’s flows.
2. Tune Mule’s configuration settings.
2. Runtime Environment Tuning
1. Tune the Java Virtual Machine (JVM). Xmx, Xms
2. Tune the garbage collection (GC) mechanism. CMS for API’s
3. Operating System Tuning
1. Tune the ulimit.
2. Tune the TCP/IP stack.

Tune Mule’s configuration settings:- Conceptually, messages are processed by flows in three stages:

• The message being received by the inbound connector
• The message being processed
• The message being sent via an outbound connector

Synchronous processing, the same thread is used to carry the message all the way through Mule.

Asynchronous processing, the receiver thread is used only to place the message on a staged event-driven architecture (SEDA) queue, at which point the message is transferred to a flow thread, and the receiver thread is released back into the receiver thread pool so it can carry another message.

Mule Design time tuning 

Threads in Mule:

• The receiver thread, which originally receives the message, either:
• Processes the entire flow (synchronous), or Ends by writing the message to a SEDA queue (asynchronous)
• The flow thread, which processes the bulk of the flow (asynchronous)
• Dispatcher threads, which send messages to one-way endpoints (asynchronous)

Threading Profile Configuration

• Configuration level 
• Connector level 
• Flow level Flow-Processing Strategies

Configuration Level Threading Profile Setting:-

Default Threading Profile

Connector level:-

VM transport is an excellent mechanism to help break your flows into smaller, composite flows that are decoupled with in-memory queuing. This allows you to realize the benefits of SEDA at the macro level, breaking your application into smaller flows that can be individually tuned and are decoupled with VM queuing.

Flow level Flow | Processing Strategies:-

Processing Strategies in Mule ESB | Synchronous verses Asynchronous Processing Strategy

A flow processing strategy determines how Mule implements message processing for a given flow. All Mule flows have an implicit processing strategy which Mule applies automatically: either synchronous or queued-asynchronous.

Even if you do not configure a processing strategy for your flow, the flow follows a processing strategy automatically applied by Mule. The processing strategy can be:

• Synchronous Flow Strategy : Mule processes messages in a single thread.
• Queued-Asynchronous Flow Processing Strategy : Mule uses a queue to decouple the receiver thread from the rest of the flow.

The following table summarizes how Mule chooses a flow processing strategy:

Synchronous Flow Processing Strategy: This is used  to process messages in the same thread that initially received the message. After the flow receives a message, all processing, including the processing of the response, is done in that same thread.

Non-Blocking Processing Strategy: This uses an evented non-blocking processing model to process requests. A single thread still handles each incoming request, but non-blocking components return this thread to the listener thread pool. Only upon obtaining and using a new thread, can processing continue.

The following Non-Blocking strategy limitations:

• Only Message Source that supports non-blocking is the HTTP Connector Listener.
• Only Message Processor that supports non-blocking is the HTTP Connector Request.
• Only Message Exchange Pattern currently supported is Request-Response.
• Only a limited set of components are fully supported for use in non-blocking flows

Synchronous Fallback: If an unsupported component is configured in a flow that is using the non-blocking processing strategy, then the flow reverts to the synchronous processing from the point in the flow. There may be advanced cases where this is desirable to take advantage of non-blocking for part of the flow, but typically it is best to stick with the synchronous processing strategy.

Lets build a flow and specify a Non-blocking processing strategy, if processing strategy isn't set flow will default it to synchronous flow strategy

Choose the message flow and choose appropriate processing strategy from processing strategy, as shown

Processing Strategy Ref option allow use to define the thread setting and pool exhaust behaviors.

Queued-Asynchronous Flow Processing Strategy: This uses a queue to decouple the flow’s receiver from the rest of the steps in the flow.

Test and verify the behavior.

Design Mule API Portal

This is in continuation with the previous tutorial on API designer. Here will build a portal for our API using Portal. Login into anypoint platform studio. Open API Manager then Open Mule API 1.0 and in API Portal Tab, Choose Create New Portal from the dropdown.

We can change the title of the page and then provide approperiate description of the api and even upload image using the options available. Set the portal as public and page as visible.

Interface allows us to add other pages using Add option. Lets go to API reference and make it as visible page.

Click on Live Portal to view the designed Portal for our API. In API Reference page, we can test the api operations as we did in API designer

Introduction RESTful API Modeling Language (RAML) | Mule API Designer

In this walk-through, we will define api definition using RAML in Mule API designer.

Login in to anypoint platform and using API Manager create new API as MuleAPI.

Open the Mule API 1.0 and in APIDefinition Tab, click Define API in API Designer.

In the designer, we can provide the base path and then add the relevant method it supports. As shown below, we defined a get operation and response with status code and response type either json or xml and we provided sample xml response for the operation.

In the resources tab, provides mock description of the operation with example. This operation can even be tested, if mocking service is enabled.

Here is actually xml response of the operation and similarly, other operations are add to support the features of Restful Service.

Resources tab, shows complete set of operation that it supports.

All the operation definition is shown below, examples shown are mocked up xml content 

Other example of API Designer using RAML shown the feature of supporting xml and json type of messages

Sample RAML for easy understanding, example json and xml has been removed for better alignment.

#%RAML 0.8

baseUri: https://mocksvc.mulesoft.com/mocks/af7204ad-81ec-462c-b3b8-f58485a4bfc0

title: MuleAPI

version: 1.0

/Orders:

  get:

    queryParameters:

      Event:

        enum: [101, 201, 501]

    responses:

      200:

        body:

          application/xml:

            example:

              ""

          application/json:

           example: |

                            

  post:

    body:

      application/xml:

        example: 

          ""  

      application/json:

        example: |

          

    responses:

      201:

        body:

          application/xml:

            example: 

              "" 

          application/json:

            example: |

              

  

  /{ID}:

    get:

      responses:

        200:

          body:

            application/xml:

              example:

                ""

            application/json:

              example: |

                

Custom Tracking in Mulesoft

In this walk through, we would explorer on how to track business milestones and information in  mule application. 

Let develop a simple mule flow that receives a order request in json and does something processing and acknowledges back the client with ack. 

 In order to track certain business events, we can use custom tracking components,lets use a message enricher to do the custom tracking, this would avoid unnecessary updates to original payload content. Inside message enricher, we will have processor chain which would contain Json To Object transformer, Custom Tracking and a Logger.

Drag and drop a Custom Business Event and configure the metadata from the properties as shown below

 Here is the complete mule flow

Run the project and test the mule application, Business Event aren't logged in the console but these are visible in MMC Business Event Tab.

Batch Processing in Mulesoft | ETL

Lets understand why batch processing is required and how this can be achieved in mulesoft. Batch processing is particularly useful when working with following scenarios:

• Handling large quantities of incoming data from APIs into legacy systems.
• Extracting, transforming, and loading (ETL) information into the destination system (i.e., uploading CSV or flat file data into the Hadoop system).
• Engineering near real-time data integration (i.e., between SaaS applications).
• Integrating datasets — small or large, streaming or not — to parallel process records.

Batches elegantly handle any record-level failures that might occur in processing so as to prevent failure of a complete batch job.

A batch job is a top-level element in Mule which exists outside all Mule flows. Batch jobs split large messages into records which Mule processes asynchronously in a batch job; just as flows process messages, batch jobs process records.

A batch job contains one or more batch steps which, in turn, contain any number of message processors that act upon records as they move through the batch job. During batch processing, you can use record-level variables (recordVars) and MEL expressions to enrich, route or otherwise act upon records.

A batch job executes when triggered by either a batch executor in a Mule flow or a message source in a batch-accepting input.

When triggered, Mule creates a new batch job instance. When all records have passed through all batch steps, the batch job instance ends and the batch job result is summarized in a report to indicate which records succeeded and which failed during processing.

In this walk through, we will build a mule application that fetches data from database and process those in batch steps and loads valid records in a destination table.

At the end of walk-through, we would build the below mule application.

Now lets bring in a batch scope and drag database and set payload inside the batch scope input area and configuration are shown below,

To avoid unnecessary changes on the payload message enricher is used, so that information is stored in variable. 1st enricher  simply updates the payload this to read the loan id to record variable and in the 2nd enricher, the status of the record is updates in DB table and the result is stored in record variable.

Lets bring in choice router and if the loan type is termed as invalid, we would stop the process by throwing an exception using groovy component. On the default flow, we log the LoanID.

As a final step of batch, we will insert the valid loan records to another table using DB connector as shown below and on completion phase we log the payload.

Now lets build another simple mule flow that invokes the batch using batch execution.

Lets run the project and test the application. Note, Batch Scope Max Failed Records is set as -1. Let load the data in source LoanApp table as shown and target LoanDetails table would be loaded by the batch job invoked by mule application.

Lets send a Get request in POST-Man as shown, this would invoke the job and each records would process and the batch result is rendered back to client.

Flow logs show, out of 4 records 2 processed successfully and 2 failed.

Lets modify the flow to handle failure records, bring a batch step before completion tab and configure the batch step property to process failure records and place set payload and for each scope as shown. In order to access the failure instances, we need to use #[getStepException] and loop through each failure using  #[payload.values()].

Inside For Each scope, add a logger for log loanID and then using DB connector, we would update the ProcessFlag to E indication exception occurred.

Lets re-run the project and test it. From Post-man, lets place a get request and result is rendered back as shown below and the batch process updated the ProcessFlag approperiately and loaded the LoadDetails table for valid loan types. Before placing a request, as you can see, ProcessFlag is set to N in LoanApp table and LoanDetails Table has been truncated.

Entries of source and target table before and after the process along with flow logs are shown below.