Tibco EMS from zero to hero

Intro

In an enterprise environment, there are lots of applications that need to talk to each other, there are enterprise patterns that allow such scenarios. One of such patterns is messaging. TibcoEMS could be utilized to implement messaging patterns and establish communication channels between multiple applications in your enterprise.

I would try to cover basic aspects of TibcoEMS (queues, topics, durables, connection factories), permission model, bridging and routing it should be enough for more than 90% of use-cases. If you want to go deep then official documentation is your best friend.

Please note, this article is not intended to replace the official guide but rather to describe the key concepts and give the reader as fast as a possible introduction to the topic.

Obtaining TibcoEMS

Navigate to Tibco Messaging / Downloads and search for “TIBCO Enterprise Message Service”. Accept the license agreement and download the version for your operating system.

I will be using the Community Edition version for Windows in my examples.

To start the broker please navigate to the folder where you have installed TibcoEMS and run tibemsd.exe located in the bin folder.

$ tibemsd.exe

If you run tibemsd.exe for the first time Windows Defender window would popup as TibcoEMS broker would want to listen on TCP 7222 port on all interfaces (ipv4 and ipv6).

In the console window, you will see messages from the Tibco EMS daemon.

Messaging basics

Messaging systems usually have message producers and message consumers that are “exchange” messages. Also, there are several ways how messages could be sent, but from a high-level perspective there are three:

• one producer - one consumer
• one producer - many consumers
• many producers - many consumers

Message model

Tibco EMS is compliant with JMS specifications, so we would look at JMS quickly to understand the concepts. JMS is a Java application programming interface (API) for message-oriented middleware. It provides generic messaging models, able to handle the producer-consumer problem, that can be used to facilitate the sending and receiving of messages between software systems.

JMS supports the following models:

• point-to-point (queues)
• publish and subscribe (topics)

Point-to-point communication

There is one producer for the message and one consumer. There might be multiple producers sending data to a single queue, and there might be multiple consumers reading from the queue, but every message could be processed by only one consumer. All messages are stored in the queue until they are received (this look very similar to a durable topic subscription).

Publish and subscribe communication

In this model producer (we can call it publisher here) is sending a message to the broker and broker broadcasting the message to all consumers (we can call them subscribers). There might be multiple producers sending data to a single topic. Every message would be broadcasted to all consumers subscribed to this topic.

Message acknowledgement modes

It is important to understand the acknowledgement mechanism and behaviour as it would affect the delivery process, duplicate message appearance, and overall performance. There is no silver bullet here and in every scenario, we need to select appropriate behaviour based on the requirements.

1. Message producer sends the message to Message Broker
2. Message broker sends confirmation (for persistent messages)
3. Message consumer receives the message from Message Broker
4. Message consumer sends an acknowledgement
5. Message broker might notify Message consumer about receiving of acknowledgement

There are a set of session modes (defining the acknowledgement behaviour) for non-transacted sessions:

CLIENT_ACKNOWLEDGE

consumer confirm that all messages delivered so far within a session are acknowledged. This is an explicit call of messaging API.

AUTO_ACKNOWLEDGE

session on the consumer side auto-confirm every message received. No explicit API call is required.

DUPS_OK_ACKNOWLEDGE

session is in “lazy” mode to acknowledge received messages. The consumer may delay acknowledgement until a convenient time. A broker might redeliver messages or send duplicates if a client was not reachable.

NO_ACKNOWLEDGE

once the message is sent to the client all information about it is destroyed on the broker side.

EXPLICIT_CLIENT_ACKNOWLEDGE

a consumer may acknowledge a particular message (not all messages received so far). Useful in case of inserting messages to the database one by one or in chunks.

EXPLICIT_CLIENT_DUPS_OK_ACKNOWLEDGE

similar to standard mode DUPS_OK_ACKNOWLEDGE, but acknowledge possible for individual messages (not all messages received so far)

In my practice I was using the following types:

• CLIENT_ACKNOWLEDGE - in cases when I need to consume messages and store them in batch mode to the database. For example, we receive 1000 messages and then store them into the database table in a single transaction, then acknowledge once to confirm that we have processed all received (1000 messages). The code would block and wait for transaction completion.
• EXPLICIT_CLIENT_ACKNOWLEDGE - in the case when every message cause massive calculations in the system and some messages could be processed quickly (simple messages, processing time around 1 sec), and some could be processed slowly (complex messages, processing time might be 5-60 sec), as we don’t care about the order of messages here, we acknowledge every message once processing is complete independently. Such behaviour could not be simply implemented using standard JMS session modes and it is available only in TibcoEMS.
• NO_ACKNOWLEDGE - if data is streamed to you and you don’t care about missed messages or retransmission, it works like UPD protocol but for messages.

Topics

The topic is the broadcasting mechanism in TibcoEMS, which means all the messages coming in the topic (producer side) would be multiplexed to all consumers (subscribers).

There are two types of subscription to the topic:

• standard consumer
• durable consumer

Creating topic from the server console

Syntax:

create topic <topic name>

Example:

create topic mytopic

Standard consumer

The standard consumer would receive a message only if it is currently connected to the topic, which means in case of network issues or maintenance works on the consumer side all copies of the messages would be lost. Depending on your use case such loss of messages might be acceptable if you care about only real-time data and you don’t care about the historical view. On the other hand, if you want to be able to consume all the messages later, let’s say after network connectivity have been restored, you would need to switch to a durable topic subscription. Other online consumers would receive their copies of original messages independently from disconnected one.

Durable consumer

Durable subscribers allow to decouple producer and consumer in time, in this case, all messages generated by the producer would be stored in a special place called “durable” and they would be accumulated over time in case a consumer is disconnected for some reason.

The creating durable subscription would either create a new durable subscription (user should have durable permission) or would attach a client to previously created durable subscription (user should have use_durable permission).

Creating durable from the server console

Syntax:

create durable <topic name> <durable name> clientId=<client ID> 

Example:

create durable mytopic dur.client1.mytopic clientId=client1

Please note that durable name has clientId encoded as part of the name, such convention allows you quickly understand the relationship between durable, topic, and clientId

Shared subscription

In TibcoEMS 8.x new mode has been introduced - shared subscription for topics. This mode allows sharing work (within-group) while subscribing to the topic. This functionality is similar to Kafka consumer groups. The shared subscription works like a queue connected via bridge to an upstream topic, but the client uses topic API.

Static and Dynamic durable subscriptions

In most cases, a number of applications in an enterprise environment is more or less static which means it is possible to define all durable subscriptions upfront to control access to the data. In this scenario, an administrator could define all objects at the time of onboarding (topic, clientId, durable, connection factory) and restrict access to the topic. Such a setup, when everything is defined upfront, is called a static subscription because all objects already exist.

On the other hand, there are scenarios when a number of consumers could be dynamic by nature, so it would be not possible to create all things upfront. In this case, the client can request the broker to create dynamic durable for his clientId on a fly, if it is allowed by permission broker would generate new durable for the topic and messages would start to accumulate in case of client disconnect for some time. If you use dynamic durable subscriptions you need to monitor closely what is happening with the broker and there should be an automatic housekeeping mechanism implemented to get rid of unused durable subscriptions from time to time.

I highly recommend using only static durable subscriptions, otherwise, there are high chances that misconfigured clients would connect and create dynamic durable subscriptions and leave it forever, so messages would infinitely accumulate thus affecting the whole broker.

Queues

The queue is the mechanism of delivering messages one by one, which means messages are not duplicated for every consumer attached to the queue. There are several use-cases for queues:

• you have only a single consumer that should process messages in a particular order without losing it while disconnected (technically you can archive the same with a topic with durable subscriber)
• you have several consumers, and these consumers would split the work between each other, this is effectively a round-robin load balancing

Bridges

Bridge allow you to connect destinations (topics and queues) creating more complex topologies. The bridge is also support filtering, so it is possible to filter for every consumer independently carving out messages from the single source based on consumer requirements.

Creating bridge from the server console

Syntax:

create bridge source=<destination type>:<name> \
              target=<destination type>:<name> \
              [selector=<selector>]

<destination type> is topic or queue

Example:

create bridge source=topic:mytopic target=queue:myqueue selector="category<>'Red'"

Connection factories

Connection factory is a special object in TibcoEMS that map factory names to connection properties, this is JNDI (Java Naming and Directory Interface - is an API that provides naming and directory functionality to applications written using the Java programming language).

Why to use connection factories:

• You have many producers/consumers and you want to enforce naming conventions
• You want to segregate connection management (to do load balancing) and data management (use different brokers for different data types)
• You want to enforce secure connections via SSL

My recommendation is to use connection factories in all cases because it could enforce naming conventions and you would have consistent configuration (in case you would need to scale your setup).

Creating connection factory from the server console

Syntax:

create factory <factory name> <destination type> \
               url=<broker urls> clientId=<client Id>

<destination type> in this case is topic, durable, or generic

Example (create connection factory to set clientId):

create factory factory.client1 topic url=tcp://servername:7222 clientId=client1

There is a short form while referencing the same server, instead of specifying server name with protocol and port like url=tcp://servername:7222 you can use url=tcp://7222.

Example (create connection factory to enforce SSL connection):

create factory MySSLFactory generic url=ssl://7243 ssl_verify_host=enabled

Routing

In distributed scenarios when we have multiple locations, we can have brokers deployed to every location. To allow applications in different locations to talk to each other routing could be configured on all brokers to create a delivery network. Route forward messages only between global destinations (queues and topics), so to make routing work we need to set up global property on both servers for destination (>addprop topic mytopic global).

Routed queues have more limitations than topics, so please check the documentation before using it.

Access Control Lists

Access Control Lists (ACLs) allow to restrict access to different objects. Conceptually there is a destination (topic, durable, …), subject (user, group), and permission.

Permissions for topics:

• publish - allows to publish messages on topic
• subscribe - allows to create non-durable subscribers
• durable - allows to create durable subscribers (does not imply subscribe permission)
• use_durable - allows to create durable subscribers if and only if the specified durable already exists in the server. (does not imply subscribe permission)
• all - all user permissions

Permissions for queues:

• send - allows to send messages into the queue
• receive - allows to receive messages
• browse - allows to browse the queue (does not imply receive permission)
• all - all user permissions

Administrative permissions for destinations:

• view - allows to view information about destination
• create - allows to create destinations under this parent
• delete - allows to delete destination
• modify - allows to change properties of the destination
• purge - allows to purge the destination and/or durable subscribers on topic destinations

Granting access

Syntax:

grant <destination type> <destination name> <subject type>=<subject name> <permissions>

<destination type> is topic or queue <destination name> is name of topic or queue <subject type> is user, group <subject name> is user name or group name

Example:

grant topic mytopic user=client1 use_durable
grant queue myqueue group=mygroup1 browse

Revoking access

Syntax:

revoke <destination type> \
       <destination-name> \
       <subject type>=<subject name> \
       <permissions>

<destination type> is topic or queue <destination name> is name of topic or queue <subject type> is user, group <subject name> is user name or group name

Example:

revoke topic mytopic user=client1 subscribe
revoke queue myqueue group=mygroup1 send

Securing your objects

It is important to control access to all objects inside the broker and have no blind spots there. By default, TibcoEMS is not enforcing ACLs on objects (if you don’t specify secure property on creation) so everything is accessible by everyone (we are talking here about only authenticated users, anonymous would not be able to read any data). That means different clients of the same broker could potentially read each other messages (which is not what we want). If you’ve created some objects without secure flag there is a way to change it using addprop command.

Topics

>addprop topic mytopic secure

Queues

>addprod queue myqueue secure

Summary

TibcoEMS is not a complicated product, depending on your needs you need to know only a fraction of its capabilities. Even though this article not covering all and every aspect of TibcoEMS it is good to know:

• basics (queues, topics, durable, and connection factories) - to speak a common language with other engineers
• permission model and ACLs - to troubleshoot access issues and understand what is possible and what is not
• bridges - to simplify application code and not reinvent the wheel (broadcasting, filtering, and load balancing could be done in broker itself)
• routing - to setup globally distributed applications

Even if you would not work with TibcoEMS in your practice it makes sense to learn one or two solutions in the messaging area to become a better engineer, I would recommend looking at Kafka.

Links

• https://docs.tibco.com/pub/ems/8.6.0/doc/html - TibcoEMS Official documentation
• https://en.wikipedia.org/wiki/Jakarta_Messaging - JMS messaging