The Complex Event Processing Blog

Our colleagues at Apama share an interesting use case, tracking the body temperature of someone walking in their recent press release.

This use case is a clear example of a subfunction of complex event processing, folks in the multi-sensor data fusion field (and here at The CEP Blog) refer to as event (object) refinement, sometimes called “track and trace.”

The reason we call this processing function “event (object) refinement” is that, in the way the use case was described in the press release, the medical staff are basically tracking body temperature and comparing it to a key indicator to generate an alarm, in this case “body temperature too high.”   This is a simple event, not complex, because the level of inference is quite very low in an overall knowledge hierarchy.

For example, we cannot infer from the alarm that “body temperature too high” is caused by a previous medical condition.  There is no causality at this stage of the game.   We cannot infer from the alarm that the walker has embarked up a steep hill, and the body temperature is expected to exceed a key indicator for a period of time.

Looking at another complex event model,  the system does not (yet) combine all of the body temperatures of the entire group of walkers, correlated by the situation of an approaching thunderstorm, and infer that the walkers have increased their pace because they don’t want to be caught in a driving rainstorm with high winds.

In other words, tracking a single object like “body temperature” is a basic-step in a CEP application, but not really a CEP application yet, because to really be a complex event, there should be some inference of higher knowledge, or estimated situation.    For example, tracking and tracing the position of an aircraft is good data, but being able to infer the complex situation “potential mid-air crash” between two airplanes is better (defining a complex event vs simply tracking state changes).

Steam processing engines are well suited for track and trace processing of individual event objects, like a walker’s body temperature, or a similar temperature monitoring application from a network device, as demonstrated by the Apama use case.  Tracking events such as “temperature in an object reaches critical threshold” have been going on for decades, in your network, in your car,  in your washing machine, in a spacecraft, just about everywhere we sense-and-respond to temperature changes.

The real marvel of this application was not the event processing on the back end, but in the sensor network, comprised of the human body, an RFID sensor, and a transmission network to a centeralized data collection facility.

Filed under: CEP News and Events, CEP Terminology, CEP Tutorials, Complex Event, Complex Event Processing, Event Processing, Event Stream Processing, Event-Driven Architecture, Progress Apama, Situation Models, Systems Engineering, Use Cases | No Comments »

Recently I had a nice conversation with the head of Asia-Pacific of an international company about how to succeed in Thailand. I explained how businesses in Thailand do not respond well to companies that come to Thailand with no experience, track record or support infrastructure here in the Kingdom. I also explained how Thailand has a strong cultural tradition around “the teacher culture,” where teachers are considered much higher than mere consultants and integrators.

The conversation went well, I thought, until I received a call from another person in the company who proceeded to tell me how to do business in Thailand and how to determine the target market, and how to set up sales. Now mind you, I had already explained that there would be no immediate sales opportunities for a few years, realistically, and that this was a long term initiative, designed around a solid education and training program - build infrastructure first. From a strong education and training program, the market would become clear.

This is such a simple win-win-win situation, but companies do not seem to understand it. They just want to exploit every contact, event situation, for a quarterly sell. Why not take the long view as well, since it does not cost you any money?

The guy on the other end of the phone would have nothing to do with our way of thinking in Thailand. He seemed to be pushing to insure pre-sales contact immediately. Instead of supporting us, he wanted to manage us from overseas!! We asked for support to build their brand, what they seemed to offer was management by proxy!

Folks, this will not work in Thailand (or most Asia countries).

If you want to tap into the fast growing Asia market, leave behind your aggressive New York or Silicon Valley sales guns and forceful presale tactics, where you are content to find an opening, exploit it, make a sale, and report the sale on your quarterly report. You can get aggressive when you have built a sustainable infrastructure. The same is true in Japan, not only Thailand.

In Asia, do as the Asians. In Rome, do as the Romans. In Thailand, do as the Thais. In Japan, do as the Japanese.

It is easy to make money in Thailand (and other Asia countries) if you follow their way of business. Educate, teach, build a workforce, build a sustainable infrastructure on the ground, and then sell, sell, sell.

Granted, many companies do not have resources to do this overseas. In that case, enable your partners to do it and let them build the business; don’t manage them, support them.

Filed under: Asia Pacific, Complex Event Processing, Thailand | No Comments »

According to our friend Wikipedia, the Straw Man argument is a red-herring fallacy where one party in a debate describes a position that, on the surface, resembles an opponent’s actual view but is easier to refute.  Then, in counterpoint, the debating partner attributes an easily refutable position to the opponent (for example, deliberately overstating the opponent’s position). Wikipedia says:

1. Person A has position X.

2. Person B ignores X and instead presents position Y.
Y is a distorted version of X and can be set up in several ways, including:

1. Presenting a misrepresentation of the opponent’s position and then refuting it, thus giving the appearance that the opponent’s actual position has been refuted.
2. Quoting an opponent’s words out of context — i.e., choosing quotations that are not representative of the opponent’s actual intentions.
3. Presenting someone who defends a position poorly as the defender and then refuting that person’s arguments, thus giving the appearance that every upholder of that position, and thus the position itself, has been defeated.
4. Inventing a fictitious persona with actions or beliefs that are criticized, such that the person represents a group of whom the speaker is critical.
5. Oversimplifying an opponent’s argument, then attacking the simplified version.

3. Person B attacks position Y.

4. Person B draws a conclusion that X is false/incorrect/flawed.
This sort of “reasoning” is fallacious because attacking a distorted version of a position simply does not constitute an attack on the position itself.

For example, there has been some lively discussions recently around the notion that CEP is overhyped.

Debate:      “CEP is Overhyped.”

Person A:   “CEP has been overhyped.”

Person B:     “CEP is just hype.”

The point of the discussion by person A was to point out that CEP has been overhyped.  Person B has exaggerated this to a harder to defend position, “CEP is mere hype.” or “CEP is just hype.”

From the customer perspective, I don’t think that fallacies and red-herring arguments are good for CEP.   Believe me, if we could take an “out of the box” stream processing rules-engine and bolt it on to a network and insure a client it would detect complex fraud, or diagnose network faults accurately, and not put my entire professional reputation on the line, I would do it in a heartbeat.

It is not the speed of the an engine which makes a good CEP engine, it is the capability of the analytics to deliver high-quality, high-confidence complex event detection in real-time.

Filed under: Complex Event Processing, Cybersecurity, Event Stream Processing, Systems Engineering, Threats and Vulnerabilities | No Comments »

I am back at the glaring computer screen after a day in Lamphun, Northern Thailand, hanging out will my friends who are preparing for a Bonsai tree competition.  I spent the day eating Thai and Chinese food and relaxing in a lounge chair under imported blue palm trees with the sound of exotic birds making background music to keep me entertained.

Back to CEP and EPTS, there are folks who appear to believe they may define “CEP” by the current use cases from self-described CEP vendors. Frankly speaking, I am puzzled by the bottom-up approach.

The bottom-up approach is a bit like saying “We have a lot of prototype rockets being built, so let’s define the future of space travel based on the prototypes!”

It really makes little sense, at least to me, to attempt to define CEP based on what the current generation products (self-described CEP products) are capable of doing.

From my perspective, it would be more beneficial to customers to define the types of complex events (and situations) businesses need to detect in real-time and match the technologies and solution architectures to detect those events, in real-time, with high confidence.

A lot of this “top down thinking” has been already done.

IT businesses need to detect operational threats and problems, and be able to pinpoint, with very high accuracy, where the problem is in a complex network, for example.  This problem remains mostly unsolved with a very low signal-to-noise ratio.

Also, most businesses would like to detect fraud and other criminal activity on their network before the activities adversely impacts their business.   This problem remains unsolved for most companies.

Scientific researchers seek models of weather, epidemiology, and so much more; and they need event processing solutions to obtain situational knowledge into current events and predict future ones.  We know how difficult predicting the weather can be!

Folks on the ground need to model urban traffic as events and design better event-driven traffic models and solutions.

The list of important event processing challenges we face go on and on.

While I see some merit in the bottom-up approach, it is better for users to define what are practical “complex event” related problems and then look for the solutions, vs. define the solution and then look for the problem.

From a strategic perspective,  self-fulfilling CEP use case studies are interesting, but they should not limit the vision, definition, and future of processing complex events.

Be careful of use case fallacies.

Filed under: Complex Event, Complex Event Processing, EPTS, Event Processing, Event Processing Technical Society, Thailand, Use Cases | No Comments »

Catching up on the blogs, I couldn’t help but comment on, Is CEP Mature? Or a Curious Case of Information Asymmetry by Mark Tsimelzon, President & CTO, Coral8.  Mark says,

“I know for a fact that every major CEP vendor has several dozen paying customers.”

Somehow Mark, I don’t find a dozen paying customers by the top CEP vendors very impressive.

Then, as to somehow justify the lack of public reference clients, Mark takes the position of a Coral8 customer and says,

“We believe that the use of Coral8 gives us a strategic advantage over our competitors. Why would we want to clue them in?”

Naturally, the same thing could have been said about the first desktop computer, or the first back-office banking system, or the first calculator, or the first telephone, frankly speaking.

Of course, when the technology is mature, then it is “Hey we have lots of computers!” “Hey, look at my fully functional sexy iPhone!” “We have the best back office banking systems on the planet by <insert your favorite big vendor here>!”

Well, all this CEP Solution Secrecy (CEPSS) might just be similar to why the government keeps many IT projects a secret;  the main reason is so we don’t know how much taxpayer money they are spending!

So, folks, the debate counterpoint that there is some “Secret Life of CEP” and that the CEP solutions today are somehow changing the way C-Level executives, and corporate America, thinks is just wishful thinking.

Companies don’t need to keep their strong technical solutions a secret. Like, Wow! I am using Coral8 and it is so impressive that I have to keep it TOP SECRET.  (Sorry Mark, nothing personal, you simply gave me a big red target and painted “fire when ready” on it)

Note:  I happen to like Coral8, and Coral8 Studio, as an event stream processing platform.

Back on point, I consider my laptop and cellphone more indispensable than most of the first generation rule-based stream processing engines out there today, and I am sure most CEOs agree.

The Secret Life of CEP….   you have to just love it

Filed under: Complex Event Processing, Coral8, Event Stream Processing, Humor, Use Cases | 7 Comments »

In  CEP as a Discipline,  David Luckham wrote: 

“Actually, it is fair to say that some of CEP can be found in other disciplines. Event processing has been going on in one form or another, for the past 50 years. Simulation, Networking, Active DBs, Middleware.

{ …. }

CEP has only just begun. The foundations are unexplored. Its an open field of research issues.”

Actually, on slide 12 of this presentation from 2006 Processing Patterns for PredictiveBusiness, we show that the foundations for complex event processing have been in place for many years and in many disciplines such as multisensor data fusion, control theory, sensor management, planning, correlation, estimation, tracking, information fusion, data fusion, data mining and more.

One obvious problem (or at least obvious to many of us) with the current group think marketing CEP is that many have ignored the established foundations for event processing and complex event processing that have been mature for many decades. It is not very efficient (nor good for customers) to pick a phrase, or concept, like “CEP” and ignore the relevant multiple disciplines that have been used to solve complex classes of distributed event processing problems for decades.

Therefore, “CEP has only begun” is only true for those who have ‘drank the CEP koolaid” and do not understand (yet) that they are “reinventing the event processing wheel” and ignoring (by accident or purposely, I have no idea of the motives) the prior-art and/or selectively picking the prior art or research associated with their company, byline, favorite researcher, CEO, etc. This is a fundamental issue (and constraint) with CEP, in my opinion. Complex event processing does not stand alone as an art or a science, nor should it, nor should it be based on single dimensional, or small groups of single dimensional, technologies.

If you want to see many of the foundations of CEP, you don’t need to go much further than slide 12 of this  presentation from 2006, Processing Patterns for PredictiveBusiness.

Based on my observation, it reminds me of a small group of folks on a discovery mission where their ship lands on the shore of a distant land and they call this “new land” — “CEP” because they feel they have discovered a new land.  Nevermind the big cities that already exist or the many people already “in the fields” of their new land.  These ”CEP explorers” are seemingly in some kind of modern day epic struggle to define themselves as “discoverers” or “founders” and they are coming up with new names of the lakes, rivers, streams and mountains that defined the landscape long before their ship arrived.

Note: It is encouraging to see folks slowly “catching up”…. maybe in a few years we will move CEP beyond the “not invented here” mind share that we see today.

Also note that, recently we saw a flurry of posts where many people rightly stated that “CEP was overhyped” - but then in rebuttal the EPTS community leaders came back with “Is CEP a mere hype?” or “Is CEP a hype?”. spinning the discussion to an extreme position that is wildly different than “CEP is Overhyped”.   

Filed under: Artificial Intelligence, Blackboard Architecture, Business Optimization, Business Process Management, Business Rules, CEP Terminology, CEP Tutorials, Complex Event, Complex Event Processing, Cybersecurity, EPTS, Event Processing, Event Processing Technical Society, Predictive Business, Process Optimization, Scheduling, Sensor Fusion, Systems Engineering | 2 Comments »

It was not too long ago that I penned Keyloggers: Why Banks Need Two-Factor Authentication. In that post, I briefly mentioned how a number of banks in Thailand use inexpensive SMS-based two-factor authentication (2FA) with one-time password (OTP) to authenticate transactions.

One of my favorite banks in Thailand is K-Bank. With K-Bank I can simply walk up to an ATM machine and pay a mobile phone bill, purchase mutual funds, buy insurance, or transact an ever-growing list of services payable at the modern and sleek K-Bank ATM.

For example, tomorrow I fly to Chiang Mai in Northern Thailand (Nok Air) and found K-Bank’s service amazingly better than in the US. For example, I booked my flight as usual (over the phone, but could have used the Internet) and told the reservation agent I was going to pay by ATM. He simply gave me a PayCode and told me I had three hours to go to the ATM and enter the PayCode to perfect my reservation.  I also got the PayCode via SMS.  This gave me the time I needed to make sure I had booked the perfect boutique hotel in Chiang Mai, the Fern Paradise.

Then, I went out into the beautiful Thai weather and completely my airline reservation at the ATM machine; which also printed out a receipt with my flight details and reservation number.

It sometimes amazes me how much further advanced some services are in Thailand compared to the US. To me, it feels more secure not to use an on-line payment center or give out my credit card details over the phone. I can simply book a ticket, take a PayCode, and complete the transaction at a nice modern, shiny, K-Bank ATM machine.

Who knows, maybe soon I can select the perfect window seat at the ATM and the receipt will act as my boarding pass!

Filed under: Asia Pacific, Business Activity Monitoring, Business Event Processing, Business Process Management, Complex Event Processing, Process Optimization, Risk Management, Thailand | No Comments »

We have seen a lot of discussions of cloud computing in the news recently, as a technology to permit “users to access technology-enabled services without knowledge of, expertise with, nor control over the technology infrastructure that supports them.”   This sounds great doesn’t it?!   Users with little to no IT expertise can log into the cloud and launch 8 instances of a server with the equivalence of 16 high performance CPU cores.   However, as we all know, all things, including cool technologies, have the potential for both good and evil, opportunity or threat; and cloud computing is no different.

It just so happens that I have been experimenting with Amazon Elastic Computing Services (EC2), documented in Computing in the Clouds with AWS over at The CEP Blog.  The server over at The UNIX and Linux Forums has been experiencing some very hardware-limited, high load averages recently. We thought we should take a look at moving the forum server up to the clouds.   

Then, a fellow system admin over at the forums suggested that maybe some rogue bots were causing high server loads; so I wrote a one-line command to do a bit of real-time spider hunting in the Apache2 logfiles.  Surprise!  I found there were a number of rogue, hungry spiders that would not follow our robots.txt directive not to crawl the site.   One of the bots was from Russia, one was from China, and another one was from Korea.  There were spiders from places I never heard of, all consuming precious  resources and denying our users!

So, I did what any Linux admin would do. I used iptables to block the networks of these rogue, hungry, spiders (sorry I was not very kind to these cyber creatures).  It probally comes to no surprise at this point in the story that four of the spiders were from the Amazon EC2 cloud.  Here is a sample of the output from iptables -L:

root@www:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP all — ec2-67-202-45-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-243-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-197-0.compute-1.amazonaws.com/24
DROP all — ec2-75-101-213-0.compute-1.amazonaws.com/24

Well, imagine a not-so-distant future dystopian world where criminals or terrorists want to launch a massive denial-of-service attack against some critical infrastructure, like the root DNS servers, or an attack against major financial institutions, military or e-commerce sites.   

First, the bad guys create an instance of powerful operating system with a malicious network application, they test it, and they place it the cloud (without invoking the instance, paying a very small storage fee, no computing time fee) and they wait.   Then, at the precise moment of their planned attack, they launch 128 instances each with the equivalence of whatever is the mega-platform at the time, and just blast away at their attack target(s).    Even more damaging, they do this from many cloud computing infrastructures.  (Note: The cost of the attack is minimal because the criminals are only charged a few pennies an hour for each running instance and the attack runs an hour or two.)

My experience with cloud computing, which is still maturing, is that cloud computing has great promise for both good and evil.  The very real example of the “spiders from the clouds” is a harmless enough story of folks using a cloud computing infrastructure for web crawling, perhaps hoping to be the next Google billionaires. 

On the other hand, cloud computing brings with it an emerging and growing danger for the misuse of the power of cloud computing infrastructures.   The misuse could be malicious, or accidental, but never-the-less, the danger is real.

What an interesting world we have created!  Who would have ever dreamed 10 years ago that we could be attacked by ……

#include <horror_movie_sounds.mp3>

…. Spiders from the Clouds.

Reprinted by permission from The Attack of the Spiders from the Clouds by Tim Bass, CISSP

Filed under: Cloud Computing, Complex Event Processing, Cybersecurity, Event Cloud, Event Processing, Financial Services, Threats and Vulnerabilities, Use Cases, Virtualization | 4 Comments »

Paul Vincent, ex-colleague at TIBCO, kindly responds to A Brief Introduction to Blackboard Architectures with Blackboards for Complex Event Processing. Paul correctly mentions that TIBCO’s BusinessEvents software is an excellent scheduling component in a blackboard systems architecture.

However, I should briefly clarify Paul’s note that “blackboard systems historically used a single memory model (i.e. multiple threads or processes using a single machine’s memory model)“.

In fact, there were many blackboard systems, some more than a decade old, that used a distributed memory data-model. What I think Paul meant to say, and my apologies to Paul for being so literal, is that “blackboard systems originally used a single memory model (i.e. multiple threads or processes using a single machine’s memory model)”

John McManus, former CTO of NASA, wrote an excellent PhD dissertation in 1992, Design and Analysis Techniques for Concurrent Blackboard Systems. John’s thesis, now more than 16 years old, examined many details of concurrent blackboards where memory is distributed. For example, refer to Figure 2.3. Distributed Blackboard System with Distributed Blackboard Data Structure, page 36 of John’s dissertation.

Quoting directly from page 37 of John’s disseration;

Rice, Aiello and Nii [20] present several options for gaining speedups in a distributed blackboard system.

• 1) Eliminate the centralized scheduling mechanism
• 2) Optimize system design for a distributed memory, message-passing hardware
• 3) Distribute the data across the blackboard to reduce hotspots

Quoting further from the same page;

Poligon [21] is based on a distributed memory hardware model when each processor is viewed as a blackboard node. They define a blackboard node as follows: “a blackboard node is a process on a processor, surrounded by a collection of processors able to service its requests to execute rules.” [22] The implicit assumption in this definition is that all knowledge sources are rule–based systems. This assumption may severely limit the performance of systems implemented using Poligon, and limits the types of problems it is suited to address.

In Blackboards for Complex Event Processing, Paul concludes,

“One suspects the blackboard systems domain and terminology is overdue some updates thanks to developments in the Complex Event Processing space.”

If you look at the historical literature, I would say that the following restatement is more accurate:

“The CEP domain and terminology is overdue some updates because folks working in CEP did not reference or incorporate the advanced event processing prior art in a number of very important areas, blackboard systems being only one.”

On the other hand, commercial off-the-shelf rule-processing technology such as TIBCO’s BusinessEvents (BE), advances the ability to economically implement myriad complex problems that blackboard systems are designed to address.

Filed under: Agents, Analytics, Blackboard Architecture, CEP Tutorials, Complex Event, Complex Event Processing, Event Processing Modelling, Scheduling, Systems Engineering, TIBCO | 2 Comments »

I am often asked pointed questions (mostly from the stream processing crowd) like, ” What product does CEP?”  Sometime it seems my answer determines the fate of that relationship, as my feet are grilled over the CEP-fire to be beat of jungle drums!  The amount of money I have lost in deals that did not go through because I refused to sprinkle holy water on a vendor’s product and call it “CEP” is staggering, quite frankly.

CEP describes an architecture, just like SOA describes an architecture and just like EDA describes an architecture.

For example, you do not buy an SOA.   An SOA describes an architectural style of programming via components that are involved as services in a distributed network architecture - a service-oriented, or service-based architecture.

The concept of CEP does not have “the A-word” like SOA and EDA, but none-the-less, CEP describes an architecture, not a product.   Do not make the mistake of thinking in terms of “buying CEP”, just like you do not buy an SOA or an EDA.  You think, plan and design in terms of CEP, just like you should do in an SOA or EDA.  These are constructs, not products.

In other words, for “true CEP” you need a number of components, some of the components might be in the architectural style of SOA, others might be in the architectural style of EDA.   Your solution architecture for solving a complex event processing problem might have request-reply transactions, or it might have fire-and-forget messages.  You might have a Neural Networking component for analytics and a rules component for filtering, mediation and scheduling.  You might even have a stream processing component performing as a high performance filter and pattern matcher on streaming data where the output is forwarded to a Bayesian Classifer for further processing.

My key message in this post is that CEP requires a number of technologies to solve complex distributed computing problems.   Do not be fooled into thinking that a single product is “CEP” no more than a single product is SOA or EDA.

Filed under: Adapters, Analytics, CEP Tutorials, Complex Event, Complex Event Processing, EAI ESB & SOA, EDA, Event Processing, Event Stream Processing, Systems Engineering | 3 Comments »