Big Data is getting bigger with IoT; Red Hat JBoss Middleware can help

Database Trends and Applications magazine (DBTA) announced its 2015 Readers' Choice Award winners recently, and two Red Hat JBoss Middleware products were named finalists: JBoss Data Grid, for Best In-Memory Database, and JBoss Data Virtualization, for Best Data Virtualization Solution.

We are thankful to be recognized by our peers and proud to be considered among the leaders in these areas. Red Hat's open source data-oriented middleware technologies are used by organizations around the world to help unify and propel data in support of applications that are more responsive and aware, through real-time data management and integration.

One of the macro-level trends fueling the interest and need for these types of technologies is the burgeoning Internet of Things (IoT) market and its impact on big data. The "things" in IoT all generate great volumes of data—often at more frequent intervals that an enterprise [is required to manage. Managing, analyzing, and acting on this data are not trivial tasks. Organizations may face the challenge of having terabytes (TBs) of IoT data being generated on a daily basis; while their ability to extract value from this data in a timely manner is shrinking. This is where IoT aligns with big data, and where it becomes evident that the two trends merge into what some have begun to call the “Internet of Big Data.” This requires a different kind of architecture to overcome this high-volume and high-velocity data challenge.

JBoss Data Grid is an In-Memory Data Grid (IMDG) solution. IMDGs enable the storage of tens of terabytes of data, faster response times, and almost instant analytics. This makes it feasible to process IoT data at nearly the same speed that it is generated. Additionally, IMDG solutions can be combined with streaming analytics technologies, such as a complex event processing (CEP) engine, to analyze data in real time and generate insights or take actions. IMDGs can also support analytical frameworks and tools to directly run insights on the data stored in-memory. Data can also be shared with other applications if the IMDG supports other connectivity options.

Writing for TechTarget's WhatIs.com, Stan Gibilisco digs deeper into the benefits of using an IMDG:

"IMDGs can support hundreds of thousands of in-memory data updates per second, and they can be clustered and scaled in ways that support large quantities of data. Specific advantages of IMDG technology include:

• Enhanced performance because data can be written to, and read from, memory much faster than is possible with a hard disk.
• The data grid can be more easily scaled, and upgrades can be more easily implemented.
• A key/value data structure, rather than a relational structure, provides greater flexibility for application developers.
• The technical advantages provide business benefits in the form of faster decision making, greater productivity, and improved customer service.”

Gibilsco also notes how IMDG can demonstrate its worth in areas ranging from banking to e-commerce.

At the same time, integrating data from many IoT sources, particularly when those sources are varied in nature and highly distributed, can pose significant challenges. Physical consolidation of all data to a centralized server for integration often is no longer practical, whether due to cost, technical difficulty, or even regulatory requirements, and organizations are now looking to data virtualization as an alternative to integrate widely dispersed data.

Data virtualization helps reduce complexity and allows business and IT to work closer together by offering agile development, a secure virtual data layer, and real-time data access and provisioning. Data virtualization addresses organizations’ need to achieve a comprehensive view of their data, while avoiding the need to rip and replace physical systems already in place.

Together, JBoss Data Grid and JBoss Data Virtualization allow historical data to be accessed more easily and correlated with more recent data by loading up the history in-memory and provisioned via an SQL interface to enable usage of familiar analytical tools.

As organizations step into IoT, they must understand its symbiotic relationship with big data. For IoT deployments to really make an impact, they must provide useful or actionable services. Just like with any big-data play, merely collecting the data isn't enough. The data must be processed, integrated, and analyzed to glean insights, and those insights must drive actionable steps that can improve the business.

“Once the Internet of things gets rolling, stand back,” warns Howard Baldwin, writing for Forbes. “We’re going to have data spewing at us from all directions – from appliances, from machinery, from train tracks, from shipping containers, from power stations... If that doesn’t get you thinking about how to handle real-time data feeds, nothing will.”

“But here’s a suggestion,” he adds. “Start now.”