Menu

IoT FEATURE NEWS

When Data Lakes Become Data Oceans, Approximation Will Rise

By

The energy continuing to form around the IoT is now changing as the sheer number of endpoints continues to grow dramatically, the sophistication of applications including analytics and algorithms continues to mature, and enterprises move beyond Proof of Concept to initial implementations that are expected to scale locally, regionally and globally.

Looking back at the “network effect” that drove the demand for broadband at the turn of the century, experts are scrambling to figure out when the IoT will scale as massively as mobile devices, applications and as-a-service cloud-based solutions. Because IoT has traditionally been thought of “tiny transactions” – which has caused many Tier One telecom companies, for example, to not invest due to the low impact on “data consumption” – has the industry not done enough to forecast the demand for data processing as the industry matures?

Don DeLoach, CEO of Infobright, which has been providing a high performance analytic database platform for The Internet of Things for over five years, has an inside view of what their partners and customers are seeing, as well as a broad industry view based on his participation in a number of industry forums and communities.

“The number of new companies entering the IoT space today, and the collective billions of dollars being invested by tech giants including Intel, Microsoft, IBM, Accenture, Google, Amazon and more, are leading indicators that the IoT is moving into a new era where systems will be deployed in a more pervasive way,” DeLoach said in a recent interview. “While we’ve all had those moments where we’ve wondered if IoT would ever truly progress beyond smaller, experimental and frankly tactical solutions – we are now seeing serious traction including with highly intelligent and hyper-automated approaches, including AI that can deliver massive productivity gains.”

Because Infobright provides data processing software and infrastructure specifically focused on enabling the rapid analysis of machine generated data, they are part of the most advanced applications performing interactive, real-time, complex queries. According to DeLoach, they are outperforming traditional data processing approaches from competitors like Oracle, because their team of data scientists and developers have developed an architecture that enables faster, lighter and resource-friendly processing whether locally or within the data lakes.

The Rising Tide Will Lift All Boats – But Some Boats Will Sail Faster Than Others
“As we watch those data lakes rise in size and volume, we see data oceans on the horizon which could become impossible to navigate if we don’t, as an industry, think ahead now on how we are going to accommodate a world where data is moving from batch-processed to real time – simply because the business value derived from real-time analysis, automation and decision making triggers is so extreme. Cost savings, improved productivity? Either way,” DeLoach explained, “businesses are seeing a very positive impact to the top and bottom lines.”

DeLoach talked about exact queries vs. approximate queries, explaining that running exact queries across a number of tables in a high volume data lake creates long running queries. When one adds in more analytics, more algorithms, more correlation – these queries can take hours. Today’s data processing techniques do not make this easy to do.

“The holy grail in IoT has always been turning insight into action, and our customers are getting there very steadily, while layering automation into that action,” DeLoach said. “We set out on a course to address this, and working with our customers came to our own insight that it is better to create statistical models that can be queried for high value approximate results, in a shorter period of time, rendering the same or nearly the same business outcome.”

Traditional data processing models just won’t fly in the next era of IoT, DeLoach believes, pointing to approaches breaking down at scale. “With enough computing resources time and money you can query these large volumes of data, but companies should ask themselves if they really need to spend that money and in effect create what could be a pile of technical debt in the near future.”

“Take what is technologically possible and make it truly practical.”
DeLoach explained that most people believe that data lakes and NoSQL technologies have this challenge covered, but this has limitations, according to DeLoach when there are multiple tables and comlex queries. “For instance, in investigative analytics, queries are typically joined in a query chain, where the result of query 9 forms the basis for query 10, and so on. We have validated that equivalent insights can be gained from a high value approximation that doesn’t bog things down.”

By creating statistical models over all the data that answers most questions relieves demands on network capacity as well as compute, when only statistical models are transferred.

At The Edge, Fog and Fuzzy Approximation Goes Beyond Best Efforts
“Edge or Fog computing is tightly related to this,” DeLoach said, “The main role of approximation will be to augment the operation (and productivity) of the centralized data lake. That said, we have also been working with industry experts on a related concept, called ‘First Receiver,’ which establishes consolidated data ownership, primacy and local management enablement first – where business decision making is likely more valuable in real time. After the First Receiver processes and analyzes the data, it can be shipped via APIs in a lighter fashion into distributed systems in the cloud. For example, a local franchise owner can run a retail shop more efficiently – controlling temperature, lighting, security, cameras and more, while also sharing information with a regional, national or global manager.”

While approximation, First Receiver, and local/distributed data, analytics and automation solutions aren’t valid for all applications, DeLoach forsees uptake in areas including demand forecasting, security sweeping, yield management, profiling, and behavioral modeling.

Emil Berthelsen, Principal Analyst at Machina Research, has published a note on the emerging “First Receiver” approach, and said “this architecture will give enterprises the ability to manage and publish data from connected products, making it available both to subscribing vendors and potentially to third-party service providers. ‘First Receiver’ data gateways can be managed by the enterprise leading to more control and business value as they own and can leverage local data in real time, while also contributing that data to the cloud for more distributed applications.”

“It’s truly and exciting time to be working alongside pioneers in the IoT industry, and the new entrepreneurs coming in with some revolutionary and meaningful applications. Those of us who take responsibility for ensuring the data and analytics processing is running smoothly – and cost efficiently – are more motivated than ever when we see the impact IoT will have on business and society – even more than we could have imagined as recently as five years ago.”

While DeLoach is traveling and unable to attend the IoT Evolution Conference and Expo next week, Andy Asava, founder and CEO of Qualesce, his professional colleague, will be speaking on a panel where he will talk about the First Receiver vision as part of the Fog Computing, Analytics & Data track, Wednesday, July 13, starting at 3 PM. 




Edited by Ken Briodagh
Get stories like this delivered straight to your inbox. [Free eNews Subscription]
SHARE THIS ARTICLE
Related Articles

Rising Edge Computing Investments to Reach $350B by 2027, According to IDC

By: Alex Passett    3/27/2024

Worldwide spending on edge computing is expected to surge (and then keep going) for the foreseeable future, according to the International Data Corpor…

Read More

ZEDEDA Adds Lisa Edwards as New Board Member, Seeks Opportunities to Strengthen Operations and Scale

By: Alex Passett    3/26/2024

Earlier this morning, ZEDEDA announced the addition of Lisa Edwards to its board of directors.

Read More

An Existing IoT Collab, Emboldened: Digi International and Telit Cinterion Transform Solutions with 5G RedCap Integration

By: Alex Passett    3/25/2024

The ongoing industry collaboration between Digi International and Telit Cinterion signals strong support for the mainstream showcasing of 5G for IoT a…

Read More

Telit Cinterion's 5G LGA Modules, Powered by Snapdragon from Qualcomm, to Create a Big Leap in IoT Connectivity

By: Alex Passett    3/25/2024

Telit Cinterion recently unveiled its FE990B34/40 LGA family of modules, powered by the Snapdragon X72 5G Modem-RF System from Qualcomm Technologies, …

Read More

Embracing Innovation in Mining: The Role of Network-Aware Applications in the Digital Transformation

By: Special Guest    3/21/2024

Shabodi leverages private 5G network capabilities and enables the development of network-aware applications to enhance operational efficiency, automat…

Read More