When it Comes to IoT Innovations, Array of Things Continues to Achieve

In this series, we are sharing the stories, accomplishments, and vision of the most successful IoT Innovators in the world. When we approached Charlie Catlett about profiling him as part of this ongoing set of stories, he asked us instead to cover the team at Array of Things – the experts across different disciplines he works with to advance some of the most meaningful implementations of connected systems we’ve seen to date.

The Array of Things is a collaborative effort among leading scientists, universities, local government, and communities to collect real-time data on the urban environment, infrastructure, and activity for research and public use, their GitHub page explains.

The project is led by Catlett, a senior computer scientist in Argonne’s Mathematics and Computer Science (MCS) Division and director of the University of Chicago Mansueto Institute’s Urban Center for Computation and Data.

The underlying open software and hardware platform, known as Waggle, was developed at Argonne National Laboratory by a team led by Pete Beckman, a senior computer scientist at Argonne and co-director of the Northwestern-Argonne Institute for Science and Engineering. The Waggle team includes Catlett along with additional Argonne computer scientists Rajesh Sankaran and Nicola Ferrier. The custom enclosure for the sensor nodes was developed by Product Development Technologies, based on early designs from Douglas Pancoast and Satya Mark Basu of the School of the Art Institute of Chicago. The project is executed in partnership with the City of Chicago.

The Array of Things project started up in 2013, with participation from scientists at academic institutions including Northern Illinois University, University of Illinois at Chicago, University of Illinois at Urbana-Champaign, DePaul University, Illinois Institute of Technology, Purdue University, University of Notre Dame, Vanderbilt University, the Santa Fe Institute, University College London, and the Institute for Advanced Architecture of Catalonia.

AT&T is the project’s communications partner, providing all AoT connectivity for Chicago. Technical advice and support come from a growing number of industry partners, including Cisco, Microsoft, Schneider Electric, Intel, Motorola Solutions, Crown Castle Communications, and Zebra Technologies.

Array of Things is funded by the National Science Foundation, the Chicago Innovation Fund, the University of Chicago, and Argonne National Laboratory.

Here’s what the Fab Four of the IoT (Pete, Rajesh, Charlie, and Nicola) collectively shared after we posed our questions.

What drew you to IoT? When were you first introduced to IoT? 

Each of us come at this from different directions.  Raj’s Ph.D. topic was in application of network of sensors and actuators in building among other things, interaction devices, prior to IoT going mainstream.  Pete’s interest in wireless sensors, also pre-IoT popularity, was partly their potential in environmental applications, but with his focus on high-performance computing and operating systems, he was also drawn to the potential for “edge computing.”   For Nicola, the interesting challenge was one of necessity - there would need to be much smarter devices to handle the data generated by sophisticated sensors, such as multi-spectral cameras. Charlie worked on the early Internet in the 1980s and through the 1990s on distributed applications, so began thinking about IoT and wireless devices in the late 1990s to make physical objects and places programmable.

Though we had worked together on other projects, our interests converged around 2011. At that time, we heard that the City of Chicago was going to replace 300,000 street lights with new LED fixtures, and we thought that perhaps there would be a science opportunity if we could piggyback on the project to install some sort of sensors at the same time.  We started talking to the Mayor’s office about this while holding scientific workshops to explore what functionality a street-pole-mounted device might provide in order to pursue questions ranging from air quality to traffic safety, from the impact of the lake on weather to the relationship between urban green spaces and urban heat islands.  

At each of about a half dozen of these workshops, we would ask rooms full of scientists the same question, “if you could place a few hundred gadgets around a major city like Chicago, what would they do to help you answer the questions you are asking?”  We had similar discussions with community groups and policymakers in the City of Chicago - what are the challenges you are facing, and are there measurements we could provide that would be useful to understanding and addressing those challenges?

We proposed the idea of hundreds of devices, measuring many variables, as a city-scale IoT infrastructure, providing open and free data, sort of like an array telescope (where many identical instruments, pointed in the same direction, provide a higher resolution picture than any of the individuals can provide).  We started calling it the “Array of Things.”

The answers to the questions we asked scientists—what would these devices do—fed into the design process in two ways.  First, it gave us a list of several dozen sensors that would provide measurements of the city to help scientists (and policymakers) answer the questions they are asking. Second, it revealed the need to measure things for which there are no electronic sensors, such as the flow of vehicles in an intersection, or the number of people who are using a public park and the patterns of use over time. We decided that the best way to provide such measurements would involve analysing the images and other data within the sensor nodes, or “at the edge.”  This took the project in a direction of needing to create a resilient, remotely programmable computing system so that scientists could write programs, such as machine learning computer vision, to analyse images and other data at the edge.

What was your first IoT project? 

Nicola’s early IoT work was to monitor and analyze data related to agricultural and environmental applications, for instance, using images to explore the diversity and health of plant life, particularly as related to external effects such as related to invasive species, weather patterns, or human activity. 

Charlie began using sensors to make a vacation home “smart” in the late 1990s, first with a thermostat connected to a Radio Shack auto-dialer so he’d get a call if the furnace failed. Over several technological generations, this system evolved into a crawlspace temperature monitor that texts him if there is danger of pipes freezing, and a basement sump pump activity monitor that tells him when the pump is working overtime (or fails!).

While in graduate school, Raj discovered the wonderful world of the Linux operating system, and started deploying sensors in his home to automate music playback.  His automated system read him the weather when he returned home from classes, loaded playlists based on the weather, automated shutdown and restart of his home server, and tuned multi-room audio setups (in his modest sized apartment) to support hands-free in-person-like conversations with his family over Skype.

What are you doing/working on today?

Today the team is focused on an ambitious follow-on to the Chicago Array of Things and the underlying technology platform, Waggle.  Together the team is leading a new collaborative effort called “SAGE,” involving scientists from around the world to not only improve the way we measure cities but to extend to ecological, environmental, and emergency management applications.  This is where their individual skills and interests weave together.  

Pete is designing the overall architecture, including how people will develop and run software across many types of devices, from IoT at the edge to central high-performance cloud systems.  

Raj is rolling up the lessons learned from four iterations of the technology platform -- from individual circuit boards to sensors and processing devices -- to create a more modular, generalized version of the Array of Things nodes that can be placed in cities or in wilderness settings, and can be attached to other instruments such as weather radar or multi-spectral imaging devices.  

Nicola is developing new machine learning capabilities that will take advantage of major improvements in the computing power of the Waggle nodes, initially looking at detecting conditions such as flooding, but ultimately analyzing the flow of traffic and the types of vehicles involved, for instance to understand the impact of an at-grade rail crossing on vehicle traffic.

Charlie is focused on two facets of expanding and improving the Array of Things and the deployment of Waggle in various non-urban settings.  The first involves engaging with end-users to understand what measurements they need, translating these into either selection of sensors or development of machine learning features.  The second is building external partnerships to ensure that the data is accessible not only to scientists but to the general public and to application developers.

What are your predictions for IoT?

Charlie: I don’t think any of us would say we are futurists, but for most of our careers, we have tried to engineer a sort of time machine approach.  We know that things like computation, storage, and communications grow on exponentials, and in some sense, this means that the kind of hardware technology you can buy for $5,000 today will only cost $100 in a few short years.  So we look at a Waggle node that costs a few thousand dollars and ask what we might do with devices like this if they cost $100, and you could put them all over a city, or put multiples of them in a home or factory?  Waggle nodes supporting the Array of Things project can see, hear, feel vibrations, and smell chemicals (they can’t taste, yet).  With high-performance computing hardware they are able to think, and with AI algorithms they should be able to learn.  In a few years you’ll have these in your home - what are your predictions for how you’d use them?

Pete: Machine learning and artificial intelligence is already transforming our world.  As edge computing continues to become more powerful, a new programmable continuum will evolve – machine learning software will be running everywhere, from the edge to the cloud.  Everything from cities to factories will become programmable.  We have to start imagining what the operating system of a future city or farm or school might be capable of when everything machine learning and artificial intelligence permeates all forms of physical infrastructure.  Today, much of the industry is focused on deployment.  Tomorrow we will be focusing on orchestration, cybersecurity, and autonomy of nearly every infrastructure.

Where do you see the most value created?  

Nicola: For science, the ability to acquire data with high spatial-temporal resolution and the ability to extract “the most important” information from the data is a huge value for many disciplines. 

Charlie: Apart from the hardware or software we are installing at any given point in time, we have created a valuable partnership in Chicago that spans universities, a national laboratory, and many city departments and community groups.  I think this is more valuable than the data or the specific technical systems on the poles because these partnerships fuel the ideas behind what is on the poles, and behind what you can do with the data.

Raj: For me the greatest value will come when the vast amount of data and insights generated through these smart and ubiquitous devices are channelled through avenues that create a long-lasting positive impact on our urban and rural landscapes. That is, when the application of IoT generated knowledge can be applied to areas such as improving the quality of life in our cities and preserving the ecological diversity in the wild.

What are the most inspiring/meaningful IoT initiatives you’ve seen?  

We are inspired by Chicago’s Array of Things project!

In your opinions, what is the most significant risk associated with IoT?  

Nicola: Having “everything” networked and powered seems to demand an increase in the global energy needs, and the world needs to figure out how to provide energy – if every device needs a battery that needs rare earth materials, it won’t be possible.  And fossil fuels are a finite resource. 

Charlie: We are all very concerned about the ethical use of information coming from IoT devices, whether it is the use of images, sound, or patterns of use.  In the Array of Things project, we were very careful putting boundaries around what could be done with the devices and with the data, with policies that included accountability.  When we say we will not stream images from the nodes this is something we can enforce, and that is auditable.  Since images are deleted after we analyze them, there is no opportunity for someone to come along later and change the terms of use, because the data no longer exists!

Raj: The not-so-easily recoverable electronic waste and the burden of mining for materials are of main concern to me. Cost-effective and easily-deployable IoT solutions are here to stay, and will drive up the uptake of these devices to a point where the total number of devices will be orders of magnitude greater than the number of humans on the planet. We must invest in technologies that are friendly and allow better recycling of material, and not just down-cycling, or else the end-of-life burden of these devices will stymie their adoption.

What is one piece of advice that you would offer a business leader that is interested in IoT?

Raj: IoT is truly greater than the sum of its parts. If employed judicially, there is immense power in applying it to some of the biggest challenges we are facing today and greatly improving our quality of life. So, choose the problems you want to tackle carefully, and put together an excellent team of hardware, software and mechware specialists who have scars from deploying systems in the wild. And, oh, since the devices can get close to people, they are likely exposed to and can collect more private data. Yes, private data needs to remain private (even from automated algorithms) as the name suggests. That is how you build trust with folks who use your products.

Charlie: I don’t know if I could narrow it down to one thing.  You want to be sure that you own the data that comes out of the systems you buy, and you want to be sure you are not constrained to use only tools from the vendor to analyze it.  You’ll want to mix the data with data from other systems and use the best tools available, wherever you can get them.  There are also increasing numbers of systems out there with lots of different sensors, and before you decide you want to deploy thousands of sensors across a factory or a city you need to understand the quality of the data.  You can’t get that from the data sheets for the individual parts because many factors are involved, including how the sensors are integrated and housed, how their data is sampled, and how those sensors hold up over time.

The opportunities are profound, and this team is clearly up to the challenge. When we spoke with Catlett, he said, “Imagine what we’ll be able to do in a city in ten years that we cannot do today. We’re building IoT super devices that will measure their surroundings in different ways including the use of AI. While we certainly have a vision and set of views, we also are makers and doers, building the actual hardware with a complementary set of expertise and talents ranging from the design of circuit boards, to writing firmware and operating systems, and from developing system architectures to autonomous robotics and computer vision.”

Catlett sees Array of Things and their partners as” continually asking what can be enabled by technologies on the horizon, and how we might create systems and services that allow people to explore those possibilities.  Array of Things and the Waggle platform are not smart city products, but rather these are platforms intended to support exploration. We are just getting started in IoT as the hardware, software, applications, clouds, and networks advance.”

You can learn more about Array of Things here.