Menu

IoT FEATURE NEWS

Prediction in the Age of Big Data: The Science Behind Recommendation Systems

By

Recommendation systems have become a primary way to discover relevant information from vast amounts of data. Netflix uses algorithms to make movie recommendations. Amazon uses them to tailor product recommendations. And the list goes on.   

But as the volume, variety, and velocity of data continues to increase, customers’ preferences are only going to become more complex and difficult to compute. Familiar approaches may become outdated, even obsolete, as people change the way they make decisions, exhibit preferences or take actions. 

How can data scientists extract meaningful insights and accurately predict customers’ preferences in the age of big data? Below are three simple steps to help ensure success.

Step 1: Define the Problem
A recommendation system cannot do its job properly without the right data to derive insights from. Therefore, prior to designing the system, data scientists must first figure out what problem they are looking to solve. Once that question is determined, data needs to become more apparent. 

For example, an apparel retailer’s problem may be figuring out which products to show customers when they are visiting the e-commerce site. In this case, the retailer will not only need data about the specific customer, their likes and dislikes, etc., but also how they compare to others. That data can be found through a customer’s browsing history, past purchases, and other online or mobile activity.  Data requirements will, of course, change if other variables, such as operational constraints, budgets, and supplier schedules, need to be taken into account before choosing which products to showcase. Put simply: the question of what data to collect depends on what challenge organizations are looking to solve. 

Step 2: Work with the Dataset
Once the right data is gathered, the next step toward building a successful recommendation system is to develop a systematic approach for processing and filtering data at scale. There are three types of filtering: collaborative, user-based, and a hybrid approach. Many companies, such as Netflix, use a hybrid approach. Other successful brands, including Amazon and Spotify, rely on a collaborative filtering model, which takes into account the users browsing history, likes, purchases and rating before providing a recommendation.

For example, a movie-streaming service lets users rate movies they've seen. In order to generate personalized recommendations, the algorithm will first look at how other user ratings overlap and then assign similarity scores. Those similarity scores would then be weighted as ratings and are aggregated to predict an individual’s response to a particular movie. In general, the more information a collaborative-filtering algorithm has about users' preferences, the more accurate its predictions will be. This also requires being equipped with tools and technologies that can handle high dimensional datasets.  

Step 3: Convert Insights to Actions
Another key step toward building a successful recommendation system is having access to a skilled team of data scientists and statisticians who can apply appropriate tools and identify appropriate statistical methods as well as behavioral models to develop data-processing algorithms. They can then design human-friendly interfaces that can collect useful data and subsequently deliver decisions.

In order to know which model or approach to implement, it is important to understand a) how modern use cases work and b) the limitations of certain systems. Professional education programs can fill this need by exposing data scientists to concepts outside their own domain and allowing them to interact with practitioners of other disciplines across industries. 

There are a variety of approaches available to build recommendation systems – but it isn’t always necessary for data scientists to start from scratch when building their own. The vast array of publicly available Recommenders (in all sorts of programming environments) can be used as a foundation. Some examples are: RecommenderLab for R, Graphlab-Create for Python, and Apache Spark's Recommendation module

If there is one thing that’s clear, it is that businesses no longer need to rely on intuition when it comes to choosing which products to spotlight for customers. Recommendation systems can now effectively be used to accurately determine which products customers are most likely to buy. And the success of modern companies like Amazon and Netflix is proof using a recommendation system can have a big impact on business. Prediction is going to get more important as big data and technologies evolve. Data scientists must move quickly to acquire the skills and knowledge necessary to keep pace so they can innovate as well, and improve the performance of their own organizations.   

Devavrat Shah, co-director of the Data Science: Data to Insights course, is a professor in MIT’s Department of Electrical Engineering and Computer Science, director of the SDSC, and a core faculty member at the IDSS. He is also a member of MIT’s Laboratory for Information and Decision Systems (LIDS) and the Operations Research Center (ORC). 




Edited by Ken Briodagh
Get stories like this delivered straight to your inbox. [Free eNews Subscription]


SHARE THIS ARTICLE
Related Articles

Rising Edge Computing Investments to Reach $350B by 2027, According to IDC

By: Alex Passett    3/27/2024

Worldwide spending on edge computing is expected to surge (and then keep going) for the foreseeable future, according to the International Data Corpor…

Read More

ZEDEDA Adds Lisa Edwards as New Board Member, Seeks Opportunities to Strengthen Operations and Scale

By: Alex Passett    3/26/2024

Earlier this morning, ZEDEDA announced the addition of Lisa Edwards to its board of directors.

Read More

An Existing IoT Collab, Emboldened: Digi International and Telit Cinterion Transform Solutions with 5G RedCap Integration

By: Alex Passett    3/25/2024

The ongoing industry collaboration between Digi International and Telit Cinterion signals strong support for the mainstream showcasing of 5G for IoT a…

Read More

Telit Cinterion's 5G LGA Modules, Powered by Snapdragon from Qualcomm, to Create a Big Leap in IoT Connectivity

By: Alex Passett    3/25/2024

Telit Cinterion recently unveiled its FE990B34/40 LGA family of modules, powered by the Snapdragon X72 5G Modem-RF System from Qualcomm Technologies, …

Read More

Embracing Innovation in Mining: The Role of Network-Aware Applications in the Digital Transformation

By: Special Guest    3/21/2024

Shabodi leverages private 5G network capabilities and enables the development of network-aware applications to enhance operational efficiency, automat…

Read More