I page single space about article review and self reflection.
ANALYTICS
Why You’re Not Getting Value
from Your Data Science
by
Kalyan Veeramachaneni
DECEMBER 07, 2016
Businesses today are constantly generating enormous amounts of data, but that doesn’t always
translate to actionable information. Over the past several years, my research group at MIT and I
have sought answers to a fundamental question: Wh
at would it take for businesses to realize the
full potential of their data repositories with machine learning?
As we worked to design machine learning–based solutions with a variety of industry partners, we
were surprised to find that the existing answers to this question often didn’t apply. Wh
y?
INSIGHT CENTER
The Next Analytics Age
SPONSORED BY SAS
Harnessing the power of machine learning and other
technologies.
First, whenever we spoke with machine learning experts (data scientists focused on training and
testing predictive models) about the most difficult part of their job, they said again and again,
“the data is a mess.” Initially, taking that statement literally, we imagined it referred to well-
known issues with data — missing values or a lack of coherence across databases. But as we dug
deeper, we realized the problem was slightly different. In its rawest form, even clean data is too
overwhelming and complex to be understood at first glance, even by experts. It has too many
tables and fields and is often collected at a very high granularity (for example, online clickstreams
generate new data with every click, and sensor data is collected at 125 observations per second).
Machine learning experts are used to working with data that’s already been aggregated into
useful variables, such as the number of website visits by a user, rather than a table of every action
the user has ever taken on the site.
At the same time, we often heard business
experts complain that “we have a lot of data and
we are not doing anything with it.” Further
investigation revealed that this was not strictly
correct either. Instead, this frustration stems
from two problems. For one thing, due to the
time it takes to understand, formulate, and
process data for a machine learning problem, machine learning experts often instead focus on the
later parts of the pipeline—trying different models, or tuning the hyperparameters of the model
once a problem is formulated, rather than formulating newer predictive questions for different
business problems. Therefore, while business experts are coming up with problems, machine
learning experts cannot always keep up.
For another, machine learning experts often didn’t build their work around the final objective—
deriving business value. In most cases, predictive models are meant to improve efficiency,
increase revenue, or reduce costs. But the folks actually working on the models rarely ask “what
value does this predictive model provide, and how can we measure it?” Asking this question
about value proposition often leads to a change in the original problem formulation, and asking
such questions is often more useful than tweaking later stages of the process. At a recent panel
filled with machine learning enthusiasts, I polled the audience of about 150 people, asking “How
many of you have built a machine learning model?” Roughly one-third raised their hands. Next, I
asked, “How many of you have deployed and/or used this model to generate value, and evaluated
it?” No one had their hand up.
In other words, the machine learning experts wanted to spend their time building models, not
processing massive datasets or translating business problems into prediction problems. Likewise,
the current technological landscape, both commercial and academic, focuses on enabling more
sophisticated models (via Latent variable models), scaling model learning algorithms (via
distributed compute), or fine-tuning (via Bayesian hyper optimization)—essentially all later
stages of the data science pipeline. However, in our experience, we found this focus to be
misplaced.
If companies want to get value from their data, they need to focus on accelerating human
understanding of data, scaling the number of modeling questions they can ask of that data in a
short amount of time, and assessing their implications. In our work with companies, we
ultimately decided that creating true impact via machine learning will come from a focus on four
principles:
Stick with simple models:
We decided that simple models, like logistic regression or those based
on
random forests
or decision trees, are sufficient for the problems at hand. The focus should
instead be on reducing the time between the data acquisition and the development of the first
simple predictive model.
Explore more problems:
Data scientists need the ability to rapidly define and explore multiple
prediction problems, quickly and easily. Instead of exploring one business problem with an
incredibly sophisticated machine learning model, companies should be exploring dozens,
building a simple predictive model for each one and assessing their value proposition.
Learn from a sample of data—not all the data:
Instead of focusing on how to apply distributed
computing to allow any individual processing module to handle big data, invest in techniques
that will enable the derivations of similar conclusions from a data subsample. By circumventing
the use of massive computing resources, they will enable the exploration of more hypotheses.
Focus on automation:
To achieve both
reduced time to first model
and
increased rate of
exploration
, companies must automate processes that are normally done manually. Over and over
across different data problems, we found ourselves applying similar data processing techniques,
whether it was to transform the data into useful aggregates, or to prepare data for predictive
modeling—it’s time to streamline these, and to develop algorithms and build software systems
that do them automatically.
This acute understanding of how data scientists interact with data and where the bottlenecks are
led us to launch “The Human-Data Interaction Project” at MIT, focusing on the goals listed above.
Our target is rapid exploration of predictive models, and to actually put them to use by solving
real problems in real organizations. These models will be simple, and automation will enable
even naive users to develop hundreds if not thousands of predictive models within hours—
something that, today, takes experts entire months.
Kalyan Veeramachaneni
is a Principal research scientist at the
Laboratory for Information and Decision
systems (LIDS)
at MIT, where he directs a research group called
Data-to-AI
. He is also cofounder of
PatternEx
and
Feature Labs
. His website is
http://www.kalyanv.org
.
This article is about
ANALYTICS
FOLLOW
THIS TOPIC
Comments
Leave a Comment
P O S T
5
COMMENTS
David Relyea
8 hours ago
"machine learning experts not building models around business value": This should have been your first point. Out
of all the data scientists I've met and interviewed, I'd estimate less than 10% do this. It's a staggering failing for
teams that are otherwise rather intelligent.
"stick with simple models": Yes, for a proof of concept. Absolutely not, if you're rolling something into production.
There are most certainly data scientists who get lost in the weeds of model complication, but I've seen too many
teams be blind-sided by consultants who out-perform the entire team with off-the-shelf NN technology. (I've
done this to more than one team myself.) You should use a RF or a linear model to begin, and once you've
demonstrated business value, absolutely run a more complicated model to draw out the rest of the value. Don't
let them spend *that* much time on it (a few days to a week should suffice). If they need more time, hire better
REPLY
0
0
data scientists. And on a similar point...
"At a recent panel filled with machine learning enthusiasts": This is what you should have focused another third of
the article on. I've interviewed almost 50 people at this point for DS positions (and have seen probably 200
resumes), and the majority of self-professed "data scientists" are enthusiasts rather than professionals. If you
were speaking to 150 people who hadn't put a model into production (or at least evaluated the model's
performance on whatever KPIs it addresses), you were speaking to 150 people who weren't data scientists.
Overall, the article makes good points (especially about companies' need to hire good data teams to organize
data). However, please tell people to have high standards when they hire data scientists, and please don't tell
business leaders to implement simple models. When it comes to improving the bottom line, that's the wrong
advice.
POSTING GUIDELINES
We hope the conversations that take place on HBR.org will be energetic, constructive, and thought-provoking. To comment, readers must
sign in
or
register
. And to ensure the quality of the discussion, our moderating team will review all comments and may edit them for clarity, length, and
relevance. Comments that are overly promotional, mean-spirited, or off-topic may be deleted per the moderators' judgment. All postings become
the property of Harvard Business Publishing.
JOIN THE CONVERSATION
Answers
Status NEW
Posted 24 Jul 2017 08:07 AM
My Price 10.00
----------- He-----------llo----------- Si-----------r/M-----------ada-----------m -----------Tha-----------nk -----------You----------- fo-----------r u-----------sin-----------g o-----------ur -----------web-----------sit-----------e a-----------nd -----------acq-----------uis-----------iti-----------on -----------of -----------my -----------pos-----------ted----------- so-----------lut-----------ion-----------. P-----------lea-----------se -----------pin-----------g m-----------e o-----------n c-----------hat----------- I -----------am -----------onl-----------ine----------- or----------- in-----------box----------- me----------- a -----------mes-----------sag-----------e I----------- wi-----------ll
Not Rated(0)