I get that question from some of my officemates. How can people appreciate “Big Data Technologies”. Is there a demo to make people appreciate “Big Data” or “Hadoop”? How can we sell “Data Lakes vs Data Warehouses”.
First, it’s hard to appreciate a piece of technology without knowing what pain it actually solves. Hadoop solves the pain of computing by presenting this conventional way of doing Distributed Computing.
Second, the term “Big Data” is being passed around too much, it’s getting too abstracted. We need to get a little lower down the chain, closer to real things and tasks entailing “doing Big Data”. As I said before, I feel that “Big Data” is a marketing term thrown for problems/solutions revolving around “Distributed Computing and Storage”, much like how “Cloud” is for “Remote Servers, Services and Applications”. At the tip of these solutions is something called Hadoop. Now, Hadoop is more of a framework, accompanied with a set of libraries to implement that framework. At the current state of the software industry, there’s no single piece of technology that magically translates your data into wonderful visualizations and actionable insights that’s instantly aligned to making your business more money.
People have been blinded by technology. We’re so impressed by “Big Data” and all these buzzwords revolving around technology. We forget that technology isn’t meant to replace humans and how we process information, but it was made to complement us:
The other buzzword that epitomizes a bias toward substitution is “big data”. Today’s companies have an insatiable appetite for data, mistakenly believeing that more data always creates more value. But big data is usually dumb data. Computers can find patterns that elude humans, but they don’t know how to compare patterns from different sources or how to interpret complex behaviors. Actionable insights can only come from a human analyst (or the kind of generalized artifical intelligence that exists only in science fiction).
We have let ourselves become enchanted by big data only because we exoticize technology. We’re impressed with small feats accomplished by computers alone but we ignore achievements from complementarity because the human contribution makes them less uncanny.
— Peter Thiel (Founder of Paypal and Palantir), from “Zero To One”
Big Data technologies such as Hadoop, HBase, NiFi, Tableau and Spark are tools for Developers and Analysts to make their lives easier. It makes it easier for them to crunch numbers, transform data, and other stuff to make those charts and visualizations the business guys are looking for.
That’s why it’s hard to sell them to business guys as some sort of “magic box”. There’s no “demo” where they can instantly “appreciate it” without human intervention.
It’s like saying “How can executives/business guys ‘appreciate’ Ruby on Rails?”.
It’s hard to sell that to them.
Demos with fancy charts won’t work on most business guys: “Hey, our current IT Team can do that with their databases and BI tools. What’s new?”
So, “how can executive/business guys ‘appreciate’ Big Data?”. Take this with a grain of salt though, I’m not a biz dev guy: Sell it to their VP of Engineering, their developers, their technical team. Naturally they’ll build great apps and dashboards on top of it. It’ll sell itself to the rest of the organization.
Disclaimer: My thoughts written here are not of my employers. These are solely my opinions and they’re not tied to what AGSX stands for.