Big Data: 8 Trend - Revision history

Onnowpurbo: /* Deep learning */

2023-02-13T03:02:17Z

Deep learning

Onnowpurbo: /* Big data lakes */

2023-02-13T02:22:04Z

Big data lakes

Onnowpurbo: /* In-memory analytics */

2015-11-08T23:18:58Z

In-memory analytics

Onnowpurbo: /* Deep learning */

2015-11-08T23:16:55Z

Deep learning

Onnowpurbo: /* More, better NoSQL */

2015-11-08T23:10:40Z

More, better NoSQL

Onnowpurbo: /* SQL on Hadoop: Faster, better */

2015-11-08T23:08:11Z

SQL on Hadoop: Faster, better

Onnowpurbo at 22:36, 8 November 2015

2015-11-08T22:36:41Z

Onnowpurbo at 00:54, 6 November 2015

2015-11-06T00:54:12Z

Onnowpurbo: New page: Sumber: http://www.cio.com/article/2838172/data-analytics/8-big-trends-in-big-data-analytics.html Bill Loconzolo, vice president of data engineering at Intuit, jumped into a data lake w...

2015-11-06T00:53:52Z

New page: Sumber: http://www.cio.com/article/2838172/data-analytics/8-big-trends-in-big-data-analytics.html Bill Loconzolo, vice president of data engineering at Intuit, jumped into a data lake w...

New page

Sumber: http://www.cio.com/article/2838172/data-analytics/8-big-trends-in-big-data-analytics.html

Bill Loconzolo, vice president of data engineering at Intuit, jumped into a data lake with both feet. Dean Abbott, chief data scientist at Smarter Remarketer, made a beeline for the cloud. The leading edge of big data and analytics, which includes data lakes for holding vast stores of data in its native format and, of course, cloud computing, is a moving target, both say. And while the technology options are far from mature, waiting simply isn’t an option.
resume makeover executive

“The reality is that the tools are still emerging, and the promise of the [Hadoop] platform is not at the level it needs to be for business to rely on it,” says Loconzolo. But the disciplines of big data and analytics are evolving so quickly that businesses need to wade in or risk being left behind. “In the past, emerging technologies might have taken years to mature,” he says. “Now people iterate and drive solutions in a matter of months — or weeks.” So what are the top emerging technologies and trends that should be on your watch list — or in your test lab? Computerworld asked IT leaders, consultants and industry analysts to weigh in. Here’s their list.

==Big data analytics in the cloud==

Hadoop, a framework and set of tools for processing very large data sets, was originally designed to work on clusters of physical machines. That has changed. “Now an increasing number of technologies are available for processing data in the cloud,” says Brian Hopkins, an analyst at Forrester Research. Examples include Amazon’s Redshift hosted BI data warehouse, Google’s BigQuery data analytics service, IBM’s Bluemix cloud platform and Amazon’s Kinesis data processing service. “The future state of big data will be a hybrid of on-premises and cloud,” he says.

Smarter Remarketer, a provider of SaaS-based retail analytics, segmentation and marketing services, recently moved from an in-house Hadoop and MongoDB database infrastructure to the Amazon Redshift, a cloud-based data warehouse. The Indianapolis-based company collects online and brick-and-mortar retail sales and customer demographic data, as well as real-time behavioral data and then analyzes that information to help retailers create targeted messaging to elicit a desired response on the part of shoppers, in some cases in real time.

Redshift was more cost-effective for Smart Remarketer’s data needs, Abbott says, especially since it has extensive reporting capabilities for structured data. And as a hosted offering, it’s both scalable and relatively easy to use. “It’s cheaper to expand on virtual machines than buy physical machines to manage ourselves,” he says.

For its part, Mountain View, Calif.-based Intuit has moved cautiously toward cloud analytics because it needs a secure, stable and auditable environment. For now, the financial software company is keeping everything within its private Intuit Analytics Cloud. “We’re partnering with Amazon and Cloudera on how to have a public-private, highly available and secure analytic cloud that can span both worlds, but no one has solved this yet,” says Loconzolo. However, a move to the cloud is inevitable for a company like Intuit that sells products that run in the cloud. “It will get to a point where it will be cost-prohibitive to move all of that data to a private cloud,” he says.

==Hadoop: The new enterprise data operating system==

Distributed analytic frameworks, such as MapReduce, are evolving into distributed resource managers that are gradually turning Hadoop into a general-purpose data operating system, says Hopkins. With these systems, he says, “you can perform many different data manipulations and analytics operations by plugging them into Hadoop as the distributed file storage system.”

What does this mean for the enterprise? As SQL, MapReduce, in-memory, stream processing, graph analytics and other types of workloads are able to run on Hadoop with adequate performance, more businesses will use Hadoop as an enterprise data hub. “The ability to run many different kinds of [queries and data operations] against data in Hadoop will make it a low-cost, general-purpose place to put data that you want to be able to analyze,” Hopkins says.
Hybrid cloud: Building a better foundation for business growth

Intuit is already building on its Hadoop foundation. “Our strategy is to leverage the Hadoop Distributed File System, which works closely with MapReduce and Hadoop, as a long-term strategy to enable all types of interactions with people and products,” says Loconzolo.

==Big data lakes==

Traditional database theory dictates that you design the data set before entering any data. A data lake, also called an enterprise data lake or enterprise data hub, turns that model on its head, says Chris Curran, principal and chief technologist in PricewaterhouseCoopers’ U.S. advisory practice. “It says we’ll take these data sources and dump them all into a big Hadoop repository, and we won’t try to design a data model beforehand,” he says. Instead, it provides tools for people to analyze the data, along with a high-level definition of what data exists in the lake. “People build the views into the data as they go along. It’s a very incremental, organic model for building a large-scale database,” Curran says. On the downside, the people who use it must be highly skilled.
Chris Curran, principal and chief technologist, PricewaterhouseCoopers’ U.S. advisory practice

"People build the views into the data as they go along. It’s a very incremental, organic model for building a large-scale database," says PwC's Chris Curran.

As part of its Intuit Analytics Cloud, Intuit has a data lake that includes clickstream user data and enterprise and third-party data, says Loconzolo, but the focus is on “democratizing” the tools surrounding it to enable business people to use it effectively. Loconzolo says one of his concerns with building a data lake in Hadoop is that the platform isn’t really enterprise-ready. “We want the capabilities that traditional enterprise databases have had for decades — monitoring access control, encryption, securing the data and tracing the lineage of data from source to destination,” he says.

==More predictive analytics==

With big data, analysts have not only more data to work with, but also the processing power to handle large numbers of records with many attributes, Hopkins says. Traditional machine learning uses statistical analysis based on a sample of a total data set. “You now have the ability to do very large numbers of records and very large numbers of attributes per record” and that increases predictability, he says.

The combination of big data and compute power also lets analysts explore new behavioral data throughout the day, such as websites visited or location. Hopkins calls that “sparse data,” because to find something of interest you must wade through a lot of data that doesn’t matter. “Trying to use traditional machine-learning algorithms against this type of data was computationally impossible. Now we can bring cheap computational power to the problem,” he says. “You formulate problems completely differently when speed and memory cease being critical issues,” Abbott says. “Now you can find which variables are best analytically by thrusting huge computing resources at the problem. It really is a game changer.”

“To enable real-time analysis and predictive modeling out of the same Hadoop core, that’s where the interest is for us,” says Loconzolo. The problem has been speed, with Hadoop taking up to 20 times longer to get questions answered than did more established technologies. So Intuit is testing Apache Spark, a large-scale data processing engine, and its associated SQL query tool, Spark SQL. “Spark has this fast interactive query as well as graph services and streaming capabilities. It is keeping the data within Hadoop, but giving enough performance to close the gap for us,” Loconzolo says.

==SQL on Hadoop: Faster, better==

If you’re a smart coder and mathematician, you can drop data in and do an analysis on anything in Hadoop. That’s the promise — and the problem, says Mark Beyer, an analyst at Gartner. “I need someone to put it into a format and language structure that I’m familiar with,” he says. That’s where SQL for Hadoop products come in, although any familiar language could work, says Beyer. Tools that support SQL-like querying let business users who already understand SQL apply similar techniques to that data. SQL on Hadoop “opens the door to Hadoop in the enterprise,” Hopkins says, because businesses don’t need to make an investment in high-end data scientists and business analysts who can write scripts using Java, JavaScript and Python — something Hadoop users have traditionally needed to do.

These tools are nothing new. Apache Hive has offered a structured a structured, SQL-like query language for Hadoop for some time. But commercial alternatives from Cloudera, Pivotal Software, IBM and other vendors not only offer much higher performance, but also are getting faster all the time. That makes the technology a good fit for “iterative analytics,” where an analyst asks one question, receives an answer, and then asks another one. That type of work has traditionally required building a data warehouse. SQL on Hadoop isn’t going to replace data warehouses, at least not anytime soon, says Hopkins, “but it does offer alternatives to more costly software and appliances for certain types of analytics.”

==More, better NoSQL==

Alternatives to traditional SQL-based relational databases, called NoSQL (short for “Not Only SQL”) databases, are rapidly gaining popularity as tools for use in specific kinds of analytic applications, and that momentum will continue to grow, says Curran. He estimates that there are 15 to 20 open-source NoSQL databases out there, each with its own specialization. For example, a NoSQL product with graph database capability, such as ArangoDB, offers a faster, more direct way to analyze the network of relationships between customers or salespeople than does a relational database. “These databases have been around for a while, but they’re picking up steam because of the kinds of analyses people need,” he says. One PwC client in an emerging market has placed sensors on store shelving to monitor what products are there, how long customers handle them and how long shoppers stand in front of particular shelves. “These sensors are spewing off streams of data that will grow exponentially,” Curran says. “A NoSQL key-value pair database such as Redis is the place to go for this because it’s special-purpose, high-performance and lightweight.”

==Deep learning==

Deep learning, a set of machine-learning techniques based on neural networking, is still evolving but shows great potential for solving business problems, says Hopkins. “Deep learning . . . enables computers to recognize items of interest in large quantities of unstructured and binary data, and to deduce relationships without needing specific models or programming instructions,” he says.

In one example, a deep learning algorithm that examined data from Wikipedia learned on its own that California and Texas are both states in the U.S. “It doesn’t have to be modeled to understand the concept of a state and country, and that’s a big difference between older machine learning and emerging deep learning methods,” Hopkins says.

“Big data will do things with lots of diverse and unstructured text using advanced analytic techniques like deep learning to help in ways that we only now are beginning to understand,” Hopkins says. For example, it could be used to recognize many different kinds of data, such as the shapes, colors and objects in a video — or even the presence of a cat within images, as a neural network built by Google famously did in 2012. “This notion of cognitive engagement, advanced analytics and the things it implies . . . are an important future trend,” Hopkins says.

==In-memory analytics==

The use of in-memory databases to speed up analytic processing is increasingly popular and highly beneficial in the right setting, says Beyer. In fact, many businesses are already leveraging hybrid transaction/analytical processing (HTAP) — allowing transactions and analytic processing to reside in the same in-memory database.

But there’s a lot of hype around HTAP, and businesses have been overusing it, Beyer says. For systems where the user needs to see the same data in the same way many times during the day — and there’s no significant change in the data — in-memory is a waste of money.

And while you can perform analytics faster with HTAP, all of the transactions must reside within the same database. The problem, says Beyer, is that most analytics efforts today are about putting transactions from many different systems together. “Just putting it all on one database goes back to this disproven belief that if you want to use HTAP for all of your analytics, it requires all of your transactions to be in one place,” he says. “You still have to integrate diverse data.”

Moreover, bringing in an in-memory database means there’s another product to manage, secure, and figure out how to integrate and scale.

For Intuit, the use of Spark has taken away some of the urge to embrace in-memory databases. “If we can solve 70% of our use cases with Spark infrastructure and an in-memory system could solve 100%, we’ll go with the 70% in our analytic cloud,” Loconzolo says. “So we will prototype, see if it’s ready and pause on in-memory systems internally right now.”
Staying one step ahead

With so many emerging trends around big data and analytics, IT organizations need to create conditions that will allow analysts and data scientists to experiment. “You need a way to evaluate, prototype and eventually integrate some of these technologies into the business,” says Curran.

“IT managers and implementers cannot use lack of maturity as an excuse to halt experimentation,” says Beyer. Initially, only a few people — the most skilled analysts and data scientists — need to experiment. Then those advanced users and IT should jointly determine when to deliver new resources to the rest of the organization. And IT shouldn’t necessarily rein in analysts who want to move ahead full-throttle. Rather, Beyer says, IT needs to work with analysts to “put a variable-speed throttle on these new high-powered tools.”

==Referensi==

* http://www.cio.com/article/2838172/data-analytics/8-big-trends-in-big-data-analytics.html

@@ Line 34: / Line 34: @@
 ==Deep learning==
-Deep learning, sebuah teknik machine-learning berbasis pada neural networking, yang masih berkembang dan memperlihatkan potensi yang sangat besar untuk menyelesaikan banyak masalah di bisnis. Dalam Deep learning membuka kemungkinan komputer untuk mengetahui hal yang menarik dalam jumlah besar dari sebuah data binary yang tidak terstruktur, untuk me-reduksi hubungan yang ada tanpa membutuhkan model yang spesifik atau instruksi pemrograman.
+Deep learning, sebuah teknik machine-learning berbasis pada neural network, yang masih berkembang dan memperlihatkan potensi yang sangat besar untuk menyelesaikan banyak masalah di bisnis. Dalam Deep learning membuka kemungkinan komputer untuk mengetahui hal yang menarik dalam jumlah besar dari sebuah data binary yang tidak terstruktur, untuk me-reduksi hubungan yang ada tanpa membutuhkan model yang spesifik atau instruksi pemrograman.
-Contoh, algoritma deep learning akan dapay menganalisa data dari Wikipedia dan mengetahui bahwa Aceh dan Bali adalah sebuah propinsi di Indonesia tanpa perlu mengetahui konsep dan model sebuah negara dan propinsi.
+Contoh, algoritma deep learning akan dapat menganalisa data dari Wikipedia dan mengetahui bahwa Aceh dan Bali adalah sebuah propinsi di Indonesia tanpa perlu mengetahui konsep dan model sebuah negara dan propinsi.
 ==In-memory analytics==

@@ Line 13: / Line 13: @@
 Artinya untuk sebuah perusahaan? Karena SQL, MapReduce, in-memory, stream processing, graph analytics dan berbagai beban akan dapat di jalankan oleh Hadoop dengan kinerja yang baik, semakin banyak usaha yang akan menggunakan Hadoop sebagai enterprise data hub.
-==Big data lakes==
+==Danau Big data==
-Teori database tradisional men-dikte agar kita mendisain data set sebelum memasukan daya. Dalam sebuah data lake (danau daya), kita memutar balikan cara ini. Artinya, kita akan mengambil semua sumber daya dan memasukan semua ke repository Hadoop yang besar, dan kita tidak men-disain data model sebelumnya.
+Teori database tradisional men-dikte agar kita mendisain data set sebelum memasukan daya. Dalam sebuah data lake (danau resource), kita memutar balikan cara ini. Artinya, kita akan mengambil semua sumber daya dan memasukan semua ke repository Hadoop yang besar, dan kita tidak men-disain data model sebelumnya.
 Kita memberikan tool untuk orang melakukan analisa data, dengan definisi high-level akan data yang ada di lake (danau) tersebut. Orang akan membuat penerawangan terhadap data sambil jalan, jadi sangat perlahan, model organik untuk membuat database skala besar. Kesulitannya, orang yang melalukan ini harus sangat cakap / pandai.
 ==More predictive analytics==

← Older revision		Revision as of 00:54, 6 November 2015
Line 2:		Line 2:


−	Bill Loconzolo, vice president of data engineering at Intuit, jumped into a data lake with both feet. Dean Abbott, chief data scientist at Smarter Remarketer, made a beeline for the cloud. The leading edge of big data and analytics, which includes data lakes for holding vast stores of data in its native format and, of course, cloud computing, is a moving target, both say. And while the technology options are far from mature, waiting simply isn’t an option.	+	Bill Loconzolo, vice president of data engineering at Intuit, jumped into a data lake with both feet. Dean Abbott, chief data scientist at Smarter Remarketer, made a beeline for the cloud. The leading edge of big data and analytics, which includes data lakes for holding vast stores of data in its native format and, of course, cloud computing, is a moving target, both say. And while the technology options are far from mature, waiting simply isn’t an option.
−	~~resume makeover executive~~
−

	“The reality is that the tools are still emerging, and the promise of the [Hadoop] platform is not at the level it needs to be for business to rely on it,” says Loconzolo. But the disciplines of big data and analytics are evolving so quickly that businesses need to wade in or risk being left behind. “In the past, emerging technologies might have taken years to mature,” he says. “Now people iterate and drive solutions in a matter of months — or weeks.” So what are the top emerging technologies and trends that should be on your watch list — or in your test lab? Computerworld asked IT leaders, consultants and industry analysts to weigh in. Here’s their list.		“The reality is that the tools are still emerging, and the promise of the [Hadoop] platform is not at the level it needs to be for business to rely on it,” says Loconzolo. But the disciplines of big data and analytics are evolving so quickly that businesses need to wade in or risk being left behind. “In the past, emerging technologies might have taken years to mature,” he says. “Now people iterate and drive solutions in a matter of months — or weeks.” So what are the top emerging technologies and trends that should be on your watch list — or in your test lab? Computerworld asked IT leaders, consultants and industry analysts to weigh in. Here’s their list.

@@ Line 41: / Line 41: @@
 ==In-memory analytics==
-The use of in-memory databases to speed up analytic processing is increasingly popular and highly beneficial in the right setting, says Beyer. In fact, many businesses are already leveraging hybrid transaction/analytical processing (HTAP) — allowing transactions and analytic processing to reside in the same in-memory database.
+Penggunaan in-memory database sangat mempercepat proses analitik dan saat ini semakin populer dan sangat menguntungkan jika di set dengan benar. Banyak usaha menjalankan hybrid transaction/analytical processing (HTAP) — yang memungkinkan proses transaksi dan analitik dilakukan pada in-memory database yang sama.
 ==Referensi==
 * http://www.cio.com/article/2838172/data-analytics/8-big-trends-in-big-data-analytics.html

@@ Line 31: / Line 31: @@
 ==More, better NoSQL==
-Alternatives to traditional SQL-based relational databases, called NoSQL (short for “Not Only SQL”) databases, are rapidly gaining popularity as tools for use in specific kinds of analytic applications, and that momentum will continue to grow, says Curran. He estimates that there are 15 to 20 open-source NoSQL databases out there, each with its own specialization. For example, a NoSQL product with graph database capability, such as ArangoDB, offers a faster, more direct way to analyze the network of relationships between customers or salespeople than does a relational database. “These databases have been around for a while, but they’re picking up steam because of the kinds of analyses people need,” he says. One PwC client in an emerging market has placed sensors on store shelving to monitor what products are there, how long customers handle them and how long shoppers stand in front of particular shelves. “These sensors are spewing off streams of data that will grow exponentially,” Curran says. “A NoSQL key-value pair database such as Redis is the place to go for this because it’s special-purpose, high-performance and lightweight.”
+Berbagai alternatif dari SQL-based relational database, di sebut NoSQL (kependekan dari “Not Only SQL”) database, saat ini memperoleh popularitas sebagai tool untuk aplikasi analitik yang spesifik.
 ==Deep learning==

@@ Line 27: / Line 27: @@
 ==SQL on Hadoop: Faster, better==
-If you’re a smart coder and mathematician, you can drop data in and do an analysis on anything in Hadoop. That’s the promise — and the problem, says Mark Beyer, an analyst at Gartner. “I need someone to put it into a format and language structure that I’m familiar with,” he says. That’s where SQL for Hadoop products come in, although any familiar language could work, says Beyer. Tools that support SQL-like querying let business users who already understand SQL apply similar techniques to that data. SQL on Hadoop “opens the door to Hadoop in the enterprise,” Hopkins says, because businesses don’t need to make an investment in high-end data scientists and business analysts who can write scripts using Java, JavaScript and Python — something Hadoop users have traditionally needed to do.
+Jika anda seorang coder yang pandai dan ahli matematika, anda dapat memasukan semua data dan melakukan analisa di Hadoop. Agar memudahkan orang maka dikembangkan SQL untuk Hadoop. Apache Hive memberikan kemungkinan untuk menggunakan SQL-like query untuk Hadoop.
 ==More, better NoSQL==

@@ Line 1: / Line 1: @@
 Sumber: http://www.cio.com/article/2838172/data-analytics/8-big-trends-in-big-data-analytics.html
 ==Big data analytics in the cloud==
-Hadoop, a framework and set of tools for processing very large data sets, was originally designed to work on clusters of physical machines. That has changed. “Now an increasing number of technologies are available for processing data in the cloud,” says Brian Hopkins, an analyst at Forrester Research. Examples include Amazon’s Redshift hosted BI data warehouse, Google’s BigQuery data analytics service, IBM’s Bluemix cloud platform and Amazon’s Kinesis data processing service. “The future state of big data will be a hybrid of on-premises and cloud,” he says.
+Hadoop, framework dan tool untuk memproses set data yang sangat besar, pada awalnya dirancang untuk bekerja pada kluster mesin fisik. Sekarang semakin banyak teknologi yang tersedia untuk pengolahan data di Cloud. Lebih murah untuk memperluas pada mesin virtual dari membeli mesin sendiri secara fisik dan mengelolanya sendiri.
 ==Hadoop: The new enterprise data operating system==
-Distributed analytic frameworks, such as MapReduce, are evolving into distributed resource managers that are gradually turning Hadoop into a general-purpose data operating system, says Hopkins. With these systems, he says, “you can perform many different data manipulations and analytics operations by plugging them into Hadoop as the distributed file storage system.”
+Distributed analytic frameworks, seperti MapReduce, berevolusi menjadi distributed resource manager yang perlahan akan membuat Hadoop menjadi general-purpose data operating system. Kita dapat melakukan berbagai manipulasi data maupun operasi analitik dengan memasukan data ke Hadoop sebagai distributed file storage system.
-What does this mean for the enterprise? As SQL, MapReduce, in-memory, stream processing, graph analytics and other types of workloads are able to run on Hadoop with adequate performance, more businesses will use Hadoop as an enterprise data hub. “The ability to run many different kinds of [queries and data operations] against data in Hadoop will make it a low-cost, general-purpose place to put data that you want to be able to analyze,” Hopkins says.
+Artinya untuk sebuah perusahaan? Karena SQL, MapReduce, in-memory, stream processing, graph analytics dan berbagai beban akan dapat di jalankan oleh Hadoop dengan kinerja yang baik, semakin banyak usaha yang akan menggunakan Hadoop sebagai enterprise data hub.
 ==Big data lakes==
-Traditional database theory dictates that you design the data set before entering any data. A data lake, also called an enterprise data lake or enterprise data hub, turns that model on its head, says Chris Curran, principal and chief technologist in PricewaterhouseCoopers’ U.S. advisory practice. “It says we’ll take these data sources and dump them all into a big Hadoop repository, and we won’t try to design a data model beforehand,” he says. Instead, it provides tools for people to analyze the data, along with a high-level definition of what data exists in the lake. “People build the views into the data as they go along. It’s a very incremental, organic model for building a large-scale database,” Curran says. On the downside, the people who use it must be highly skilled.
+Teori database tradisional men-dikte agar kita mendisain data set sebelum memasukan daya. Dalam sebuah data lake (danau daya), kita memutar balikan cara ini. Artinya, kita akan mengambil semua sumber daya dan memasukan semua ke repository Hadoop yang besar, dan kita tidak men-disain data model sebelumnya.
-"People build the views into the data as they go along. It’s a very incremental, organic model for building a large-scale database," says PwC's Chris Curran.
+Kita memberikan tool untuk orang melakukan analisa data, dengan definisi high-level akan data yang ada di lake (danau) tersebut. Orang akan membuat penerawangan terhadap data sambil jalan, jadi sangat perlahan, model organik untuk membuat database skala besar. Kesulitannya, orang yang melalukan ini harus sangat cakap / pandai.
 ==More predictive analytics==
-With big data, analysts have not only more data to work with, but also the processing power to handle large numbers of records with many attributes, Hopkins says. Traditional machine learning uses statistical analysis based on a sample of a total data set. “You now have the ability to do very large numbers of records and very large numbers of attributes per record” and that increases predictability, he says.
+Dengan big data, analis akan mempunyai lebih banyak data untuk bekerja, juga processing power untuk menangani record yang besar dengan banyak atribut. Mesin learning tradisional menggunakan analisa statistik berbasis pada sample dari total data set. Sekarang kita mempunyai kemampuan menganalisa record yang sangat besar dengan banyak atribut per record.
 ==SQL on Hadoop: Faster, better==