Spark and Hadoop – replace or complement?

I recently read a survey report from Typesafe of 2,136 respondents who were asked about Spark vs Hadoop. You can see the report here for yourself (registration required) https://info.typesafe.com/COLL-20XX-Spark-Survey-Report_LP.html?lst=RW&lsd=COLL-20XX-Spark-Survey-Trends-Adoption-Report

The most interesting part of the report for me was that 78% of respondents were using Spark for fast processing of BATCH data sets! Think about that. Spark can work with HDFS as the persistent data store but Spark is really good at processing streaming, transactional data – but – most respondents are just using it to make batch go faster.

This is our experience too – when we talk to customers they want to consider Spark, they know they have to think about future use cases which will almost certainly involve streaming data, transactional data sets and – most importantly – real time analytics and machine learning. But – for now, even ten years after Doug Cutting and Mike Cafarella invented Hadoop – we are still seeing the vast majority of use case focused on batch processing. It really is – back to the 80’s!

So – in my view – Spark is not replacing Hadoop but is simply complementing what is already out there. What do you think?

This Aptuz blog also summarizes neatly the Spark vs Hadoop discussion http://aptuz.com/blog/is-apache-spark-going-to-replace-hadoop/

If you have additional questions, get in touch with us!

7 + 12 =

EXCELERATE SYSTEMS

Headquartered in Redmond, Washington, Excelerate Systems operates in the United States, Canada, Latin America, Europe, Australia and New Zealand.

Corporate Head Quarters

  2205 152nd Avenue NE
Redmond, WA 98052
USA

 +1.(425).605.1289

European Head Office (France)

  Les Bureaux du Lac II Rue Robert Caumont, imm P 33049 Bordeaux         Cedex – France

 +33 (0)5 56.07.23.33

Latin America & The Caribbean

Córdoba No. 42 Int. 807, Col. Roma Norte, Cuauhtémoc, C.P. 06700, Ciudad de México

 +52 (55) 5255-1329

CONTACT INFORMATION

Corporate Head Quarters
  2205 152nd Avenue NE
Redmond, WA 98052
USA

 +1.(425).605.1289

Euope
  Les Bureaux du Lac II Rue Robert Caumont, imm P 33049 Bordeaux         Cedex – France

 +33 (0)5 56.07.23.33

Latin America & The Caribbean

Córdoba No. 42 Int. 807, Col. Roma Norte, Cuauhtémoc, C.P. 06700, Ciudad de México

+52 (55) 5255-1329

Search Guard is a trademark of floragunn GmbH, registered in the U.S. and in other countries. Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries. Open Distro for Elasticsearch is licensed under Apache 2.0. All other trademark holders rights are reserved.

By continuing to use the site, you agree to the use of cookies. More information ?

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close