Apache Spark

Apache Spark was built specifically to fix the massive, glaring flaw in Apache Hadoop: speed. Hadoop's MapReduce engine was brilliant, but it physically wrote the data to the hard drive after every single step of the calculation, making it incredibly slow for complex Machine Learning algorithms. Spark introduced the concept of "In-Memory" processing. Instead of writing data to the slow physical hard drive, Spark loads the massive datasets directly into the ultra-fast RAM of the server cluster. Because RAM is mathematically thousands of times faster than a hard drive, Spark can process massive datasets up to 100x faster than Hadoop MapReduce. Because of its blinding speed, Spark completely conquered the Machine Learning and Data Science world. It provides native, highly optimized libraries (MLlib) for training massive artificial intelligence models. A data scientist can use Spark to ingest 10 years of credit card transaction data, train a complex fraud-detection neural network, and deploy it into production, all within a single unified framework.