How To Install Hadoop And Spark On Windows

Apache Spark is a lightning-fast unified analytics engine used for cluster calculating for big data sets similar BigData and Hadoop with the aim to run programs parallel across multiple nodes. It is a combination of multiple stack libraries such as SQL and Dataframes, GraphX, MLlib, and Spark Streaming.

Spark operates in 4 different modes:

1. Standalone Fashion: Here all processes run within the same JVM process.
2. Standalone Cluster Fashion: In this mode, information technology uses the Job-Scheduling framework in-built in Spark.
3. Apache Mesos: In this mode, the work nodes run on various machines, but the driver runs just in the master node.
4. Hadoop YARN: In this way, the drivers run inside the awarding'southward master node and is handled past YARN on the Cluster.

In This article, we will explore Apache Spark installation in a Standalone manner. Apache Spark is developed in Scala programming language and runs on the JVM. Java installation is 1 of the mandatory things in spark. So let'due south start with Java installation.

Installing Coffee:

Pace 1: Download the Java JDK.

Stride ii: Open up the downloaded Java SE Development Kit and follow along with the instructions for installation.

Step three: Open up the surroundings variable on the laptop past typing it in the windows search bar.

Set JAVA_HOME Variables:

To set the JAVA_HOME variable follow the below steps:

• Click on the User variable Add JAVA_HOME to PATH with value Value: C:\Program Files\Java\jdk1.viii.0_261.
• Click on the Arrangement variable Add C:\Program Files\Java\jdk1.8.0_261\bin to PATH variable.
• Open command prompt and type "coffee –version", it will show bellow appear & verify Coffee installation.

Installing Scala:

For installing Scala on your local machine follow the below steps:

Step i: Download Scala.

Stride two: Click on the .exe file and follow along instructions to customize the setup according to your needs.

Step three: Take the agreement and click the side by side button.

Set environmental variables:

• In User Variable Add together SCALA_HOME to PATH  with value C:\Program Files (x86)\scala.
• In System Variable Add together C:\Program Files (x86)\scala\bin to PATH variable.

Verify Scala installation:

In the Control prompt employ the below command to verify Scala installation:

scala

Installing Spark:

Download a pre-built version of the Spark and extract it into the C bulldoze, such as C:\Spark. Then click on the installation file and follow along the instructions to set up upward Spark.

Set environmental variables:

• In User variable Add together SPARK_HOME to PATH with value C:\spark\spark-2.four.half dozen-bin-hadoop2.7.
• In Organisation variable Add%SPARK_HOME%\bin to PATH variable.

Download Windows Utilities:

If you wish to operate on Hadoop information follow the below steps to download utility for Hadoop:

Step 1: Download the winutils.exe file.

Pace 2: Copy the file to  C:\spark\spark-1.6.1-bin-hadoop2.6\bin.

Step iii: Now execute "spark-shell" on cmd to verify spark installation equally shown below:

Source: https://www.geeksforgeeks.org/install-apache-spark-in-a-standalone-mode-on-windows/

Posted by: farrellfroma1981.blogspot.com

Share this post