Spark On Zos

In this journey we demonstrate running an analytics application using Spark on z/OS. Apache Spark on z/OS is in-place, optimized abstraction and real-time analysis of structured and unstructured enterprise data
Alternatives To Spark On Zos
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Spark35,3482,394882a day ago46May 09, 2021223apache-2.0Scala
Apache Spark - A unified analytics engine for large-scale data processing
Cookbook11,362
3 months ago108apache-2.0
The Data Engineering Cookbook
God Of Bigdata7,992
4 days ago2
专注大数据学习面试,大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...
Zeppelin5,98132237 days ago2June 21, 2017134apache-2.0Java
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
Sparkinternals4,665
a year ago27
Notes talking about the design and implementation of Apache Spark
Bigdl4,18110a day ago16April 19, 2021717apache-2.0Jupyter Notebook
Fast, distributed, secure AI for Big Data
Iceberg4,076
a day ago4May 23, 20221,312apache-2.0Java
Apache Iceberg
Tensorflowonspark3,849
521 days ago32April 21, 202211apache-2.0Python
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
Koalas3,2281124 months ago47October 19, 2021109apache-2.0Python
Koalas: pandas API on Apache Spark
Spark Nlp3,16322a day ago90March 05, 202136apache-2.0Scala
State of the Art Natural Language Processing
Alternatives To Spark On Zos
Select To Compare


Alternative Project Comparisons
Readme

Data Analysis using Spark on zOS and Jupyter Notebooks

Apache Spark is an open-source cluster-computing framework. Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, and S3. And you can use it interactively from the Scala, Python and R shells. You can run Spark using its standalone cluster mode, on an IaaS, on Hadoop YARN, or on container orchestrators like Apache Mesos.

z/OS is an extremely scalable and secure high-performance operating system based on the 64-bit z/Architecture. z/OS is highly reliable for running mission-critical applications, and the operating system supports Web- and Java-based applications.

In this jurney we demonstrate running an analytics application using Spark on z/OS. Apache Spark on z/OS is in-place, optimized abstraction and real-time analysis of structured and unstructured enterprise data which is powered by z Systems Community Cloud.

z/OS Platform for Apache Spark includes a supported version of Apache Spark open source capabilities consisting of the ApacheSpark core, Spark SQL, Spark Streaming, Machine Learning Library (MLib) and Graphx.It also includes optimized data access to a broad set of structured and unstructured data sources through Spark APIs. With this capability, traditional z/OS data sources, such as IMS™, VSAM, IBM DB2®, z/OS, PDSE, or SMF data, can be accessed in a performance-optimized manner with Spark

This analytics example uses data stored in DB2 and VSAM tables, and a machine learning application written in Scala. The code also uses open-source Jupyter Notebook to write and submit Scala code to your Spark instance, and view the output within a web GUI. The Jupyter Notebook is commonly used in data analytics space for data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more.


Included Components

The scenarios are accomplished by using:

Prerequisites

Register at z Systems Community Cloud for a trial account. You will receive an email containing credentials to access the self-service portal. This is where you can start exploring all the available services.

Steps

Part A: Use Self-service Dashboard

  1. Start your Spark Cluster
  2. Upload the DB2 and VSAM data
  3. Submit a Scala program to analyze the data
  4. Launch Spark GUI to view the submitted job

Part B: Work with Jupyter Notebook

  1. Launch Jupyter Notebook and connect to Spark
  2. Run Jupyter Notebook cells to load data and perform analysis

Part A: Use Self-service Dashboard

1. Start your Spark Cluster

1.Open a web browser and enter the URL to access the z Systems Community Cloud self-service portal.



2.Enter your Portal User ID and Portal Password, and click ‘Sign In’.

3.You will see the home page for the z Systems Community Cloud self-service portal.

  • Click on ‘Try Analytics Service’


4.You will now see a dashboard, which shows the status of your Apache Spark on z/OS instance.

At the top of the screen, notice the ‘z/OS Status’ indicator, which should show the status of your instance as ‘OK’.

In the middle of the screen, the ‘Spark Instance’, ‘Status’, ‘Data management’, and ‘Operations’ sections will be displayed. The ‘Spark Instance’ section contains your individual Spark username and IP address.

Below the field headings, you will see buttons for functions that can be applied to your instance.
GUI

The following table lists the operation for each function:


5.If it is the first time for you to try the Analytics Service on zOS, you must set a new Spark password.

  • Click ‘Change Password’


6.Confirm your instance is Active. If it is ‘Stopped’, click ‘Start’ to start it.

2. Upload the DB2 and VSAM data

1.Go to cloud4z/spark and download all the sample files.

2.Load the DB2 data file :

  • Click ‘Upload Data’
  • Select and load the DB2 DDL file
  • Select and load the DB2 data file
  • Click ‘Upload’

“Upload Success” will appear in the dashboard when the data load is complete. The VSAM data for this exercise has already been loaded for you. However, this step may be repeated by loading the VSAM copybook and VSAM data file you downloaded, from your local system.

3. Submit a Scala program to analyze the data

Submit a prepared Scala program to analyze the data.

  • Click ‘Spark Submit’
  • Select your Spark Demo JAR file
  • Specify Main class name ‘com.ibm.scalademo.ClientJoinVSAM’
  • Enter the arguments: ‘Spark Instance Username’ ‘Spark Instance Password’
  • Click ‘Submit’
    Note: The arguments suggest you need to login to the GUI to view the job results.

“JOB Submitted” will appear in the dashboard when the program is complete. This Scala program will access DB2 and VSAM data, perform transformations on the data, join these two tables in a Spark dataframe, and store the result back to DB2.

4. Launch Spark GUI to view the submitted job

Launch your individual Spark worker output GUI to view the job you just submitted.

  • Click ‘Spark UI’
  • Click on the ‘Worker ID’ for your program in the ‘Completed Drivers’ section.
  • Log in with your Spark username and Spark password. The ones mentioned in step 6.
  • Click on ‘stdout’ for your program in the ‘Finished Drivers’ section to view your results.

!

Part B: Work with Jupyter Notebook

5. Launch Jupyter Notebook and connect to Spark

Jupyter Notebook tool that is installed in the dashboard. This tool will allow you to write and submit Scala code to your Spark instance, and view the output within a web GUI.

1.Launch the Jupyter Notebook service in your browser from your dashboard.

  • Click on ‘Jupyter’.
    You will see the Jupyter home page.

Jupyter

The prepared Scala program in this level will access DB2 and VSAM data, perform transformations on the data, join these two tables in a Spark dataframe, and store the result back to DB2. It will also perform a logistic regression analysis and plot the output.

  1. Double click the Demo.jpynb file.

Jupyter File Select

The Jupyter Notebook will connect to your Spark on z/OS instance automatically and will be in the ready state when the Apache Toree –Scala indicator in the top right hand corner of the screen is clear.

6. Run Jupyter Notebook cells to load data and perform analysis

The Jupyter Notebook environment is divided into input cells labelled with ‘In [#]:’.

6.1 Load VSAM and DB2 data into Spark and perform a data transformation

Run cell #1 - The Scala code in the first cell loads the VSAM data (customer information) into Spark and performs a data transformation.

  • Click on the first ‘In [ ]:’
    The left border will change to blue when a cell is in command mode, as shown below.

  Before running the code, make the fllowing changes:

  • Change the value of zOS_IP to your Spark IP address.
  • Change the value of zOS_USERNAME to your Spark username and the value of zOS_PASSWORD to your Spark password.

Change code

  • Click the run cell button indicated by the red box as shown below

The Jupyter Notebook connection to your Spark instance is in the busy state when the Apache Toree –Scala indicator in the top right hand corner of the screen is grey.


When this indicator turns clear, the cell run has completed and returned to the ready state.
The output should be similar to the following:


Run cell #2 - The Scala code in the second cell loads the DB2 data (transaction data) into Spark and performs a data transformation.

  • Click on the next ‘In [ ]:’ to select the next cell
  • Click the run cell button The output should be similar to the following:

6.2 Join the VSAM and DB2 data into dataframe in Spark

Run cell #3 - The Scala code in the third cell joins the VSAM and DB2 data into a new ‘client_join’ dataframe in Spark.

  • Click on the next ‘In [ ]:’ to select the next cell
  • Click the run cell button The output should be similar to the following:

6.3 Create a logistic regression dataframe and plot it

Run cell #4 - The Scala code in the fourth cell performs a logistic regression to evaluate the probability of customer churn as a function of customer activity level. The ‘result_df’ dataframe is also created, which is used to plot the results on a line graph.

  • Click on the next ‘In [ ]:’ to select the next cell
  • Click the run cell button The output should be similar to the following:


Run cell #5 - The Scala code in the fifth cell plots the ‘plot_df’ dataframe.

  • Click on the next ‘In [ ]:’ to select the next cell
  • Click the run cell button
    The output should be similar to the following:

6.4 Get statistical data

a. The number of rows in the input VSAM dataset

  • println(clientInfo_df.count()) Result should be 6001. b. The number of rows in the input DB2 dataset
  • println(clientTrans_df.count()) Result should be 20000. c. The number of rows in the joined dataset
  • println(client_df.count()) Result should be 112.

Reference

IBM z/OS Platform for Apache Spark - http://www-03.ibm.com/systems/z/os/zos/apache-spark.html

Popular Apache Projects
Popular Spark Projects
Popular Web Servers Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Jupyter Notebook
Scala
Cloud Computing
Apache
Operating System
Spark
Apache Spark
Ibmcode