PropGBench | Install GraphFrames on Jetstream2

1. System Information

The virtual machine that we used is built from the Big_Data_Hadoop-NEW_IMAGE image on Jetstream2 with a 16-core AMD EPYC-Milan CPU using x86_64 architecture, 60 GB RAM, and 60 GB storage. The operating system is Ubuntu 22.04.5 LTS with the Linux 6.8.0-90-generic kernel. This guide details the end-to-end installation of Spark 3.4.4 on Jetstream2, including Delta Lake support and GraphFrames analytics

2. Installation & Configuration

Spark & Delta Lake Installation

We run all these following commands only on the master node, however, for some configuration files, they need to be manually added to the worker node configuration paths. First, download and extract the Spark binaries compatible with Hadoop 3.4.4.

bash

wget https://dlcdn.apache.org/spark/spark-3.4.4/spark-3.4.4-bin-hadoop3.tgz
tar -xvzf spark-3.4.4-bin-hadoop3.tgz
rm spark-3.4.4-bin-hadoop3.tgz

Environment Variables

Update your ~/.bashrc to define system paths.

bash

export SPARK_HOME=/home/exouser/spark-3.4.4-bin-hadoop3
export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH

Apply changes across the cluster. Make sure to also broadcast to all datanodes.

Spark Environment

Define master host and Java/Python paths. We need to modify the $SPARK_HOME/conf/spark-env.sh file to include these variables.

bash

export SPARK_MASTER_HOST=master
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PYSPARK_PYTHON=/usr/bin/python3
export PYSPARK_DRIVER_PYTHON=/usr/bin/python3

We also need to modify the $SPARK_HOME/conf/spark-defaults.conf file to add these lines.

properties

spark.master                     spark://master:7077
spark.eventLog.enabled           true
spark.eventLog.dir               file:///home/exouser/spark-events
spark.history.fs.logDirectory    file:///home/exouser/spark-events

We also need to modify the worker list so that the worker nodes can find the master node. $SPARK_HOME/conf/workers file to add the machine names of your worker nodes.

properties

[WORKER_NODE_NAME1]
[WORKER_NODE_NAME2]
[WORKER_NODE_NAME3]
...

After editing these configuration files, they need to be broadcasted to all worker nodes.

Firewall Configuration

Open ports for Master UI (8080), Application UI (4040), and History Server (18080).

bash

sudo ufw allow 8080/tcp
sudo ufw allow 4040/tcp
sudo ufw allow 18080/tcp
sudo ufw reload

Start Spark

Start the Spark master and worker daemons.

bash

start-all.sh

You can check the Spark UI at http://[MASTER_IP_ADDRESS]:8080.

3. Running GraphFrames

Initialize PySpark with the GraphFrames and Delta Lake packages.

python

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .appName("GraphFrames Example") \
    .config("spark.jars.packages", "graphframes:graphframes:0.8.3-spark3.4-s_2.12") \
    .getOrCreate()

Graph Construction Example

Defining vertices and edges manually

python

from graphframes import GraphFrames
vertices = spark.createDataFrame([("a", "Alice", 34), ("b", "Bob", 36)], ["id", "name", "age"])
edges = spark.createDataFrame([("a", "b", "friend")], ["src", "dst", "relationship"])
g = GraphFrames(vertices, edges)