Pyspark is not working in my macos

See more:

Java

Python

macOS

spark

I just bought a new Macbook M1 and i am having a hard time to set up pyspark in MacOs , it would be appreciated if anyone can help me in this as i am completely new to Mac.
I have followed many online tutorials and instructions but for some reason it is not working.


when i run spark -shell , it does not work neither when i try to work on juypter notebook ( although i "pip installed pyspark")
Here is the code i run in my Juypter notebook


import os
import findspark
import pyspark
import time
import operator
from pyspark import SparkConf
from pyspark import SparkContext

conf = SparkConf()
conf.setMaster("local")
conf.setAppName("spark-basic")
sc = SparkContext(conf = conf)

I have this error
**Exception: Java gateway process exited before sending its port number**

i appreciate your help.
Thanks



[1]: https://i.stack.imgur.com/xcs8p.png
[2]: https://i.stack.imgur.com/IIIi4.png

What I have tried:

I installed anconda,
I downloaded Spark file and put it in the correct path.
I did install Java Azum Arm64 (https://www.azul.com/downloads/zulu-community/)
I set my my environment like the attached picture.

Here is what i have in my .dash_profile

export SPARK_PATH=~/spark-1.6.0-bin-hadoop2.6
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'
export JAVA_HOME=$(/Library/Java/JavaVirtualMachines/jdk-16.0.1.jdk/Contents/Home)
export PATH="/Users/muhashahada/opt/anaconda3/bin:$PATH"
~