{"id":419,"date":"2018-03-09T16:23:40","date_gmt":"2018-03-09T16:23:40","guid":{"rendered":"http:\/\/www.igfasouza.com\/blog\/?p=419"},"modified":"2021-04-27T10:49:59","modified_gmt":"2021-04-27T09:49:59","slug":"jupyter-notebook","status":"publish","type":"post","link":"http:\/\/www.igfasouza.com\/blog\/jupyter-notebook\/","title":{"rendered":"Jupyter Notebook"},"content":{"rendered":"

\"\"<\/p>\n

How\u2019s it going there?<\/b><\/p>\n

Jupyter Notebook<\/a> is a popular application that enables you to edit, run and share Python code into a web view. It allows you to modify and re-execute parts of your code in a very flexible way. That\u2019s why Jupyter is a great tool to test and prototype programs.<\/p>\n

Apache Spark<\/a> is a fast and powerful framework that provides an API to perform massively distributed processing over resilient sets of data.<\/p>\n

Get Started with Spark and Jupyter together.<\/p>\n

Install Spark<\/p>\n

visit the Spark downloads page<\/a>. Select the latest Spark release, a prebuilt package for Hadoop, and download it directly.
\nUnzip it and move it to your \/opt folder:<\/p>\n

1
2
<\/div><\/td>
$ tar<\/span> -xzf<\/span> spark-1.2.0-bin-hadoop2.4.tgz
\n$ mv<\/span> spark-1.2.0-bin-hadoop2.4 \/<\/span>opt\/<\/span>spark-1.2.0<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Create a symbolic link:<\/p>\n

1
<\/div><\/td>
$ <\/span>ln<\/span> -s<\/span> \/<\/span>opt\/<\/span>spark-1.2.0 \/<\/span>opt\/<\/span>spark<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

This way, you will be able to download and use multiple Spark versions.<\/p>\n

Finally, tell your bash (or zsh, etc.) where to find Spark. To do so, configure your $PATH variables by adding the following lines in your ~\/.bashrc (or~\/.zshrc) file:<\/p>\n

1
2
<\/div><\/td>
$ export<\/span> SPARK_HOME<\/span>=\/<\/span>opt\/<\/span>spark
\n$ export<\/span> PATH<\/span>=$SPARK_HOME<\/span>\/<\/span>bin:$PATH<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Install Jupyter<\/p>\n

1
<\/div><\/td>
$ <\/span>pip install<\/span> jupyter<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

You can run a regular jupyter notebook by typing:<\/p>\n

1
<\/div><\/td>
$ <\/span>jupyter notebook<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

There are two ways to get PySpark available in a Jupyter Notebook:<\/p>\n

1 – Configure PySpark driver to use Jupyter Notebook: running pyspark will automatically open a Jupyter Notebook
\n2 – Load a regular Jupyter Notebook and load PySpark using findSpark package<\/p>\n

Option 1:<\/strong><\/p>\n

Update PySpark driver environment variables: add these lines to your~\/.bashrc (or ~\/.zshrc) file.<\/p>\n

1
2
<\/div><\/td>
$ export<\/span> PYSPARK_DRIVER_PYTHON<\/span>=jupyter
\n$ export<\/span> PYSPARK_DRIVER_PYTHON_OPTS<\/span>='notebook'<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Restart your terminal and launch PySpark again:<\/p>\n

1
<\/div><\/td>
$ <\/span>pyspark<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Now, this command should start a Jupyter Notebook in your web browser. Create a new notebook by clicking on \u2018New\u2019 > \u2018Notebooks Python [default]\u2019.<\/p>\n

Option 2:<\/strong><\/p>\n

Use findSpark package to make a Spark Context available in your code.<\/p>\n

findSpark package is not specific to Jupyter Notebook, you can use this trick in your favorite IDE too.
\nTo install findspark:<\/p>\n

1
<\/div><\/td>
$ <\/span>pip install<\/span> findspark<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Launch a regular Jupyter Notebook:<\/p>\n

1
<\/div><\/td>
$ <\/span>jupyter notebook<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

In your python code you need to add:<\/p>\n

1
2
<\/div><\/td>
import<\/span> findspark
\nfindspark.init<\/span>(<\/span>\u201c\/path_to_spark\u201d)<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Now you can try out and see. I hope this guide will help you easily get started with Jupyter and Spark<\/p>\n

Here is a python code example to test:<\/p>\n

1
2
3
4
5
6
7
8
9
10
11
12
13
<\/div><\/td>
import<\/span> findspark
\nfindspark.init<\/span>(<\/span>\u201c\/opt\/spark-1.4.1-bin-hadoop2.6\/\u201d)<\/span>
\nimport<\/span> pyspark
\nimport<\/span> random<\/span>
\nsc =<\/span> pyspark.SparkContext<\/span>(<\/span>appName=<\/span>"Pi"<\/span>)<\/span>
\nnum_samples =<\/span> 100000000<\/span>
\ndef<\/span> inside(<\/span>p)<\/span>:    
\n x,<\/span> y =<\/span> random<\/span>.random<\/span>(<\/span>)<\/span>,<\/span> random<\/span>.random<\/span>(<\/span>)<\/span>
\n return<\/span> x*x + y*y <<\/span> 1<\/span>
\ncount =<\/span> sc.parallelize<\/span>(<\/span>range<\/span>(<\/span>0<\/span>,<\/span> num_samples)<\/span>)<\/span>.filter<\/span>(<\/span>inside)<\/span>.count<\/span>(<\/span>)<\/span>
\npi =<\/span> 4<\/span> * count \/ num_samples
\nprint<\/span>(<\/span>pi)<\/span>
\nsc.stop<\/span>(<\/span>)<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Apache Toree<\/a> is a kernel for the Jupyter Notebook platform providing interactive access to Apache Spark.<\/p>\n

Install Toree.<\/p>\n

1
<\/div><\/td>
$ <\/span>sudo<\/span> pip install<\/span> toree<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Configure<\/p>\n

Set SPARK_HOME to point to the directory where you downloaded and expanded the Spark binaries.<\/p>\n

1
2
3
4
<\/div><\/td>
$ SPARK_HOME<\/span>=$HOME<\/span>\/<\/span>Downloads\/<\/span>spark-x.x.x-bin-hadoopx.x
\n
\n$ jupyter toree install<\/span> \\
\n  --spark_home=$SPARK_HOME<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Start notebook.<\/p>\n

1
<\/div><\/td>
$ <\/span>jupyter notebook<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Test<\/p>\n

Point browser to http:\/\/localhost:8888.
\nThen open a new notebook using New > Toree.<\/p>\n

Test notebook with simple Spark Scala code.<\/p>\n

1
2
3
4
<\/div><\/td>
sc.parallelize<\/span>(<\/span>1<\/span> to 100<\/span>)<\/span>.
\n  filter<\/span>(<\/span>x =><\/span> x %<\/span> 2<\/span> ==<\/span> 0<\/span>)<\/span>.
\n  map<\/span>(<\/span>x =><\/span> x *<\/span> x)<\/span>.
\n  take<\/span>(<\/span>10<\/span>)<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Here you can use tab for auto-complete.<\/p>\n

To run Jupyter with R
\nInstall IRkernel<\/p>\n

1
<\/div><\/td>
$ <\/span>conda install<\/span> -c<\/span> r ipython-notebook r-irkernel<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

You can now open R and Install some necessary packages used by R kernel on ipython notebook<\/p>\n

1
<\/div><\/td>
install.packages<\/span>(<\/span>c(<\/span>'rzmq'<\/span>,<\/span>'repr'<\/span>,<\/span>'IRkernel'<\/span>,<\/span>'IRdisplay'<\/span>)<\/span>,<\/span> repos =<\/span> 'http:\/\/irkernel.github.io\/'<\/span>,<\/span> type<\/span> =<\/span> 'source'<\/span>)<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

After the packages are successfully downloaded and installed.
\nType this and quit<\/p>\n

1
2
3
<\/div><\/td>
IRkernel::installspec(<\/span>)<\/span>
\n
\nquit(<\/span>)<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Start the notebook and check new -> R<\/p>\n

You can install Jupyter on Raspberry Pi<\/p>\n

1
2
3
4
5
<\/div><\/td>
$ apt-get install<\/span> python3-matplotlib
\n$ apt-get install<\/span> python3-scipy
\n$ pip3 install<\/span> --upgrade<\/span> pip
\n$ reboot
\n$ sudo<\/span> pip3 install<\/span> jupyter<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

To start<\/p>\n

1
<\/div><\/td>
$ <\/span>jupyter-notebook<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Simple Python example:<\/p>\n

1
2
3
4
5
6
<\/div><\/td>
import<\/span> pyspark
\nsc =<\/span> pyspark.SparkContext<\/span>(<\/span>'local[*]'<\/span>)<\/span>
\n
\n# do something to prove it works<\/span>
\nrdd =<\/span> sc.parallelize<\/span>(<\/span>range<\/span>(<\/span>1000<\/span>)<\/span>)<\/span>
\nrdd.takeSample<\/span>(<\/span>False<\/span>,<\/span> 5<\/span>)<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Simple R example:<\/p>\n

1
2
3
4
5
6
7
<\/div><\/td>
library(<\/span>SparkR)<\/span>
\n
\nas<\/span> <<\/span>- sparkR.session<\/span>(<\/span>"local[*]"<\/span>)<\/span>
\n
\n# do something to prove it works<\/span>
\ndf <<\/span>- as<\/span>.DataFrame<\/span>(<\/span>iris)<\/span>
\nhead(<\/span>filter<\/span>(<\/span>df,<\/span> df$Petal_Width ><\/span> 0.2<\/span>)<\/span>)<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Simple Scala example:<\/p>\n

1
2
<\/div><\/td>
val<\/span><\/a> rdd =<\/span> sc.parallelize<\/span>(<\/span>0<\/span> to 999<\/span>)<\/span>
\nrdd.takeSample<\/span>(<\/span>false<\/span><\/a>, 5<\/span>)<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

<\/p>\n

Use the pre-configured SparkContext in variable sc.<\/p>\n

Links<\/strong><\/p>\n

Apache Torre
\nhttps:\/\/github.com\/asimjalis\/apache-toree-quickstart<\/a><\/p>\n

R on Jupyter
\nhttps:\/\/discuss.analyticsvidhya.com\/t\/how-to-run-r-on-jupyter-ipython-notebooks\/5512\/2<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

How\u2019s it going there? Jupyter Notebook is a popular application that enables you to edit, run and share Python code into a web view. It allows you to modify and re-execute parts of your code in a very flexible way.… Continue Reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":829,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-419","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts\/419","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/comments?post=419"}],"version-history":[{"count":14,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts\/419\/revisions"}],"predecessor-version":[{"id":1193,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts\/419\/revisions\/1193"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/media\/829"}],"wp:attachment":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/media?parent=419"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/categories?post=419"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/tags?post=419"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}