• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

kleinee/jns: jupyter notebook and lab on Raspberry Pi

原作者: [db:作者] 来自: 网络 收藏 邀请

开源软件名称:

kleinee/jns

开源软件地址:

https://github.com/kleinee/jns

开源编程语言:

Shell 89.5%

开源软件介绍:

hardware os

python nodejs Julia R

jupyter piwheels

Jupyter Notebook & Lab Server on Raspberry Pi

Intro

Project Jupyter not only revolutionizes data-heavy research across domains - it also boosts personal productivity for problems on a much smaller scale. Due to openness it is an amazing platform for exploring concepts and learning new things.

I started setting up a Jupyter Notebook Server on a Raspberry Pi following this blog post by Arun Durvasula. Convinced of the potential of the platform I followed the development.

My personal exercise soon taught me a great deal about the underlying architcture. Given Jupyter's complexity, speed of growth and scale, it is remarkable that such a system runs fine on a Raspberry Pi.

This repository isn't really anything genuine: I owe big thanks to many contributors in the Jupyter, Raspberry Pi, Linux, Python, Julia and greater Open Source communities for providing all the beautiful building blocks - small and large.

What is new?

  • Rather than installing the latest version of Python as I did the past, I decided that the new version would use the latest Python 3 version supported in Raspbian - as of this writing Python 3.5.3.
  • Whilst this seems to be a step backwards, it is a in fact a giant step forward as you benefit from significant installation speedups made possible by the recently released piwheels project.
  • The scripts work across the entire range of Raspberry Pis - including the early models with just 256MB of memory.
  • Python support for GPIO, Sense HAT and PICAMERA is installed without the earlier worries of breaking things on system level.
  • All Python modules are pip installed into a virtual environment following advice found online: You should never use sudo pip install -NEVER. Well I did this in the past and it certainly had me and users confused. We have to learn certain things the hard way to really appreciate the benefits of doing them right. It is worth reading up on this in this blogpost.
  • You now install Python packages into a virtual environment created with venv using a requirements.txt file. This really achieves a more maintainable setup, opens up more possibilities and hopefully makes this project more useful for the Raspberry Pi and Jupyter communities.
  • Python 3, Julia and Bash kernels are installed and configured during installation.
  • JupyterLab is installed and ready to use.

What do you need to follow along?

  • a Raspberry Pi model of your choice complete with micro-usb power-supply - I recommend a Raspberry Pi 3 but the setup should work across the entire range of Pi models, perhaps with the exception of the very early models that featured only 256MB of memory. I tested on a ZeroW, 1, 2 and 3.
  • a (micro) SD card with 16GB capacity or more to suit your Pi model with Raspbian Stretch Lite installed and configured to permit access via ssh as user pi.
  • an ethernet or wifi connection for the Pi
  • internet access on the Pi
  • a computer to carry out the installation connected to the same network as the Pi
  • LESS TIME THAN EVER BEFORE due to the recent release of piwheels. Users new to this project might argue that the setup is still time-consuming. Believe me: In the past 6 hours+ were not uncommon and installing the system on a Raspberry Pi 1 was not impossible but required quite some patience and time. Note that some packages listed in requirements.txt may not yet be available as Python wheels. Such packages are then built from source and this takes some time...

Installation

IMPORTANT NOTE on fresh installations

  • An increasing number of users seem to install on top of images that have 'nodejs' already installed.

  • The scripts in this repository were initially designed to work based on Raspbian Stretch Lite as a starting point with the intention to run the server headless in order to maximise memory available for data analysis.

  • One such starting point is the desktop version of Raspbian Stretch which comes withnodejs (and git) pre-installed. conf_jupyter.sh explained later now checks for the existence of node and only installs it if not yet present on the system.

  • For the scripts to run properly on the desktop version of Raspbian or any other startingpoint with node installed, it is necessary that node is version 5 or higher !!!

  • If you start with a fresh Raspbian Stretch Desktop image, you can uninstall node using apt purge nodejs and then execute the scripts.

First boot with fresh SD card

  • ssh into your Raspberry Pi with the the fresh install of Raspbian Stretch Lite as user pi. Then run sudo raspi-config and set the memory split to 16MB, expand the file-system and set a new password for the user pi. When done, reboot and log in again via ssh.

  • If not yet present, install git:

sudo apt install -y git
  • To increase the size of swap_file to 2048MB run:
sudo sed -i -e 's/CONF_SWAPSIZE=100/CONF_SWAPSIZE=2048/' /etc/dphys-swapfile
sudo /etc/init.d/dphys-swapfile stop
sudo /etc/init.d/dphys-swapfile start
  • With preparations out of the way clone this repository into the home directory of user pi
git clone https://github.com/kleinee/jns
  • Change into the new directory ~/jns/scripts just created with git:
cd ~/jns/scripts

Technically you can now run sudo ./inst_jns.sh which is the installer script that combines the steps described below. If you follow along I assume that you run all scripts from inside the directory ~/jns/scripts.

You might not want all features on your system. Feel free to edit `inst_jns.sh' to suit your requirements.

Install required Raspbian packages with apt

sudo ./prep.sh

A couple of packages from the Raspbian repository are required during installation and later for a some Python packages to work properly. The script just fetches these packages and installs them.

#!/bin/bash
# script name:     prep.sh
# last modified:   2018/09/09
# sudo:            yes

script_name=$(basename -- "$0")

if ! [ $(id -u) = 0 ]; then
   echo "usage: sudo ./$script_name"
   exit 1
fi

apt update && apt -y upgrade
apt -y install pandoc
apt -y install libxml2-dev libxslt-dev
apt -y install libblas-dev liblapack-dev
apt -y install libatlas-base-dev gfortran
apt -y install libtiff5-dev libjpeg62-turbo-dev
apt -y install zlib1g-dev libfreetype6-dev liblcms2-dev
apt -y install libwebp-dev tcl8.5-dev tk8.5-dev
apt -y install libharfbuzz-dev libfribidi-dev
apt -y install libhdf5-dev
apt -y install libnetcdf-dev
apt -y install python3-pip
apt -y install python3-venv
apt -y install libzmq3-dev
apt -y install sqlite3 

Install required Python 3 packages with pip

./inst_stack.sh
  • This creates a virtual Python 3 environment '/home/pi/.venv/jns' and activates it temporarily
  • It then updates pip3 to the latest version available from the Python package repository before it processes the requirements.txt file line by line.
  • This is a workaround to prevent pip from failing if one or more requirements listed fail to install.
#!/bin/bash
# script name:     inst_stack.sh
# last modified:   2018/01/14
# sudo: no

script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"

if [ $(id -u) = 0 ]
then
   echo "usage: ./$script_name"
   exit 1
fi

if [ ! -d "$venv" ]; then
  python3 -m venv $env
fi

# activate virtual environment
source $env/bin/activate

pip3 install pip==9.0.0
pip3 install setuptools
pip3 install -U pip

cat requirements.txt | xargs -n 1 pip3 install

Configure Jupyter

./conf_jupyter.sh

With this script you generate a jupyter notebook configuration directory and in it a file called jupyter_notebook_config.py that holds the configuration settings for your notebook / lab server. You also create a folder notebooks in the home directory of user pi as the notebook_dir for your server. In the configuration file, you apply the following changes:

  • tell jupyter not to sart a browser upon start - we access the server from a remote machine on the same network
  • set the IP address to '*'
  • set the port for the notebook server to listen to 8888
  • enable mathjax for rendering math in notebooks
  • set the notebook_dir to ~/notebooks
  • use the password hash for the default server password jns

NOTE: This setup still uses password authentication. If you prefer token-based authentication, you have to change settings in the config file /home/pi/.jupyter/jupyter_notebook_config.py. Documentation of possible configuration settings can be found here.

After the basic configuration the script activates the bash kernel and activates extensions for Jupyter Notebook and JupyterLab. At the JupyterLab end this requires intstallation of node followed by installation of the underlying JS infrastructure which is a bit time-consuming but ultimately allows you to use ipywidgets, bqplot and potentially other extensions.

#!/bin/bash
# script name:     conf_jupyter.sh
# last modified:   2018/09/09
# sudo:            no

script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"

if [ $(id -u) = 0 ]
then
   echo "usage: ./$script_name"
   exit 1
fi

# activate virtual environment
source $env/bin/activate

# generate config and create notebook directory
# if notebook directory exists, we keep it (-p)
# if configuration file exeists, we overwrite it (-y)

jupyter notebook -y --generate-config
cd $home
mkdir -p notebooks

target=~/.jupyter/jupyter_notebook_config.py

# set up dictionary of changes for jupyter_config.py
declare -A arr
app='c.NotebookApp'
arr+=(["$app.open_browser"]="$app.open_browser = False")
arr+=(["$app.ip"]="$app.ip ='*'")
arr+=(["$app.port"]="$app.port = 8888")
arr+=(["$app.enable_mathjax"]="$app.enable_mathjax = True")
arr+=(["$app.notebook_dir"]="$app.notebook_dir = '/home/pi/notebooks'")
arr+=(["$app.password"]="$app.password = 'sha1:5815fb7ca805:f09ed218dfcc908acb3e29c3b697079fea37486a'")

# apply changes to jupyter_notebook_config.py

for key in ${!arr[@]};do
    if grep -qF $key ${target}; then
        # key found -> replace line
        sed -i "/${key}/c ${arr[${key}]}" $target
    else
        # key not found -> append line
        echo "${arr[${key}]}" >> $target
    fi
done

# install bash kernel
python3 -m bash_kernel.install

# install extensions
jupyter serverextension enable --py jupyterlab
jupyter nbextension enable --py widgetsnbextension --sys-prefix
jupyter nbextension enable --py --sys-prefix bqplot

# activate clusters tab in notebook interface
/home/pi/.venv/jns/bin/ipcluster nbextension enable --user

# install nodejs and node version manager n
# if node is not yet installed
if which node > /dev/null
    then
        echo "node is installed, skipping..."
    else
        # install nodejs and node version manager n
        cd ~/jns
        # fix for issue #22
        # install nodejs and node version manager n
        # see: https://github.com/mklement0/n-install
        curl -L https://git.io/n-install | bash -s -- -y lts
fi

# install jupyter lab extensions
bash -i ./inst_lab_ext.sh

The script inst_lab_ext.sh - introduced by @Kevin--R to fix issue#23 has the following content:

#!/bin/bash
# script name:     inst_lab_ext.sh
# last modified:   2019/04/06
# sudo:            no

script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"

if [ $(id -u) = 0 ]
then
   echo "usage: ./$script_name"
   exit 1
fi

. /home/pi/.bashrc
. $env/bin/activate
jupyter lab clean
jupyter labextension install @jupyter-widgets/jupyterlab-manager --no-build
jupyter labextension install bqplot --no-build
jupyter labextension install jupyterlab_bokeh --no-build
jupyter labextension install jupyter-leaflet --no-build
jupyter lab build

Start and access your server

Activate the virtual environment

Since you used a virtual environment to install Python modules, you need to activate this environment before you can start your server:

source /home/pi/.venv/jns/bin/activate

The prompt will change to indicate successfull activation preceding pi@hostname: with the envireonment name - in case pf this setup (jns). With hostname set to zerow it looks like this:

(jns) pi@zerow:~ $

Before you proceed

After installation completes, you will still need to activate the change made to ~\.bashrc when node was installed before doing anything that requires node.

You can be accomplish this by any of the following:

  • reboot
  • logout and log back in
  • call . ~/.bashrc from the command line

That's the reason for this warning during node installation:

  IMPORTANT: OPEN A NEW TERMINAL TAB/WINDOW or run `. /home/pi/.bashrc`
             before using n and Node.js.

You can see this by running the following commands after your installation completes:

pi@test-pi:~/jns $ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games
pi@test-pi:~/jns $ . ~/.bashrc
pi@test-pi:~/jns $ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games:/home/pi/n/bin
pi@test-pi:~/jns $ 

If you look at your $PATH environment variable and see /home/pi/n/bin you are ready to use node.

Also note that if you uninstall node with n-uninstall /home/pi/n/bin will remain in your $PATH environment variable until you reboot or logout and log back in.

Start the server

To start your server just type jupyter notebook or jupyter lab

Access the server

To access your server form a webbrowser on a computer running on the same network as your Raspberry Pi, just open a browser and use the Pi's IP address / port 8888 as the url.

xxx.xxx.xxx.xxx:8888

Change `xxx.xxx.xxx.xxx' to the IP address of the Raspberry Pi.

Login

During the configuration the default password for the server was set to jns. You can change this by typing:

(jns) pi@zerow:~ $ jupyter notebook password
Enter password:  ****
Verify password: ****

Install TeX (optional)

sudo ./inst_tex.sh
  • TeX (and Pandoc) are used under the hood to convert Jupyter notebooks to other formats including PDF.
  • Whilst not strictly necessary if no PDF export is rquired, I still recommend to run this step.
#!/bin/bash
# script name:     inst_tex.sh
# last modified:   2018/03/11
# sudo:            yes

script_name=$(basename -- "$0")

if ! [ $(id -u) = 0 ]; then
   echo "usage: sudo ./$script_name"
   exit 1
fi

#------------------------------------------------------
apt install -y texlive-xetex
apt install -y latexmk
#------------------------------------------------------

Install Julia and the IJulia kernel (optional)

  • Julia is a relatively new high-level, high-performance dynamic programming language for numerical computing trying to combine the ease of Python with the speed of C. Thanks to the efforts of the Raspberry Pi community Julia 0.6.0 is available in the Raspbian Stretch Repository. It is really worth a try as the language is a rising star in scientific computing.

  • IJulia is the kernel required for Jupyter Notebook / JupyterLab. Backgroud information on Julia on the Raspberry Pi can be found here.

Alternative 1: Julia 1.1.0 (RECOMMENDED)

sudo ./inst_julia-1.1.0.sh
  • NOTE tha the installer assumes that *** Julia IS NOT yet installed***. If it is and you want to proceed with installing Julia-1.1.0, I suggest to remove the existing Julia installation before you proceed.

  • Julia 1.1.0 binaries were cross-compiled by Mr Satoshi Terasaki (for more information follow this link) and Mr Satoshi thankfully hosts his binaries here.

  • We need to download a large file from Google Drive and need to install the right binary based on the CPU architecture. The download contains binaries for both architectures.

The Download Helper

As per comments in the script this is literally a 1:1 copy of code found on stack overflow - adjustments were only necessary to set the FILE_ID and the DESTINATION as required in the context of this repository. The helper is called by the installer script and is not meant to be exceuted manually.

#!/home/pi/.venv/jns/bin/python

#
# last modified 2019/05/26
#
# Python helper script to download Julia 1.1.0 binaries
# not meant to be executed manually
# https://stackoverflow.com/questions/38511444/python-download-files-from-google-drive-using-url
#

FILE_ID = '1fj6pNAJgmUD7bsSXqh8ocC1wESx8jkRh'
DESTINATION = './julia-1.1.0-arm32bit.zip'

import requests

def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download"

    session = requests.Session()

    response = session.get(URL, params = { 'id' : id }, stream = True)
    token = get_confirm_token(response)

    if token:
        params = { 'id' : id, 'confirm' : token }
        response = session.get(URL, params = params, stream = True)

    save_response_content(response, destination)    

def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            return value

    return None

def save_response_content(response, destination):
    CHUNK_SIZE = 32768

    with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)

if __name__ == "__main__":
    file_id = FILE_ID
    destination = DESTINATION
    download_file_from_google_drive(file_id, destination)

The Installer

Note that the code assumes that julia is not present.

  • installs necessary dependencies as suggested by Mr Terasaki
  • uses the downlaod helper to downlaod his binaries
  • detects the CPU architecture and installs the matching julia binary
  • creates a soft link in /usr/local/bin
  • installs the the IJulia kernel
#!/bin/bash
# script name:     inst_julia-1.1.0.sh
# last modified:   2019/05/26
# sudo:            yes

SCRIPT_NAME=$(basename -- "$0")
JNS_USER='pi'
HOME_DIR="/home/$JNS_USER"
ENV="$HOME_DIR/.venv/jns"

JULIA_HOME=$HOME_DIR/julia/

if ! [ $(id -u) = 0 ]; then
   echo "usage: sudo ./$SCRIPT_NAME"
   exit 1
fi

#
# apt install dependencies
#
apt install -y build-essential 
apt install -y libatomic1
apt install -y gfortran 
apt install -y perl 
apt install -y wget
apt install -y m4
apt install -y cmake 
apt install -y pkg-config
apt install -y libopenblas-base libopenblas-dev
apt install -y libatlas3-base libatlas-base-dev
apt install -y liblapack-dev
apt install -y libmpfr-dev libgmp3-dev
apt install -y libgfortran3

#
# download and install julia based on architecture
#
su pi <<ONE
    cd $HOME_DIR
    . $ENV/bin/activate
    ./dnld_julia-1.1.0-arm32bit.py
    unzip ./julia-1. 

鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
NHDaly/PaddleBattle.jl: Example game written in julia using SDL2发布时间:2022-07-09
下一篇:
Keras: the Python deep learning API发布时间:2022-07-09
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap