Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
Here I will use NiFi to create a 30 seconds scheduler to retrieve the CitiBike’s Station Feed. The data is in the JSON format:
In last post, we have set up a
jupyterhub in CEntOS, the same installation guide can be found for Ubuntu-like system.
In this post, we will discuss how to make
jupyterhub work with
Step 1: create a kernel directory in the user’s home folder
Step 2: create and edit kernel file
Create & Edit the file:
Put the script below into the file:
Please replace the
<spark_dir> with the location of the spark:
/opt/mapr/spark/spark-1.4.1 in my example.
Put lauching arguments in
PYSPARK_SUBMIT_ARGS, as shown here create a local
--master yarn-client pyspark-shell in my case for
Step 3: launch jupyterhub and create a spark notebook with kernel spark-*.*.*
Optional Step: copy the file to every spark user
The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text, just like RStudio’s rmarkdown. The project comes from IPython which now becomes a kernel for
Jupyterhub is a multi-user server that manages and proxies multiple instances of the single-user Jupyter notebook server.
The ubuntu-like installation guide can be found here. And here goes the installation steps for CEntOS 7.
CEntOS (the community enterprise operating system) is based on RedHat distribution of Linux. The reputation of the stability makes it a first choice for enterprise grade server.
Here is some simple configuration before compiling and other installation.
Add EPEL repo
Check here for the latest
su -c 'rpm -Uvh http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm'
Install Dev Tools
yum group install "Development Tools"
R and its friends
Have been changed to work on Mac OS X for a while with R, here is some documentation for configure R in Mac.
Install the missing package management for OS X
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Then retrieve the caskroom for R binary installation:
brew install caskroom/cask/brew-cask
Then install R:
brew install Caskroom/cask/r
Use a browser for R help document viewing:
options(help_type = "html")in the .rprofile
Internalisation of R, run the code in terminal:
defaults write org.R-project.R force.LANG en_US.UTF-8
The following mind maps illustrate the regression models from the book Applied Predictive Modeling
This is the summary mind note from the Applied Predictive Modeling. This part shows the model building process and details the pre-processing strategies.
JQuery like manipulation
Choose elements and bind the data
var body = d3.select("body"); //selection
var dataset = ["I like dog","I like cat","I like snake"];
SVG Canvas & Scales & Axis
D3.js uses SVG as canvas to draw.
var width = 300; //canvas width
var svg = d3.select('body')
delay() which is used for a transformation delay.
A list of bult-in layout:
Apache Spark in a distributed in-memory cluster computing system. Many people including me like to use Spark in python with IPython for a data analysis purpose.
But unfortunately the configuration is always a little bit tricky for the moment.
Steps to follow:
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz && tar -zxvf spark-1.5.1-bin-hadoop2.6.tgz
SPARK_HOMEto the unzipped folder, don’t forget to source the .bashrc or .zshrc.
pip install findspark.
That’s it, go play with the
I recently purchased a new add-on for Raspberry. With a successful advertisement of sending the Sense HAT to Internation Space Station, now the widget is ready for purchase: link, online shopping, french store.
To begin with the little board, all you have to do is to run the following command in order to install the driver:
sudo apt-get update
A reboot of the raspberry will give you access to the board.
Cannot get sense-hat installed
This problem seems to be related to the operating system. I have
HypriotOS installed, and the os repository does not provide
sense-hat, so I reinstall the original image of
Still not working
Make sure you have reboot your raspberry. Once the Sense HAT shows no LED after reboot, this means it is sucessfully installed.
Make sure you have enabled I2C by
sudo raspi-config, select
8. Advanced Options - I2C - Enable
With provided example, you can already play with the sense HAT.
Here is a quick start with
from sense_hat import SenseHat # import sense_hat object
So I make a deeper step, by showing a heart in order to convience my GF (lol) for the purchase:
from sense_hat import SenseHat