How heya?

Introduction

A long time I came collecting Twitter data with the idea to sell this in the future or something similar. so I decided to do some kind of basic data analyses.
I have been thinking in do something like this for a while.
The idea in this post is not to be technical and show all code details, but just tell the story around the data.
Everything is based on the fact that you can find the codes to do this easily on Google.
The go here is doing some kind of basic data analyses in a json file with tweets sample.

Twitter

For those of you unfamiliar with Twitter, it’s a social network where people post short, 140-character, status messages called tweets. Because tweets are sent out continuously, Twitter is a great way to figure out how people feel about current events.

Creating a Twitter App

First, we need to access our Twitter account and create an app. The website to do this is https://apps.twitter.com

The Twitter Streaming API

In order to make it easy to work with real-time tweets, Twitter provides the Twitter Streaming API.
There are a variety of clients for the Twitter Streaming API across all major programming languages. For Python, there are quite a few, which you can find here. The most popular is tweepy, which allows you to connect to the streaming API and handle errors properly.

User Case

I used the Tweepy API to download all the tweets containing the string #rbs6nations during the Saturday, last championship day. Obviously, not all the tweets about the event contained the hashtag and this API get just a part of all tweets that contain the hashtag, but this is a good baseline. The time frame for the download was from around 12:15 PM to 7:15 PM GMT, that is from about 15 minutes before the first match to about 15 minutes after the last match was over. In the end, about 20,000 tweets have been downloaded in JSON format, making about 78Mb of data. This was enough to have some performance problem with my Raspberry Pi Zero. I think is a good size to observe something possibly interesting.

Six Nations

As the name suggests, six teams are involved in the competition: England, Ireland, Wales, Scotland, France and Italy. Six Nations Championship.

Extracting information

There are a few fields that will be interesting to us:

• The user’s location (status.user.location). This is the location the user who created the tweet wrote in their biography.
• The screen name of the user (status.user.screen_name).
• The text of the tweet (status.text).
• The unique id that Twitter assigned to the tweet (status.id_str).
• When the tweet was sent (status.created_at).
• How many times the tweet has been retweeted (status.retweet_count).
• The tweet’s coordinates (status.coordinates). The geographic coordinates from where the tweet was sent.

Counting Terms

The first exploratory analysis that we can perform is a simple word count. In this way, we can observe what are the terms most commonly used in the dataset.
We can easily split the tweets into a list of terms.
Split by space.

This is the list of top 10 most frequent terms

[(‘ireland’, 3163), (‘england’, 2584), (‘on’, 2271), (‘…’, 2068), (‘day’, 1479), (‘france’, 1380), (‘win’, 1338), (to, 1253), (Grand, 1221), (‘and’, 1180)]

Stop-Words

In every language, some words are particularly common. While their use in the language is crucial, they don’t usually convey a particular meaning, especially if taken out of context. This is the case of articles, conjunctions, some adverbs, etc. which are commonly called stop-words. In the example above, we can see three common stop-words “to”, “and” and “on”. Stop-word removal is one important step that should be considered during the pre-processing stages. You can easily find a list of Stop-word for almost all languages in the world.

After removing the list of stop-words, we have a new result:

[(‘ireland’, 3163), (‘england’, 2584), (‘wales’, 2271), (‘day’, 1479), (‘Grand’, 1380), (‘win’, 1338), (‘rugby’, 1253), (‘points’, 1221), (‘title’, 1180), (‘#x2618;’, 1154)]

The #x2618; is an emoji text-based symbol that represents the Irish “Shamrock”. ☘

If we have a look at the most common hashtags, we need to consider that #rbs6nations will be in all tweets becouse I used as a filter, so we can exclude it from the list. This leave us with:

[(‘#engvirl’, 1701), (‘#walvfran’, 927), (‘#rugby’, 880), (‘#itavsco’, 692), (‘#ireland’, 686), (‘#’champions’, 554), (‘#ireland’, 508), (‘#rbs’, 500), (‘#6nation’, 446), (‘#england’, 406)]

We can observe that the most common hashtags, apart from #rugby, are related to the individual matches. In particular England v Ireland has received the highest number of mentions, probably being the last match of the day.

Detect languages

Here the idea is to try to detect which language is used in the tweet.

I divide by language and found some Irish. Italian and French tweets.
Something interesting to notice is that a fair amount of tweets also contained terms in French.
Apparently, the API did not recognize anything in Welsh and Scottish.

Co-occurrences

Sometimes we are interested in the terms that occur together. This is mainly because the context gives us a better insight about the meaning of a term, supporting applications such as word disambiguation or semantic similarity.

I build a co-occurrence matrix such that com [x ][y] contains the number of times the term x has been seen in the same tweet as the term y:

The results:

[((‘6’, ‘nations’), 845), ((‘champions’, ‘ireland’), 760), ((‘nations’, ‘rbs’), 742), ((‘day’, ‘ireland’), 731), ((‘grand’, ‘slam’), 674)]

We could also look for a specific term and extract its most frequent co-occurrences.

The outcome for “ireland”:

[(‘champions’, 756), (‘day’, 727), (‘nations’, 659), (‘england’, 654), (‘2018’, 638), (‘6’, 613), (‘rbs’, 585), (‘ireland’, 559), (‘#x2618;’, 526), (‘game’, 522), (‘win’, 377), (‘grand’, 377), (‘slam’, 377), (’26’, 360), (‘points’, 356), (‘wales’, 355), (‘ire’, 355), (‘title’, 346), (’15’, 301), (‘turn’, 295)]

The outcome for “rugby”:

[(‘day’, 476), (‘game’, 160), (‘ireland’, 143), (‘england’, 132), (‘great’, 105), (‘today’, 104), (‘best’, 97), (‘well’, 90), (‘ever’, 89), (‘incredible’, 87), (‘amazing’, 84), (‘done’, 82), (‘amp’, 71), (‘games’, 66), (‘points’, 64), (‘monumental’, 58), (‘strap’, 56), (‘world’, 55), (‘team’, 55), (‘wales’, 53)]

Sentiment Analysis

In order to add another layer to your analysis, you can perform sentiment analysis of the tweets.
Here the API gives a value between -1 and 1.
-1 to negative, 0 to neutral and 1 to positive.

The result is 68% positive, 12% neutral and 20% negative.

Retweet

Using the status.retweet_count I was able to create a list of the top 10 re-tweets and get the first one.
I decide to not put here the first one because was an advertisement for an alcoholic beverage. I let you guess.

And with status.coordinates I get the location of the first one:
Some of the tweets don’t contain the coordinates because the user can mark as I don’t want to give the location, so I get the most retweeted with location and with Google maps, I discovered that the location was London stadium. This Tweet came from an Android mobile.

Extract key phrases

Extract the keywords or phrases which conveys the gist of the meaning of the tweet.

The result was:
‘Rugby’, ‘Championship’ and ‘Nations’.

Code

Here you can find a Jupyter notebook with almost everything that you need to do something similar.
You can have a look at my post to start your jupyter environment

Under the Hood

• All codes that I used for this post I found on Google and I basically just make small changes to fit my scenario.
• Just with a little bit of patience, I was able to build some insights about the Six Nations.
• I had the idea to do this blog post on Ireland Storm Emma weekend, but I had problems with energy and internet.
• I did everything using a Raspberry PI Zero and I had performance problems because of the size of the Json file.
• I use Microsoft cognitive API example to do the Sentiment Analysis, Detect languages and to Extract key phrases.
• There are several similar blog post like this with same and/or differences scenarios.

Conclusion

Here I showed a simple example of Text Mining on Twitter, using some realistic data taken during a sports event. Using just Google examples, we have downloaded some data using the streaming API, pre-processed the data in JSON format and extracted some interesting terms and hashtags from the tweets.

I also introduced the concept of Counting Terms, Detect Languages, Co-occurrence, Stop-words, Sentiment Analysis, and create some interesting insight.

Now I challenge you to do something similar to your scenario.
I might accept my challenge and do something like this again. I really liked to do this.
I might add all code on the GitHub and create a Jupyter notebook as well.
I might make the jsons file available. Because of the size, I don’t know a place for free hosting!

How’s it going there?

Jupyter Notebook is a popular application that enables you to edit, run and share Python code into a web view. It allows you to modify and re-execute parts of your code in a very flexible way. That’s why Jupyter is a great tool to test and prototype programs.

Apache Spark is a fast and powerful framework that provides an API to perform massively distributed processing over resilient sets of data.

Get Started with Spark and Jupyter together.

Install Spark

visit the Spark downloads page. Select the latest Spark release, a prebuilt package for Hadoop, and download it directly.
Unzip it and move it to your /opt folder:

Create a symbolic link:

This way, you will be able to download and use multiple Spark versions.

Finally, tell your bash (or zsh, etc.) where to find Spark. To do so, configure your $PATH variables by adding the following lines in your ~/.bashrc (or~/.zshrc) file:

Install Jupyter

You can run a regular jupyter notebook by typing:

There are two ways to get PySpark available in a Jupyter Notebook:

1 – Configure PySpark driver to use Jupyter Notebook: running pyspark will automatically open a Jupyter Notebook
2 – Load a regular Jupyter Notebook and load PySpark using findSpark package

Option 1:

Update PySpark driver environment variables: add these lines to your~/.bashrc (or ~/.zshrc) file.

Restart your terminal and launch PySpark again:

Now, this command should start a Jupyter Notebook in your web browser. Create a new notebook by clicking on ‘New’ > ‘Notebooks Python [default]’.

Option 2:

Use findSpark package to make a Spark Context available in your code.

findSpark package is not specific to Jupyter Notebook, you can use this trick in your favorite IDE too.
To install findspark:

Launch a regular Jupyter Notebook:

In your python code you need to add:

Now you can try out and see. I hope this guide will help you easily get started with Jupyter and Spark

Here is a python code example to test:

Apache Toree is a kernel for the Jupyter Notebook platform providing interactive access to Apache Spark.

Install Toree.

Configure

Set SPARK_HOME to point to the directory where you downloaded and expanded the Spark binaries.

Start notebook.

Test

Point browser to http://localhost:8888.
Then open a new notebook using New > Toree.

Test notebook with simple Spark Scala code.

Here you can use tab for auto-complete.

To run Jupyter with R
Install IRkernel

You can now open R and Install some necessary packages used by R kernel on ipython notebook

After the packages are successfully downloaded and installed.
Type this and quit

Start the notebook and check new -> R

You can install Jupyter on Raspberry Pi

To start

Simple Python example:

Simple R example:

Simple Scala example:

Use the pre-configured SparkContext in variable sc.

Links

Apache Torre
https://github.com/asimjalis/apache-toree-quickstart

R on Jupyter
https://discuss.analyticsvidhya.com/t/how-to-run-r-on-jupyter-ipython-notebooks/5512/2

Alright, boss?

Grafana is an open source metric analytics & visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics but many use it in other domains including industrial sensors, home automation, weather, and process control.

https://grafana.com/
http://docs.grafana.org/guides/getting_started/

InfluxDB is an open-source time series database developed by InfluxData.
it is written in Go and optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics.

https://www.influxdata.com/

Install InfluxDB on Raspberry Pi
https://gist.github.com/boseji/bb71910d43283a1b84ab200bcce43c26

You can follow the link for more details but basically, I just run this:

For this example we don’t care about the web-based Admin user interface.

To start the server

Install Grafana on Raspberry Pi
https://github.com/fg2it/grafana-on-raspberry/wiki

You can follow the link for more details but basically, I just run this:

To start the server

With everything installed you are ready to start doing some awesome things.

I going to show two examples here:
One using sense-hat to get the temperature, pressure and humidity value
and other using HS110 to get the energy consumption

Here is a good InfluxDB getting started
https://docs.influxdata.com/influxdb/v0.8/introduction/getting_started/

You can test with

To run the code you need Install the python package first

Both examples are quite similar. Just change what I write on the table.
This code is a hack from some Google examples.

The HS110 example I just change some lines:

To run:

To setup Grafana:
http://localhost:3000

Go to Datasource->Add New and fill in with your database details
User and psw should be: “root” and “root” by default

Create a new dashboard, choose your database and on the tab matrics configure your query.

Edit the dashboard and go to metrics.

Choose the database and then configure the query.

If you succeed in create your dashboard let a comment below about what are you doing.

How’s the form?

A smart device is an electronic device, generally connected to other devices or networks via different wireless protocols such as Bluetooth, NFC, Wi-Fi, 3G, etc., that can operate to some extent interactively and autonomously.

A smart plug gives you several degrees of added control over just about any electrical appliance in your home. For one thing, it gives you remote access to switch a device on and off. Some you can also program to do so on a set schedule. Many models go a step further to give you in-depth insights about the way you use specific devices and the power they consume.

While many people buy smart plugs for their own convenience, there are plenty of ways they can make the world a better place, as well. The main one is by limiting energy waste. Sure, some models simply allow you to turn a device on or off without physically unplugging it, but others allow you to closely monitor your power usage and make positive changes toward conservation.

I have two models from TP-Link:

HS100
https://www.tp-link.com/uk/products/details/cat-5258_HS100.html

HS110
https://www.tp-link.com/uk/products/details/cat-5258_HS110.html

I found on Google this APIs for Python and Node-Js

Python
https://github.com/GadgetReactor/pyHS100

Example-1:

Example-2:

Node-js
https://github.com/plasticrake/tplink-smarthome-api

Example:

I manage to turn on or turn off my smart plug from a Tweet.

Now I can turn on or turn off something in my house just with a simple Tweet!

A post shared by igfasouza (@igfasouza) on Sep 15, 2017 at 11:00am PDT

What’s the story Rory?

The Sense HAT is an add-on board for Raspberry Pi, made especially for the Astro Pi mission – it launched to the International Space Station in December 2015 – and is now available to buy. The Sense HAT has an 8×8 RGB LED matrix, a five-button joystick and
includes the following sensors: Gyroscope, Accelerometer, Magnetometer, Barometer, Temperature sensor and Relative Humidity sensor.

www.raspberrypi.org/products/sense-hat/
www.raspberrypi.org/blog/announcing-2017-18-astro-pi/
www.astro-pi.org/

Technical Specification:
Gyroscope – angular rate sensor: +/-245/500/2000dps
Accelerometer – Linear acceleration sensor: +/-2/4/8/16 g
Magnetometer – Magnetic Sensor: +/- 4/8/12/16 gauss
Barometer: 260 – 1260 hPa absolute range (accuracy depends on the temperature and pressure, +/- 0.1 hPa under normal conditions)
Temperature sensor (Temperature accurate to +/- 2 degC in the 0-65 degC range)
Relative Humidity sensor (accurate to +/- 4.5% in the 20-80%rH range, accurate to +/- 0.5 degC in 15-40 degC range)
8×8 LED matrix display
Small 5 button joystick

Getting started with the Sense HAT

https://projects.raspberrypi.org/en/projects/getting-started-with-the-sense-hat
https://www.raspberrypi.org/documentation/hardware/sense-hat/README.md

Free Essentials book
https://www.raspberrypi.org/blog/experiment-with-the-sense-hat-in-the-new-magpi-essentials-book/

Python
https://pythonhosted.org/sense-hat/

NodeJS
https://www.npmjs.com/package/node-sense-hat

Scratch
https://www.raspberrypi.org/documentation/usage/scratch/gpio/README.md

node-red
https://www.npmjs.com/package/node-red-node-pi-sense-hat

Source code
https://github.com/RPi-Distro/python-sense-hat

Emulator
https://www.trinket.io/sense-hat

Examples
https://github.com/RPi-Distro/python-sense-hat/blob/master/examples/README.md
https://codeclubprojects.org/en-GB/sense-hat/

I have some examples in my Github as well

Story Horse?

The Raspberry Pi Zero support USB On The Go, given the processor, is connected directly to the USB port, unlike on the B, B+ or Pi 2 B, which goes via a USB hub.

Since there are so many who have experienced various problems with their Pi setup and connection I created the following steps that I used from Linux Ubuntu 16.04 with success.

With OTG mode, the process is like this:

• Format the SD card on your computer
• Tweak a few files on the SD card before ejecting it
• Plug the SD card into the Pi
• Plug the Raspberry Pi USB port into your computer
• Share Internet with the Raspberry pi device
• SSH to the Pi
• Continue any setup on the device

This is a huge improvement as we no longer need to plug in the HDMI monitor or keyboard for initial setup!

When you plug the Pi into your computer in the above example, it is emulating a virtual Ethernet device. Essentially, your computer thinks you plugged a network card into it.

Let’s go:

1 – Downloaded Raspbian Jessie Lite image from here:

https://www.raspberrypi.org/downloads/

2 – Copied image to Micro SDHC card via these instructions:

https://www.raspberrypi.org/documentation/installation/

OR

OR you can use Etcher https://etcher.io/

I recommend Etcher.io which will flash your SD card, in the same way, whichever of those Operating Systems you use on your PC/laptop.

3 – To help prevent Raspberry Pis around the globe from being hacked the RPi foundation have now disabled SSH on the default image. Just create a text file in /boot/ called ssh – it can be blank or you can type anything you want it.

To /boot add empty file: ssh

https://www.raspberrypi.org/documentation/remote-access/ssh/

To share internet from USB

To /boot/config.txt add line with: dtoverlay=dwc2

To /boot/cmdline.txt after rootwait add: modules-load=dwc2,g_ether

4 – Unmount sd card, put in Pi Zero, connect USB cable between Pi Zero power port and Linux computer.

5 – From Linux computer:

Open the nm-connection-editor (from the panel). Select Wired connection 2 (RasPi on USB) & click Edit. Under ‘IPv4 Setting’ set ‘Method’ = ‘Shared to other computers’.

6 – SSH

DONE!

Extras:
Once its booted you should now be able to open up the terminal, and using ping Raspberry pi.local you can see the RPi’s IP address.
Using this address in terminal type

https://raspberryblog.de/?p=1921

What’s the crack jack?

Inspired by the code everyday article and by my new site/blog, I decided to follow the idea.

I Googled it and I found several similar ideas like #100DaysOfCode.
I found that people that were following the #100DaysOfCode was too obsessed to just commit and have the green line on GitHub and I found several different rules as well.

With that, I decided to change the rules and create my own version of one hundred days of code. And create something that was not just about commit every day.

I decided to set a couple of rules for myself:

1. I set up a limit one hundred days.
2. I must write code every day. I can write docs, or blog posts, or other things but it must be in addition to the code that I write.
   Here I count my Blog, my site, Github, RackHank and any draft on my whiteboard.
3. It must be useful code.
4. All code must be written before midnight.
5. The code must be Open Source and up on Github.
6. I can write in different languages and can do different projects.
7. I’ve decided to code on the weekends as well

My plan is to create a habit of code every day.

I have been interested in changing the way I schedule my daily tasks. I used to get at home from work and pick up any of my side projects, whenever I could, even if I felt too tired to only grasp a few lines of code, and ultimately my projects didn’t get too far. As a good developer, I started several things and I never finished them …

Is exactly what I want to change with this!

Recently I found “Not learning at least 5 hours per week is the smoking of the 21st century.”  The 5-hour rule

With luck in one hundred days, I’m going to make a new post with the results.

The blog uses a WordPress theme and is hosted by Godaddy