ODC Appreciation Day 2018 : Tweet analysis

How’s the craic?

Last Thursday was the Oracle Developer Community ODC Appreciation Day 2018 #ThanksODC

I came across with the #ThanksODC idea Thursday afternoon and of course, I join with a blog post as well. Here

And you can check the result of #ThanksODC here

Today I decided to do my tweets analysis here and see some nice insights.

But just to clarify.

  • This is my blog, so views expressed here are my own and do not necessarily reflect the views of Oracle and I’m not contesting the result given by oracle-base.
  • The idea here is just fun.
  • I think I’m the only Brazilian and the only person in Dublin to participate. (Two points that I couldn’t validate)

You can check my old post to see an explanation about the idea and maths behind

Six Nations

Let’s check the data.

I run my script to get the tweets at 12/10/2018 14:00 Dublin time and I got 51 tweets.

13 tweets at 12/10 and 38 at 11/10.
The first tweet at 11/10 was 08:14:18 and the last one was 23:31:44.
The last tweet that I got was 12/10 at 12:31:34.

I found 46 links to a blog post
I found 8 tweets with ODC Appreciation Day 2018

Counting Terms

Top 10

[(‘Appreciation’, 31), (‘ODC’, 27), (‘Day’, 25), (‘the’, 19), (‘to’, 17), (‘for’, 14), (‘I’, 11), (‘a’, 11), (‘and’, 10), (‘in’, 9)]

without Stop-words

Top 10

[(‘Appreciation’, 32), (‘ODC’, 28), (‘Day’, 26), (‘Oracle’, 9), (‘2018’, 9), (‘Wrap’, 6), (‘posts’, 6), (‘ORDS’, 5), (‘may’, 5), (‘blog’, 5)]

Languages

All tweets are in English, but I found six blogs in Spanish.

Tweets 100% English
Blog post 40 English 6 Spanish

Sentiment Analysis

Here the API gives a value between -1 and 1.
-1 to negative, 0 to neutral and 1 to positive.

The result is 88% positive, 11% neutral and 1% negative.

Retweets

Top Retweets -> 12
@oraclebase – ODC Appreciation Day 2018 : It’s a Wrap (#ThanksODC) : https://t.co/RYOeFzyHAs https://t.co/cQsj0N5xVB

Likes

Top Likes -> 28
@HeliFromFinland – Oracle SQL Developer Data Modeler is my favourite tool #ThanksODC https://t.co/ZXr3GI2LXV

Hashtag

If we have a look at the most common hashtags, we need to consider that #ThanksODC will be in all tweets because I used as a filter, so we can exclude it from the list. This leaves us with:

2 #ThanksOTN
2 #oow18
1 #oracle
1 #SOASuite
1 #ThanksOracleBase
1 #OrclDB
1 #ThanksTim
1 #oracleace
1 #ODCAppreciationDay
1 #ThanksOTNOTN/ODC
1 #orclepm
1 #orclbi
1 #Terraform
1 #ThanksTim
1 #plsql
1 #sq
1 #SeeYouAtOpenWorld
1 #ACED

number of hashtags per tweet

1 tweet with 4 hashtags
4 tweet with 3 hashtags
3 tweet with 2 hashtags

number of @

8 @oracleace
7 @oraclebase
2 @OracleDevsLA
2 @connor_mc_d
1 @rickProdManager
1 @odtug
1 @OracleDevs
1 @joelkallman
1 @OraPubInc
1 @oravirt
1 @odevcommunity
1 @floo_bar
1 @oracle
1 @gwronald
1 @dhamijarohit
1 @wordpressdotcom
1 @FTisiot

number of @ per tweet

2 tweet with 4 @
3 tweet with 3 @
1 tweet with 2 @

Spelling mistakes

Here I remove all hashtags and I found just one mistake and was “mornging”

Source

23 Twitter Web Client
06 TweetDeck
05 WordPress.com
03 Twitter Lite
03 Twitter for Android
03 Twitter for iPhone
01 IFTTT
02 Tweetbot for Mac
01 Hootsuite Inc.
01 Twibble.io
01 dlvr.it
01 Tweetbot for iΟS
01 Twitter Ads Composer

Number of tweets per user

4 LucasBednarek
3 oraculix
2 debralilley
2 RobbieDatabee
2 Makker_nl
2 kibeha
2 oraclebase
2 ZedDBA
2 Igfasouza
2 amitzil
1 orana
1 daanbakboord
1 orclDBblogs
1 ITProjectsToday
1 EdelweissK
1 signal006
1 ritan2000
1 Nikitas_Xenakis
1 mathiasmag
1 simon_haslam
1 PeterRaganitsch
1 HeliFromFinland
1 gassenmj
1 FranckPachot
1 SvenWOracle
1 opal_EPM
1 reguchi_br
1 rodrigojorgedba
1 svilmune
1 swesley_perth
1 dw_pete
1 oraesque
1 RonEkins
1 connor_mc_d
1 Addidici
1 dan_ekb
1 rittmanmead
1 FTisiot

My preferred tweet:
I really should go to sleep 💤, but I’m too busy reading everyone’s #ThanksODC blog posts

ODC Appreciation Day 2018: Oracle Visual Builder Cloud and Oracle JET

How’s it going horse?

Today it’s #ThanksODC day and I decide to join the idea with a post about VBCS (Oracle Visual Builder Cloud).

This post idea is going to be divided into three parts;

  1. A VBCS step by step Dublin Bus app
  2. Oracle JET step by step Dublin Bus app
  3. A Comparison of the two projects

Let’s start.

Create a Mobile Application

  1. In the web browser, log in to Oracle Visual Builder Cloud Service.
  2. On the Visual Applications page, click the New button.
  3. In the Create Application dialog box, enter DublinBus in the Application Name field and Tutorial application in the Description field.
  4. The Application ID text field is automatically populated as you type based on the value you enter in Application Name.
  5. Click Finish.
  6. Click + Mobile Application (or click the + sign at the top of the tab).
  7. In the Application name field add DublinBus and in the Navigation Style choose none.
  8. Go to Service Connections and create a new one, choose “Define by endpoint”

    In the field, URL add https://data.smartdublin.ie/cgi-bin/rtpi/realtimebusinformation
    click next;
    Go to request and URL Parameters add one query parameters:
    name: stopid, type: string, default Value: 1071, check as required and click in test and send
    copy to response body.

  9. Create a variable “stopid” as string.
  10. Add an input data go in data and change to stopid.
  11. Add a button and change the label to “search”.
  12. Add a table and choose to add data:
    choose Service Connections and choose the get for you rest API

    click on next and choose: route, Duetime and origin for Primary key;
    next and finish.

  13. Create a variable bus and change the type for Array Data Provider.
  14. Select the table and go on tab data and change for the variable bus.
  15. click on the button search and create an event

    Add “Call REST Endpoint” and choose the get rest API;
    Add “Reset Variables” and choose bus;
    Add “Assign Variables” and on the Variables assign the REST to bus;

I put everything in my Github, so you can get the code and play with.

Check Tweets Spelling

Alright Boyo?

Donald Trump has been forced to correct his tweet boasting about his writing ability after it was filled with spelling mistakes.
Here

The US president posted on the social media website to defend his writing style and criticise the “Fake News” media for searching for mistakes in his tweets.

The tweet itself had a few errors: Instead of “pore over” Mr Trump wrote “pour over” and instead of “bestselling” he wrote “best selling”.

There are also question marks over how many books the former businessman has actually written.

The tweet received a number of mocking responses before it was deleted and reposted with the “pour over” error corrected.

With this and because I already have several projects using his tweets, I came up with the idea to analyze all Donald trump Tweets and check all spelling mistakes.

I work in a data analytics company and I decide to ask for suggestions about the idea.
Talking with colleagues here I decided to add a blog post.

Here I add a big thanks to my colleagues who helped me to do this analysis.
Brian Sullivan and Aishwarya Mundalik

I was collecting all Tweets from him even on fly. You can check in my GitHub a Python script to get all Tweets from a user account.

Just a small change in the code to save a csv file with tweet_id and word columns:

1
2
3
4
5
6
7
8
9
  with open('%s_tweets.csv' % screen_name, 'wb') as f:
    writer = csv.writer(f, delimiter='|')
    writer.writerow(["id","words"])
    for items in outtweets:
      index = items[0]
      out=items[1]
      word_array=out.split()
      for word in word_array:
        writer.writerow((index , word))

And the R code

The code is using the Hunspell R API to analyze the words.

First I checked Donald Trump’s tweets, and then I decided to compare against some others;
I chose Leo Varadkar and Fintan O’Toole because I’m in Ireland and I choose J.K. Rowling because she is a writer and according to the news she was one of the people to make a lot of jokes about the case.

1
2
3
4
5
6
7
8
9
10
Output ----->

> mean(TT_Final$correct/TT_Final$n) (Trunmp Tweets)
[1] 0.9645933
> mean(LT_Final$correct/LT_Final$n)  (Leo Tweets)
[1] 0.9187365
> mean(RT_Final$correct/RT_Final$n)  (J_K Rolling tweets)
[1] 0.9338411
> mean(FT_Final$correct/FT_Final$n)   (Fotoole Tweets)
[1] 0.9212394

The result was impressive, Donald Trump has the best value. This means that he has fewer mistakes than the others. The API just states whether the spelling is correct or not for each word.
Unfortunately, the API just checks words and not grammar or syntax- and for some values like single characters, the result is ‘true’ when it may not make sense.

Here we can see a sample result.

1
2
3
4
5
6
7
8
ID   WORD   RESULT
1017190186269184000   but   TRUE
1017190186269184000   it    TRUE
1017190186269184000   isn   FALSE
1017190186269184000   t    TRUE
1017190186269184000   nearly    TRUE
1017190186269184000   U    TRUE
1017190186269184000   S    TRUE

This is just a basic analysis and the result are completely dependent on the API.
It would be really nice to do a grammar or syntax analysis as well.

The website politwoops.eu follows some politicians on Twitter and they show a list of deleted tweets from each one. I manage to get the last 60 tweets from Donald Trump and the result is:

1
2
> mean(DD_Final$correct/DD_Final$n)   (Deleted Donald Trump tweets)
[1] 0.958324

This proves that he actually deleted the tweet and posted it again and that he makes some mistakes.

I just analysed the last 3000 tweets for each account.

For fun, I have a look at my tweets as well:

1
2
> mean(Igor_Final$correct/IG_Final$n)
[1] 0.6915032

And here is my defence … hehehe, Apparent IT words are not correct.

1
2
3
4
5
6
7
8
9
10
11
ID   WORD   RESULT
956884676173553664   GDG   FALSE
956884676173553664   Hackathon   FALSE
950817131503013888   Hacktoberfest   FALSE
947550776913735680   Flume   FALSE
947550776913735680   Spark   FALSE
943640519502106624   sudo   FALSE
943640519502106624   init   FALSE
943640519502106624   Brazil2018   FALSE
936178221153910784   O'Reilly's   FALSE
936178221153910784   Hadoop   FALSE

I put everything in my Github, so you can get the code and play with. Just change the Twitter account and check yourself.

Six Nations

How heya?

Introduction

A long time I came collecting Twitter data with the idea to sell this in the future or something similar. so I decided to do some kind of basic data analyses.
I have been thinking in do something like this for a while.
The idea in this post is not to be technical and show all code details, but just tell the story around the data.
Everything is based on the fact that you can find the codes to do this easily on Google.
The go here is doing some kind of basic data analyses in a json file with tweets sample.

Twitter

For those of you unfamiliar with Twitter, it’s a social network where people post short, 140-character, status messages called tweets. Because tweets are sent out continuously, Twitter is a great way to figure out how people feel about current events.

Creating a Twitter App

First, we need to access our Twitter account and create an app. The website to do this is https://apps.twitter.com

The Twitter Streaming API

In order to make it easy to work with real-time tweets, Twitter provides the Twitter Streaming API.
There are a variety of clients for the Twitter Streaming API across all major programming languages. For Python, there are quite a few, which you can find here. The most popular is tweepy, which allows you to connect to the streaming API and handle errors properly.

User Case

I used the Tweepy API to download all the tweets containing the string #rbs6nations during the Saturday, last championship day. Obviously, not all the tweets about the event contained the hashtag and this API get just a part of all tweets that contain the hashtag, but this is a good baseline. The time frame for the download was from around 12:15 PM to 7:15 PM GMT, that is from about 15 minutes before the first match to about 15 minutes after the last match was over. In the end, about 20,000 tweets have been downloaded in JSON format, making about 78Mb of data. This was enough to have some performance problem with my Raspberry Pi Zero. I think is a good size to observe something possibly interesting.

Six Nations

As the name suggests, six teams are involved in the competition: England, Ireland, Wales, Scotland, France and Italy. Six Nations Championship.

Extracting information

There are a few fields that will be interesting to us:

  • The user’s location (status.user.location). This is the location the user who created the tweet wrote in their biography.
  • The screen name of the user (status.user.screen_name).
  • The text of the tweet (status.text).
  • The unique id that Twitter assigned to the tweet (status.id_str).
  • When the tweet was sent (status.created_at).
  • How many times the tweet has been retweeted (status.retweet_count).
  • The tweet’s coordinates (status.coordinates). The geographic coordinates from where the tweet was sent.

Counting Terms

The first exploratory analysis that we can perform is a simple word count. In this way, we can observe what are the terms most commonly used in the dataset.
We can easily split the tweets into a list of terms.
Split by space.

This is the list of top 10 most frequent terms

[(‘ireland’, 3163), (‘england’, 2584), (‘on’, 2271), (‘…’, 2068), (‘day’, 1479), (‘france’, 1380), (‘win’, 1338), (to, 1253), (Grand, 1221), (‘and’, 1180)]

Stop-Words

In every language, some words are particularly common. While their use in the language is crucial, they don’t usually convey a particular meaning, especially if taken out of context. This is the case of articles, conjunctions, some adverbs, etc. which are commonly called stop-words. In the example above, we can see three common stop-words “to”, “and” and “on”. Stop-word removal is one important step that should be considered during the pre-processing stages. You can easily find a list of Stop-word for almost all languages in the world.

After removing the list of stop-words, we have a new result:

[(‘ireland’, 3163), (‘england’, 2584), (‘wales’, 2271), (‘day’, 1479), (‘Grand’, 1380), (‘win’, 1338), (‘rugby’, 1253), (‘points’, 1221), (‘title’, 1180), (‘#x2618;’, 1154)]

The #x2618; is an emoji text-based symbol that represents the Irish “Shamrock”. ☘

If we have a look at the most common hashtags, we need to consider that #rbs6nations will be in all tweets becouse I used as a filter, so we can exclude it from the list. This leave us with:

[(‘#engvirl’, 1701), (‘#walvfran’, 927), (‘#rugby’, 880), (‘#itavsco’, 692), (‘#ireland’, 686), (‘#’champions’, 554), (‘#ireland’, 508), (‘#rbs’, 500), (‘#6nation’, 446), (‘#england’, 406)]

We can observe that the most common hashtags, apart from #rugby, are related to the individual matches. In particular England v Ireland has received the highest number of mentions, probably being the last match of the day.

Detect languages

Here the idea is to try to detect which language is used in the tweet.

I divide by language and found some Irish. Italian and French tweets.
Something interesting to notice is that a fair amount of tweets also contained terms in French.
Apparently, the API did not recognize anything in Welsh and Scottish.

Co-occurrences

Sometimes we are interested in the terms that occur together. This is mainly because the context gives us a better insight about the meaning of a term, supporting applications such as word disambiguation or semantic similarity.

I build a co-occurrence matrix such that com [x ][y] contains the number of times the term x has been seen in the same tweet as the term y:

The results:

[((‘6’, ‘nations’), 845), ((‘champions’, ‘ireland’), 760), ((‘nations’, ‘rbs’), 742), ((‘day’, ‘ireland’), 731), ((‘grand’, ‘slam’), 674)]

We could also look for a specific term and extract its most frequent co-occurrences.

The outcome for “ireland”:

[(‘champions’, 756), (‘day’, 727), (‘nations’, 659), (‘england’, 654), (‘2018’, 638), (‘6’, 613), (‘rbs’, 585), (‘ireland’, 559), (‘#x2618;’, 526), (‘game’, 522), (‘win’, 377), (‘grand’, 377), (‘slam’, 377), (’26’, 360), (‘points’, 356), (‘wales’, 355), (‘ire’, 355), (‘title’, 346), (’15’, 301), (‘turn’, 295)]

The outcome for “rugby”:

[(‘day’, 476), (‘game’, 160), (‘ireland’, 143), (‘england’, 132), (‘great’, 105), (‘today’, 104), (‘best’, 97), (‘well’, 90), (‘ever’, 89), (‘incredible’, 87), (‘amazing’, 84), (‘done’, 82), (‘amp’, 71), (‘games’, 66), (‘points’, 64), (‘monumental’, 58), (‘strap’, 56), (‘world’, 55), (‘team’, 55), (‘wales’, 53)]

Sentiment Analysis

In order to add another layer to your analysis, you can perform sentiment analysis of the tweets.
Here the API gives a value between -1 and 1.
-1 to negative, 0 to neutral and 1 to positive.

The result is 68% positive, 12% neutral and 20% negative.

Retweet

Using the status.retweet_count I was able to create a list of the top 10 re-tweets and get the first one.
I decide to not put here the first one because was an advertisement for an alcoholic beverage. I let you guess.

And with status.coordinates I get the location of the first one:
Some of the tweets don’t contain the coordinates because the user can mark as I don’t want to give the location, so I get the most retweeted with location and with Google maps, I discovered that the location was London stadium. This Tweet came from an Android mobile.

Extract key phrases

Extract the keywords or phrases which conveys the gist of the meaning of the tweet.

The result was:
‘Rugby’, ‘Championship’ and ‘Nations’.

Code

Here you can find a Jupyter notebook with almost everything that you need to do something similar.
You can have a look at my post to start your jupyter environment

Under the Hood

  • All codes that I used for this post I found on Google and I basically just make small changes to fit my scenario.
  • Just with a little bit of patience, I was able to build some insights about the Six Nations.
  • I had the idea to do this blog post on Ireland Storm Emma weekend, but I had problems with energy and internet.
  • I did everything using a Raspberry PI Zero and I had performance problems because of the size of the Json file.
  • I use Microsoft cognitive API example to do the Sentiment Analysis, Detect languages and to Extract key phrases.
  • There are several similar blog post like this with same and/or differences scenarios.

Conclusion

Here I showed a simple example of Text Mining on Twitter, using some realistic data taken during a sports event. Using just Google examples, we have downloaded some data using the streaming API, pre-processed the data in JSON format and extracted some interesting terms and hashtags from the tweets.

I also introduced the concept of Counting Terms, Detect Languages, Co-occurrence, Stop-words, Sentiment Analysis, and create some interesting insight.

Now I challenge you to do something similar to your scenario.
I might accept my challenge and do something like this again. I really liked to do this.
I might add all code on the GitHub and create a Jupyter notebook as well.
I might make the jsons file available. Because of the size, I don’t know a place for free hosting!

Jupyter Notebook

How’s it going there?

Jupyter Notebook is a popular application that enables you to edit, run and share Python code into a web view. It allows you to modify and re-execute parts of your code in a very flexible way. That’s why Jupyter is a great tool to test and prototype programs.

Apache Spark is a fast and powerful framework that provides an API to perform massively distributed processing over resilient sets of data.

Get Started with Spark and Jupyter together.

Install Spark

visit the Spark downloads page. Select the latest Spark release, a prebuilt package for Hadoop, and download it directly.
Unzip it and move it to your /opt folder:

1
2
$ tar -xzf spark-1.2.0-bin-hadoop2.4.tgz
$ mv spark-1.2.0-bin-hadoop2.4 /opt/spark-1.2.0

Create a symbolic link:

1
$ ln -s /opt/spark-1.2.0 /opt/spark

This way, you will be able to download and use multiple Spark versions.

Finally, tell your bash (or zsh, etc.) where to find Spark. To do so, configure your $PATH variables by adding the following lines in your ~/.bashrc (or~/.zshrc) file:

1
2
$ export SPARK_HOME=/opt/spark
$ export PATH=$SPARK_HOME/bin:$PATH

Install Jupyter

1
$ pip install jupyter

You can run a regular jupyter notebook by typing:

1
$ jupyter notebook

There are two ways to get PySpark available in a Jupyter Notebook:

1 – Configure PySpark driver to use Jupyter Notebook: running pyspark will automatically open a Jupyter Notebook
2 – Load a regular Jupyter Notebook and load PySpark using findSpark package

Option 1:

Update PySpark driver environment variables: add these lines to your~/.bashrc (or ~/.zshrc) file.

1
2
$ export PYSPARK_DRIVER_PYTHON=jupyter
$ export PYSPARK_DRIVER_PYTHON_OPTS='notebook'

Restart your terminal and launch PySpark again:

1
$ pyspark

Now, this command should start a Jupyter Notebook in your web browser. Create a new notebook by clicking on ‘New’ > ‘Notebooks Python [default]’.

Option 2:

Use findSpark package to make a Spark Context available in your code.

findSpark package is not specific to Jupyter Notebook, you can use this trick in your favorite IDE too.
To install findspark:

1
$ pip install findspark

Launch a regular Jupyter Notebook:

1
$ jupyter notebook

In your python code you need to add:

1
2
import findspark
findspark.init(“/path_to_spark”)

Now you can try out and see. I hope this guide will help you easily get started with Jupyter and Spark

Here is a python code example to test:

1
2
3
4
5
6
7
8
9
10
11
12
13
import findspark
findspark.init(“/opt/spark-1.4.1-bin-hadoop2.6/”)
import pyspark
import random
sc = pyspark.SparkContext(appName="Pi")
num_samples = 100000000
def inside(p):    
 x, y = random.random(), random.random()
 return x*x + y*y < 1
count = sc.parallelize(range(0, num_samples)).filter(inside).count()
pi = 4 * count / num_samples
print(pi)
sc.stop()

Apache Toree is a kernel for the Jupyter Notebook platform providing interactive access to Apache Spark.

Install Toree.

1
$ sudo pip install toree

Configure

Set SPARK_HOME to point to the directory where you downloaded and expanded the Spark binaries.

1
2
3
4
$ SPARK_HOME=$HOME/Downloads/spark-x.x.x-bin-hadoopx.x

$ jupyter toree install \
  --spark_home=$SPARK_HOME

Start notebook.

1
$ jupyter notebook

Test

Point browser to http://localhost:8888.
Then open a new notebook using New > Toree.

Test notebook with simple Spark Scala code.

1
2
3
4
sc.parallelize(1 to 100).
  filter(x => x % 2 == 0).
  map(x => x * x).
  take(10)

Here you can use tab for auto-complete.

To run Jupyter with R
Install IRkernel

1
$ conda install -c r ipython-notebook r-irkernel

You can now open R and Install some necessary packages used by R kernel on ipython notebook

1
install.packages(c('rzmq','repr','IRkernel','IRdisplay'), repos = 'http://irkernel.github.io/', type = 'source')

After the packages are successfully downloaded and installed.
Type this and quit

1
2
3
IRkernel::installspec()

quit()

Start the notebook and check new -> R

You can install Jupyter on Raspberry Pi

1
2
3
4
5
$ apt-get install python3-matplotlib
$ apt-get install python3-scipy
$ pip3 install --upgrade pip
$ reboot
$ sudo pip3 install jupyter

To start

1
$ jupyter-notebook

Simple Python example:

1
2
3
4
5
6
import pyspark
sc = pyspark.SparkContext('local[*]')

# do something to prove it works
rdd = sc.parallelize(range(1000))
rdd.takeSample(False, 5)

Simple R example:

1
2
3
4
5
6
7
library(SparkR)

as <- sparkR.session("local[*]")

# do something to prove it works
df <- as.DataFrame(iris)
head(filter(df, df$Petal_Width > 0.2))

Simple Scala example:

1
2
val rdd = sc.parallelize(0 to 999)
rdd.takeSample(false, 5)

Use the pre-configured SparkContext in variable sc.

Links

Apache Torre
https://github.com/asimjalis/apache-toree-quickstart

R on Jupyter
https://discuss.analyticsvidhya.com/t/how-to-run-r-on-jupyter-ipython-notebooks/5512/2

Raspberry Pi with InfluxDB and Grafana

Alright, boss?

Grafana is an open source metric analytics & visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics but many use it in other domains including industrial sensors, home automation, weather, and process control.

https://grafana.com/
http://docs.grafana.org/guides/getting_started/

InfluxDB is an open-source time series database developed by InfluxData.
it is written in Go and optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics.

https://www.influxdata.com/

Install InfluxDB on Raspberry Pi
https://gist.github.com/boseji/bb71910d43283a1b84ab200bcce43c26

You can follow the link for more details but basically, I just run this:

1
2
3
4
5
6
7
sudo apt-get update && sudo apt install apt-transport-https curl

curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -

echo "deb https://repos.influxdata.com/debian jessie stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

sudo apt-get update && sudo apt-get install influxdb

For this example we don’t care about the web-based Admin user interface.

To start the server

1
sudo service influxdb start

Install Grafana on Raspberry Pi
https://github.com/fg2it/grafana-on-raspberry/wiki

You can follow the link for more details but basically, I just run this:

1
2
3
4
5
6
7
8
9
sudo apt-get install apt-transport-https curl

curl https://bintray.com/user/downloadSubjectPublicKey?username=bintray | sudo apt-key add -

echo "deb https://dl.bintray.com/fg2it/deb jessie main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

sudo apt-get update

sudo apt-get install grafana

To start the server

1
sudo service grafana-server start

With everything installed you are ready to start doing some awesome things.

I going to show two examples here:
One using sense-hat to get the temperature, pressure and humidity value
and other using HS110 to get the energy consumption

Here is a good InfluxDB getting started
https://docs.influxdata.com/influxdb/v0.8/introduction/getting_started/

1
influx -precision rfc3339
1
CREATE DATABASE logger

You can test with

1
SHOW DATABASES

To run the code you need Install the python package first

1
sudo pip install influxdb

Both examples are quite similar. Just change what I write on the table.
This code is a hack from some Google examples.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
import argparse
import time
import datetime
import sys
from influxdb import InfluxDBClient
from sense_hat import SenseHat

sense=SenseHat()
 
# Set required InfluxDB parameters.
host = "localhost" #Could also set local ip address
port = 8086
user = "root"
password = "root"
 
# How frequently we will write sensor data from SenseHat to the database.
sampling_period = 5

def get_args():
    '''This function parses and returns arguments passed in'''
    # Assign description to the help doc
    parser = argparse.ArgumentParser(description='Program writes measurements data from SenseHat to specified influx db.')
    # Add arguments
    parser.add_argument(
        '-db','--database', type=str, help='Database name', required=True)
    parser.add_argument(
        '-sn','--session', type=str, help='Session', required=True)
    now = datetime.datetime.now()
    parser.add_argument(
        '-rn','--run', type=str, help='Run number', required=False,default=now.strftime("%Y%m%d%H%M"))
   
    # Array of all arguments passed to script
    args=parser.parse_args()
    # Assign args to variables
    dbname=args.database
    runNo=args.run
    session=args.session
    return dbname, session,runNo
   
def get_data_points():
    # Get the three measurement values from the SenseHat sensors
    temperature = sense.get_temperature()
    pressure = sense.get_pressure()
    humidity = sense.get_humidity()
    # Get a local timestamp
    timestamp=datetime.datetime.utcnow().isoformat()
    print ("{0} {1} Temperature: {2}{3}C Pressure: {4}mb Humidity: {5}%" .format(session,runNo,
    round(temperature,1),u'u00b0'.encode('utf8'),
    round(pressure,3),round(humidity,1)))
   
    # Create Influxdb datapoints (using lineprotocol as of Influxdb >1.1)
    datapoints = [
            {
                "measurement": session,
                "tags": {"runNum": runNo,
                },
                "time": timestamp,
                "fields": {
                    "temperaturevalue":temperature,"pressurevalue":pressure,"humidityvalue":humidity
                    }
                }
            ]
    return datapoints

# Match return values from get_arguments()
# and assign to their respective variables
dbname, session, runNo =get_args()  
print "Session: ", session
print "Run No: ", runNo
print "DB name: ", dbname

# Initialize the Influxdb client
client = InfluxDBClient(host, port, user, password, dbname)
       
try:
     while True:
        # Write datapoints to InfluxDB
        datapoints=get_data_points()
        bResult=client.write_points(datapoints)
        print("Write points {0} Bresult:{1}".format(datapoints,bResult))
       
        sense.show_message("OK")
           
        # Wait for next sample
        time.sleep(sampling_period)
       
        # Run until keyboard ctrl-c
except KeyboardInterrupt:
    print ("Program stopped by keyboard interrupt [CTRL_C] by user. ")

The HS110 example I just change some lines:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
consumption = plug.get_emeter_realtime()[“power”]
    # Create Influxdb datapoints (using lineprotocol as of Influxdb >1.1)
    datapoints = [
            {
                "measurement": session,
                "tags": {"runNum": runNo,
                },
                "time": timestamp,
                "fields": {
                    "consumption":consumption
                    }
                }
            ]
    return datapoints

To run:

1
python igor.py -db=logger -sn=test1

To setup Grafana:
http://localhost:3000

Go to Datasource->Add New and fill in with your database details
User and psw should be: “root” and “root” by default

Create a new dashboard, choose your database and on the tab matrics configure your query.

Edit the dashboard and go to metrics.

Choose the database and then configure the query.

If you succeed in create your dashboard let a comment below about what are you doing.

Smart Plug TP-Link

How’s the form?

A smart device is an electronic device, generally connected to other devices or networks via different wireless protocols such as Bluetooth, NFC, Wi-Fi, 3G, etc., that can operate to some extent interactively and autonomously.

A smart plug gives you several degrees of added control over just about any electrical appliance in your home. For one thing, it gives you remote access to switch a device on and off. Some you can also program to do so on a set schedule. Many models go a step further to give you in-depth insights about the way you use specific devices and the power they consume.

While many people buy smart plugs for their own convenience, there are plenty of ways they can make the world a better place, as well. The main one is by limiting energy waste. Sure, some models simply allow you to turn a device on or off without physically unplugging it, but others allow you to closely monitor your power usage and make positive changes toward conservation.

I have two models from TP-Link:

HS100
https://www.tp-link.com/uk/products/details/cat-5258_HS100.html

HS110
https://www.tp-link.com/uk/products/details/cat-5258_HS110.html

I found on Google this APIs for Python and Node-Js

Python
https://github.com/GadgetReactor/pyHS100

Example-1:

1
2
3
from pyHS100 import Discover
for dev in Discover.discover().values():
print(dev)

Example-2:

1
2
3
4
5
from pyHS100 import SmartPlug, SmartBulb
from pprint import pformat as pf

plug = SmartPlug("192.168.XXX.XXX")
print("Current consumption: %s" % plug.get_emeter_realtime())

Node-js
https://github.com/plasticrake/tplink-smarthome-api

Example:

1
2
3
4
5
6
7
8
9
10
11
12
13
const { Client } = require('tplink-smarthome-api');

const client = new Client();
const plug = client.getDevice({host: '192.168.XXX.XXX'}).then((device)=&gt;{
device.getSysInfo().then(console.log);
device.setPowerState(true);
});

// Look for devices, log to console, and turn them on
client.startDiscovery().on('device-new', (device) =&gt; {
device.getSysInfo().then(console.log);
device.setPowerState(true);
});

 

I manage to turn on or turn off my smart plug from a Tweet.

Now I can turn on or turn off something in my house just with a simple Tweet!

A post shared by igfasouza (@igfasouza) on

Sense HAT

What’s the story Rory?

The Sense HAT is an add-on board for Raspberry Pi, made especially for the Astro Pi mission – it launched to the International Space Station in December 2015 – and is now available to buy. The Sense HAT has an 8×8 RGB LED matrix, a five-button joystick and
includes the following sensors: Gyroscope, Accelerometer, Magnetometer, Barometer, Temperature sensor and Relative Humidity sensor.

www.raspberrypi.org/products/sense-hat/
www.raspberrypi.org/blog/announcing-2017-18-astro-pi/
www.astro-pi.org/

Technical Specification:
Gyroscope – angular rate sensor: +/-245/500/2000dps
Accelerometer – Linear acceleration sensor: +/-2/4/8/16 g
Magnetometer – Magnetic Sensor: +/- 4/8/12/16 gauss
Barometer: 260 – 1260 hPa absolute range (accuracy depends on the temperature and pressure, +/- 0.1 hPa under normal conditions)
Temperature sensor (Temperature accurate to +/- 2 degC in the 0-65 degC range)
Relative Humidity sensor (accurate to +/- 4.5% in the 20-80%rH range, accurate to +/- 0.5 degC in 15-40 degC range)
8×8 LED matrix display
Small 5 button joystick

Getting started with the Sense HAT

1
sudo apt-get install sense-hat

https://projects.raspberrypi.org/en/projects/getting-started-with-the-sense-hat
https://www.raspberrypi.org/documentation/hardware/sense-hat/README.md

Free Essentials book
https://www.raspberrypi.org/blog/experiment-with-the-sense-hat-in-the-new-magpi-essentials-book/

Python
https://pythonhosted.org/sense-hat/

NodeJS
https://www.npmjs.com/package/node-sense-hat

Scratch
https://www.raspberrypi.org/documentation/usage/scratch/gpio/README.md

node-red
https://www.npmjs.com/package/node-red-node-pi-sense-hat

Source code
https://github.com/RPi-Distro/python-sense-hat

Emulator
https://www.trinket.io/sense-hat

Examples
https://github.com/RPi-Distro/python-sense-hat/blob/master/examples/README.md
https://codeclubprojects.org/en-GB/sense-hat/

I have some examples in my Github as well

Raspberry Pi Zero OTG mode

Story Horse?

The Raspberry Pi Zero support USB On The Go, given the processor, is connected directly to the USB port, unlike on the B, B+ or Pi 2 B, which goes via a USB hub.

Since there are so many who have experienced various problems with their Pi setup and connection I created the following steps that I used from Linux Ubuntu 16.04 with success.

With OTG mode, the process is like this:

  • Format the SD card on your computer
  • Tweak a few files on the SD card before ejecting it
  • Plug the SD card into the Pi
  • Plug the Raspberry Pi USB port into your computer
  • Share Internet with the Raspberry pi device
  • SSH to the Pi
  • Continue any setup on the device

This is a huge improvement as we no longer need to plug in the HDMI monitor or keyboard for initial setup!

When you plug the Pi into your computer in the above example, it is emulating a virtual Ethernet device. Essentially, your computer thinks you plugged a network card into it.

Let’s go:

1 – Downloaded Raspbian Jessie Lite image from here:

https://www.raspberrypi.org/downloads/

2 – Copied image to Micro SDHC card via these instructions:

https://www.raspberrypi.org/documentation/installation/

1
$ sudo dd bs=4M if=2017-04-10-raspbian-jessie.img of=/dev/sdb status=progress

OR

1
$ sudo unzip -p 2017-04-10-raspbian-jessie.zip | sudo dd of=/dev/sdX bs=4096

OR you can use Etcher https://etcher.io/

I recommend Etcher.io which will flash your SD card, in the same way, whichever of those Operating Systems you use on your PC/laptop.

3 – To help prevent Raspberry Pis around the globe from being hacked the RPi foundation have now disabled SSH on the default image. Just create a text file in /boot/ called ssh – it can be blank or you can type anything you want it.

To /boot add empty file: ssh

https://www.raspberrypi.org/documentation/remote-access/ssh/

To share internet from USB

To /boot/config.txt add line with: dtoverlay=dwc2

To /boot/cmdline.txt after rootwait add: modules-load=dwc2,g_ether

4 – Unmount sd card, put in Pi Zero, connect USB cable between Pi Zero power port and Linux computer.

5 – From Linux computer:

Open the nm-connection-editor (from the panel). Select Wired connection 2 (RasPi on USB) & click Edit. Under ‘IPv4 Setting’ set ‘Method’ = ‘Shared to other computers’.

6 – SSH

1
$ ssh pi@raspberrypi.local (password = raspberry)

DONE!

Extras:
Once its booted you should now be able to open up the terminal, and using ping Raspberry pi.local you can see the RPi’s IP address.
Using this address in terminal type

1
ssh pi@

https://raspberryblog.de/?p=1921

Code Everyday

What’s the crack jack?

Inspired by the code everyday article and by my new site/blog, I decided to follow the idea.

I Googled it and I found several similar ideas like #100DaysOfCode.
I found that people that were following the #100DaysOfCode was too obsessed to just commit and have the green line on GitHub and I found several different rules as well.

With that, I decided to change the rules and create my own version of one hundred days of code. And create something that was not just about commit every day.

I decided to set a couple of rules for myself:

  1. I set up a limit one hundred days.
  2. I must write code every day. I can write docs, or blog posts, or other things but it must be in addition to the code that I write.
    Here I count my Blog, my site, Github, RackHank and any draft on my whiteboard.
  3. It must be useful code.
  4. All code must be written before midnight.
  5. The code must be Open Source and up on Github.
  6. I can write in different languages and can do different projects.
  7. I’ve decided to code on the weekends as well

My plan is to create a habit of code every day.

I have been interested in changing the way I schedule my daily tasks. I used to get at home from work and pick up any of my side projects, whenever I could, even if I felt too tired to only grasp a few lines of code, and ultimately my projects didn’t get too far. As a good developer, I started several things and I never finished them …

Is exactly what I want to change with this!

Recently I found “Not learning at least 5 hours per week is the smoking of the 21st century.”  The 5-hour rule

With luck in one hundred days, I’m going to make a new post with the results.