{"id":451,"date":"2018-07-24T14:16:51","date_gmt":"2018-07-24T13:16:51","guid":{"rendered":"http:\/\/www.igfasouza.com\/blog\/?p=451"},"modified":"2021-04-27T10:51:40","modified_gmt":"2021-04-27T09:51:40","slug":"check-tweets-spelling","status":"publish","type":"post","link":"http:\/\/www.igfasouza.com\/blog\/check-tweets-spelling\/","title":{"rendered":"Check Tweets Spelling"},"content":{"rendered":"

\"\"<\/p>\n

Alright Boyo?<\/b><\/p>\n

Donald Trump has been forced to correct his tweet boasting about his writing ability after it was filled with spelling mistakes.
\nHere<\/a><\/p>\n

The US president posted on the social media website to defend his writing style and criticise the “Fake News” media for searching for mistakes in his tweets. <\/p>\n

The tweet itself had a few errors: Instead of “pore over” Mr Trump wrote “pour over” and instead of “bestselling” he wrote “best selling”.<\/p>\n

There are also question marks over how many books the former businessman has actually written.<\/p>\n

\"\"<\/a><\/p>\n

The tweet received a number of mocking responses before it was deleted and reposted with the “pour over” error corrected. <\/p>\n

With this and because I already have several projects using his tweets, I came up with the idea to analyze all Donald trump Tweets and check all spelling mistakes.<\/p>\n

I work in a data analytics company and I decide to ask for suggestions about the idea.
\nTalking with colleagues here I decided to add a blog post.<\/p>\n

Here I add a big thanks to my colleagues who helped me to do this analysis.
\nBrian Sullivan<\/a> and Aishwarya Mundalik<\/a><\/p>\n

I was collecting all Tweets from him even on fly. You can check in my GitHub<\/a> a Python script to get all Tweets from a user account.<\/p>\n

Just a small change in the code to save a csv file with tweet_id and word columns:<\/p>\n

1
2
3
4
5
6
7
8
9
<\/div><\/td>
  with<\/span> open<\/span>(<\/span>'%s_tweets.csv'<\/span> % screen_name,<\/span> 'wb'<\/span>)<\/span> as<\/span> f:
\n    writer =<\/span> csv<\/span>.writer<\/span>(<\/span>f,<\/span> delimiter=<\/span>'|'<\/span>)<\/span>
\n    writer.writerow<\/span>(<\/span>[<\/span>"id"<\/span>,<\/span>"words"<\/span>]<\/span>)<\/span>
\n    for<\/span> items in<\/span> outtweets:
\n      index =<\/span> items[<\/span>0<\/span>]<\/span>
\n      out=<\/span>items[<\/span>1<\/span>]<\/span>
\n      word_array=<\/span>out.split<\/span>(<\/span>)<\/span>
\n      for<\/span> word in<\/span> word_array:
\n        writer.writerow<\/span>(<\/span>(<\/span>index ,<\/span> word)<\/span>)<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

And the R code<\/p>\n

The code is using the Hunspell<\/a> R API to analyze the words.<\/p>\n

First I checked Donald Trump\u2019s tweets, and then I decided to compare against some others;
\nI chose Leo Varadkar and Fintan O’Toole because I\u2019m in Ireland and I choose J.K. Rowling because she is a writer and according to the news she was one of the people to make a lot of jokes about the case.<\/p>\n

1
2
3
4
5
6
7
8
9
10
<\/div><\/td>
Output -----><\/span>
\n
\n><\/span> mean(<\/span>TT_Final$correct<\/span>\/<\/span>TT_Final$n<\/span>)<\/span> (<\/span>Trunmp Tweets)<\/span>
\n[<\/span>1<\/span>]<\/span> 0.9645933<\/span>
\n><\/span> mean(<\/span>LT_Final$correct<\/span>\/<\/span>LT_Final$n<\/span>)<\/span>  (<\/span>Leo Tweets)<\/span>
\n[<\/span>1<\/span>]<\/span> 0.9187365<\/span>
\n><\/span> mean(<\/span>RT_Final$correct<\/span>\/<\/span>RT_Final$n<\/span>)<\/span>  (<\/span>J_K Rolling tweets)<\/span>
\n[<\/span>1<\/span>]<\/span> 0.9338411<\/span>
\n><\/span> mean(<\/span>FT_Final$correct<\/span>\/<\/span>FT_Final$n<\/span>)<\/span>   (<\/span>Fotoole Tweets)<\/span>
\n[<\/span>1<\/span>]<\/span> 0.9212394<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

The result was impressive, Donald Trump has the best value. This means that he has fewer mistakes than the others. The API just states whether the spelling is correct or not for each word.
\nUnfortunately, the API just checks words and not grammar or syntax- and for some values like single characters, the result is \u2018true\u2019 when it may not make sense.<\/p>\n

Here we can see a sample result.<\/p>\n

1
2
3
4
5
6
7
8
<\/div><\/td>
ID   WORD   RESULT
\n1017190186269184000<\/span>   but   TRUE
\n1017190186269184000<\/span>   it    TRUE
\n1017190186269184000<\/span>   isn   FALSE
\n1017190186269184000<\/span>   t    TRUE
\n1017190186269184000<\/span>   nearly    TRUE
\n1017190186269184000<\/span>   U    TRUE
\n1017190186269184000<\/span>   S    TRUE<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

This is just a basic analysis and the result are completely dependent on the API.
\nIt would be really nice to do a grammar or syntax analysis as well.<\/p>\n

The website politwoops.eu follows some politicians on Twitter and they show a list of deleted tweets from each one. I manage to get the last 60 tweets from Donald Trump and the result is:<\/p>\n

1
2
<\/div><\/td>
><\/span> mean(<\/span>DD_Final$correct<\/span>\/<\/span>DD_Final$n<\/span>)<\/span>   (<\/span>Deleted Donald Trump tweets)<\/span>
\n[<\/span>1<\/span>]<\/span> 0.958324<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

\"\"<\/a><\/p>\n

This proves that he actually deleted the tweet and posted it again and that he makes some mistakes.<\/p>\n

I just analysed the last 3000 tweets for each account.<\/p>\n

For fun, I have a look at my tweets as well:<\/p>\n

1
2
<\/div><\/td>
><\/span> mean(<\/span>Igor_Final$correct<\/span>\/<\/span>IG_Final$n<\/span>)<\/span>
\n[<\/span>1<\/span>]<\/span> 0.6915032<\/span><\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

And here is my defence … hehehe, Apparent IT words are not correct.<\/p>\n

1
2
3
4
5
6
7
8
9
10
11
<\/div><\/td>
ID   WORD   RESULT
\n956884676173553664<\/span>   GDG   FALSE
\n956884676173553664<\/span>   Hackathon   FALSE
\n950817131503013888<\/span>   Hacktoberfest   FALSE
\n947550776913735680<\/span>   Flume   FALSE
\n947550776913735680<\/span>   Spark   FALSE
\n943640519502106624<\/span>   sudo<\/span>   FALSE
\n943640519502106624<\/span>   init   FALSE
\n943640519502106624<\/span>   Brazil2018   FALSE
\n936178221153910784<\/span>   O'Reilly'<\/span>s   FALSE
\n936178221153910784<\/span>   Hadoop   FALSE<\/div><\/td><\/tr><\/tbody><\/table><\/div>\n

\"\"<\/a><\/p>\n

I put everything in my Github<\/a>, so you can get the code and play with. Just change the Twitter account and check yourself.<\/p>\n","protected":false},"excerpt":{"rendered":"

Alright Boyo? Donald Trump has been forced to correct his tweet boasting about his writing ability after it was filled with spelling mistakes. Here The US president posted on the social media website to defend his writing style and criticise… Continue Reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":456,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[18,31],"class_list":["post-451","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-python","tag-python","tag-twitter"],"_links":{"self":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts\/451","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/comments?post=451"}],"version-history":[{"count":10,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts\/451\/revisions"}],"predecessor-version":[{"id":1195,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/posts\/451\/revisions\/1195"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/media\/456"}],"wp:attachment":[{"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/media?parent=451"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/categories?post=451"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.igfasouza.com\/blog\/wp-json\/wp\/v2\/tags?post=451"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}