Python: Mining Twitter for GamerGate: Visualization
In the previous posting, I went over how to connect to Twitter’s streaming API using a connector app and the Tweepy Python library, as well as a quick overview of how to construct a Pandas dataframe from the tweets we’ve collected.
In this posting, we’ll extract all of the information we’ll need to use NetworkX to create a directed graph that we can visualize in Gephi of who’s retweeting whom, keeping track of the age in days and the number of followers that each user has so we can filter on those factors if we like.
First, if you don’t have NetworkX, install it with pip, and download and install Gephi.
Again, we’ll assume that our tweets are collected in a text file, “gamergate.txt”. Let’s pull the data out of the text file into a new data frame.
import json
import re
import pandas as pd
from time import gmtime, mktime, strptime
tweets_data = []
tweets_file = open(tweets_data_path, "r")
for line in tweets_file:
try:
tweet = json.loads(line)
tweets_data.append(tweet)
except:
continue
#
# Clean out limit messages, etc.
#
for tweet in tweets_data:
try:
user = tweet['user']
except:
tweets_data.remove(tweet)
for tweet in tweets_data:
try:
user = tweet['text']
except:
tweets_data.remove(tweet)
#
# See how many we wound up with
#
print len(tweets_data)
#
# Pull the data we're interested in out of the Twitter data we captured
#
rows_list = []
now = mktime(gmtime())
for tweet in tweets_data:
author = ""
rtauthor = ""
age = rtage = followers = rtfollowers = 0
#
# If it was a retweet, get both the original author and the retweeter, save the original author's
# follower count and age
#
try:
author = tweet['user']['screen_name']
rtauthor = tweet['retweeted_status']['user']['screen_name']
rtage = int(now - mktime(strptime(tweet['retweeted_status']['user']['created_at'], "%a %b %d %H:%M:%S +0000 %Y")))/(60*60*24)
rtfollowers = tweet['retweeted_status']['user']['followers_count']
except:
#
# Otherwise, just get the original author
#
try:
author = tweet['user']['screen_name']
except:
continue
#
# If this was a reply, save the screen name being replied to
#
reply_to = ""
if (tweet['in_reply_to_screen_name'] != None):
reply_to = tweet['in_reply_to_screen_name']
#
# Calculate the age, in days, of this Twitter ID
#
age = int(now - mktime(strptime(tweet['user']['created_at'], "%a %b %d %H:%M:%S +0000 %Y")))/(60*60*24)
#
# Grab this ID's follower count and the text of the tweet
#
followers = tweet['user']['followers_count']
text = tweet['text']
dict1 = {}
#
# Construct a row, add it to our list
#
dict1.update({'author': author, 'reply_to': reply_to, 'age': age, 'followers': followers, 'retweet_of': rtauthor, 'rtfollowers': rtfollowers, 'rtage': rtage, 'text': text})
rows_list.append(dict1)
#
# When we've processed all the tweets, build the DataFrame from the rows
# we've collected
#
tweets = pd.DataFrame(rows_list)
Here’s a script that will iterate through the dataframe, row by row, and construct a directed graph of who’s retweeting whom. Each directed edge represented the relationship “is retweeted by”, the higher the weight of an edge, the more person B is getting retweeted by person A. Each node represents an individual ID on Twitter, and has attributes to track the number of followers and the age of the ID in days.
import networkx as nx
#
# Create a new directed graph
#
J = nx.DiGraph()
#
# Iterate through the rows of our dataframe
#
for index, row in tweets.iterrows():
#
# Gather the data out of the row
#
this_user_id = row['author']
author = row['retweet_of']
followers = row['followers']
age = row['age']
rtfollowers = row['rtfollowers']
rtage = row['rtage']
#
# Is the sender of this tweet in our network?
#
if not this_user_id in J:
J.add_node(this_user_id, attr_dict={
'followers': row['followers'],
'age': row['age'],
})
#
# If this is a retweet, is the original author a node?
#
if author != "" and not author in J:
J.add_node(author, attr_dict={
'followers': row['rtfollowers'],
'age': row['rtage'],
})
#
# If this is a retweet, add an edge between the two nodes.
#
if author != "":
if J.has_edge(author, this_user_id):
J[author][this_user_id]['weight'] += 1
else:
J.add_weighted_edges_from([(author, this_user_id, 1.0)])
nx.write_gexf(J, 'ggrtages.gexf')
The last thing we did was to save out a GEFX file we can then read into Gephi. Start Gephi up, and open our file; we called ours “ggrtages.gexf”.
gephiScreenSnapz013
You’ll get a dialog telling you how many nodes and edges there are in the graph, whether it’s directed or not, and other information, warnings, etc. Click “OK”.
gephiScreenSnapz014
Gephi will import the GEFX file. You can now look at the information it contains by clicking on the “Data Laboratory” button at the top.
gephiScreenSnapz015
Click on the “Overview” button to start working with the network. At first, it doesn’t look like anything, since we haven’t actually run a visualization on it. Before we do, we can use some of the node attributes to color nodes a darker blue based on their age.
gephiScreenSnapz016
We can use the “Ranking” settings to color our nodes. Click on the “Select attribute” popup, and choose “age”.
gephiScreenSnapz017
You can choose difference color schemes, change the spline curve used to apply color, etc., from here as well.
gephiScreenSnapz018
Click on the “Apply” button to apply the ranking to the network. The nodes will now be colored rather than gray.
gephiScreenSnapz019
Now, we’re ready to run a visualization on our data. From the “Layout” section, let’s choose “ForceAtlas 2″—it’s fast and good at showing relationships in a network.
gephiScreenSnapz020
Press the “Run” button, and let it go for a bit. A network this size—about 10K nodes and 30K edges—settled down on my MacBook Pro within five minutes or less. When you feel it’s stabilized into something interesting, press the “Stop” button, and then click on the “Preview” button at the top.
gephiScreenSnapz022
The preview panel won’t show anything at first. Click the “Refresh” button.
gephiScreenSnapz023
Gephi will render your visualization. You can use the mouse to drag it around, and you can zoom in and out with a scroll-wheel or with the “+” and “-” buttons below.
gephiScreenSnapz024