Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- First, the data comes from [this BigQuery table of Reddit posts](https://bigquery.cloud.google.com/table/fh-bigquery:reddit_posts.2016_08) using the following query:
- SELECT subreddit, domain, url, num_comments, score, over_18, is_self FROM [fh-bigquery:reddit_posts.2016_08]
- WHERE url IN (SELECT url FROM [fh-bigquery:reddit_posts.2016_08] GROUP BY url HAVING count(url) > 1)
- What this does is select all posts (not comments) which share a URL with at least one other post. This is the idea behind the network, that shared URLs represent shared interests. You could update it for a more recent month if you want.
- To save you some time learning BigQuery, you can access the resulting CSV here (37MB compressed, 1.2M rows): https://www.dropbox.com/s/tyypnsr4wd15hey/crossposts_aug_2016_v2.csv.gz?dl=0
- Now, sadly I've lost the code I used to process this into a map. But it would have been something like this:
- Stage 1: Make a list of the subreddits appearing next to each unique URL, e.g. from these rows:
- NoMansSkyTheGame,imgur.com,http://imgur.com/a/kWgnj,7,4,false,false
- nomanshigh,imgur.com,http://imgur.com/a/kWgnj,1,1,false,false
- Make this:
- "http://imgur.com/a/kWgnj":["NoMansSkyTheGame","nomanshigh"]
- Stage 2: For each unique URL, connect all its subreddits in a round-robin, e.g.
- ["pics","EarthPorn","nature","funny"]
- Becomes this set of edges:
- pics,Earthporn
- pics,nature
- pics,funny
- EarthPorn,nature
- EarthPorn,funny
- nature,funny
- (Depending on how you're doing this, you might like to sort by alphabetical order so you always end up with 'nature,pics' instead of sometimes having 'pics,nature' in your edge list.)
- I can't remember whether or not I corrected for subreddit popularity. For instance, which of the following relationships is closer?
- * pics and EarthPorn with 50 shared posts out of 20,000 total posts, or
- * PenmanshipPorn and calligraphy with 10 shared posts out of 200 total posts?
- I'd argue it's the latter, but that's an exercise for you to play around with.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement