I downloaded the entire subreddit for /r/conspiracy using timesearch. https://github.com/voussoir/timesearch this includes all the submissions and comments to date.
I'm currently downloading the rest of the comments from 20th August. The database file is currently over 6.5gb and I estimate that it will take a few more days to download the rest of the comments for /r/conspiracy
If anyone is interested in this let me know and I will create a torrent for it and post it in the comments the magnet link.
Yeah. I think you should put it up as a torrent, at least make it available.
I would split the submissions from the comments, as different people will want different things.
And, 6.5gb of text?
Yep its a big forum. I will torrent the database as soon as I've finished downloading the rest of the comments, this should take a few more days
Yes, let's just gather a database of all people comments and then release it to the public, I'm sure that it will never be used by law enforcement or bad actors to track people down....
Just saying, you did exactly what everyone is worried about the government doing....
The fact you are even able to do this is very concerning frankly.
I'm just a innocent man trying to find the truth. Anyone is able to do a livestream of all comments and submissions on a subreddit so that even if you delete your account and the post it will be saved in the database indefinitely. Which is good because then you are able to see deleted comments and deleted submissions by moderators.
Perhaps law enforcement already have their own system for finding certain individuals online. Just look at the Snowden archives. Imagine what they have at their disposal nowadays....
Call it concerning but that's a fact, anything you post online can be with you for life. That's why its important to segregate your real life to your forum usage
Do you have any metadata about the moderators? They hide the names of the mods, and I now realize some of the mods who snuck in have censorship agendas. So I was wondering whether the scrape will contain info on bans they've inflicted.
Do you have an exmaples of posts that they banned so i can query the database for those and see if it shows the post/comment?
I wish I could help but I didn't keep good records beside general memory. However, the one mod who stood out for abuses and who gets the most complaints is Amos_Quito. It would be nice to see if his history pops out as having slammed a lot of users.
I am! Torrent would be nice ;)