I crawled the forum and made some stats - ComputerCraft Forums (archive)

Posted 23 June 2016 - 04:55 PM

Hi

I'm trying to make a packet manager for CC computers that (in the end) should be able to install any program posted there from inside the game.

I had to code something that could read the forum, parse the html, fetch topic names, topic authors, urls and whatnot, and, after that, if the user asks for it, download a specific thread, analyze it, and attempt to find a download link for the program and install it.

This is an attempt to allow any player to browse the forum ingame.

I don't really store anything so legally it should be fine, every user does its own crawling like any web browser would do. It's not bandwidth-heavy either since at first it just browses the main forum pages, not individual threads.

It's not ready yet, but I got the crawler part set up so I thought "might as well do some stats !" for the fun of it

So here's what my bot gave me;

All the links except pastebin

63% of the topics include a link to pastebin.com or some "pastebin get/run …" shell command
5.6% of the links refer to adf.ly
10% of the stuff linked on the forum now links to an http error somewhere (400, 404, 503…)
Only 41% of the stuff linked on the forum still returns an http 200 (works fine)
50% of the links are now redirections so I can't tell for sure if the stuff is still there or not but I'm sure most of it still works

Here's a graph of the most linked websites (this forum itself and pastebin excluded)

All the links and their http return code:

Spoiler


200	http://199.91.152.245/gy2ufo97dgwg/uqhdqb1l01j35at/CC-%5C%27Mine-menu%5C%27+-+Zitrone77+%5Bv1.1%5D+-+all+PCs.zip
200	http://205.196.120.125/40v5cdh1687g/t2wwe7zffcgk9ua/CC-%5C%27Mine-menu%5C%27+-+Zitrone77+%5Bv1.1%5D.zip
timeout	http://aafs01.funpic.de/files/PrintCapture
404	http://aafs01.funpic.de/files/remote.lua
404	http://aafs01.funpic.de/files/ScreenCapture
404	http://aafs01.funpic.de/files/ScreenCapture_BC
404	http://aafs01.funpic.de/files/ScreenCapture_BC
404	http://ac-get.darkdna.net/download.html
200	http://adf.ly/2516982/keykard
200	http://adf.ly/384749/dynet-10
404	http://adf.ly/391967/
200	http://adf.ly/908dS
200	http://adf.ly/98y7i
200	http://adf.ly/9CIj3
200	http://adf.ly/9CJRc
200	http://adf.ly/9CJYH
200	http://adf.ly/9CRWr
200	http://adf.ly/9pMc6
200	http://adf.ly/9pOkg
200	http://adf.ly/9vTqn
200	http://adf.ly/9vUR8
200	http://adf.ly/DEiHs
200	http://adf.ly/DNVVl
200	http://adf.ly/DP08i
200	http://adf.ly/Gfbau
200	http://adf.ly/iUHlS
404	http://adf.ly/LC1R.
200	http://ajworld.net/minecraft/cc/drive2
200	http://ajworld.net/minecraft/cc/edit2
200	http://ajworld.net/minecraft/cc/list2
404	http://ardera.funpic.de/sys/wordpress
200	http://backspace.cf/
200	http://bible.janvanrosmalen.nl/ccbible.zip
302	http://bit.ly/RVQTQi
302	http://bit.ly/VZyzVw
302	http://bit.ly/Xd5a7l
200	http://cdn.afterlifelochie.net/lua/?f=allbios-latest
200	http://controlguru.com/
200	http://craftnanny.org
301	http://creativecommons.org/licenses/by-nc/3.0/#
301	http://creativecommons.org/licenses/by-nc/3.0/
301	http://creativecommons.org/licenses/by-nc/3.0/
301	http://creativecommons.org/licenses/by-nc/3.0/
301	http://creativecommons.org/licenses/by-nc/3.0/legalcode
301	http://creativecommons.org/licenses/by-nc-nd/3.0/
301	http://creativecommons.org/licenses/by-nc-nd/3.0/
301	http://creativecommons.org/licenses/by-nc-sa/3.0/
301	http://creativecommons.org/licenses/by-nc-sa/3.0/
301	http://creativecommons.org/licenses/by-nc-sa/3.0/
301	http://creativecommons.org/licenses/by-sa/3.0/
200	http://cur.lv/mc4a
200	http://cur.lv/mc4d
200	http://cur.lv/mc4i
200	http://cur.lv/mc4i
302	http://db.tt/b7vB9ZmW
302	http://db.tt/eVP0i7ZE
200	http://dc236.4shared.com/download/a7ejRCuy/Pythagorean_Theorem_Calculator.zip?tsid=20130228-230727-57dc76ba
302	http://dl.dropbox.com/u/16610659/Minecraft/ComputerCraft/PVars
302	http://dl.dropbox.com/u/2934189/dice.zip
302	http://dl.dropbox.com/u/32207859/befactory-1.0.7z
302	http://dl.dropbox.com/u/36877135/ComputerCraft/TurtleJack
302	http://dl.dropbox.com/u/36877135/ComputerCraft/TurtleJack
302	http://dl.dropbox.com/u/36877135/ComputerCraft/WanderRound
302	http://dl.dropbox.com/u/36877135/ComputerCraft/WanderRound
302	http://dl.dropbox.com/u/48307892/House
302	http://dl.dropbox.com/u/50273341/ccSVN%20Client%2BServer.zip
302	http://dl.dropbox.com/u/50273341/LolOS%20V0.1%20PR_2.zip
302	http://dl.dropbox.com/u/53340494/Youtube%20Downloads/Piano%20-%20ComputerCraft.rar
302	http://dl.dropbox.com/u/55126910/I%20Cube%20OS.zip
302	http://dl.dropbox.com/u/59329517/Oh%20no%21%20I%20don%27t%20know%20how%20to%20use%20the%20updater%21.zip
302	http://dl.dropbox.com/u/59329517/Updater%20v0.1.zip
302	http://dl.dropbox.com/u/60138622/MC-Lua/UNityOS/Release/UNityOS_v0.2.zip
302	http://dl.dropbox.com/u/6061795/minenix/disk.zip
302	http://dl.dropbox.com/u/65278367/av
302	http://dl.dropbox.com/u/65278367/turtlebomb
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/calc
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/calcadd
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/calcdivide
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/calcmultiply
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/calcsq
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/calcsubtract
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/namegen
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/numgen
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/phonegen
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/selfdestruct
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/turtle/controller
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/turtle/house
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/turtle/houseall
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/turtle/housefence
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/turtle/housetorch
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/mods/ComputerCraft/lua/rom/programs/turtle/housewindows
302	http://dl.dropbox.com/u/66296404/Minecraft/ComputerCraft/Source%20Code.zip
302	http://dl.dropbox.com/u/66296404/Other/Race%20Pro%20Kid%27s%20CC%20Programs%20Installer.exe
200	http://download1583.mediafire.com/ocj9t4f2lqgg/m7ckcno1s5bp5km/CrafterNet+v0.1.zip
Valid name, no data record of requested type	http://downloads.xyzzy.cc/CC-RP-elevator-code-v2.zip
301	http://en.wikipedia.org/wiki/Brainfuck
301	http://en.wikipedia.org/wiki/Combinatorics
301	http://en.wikipedia.org/wiki/Digital_Fortress
301	http://en.wikipedia.org/wiki/Forth_(programming_language)
302	http://epicnessisbrettsmith21.enjin.com/
host not found	http://erayarslan.com/door
200	http://forestry.sengir.net/wiki/index.php?action=home
404	http://forums.technicpack.net/threads/customizable-player-proximity-detecting-door.45502/
200	http://foxdev.co.uk/downloads/minecraft/seg48/font_fox48
200	http://foxdev.co.uk/downloads/minecraft/seg48/seg48
200	http://foxdev.co.uk/downloads/minecraft/seg48/seg48ctl
200	http://foxdev.co.uk/downloads/minecraft/seg48/seg48world.zip
200	http://g7.byethost.com/firebox.html
200	http://g7.byethost.com/FireBoxScreenShot/
200	http://g7.byethost.com/FireBox.zip
connection refused	http://git.grimpen.net/fritz/interlocking.git/
301	http://github.com/ccLinux/kernel
301	http://github.com/ops99/LuaPrograms
302	http://goo.gl/i3gJ9
301	http://gyazo.com/c2417e9a7e278c531a57125bc4b8671c
200	http://hastebin.com/citocayigu
Valid name, no data record of requested type	http://host/door
Valid name, no data record of requested type	http://host/door/index.php
Valid name, no data record of requested type	http://host/door.txt
200	http://hyperq.web44.net/
200	http://hyperq.web44.net/
404	http://i1161.photobucket.com/albums/q506/Bamse5b/2012-10-05_133057.png
404	http://i1161.photobucket.com/albums/q506/Bamse5b/2012-10-05_133104.png
404	http://i1161.photobucket.com/albums/q506/Bamse5b/2012-10-05_133118.png
200	http://i.imgur.com/2069qxn.png
200	http://i.imgur.com/47AC6NX.png
200	http://i.imgur.com/48hyN4L.png
200	http://i.imgur.com/PSg7JKt.png
200	http://i.imgur.com/QggBF92.png
200	http://i.imgur.com/Uuancht.png
200	http://i.imgur.com/Vj8jsNt.png
200	http://i.imgur.com/XXswSTw.png
200	http://i.imgur.com/Y7CqxzR.png
200	http://i.imgur.com/YDRCXeN.png
200	http://i.imgur.com/z08Ur.png
404	http://img600.imageshack.us/img600/9328/seg48fail.png
200	http://imgur.com/a/aGJmD
404	http://imgur.com/a/Itogk
200	http://imgur.com/AtdzdEt
200	http://imgur.com/IO3rErr
200	http://imgur.com/QOppk9N
200	http://jeremy.vyska.info/2013/03/tekkit-lite-elevator-control/1469
404	http://justynthecoder.tumblr.com/
200	http://kapowcreations.com/games
200	http://kapowcreations.com/games
200	http://kapowcreations.com/games
200	http://kapowcreations.com/games
200	http://kapowcreations.com/games
Valid name, no data record of requested type	http://login.minecraft.net/?user=
200	http://lua.gts-stolberg.de/en/index.php?uml=1
200	http://lua-users.org/wiki/TutorialDirectory
200	http://mad-os.comyr.com/Mad-DL
200	http://mad-os.comyr.com/Mad-DL/Mad-DL.zip
200	http://mad-os.comyr.com/Mad-DL/Mads_Redworks_Installer.zip
404	http://master.mygame-community.de/OpenPeripheralCore-0.3.0-snapshot-1.jar
200	http://mccraftcpl.proboards.com/index.cgi?action=display&amp;board=programsboard&amp;thread=101&amp;page=1
200	http://mccraftcpl.proboards.com/index.cgi?action=display&amp;board=programsboard&amp;thread=257
200	http://mccraftcpl.proboards.com/index.cgi?board=altos&amp;action=display&amp;thread=87
connection refused	http://mcgrapeseed.com/forum/index.php
connection refused	http://mcgrapeseed.com/forum/index.php
connection refused	http://mcgrapeseed.com/forum/member.php?action=profile&amp;uid=4
host not found	http://mduk.pw/0DutCa
host not found	http://mduk.pw/Q5qwt4
301	http://mieper.de/wp-content/uploads/2012/12/2012-12-27_01.45.35-1024x542.png
301	http://mieper.de/wp-content/uploads/2012/12/2012-12-27_01.47.28-1024x542.png
200	http://minecraft.gamepedia.com/Tutorials/Basic_Logic_Gates
404	http://mirror.openshell.no/minecraft/computercraft/oddstr13s_advanced_doorlock_v2.7z
404	http://mirror.openshell.no/minecraft/computercraft/oddstr13s_advanced_doorlock_v2.tgz
404	http://mirror.openshell.no/minecraft/computercraft/oddstr13s_advanced_doorlock_v2.zip
404	http://mitchfizz05.net/misc/cc_js_imagerenderer/
301	http://m.youtube.com/index?desktop_uri=%2F&amp;gl=GB#/watch?v=LfnRpzVL0hs
301	http://openmods.info
200	http://paste42.de/7302/
timeout	http://pixeltoast.x64.me/cc/diskreceiver
404	http://pkpoison378.wix.com/agoldfish-programs
200	http://prntscr.com/1nb0dh
200	http://prntscr.com/1nb1l6
404	http://programcrafinc.webs.com/
host not found	http://pts.failreactor.com/cc/droid.zip
host not found	http://pts.failreactor.com/cc/server.zip
404	http://puu.sh/2AwuB
404	http://puu.sh/2Awvj
200	http://regex.info/blog/lua/json
200	http://rosettacode.org/wiki/Pi#Lua
200	http://s1147.photobucket.com/user/RandomShovel/slideshow/CC-Item%20ID%20Tracker
200	http://s1147.photobucket.com/user/RandomShovel/slideshow/CC-Monitor%20GUI
200	http://s1147.photobucket.com/user/RandomShovel/slideshow/CC-Startup%20Lock
200	http://s1147.photobucket.com/user/RandomShovel/slideshow/CC-Todo%20List
200	http://sam.zoy.org/wtfpl/
301	https://bitbucket.org/DrakuSoft/securbyte-computercraft/issues/new
301	https://bitbucket.org/TheVarmari/tuigo
200	https://code.google.com/p/luaforwindows/
302	https://dl.dropbox.com/s/alnuhpdvxx5hz7a/SockCraftIrc_2.1.1.zip?dl=1
302	https://dl.dropbox.com/s/hnfq0x0tddia2qf/SockCraftIrc_2.0.zip?dl=1
302	https://dl.dropbox.com/s/ufmn8cpy8gmpqap/SockCraftIrc_2.1.zip?dl=1
302	https://dl.dropbox.com/u/13281778/CC%20Images/Reactor%20Control.zip
302	https://dl.dropbox.com/u/14521842/TangentPaint
302	https://dl.dropbox.com/u/16727935/Code/dropbox.lua
302	https://dl.dropbox.com/u/26535928/Pics/bawsChat.png
302	https://dl.dropbox.com/u/26746878/ComputerCraft/Bank/v0.1.zip
302	https://dl.dropbox.com/u/36877135/ComputerCraft/pathfollower
302	https://dl.dropbox.com/u/36877135/ComputerCraft/pathfollower
302	https://dl.dropbox.com/u/43297400/Images/Other/bosschat-3d.png
302	https://dl.dropbox.com/u/43297400/Images/Other/bosschat-gears.png
302	https://dl.dropbox.com/u/43297400/Images/Other/bosschat-gradient.png
302	https://dl.dropbox.com/u/43297400/Images/Other/bosschat-idk.png
302	https://dl.dropbox.com/u/43297400/Images/Other/bosschat-mc.png
302	https://dl.dropbox.com/u/43297400/Images/Other/bosschat-rocky.png
302	https://dl.dropbox.com/u/45743579/LuaCreations/SensorCenter/sencenter
302	https://dl.dropbox.com/u/48373980/cSnake
302	https://dl.dropbox.com/u/48373980/fileshare
302	https://dl.dropbox.com/u/9049845/CC/24time
302	https://dl.dropbox.com/u/9049845/CC/branch
302	https://dl.dropbox.com/u/9049845/CC/corridor
302	https://dl.dropbox.com/u/9049845/CC/layer
302	https://dl.dropbox.com/u/9049845/CC/messages
302	https://dl.dropbox.com/u/9049845/CC/minerplacer
302	https://dl.dropbox.com/u/9049845/CC/towerbuilder
302	https://dl.dropboxusercontent.com/u/861751/imageconverter.jar
302	https://docs.google.com/open?id=0B0QSbWk85WR5cmFWWGhsc1pRMEd1SGdZSWxlR3M0QQ
301	https://docs.google.com/spreadsheets/d/1HoxhLvBzhQHx5U7ZM1A7nMbmcYesyZ4QKT8c_57qy1g/edit#gid=434245554
301	http://sdrv.ms/ZZy5CH
200	http://sealife.top-web.info/cc/sg/mods/sg_monitor.lua
301	https://gist.github.com/3314193
301	https://gist.github.com/4388010
301	https://gist.github.com/tuogex/7017200
301	https://github.com/alexandrecoc/Redworks-INSTALLER/zipball/master
301	https://github.com/ccLinux/openssh-client
301	https://github.com/ccLinux/opensshd
301	https://github.com/ChenThread/ctif
301	https://github.com/crazybmanp/Bauth
301	https://github.com/cswarm
301	https://github.com/cswarm/router
301	https://github.com/cswarm/swarmc
301	https://github.com/cswarm/swarmd
301	https://github.com/cswarm/worker
301	https://github.com/DelusionalLogic/openRedWork
301	https://github.com/DemHydraz
301	https://github.com/Doridian/ComputerCraftFTPd
301	https://github.com/downloads/Doridian/ComputerCraftFTPd/CCFTP_7.zip
301	https://github.com/downloads/ops99/LuaPrograms/tapi
301	https://github.com/downloads/ops99/LuaPrograms/trt
301	https://github.com/ElvishJerricco/Project-NewLife
301	https://github.com/Etherous/EtherWorks
301	https://github.com/FuzzyPurp/Redworks
301	https://github.com/FuzzyPurp/Redworks-FLOPPY
301	https://github.com/FuzzyPurp/Redworks-FLOPPY/zipball/master
301	https://github.com/FuzzyPurp/Redworks/zipball/master
301	https://github.com/jaranvil/CraftNanny
301	https://github.com/jaredallard
301	https://github.com/jesusthekiller/RednetControl
301	https://github.com/jhartikainen/cc-jhsh
301	https://github.com/jhartikainen/cc-jhsh/blob/master/README.md
301	https://github.com/kizz12/KBUF
301	https://github.com/Leonardoas26/RedID
301	https://github.com/LNETeam/Path
301	https://github.com/luaforge/json
301	https://github.com/lyqyd/cc-lsh
301	https://github.com/lyqyd/LyqydNet-Programs
301	https://github.com/mad1231999/Computer-Craft-Programs/blob/master/lua/bios.lua
301	https://github.com/mad1231999/Computer-Craft-Programs/blob/master/lua/rom/apis/Button
301	https://github.com/mad1231999/Computer-Craft-Programs/blob/master/lua/rom/apis/flib
301	https://github.com/mad1231999/Computer-Craft-Programs/blob/master/lua/rom/apis/ovar
301	https://github.com/mad1231999/Computer-Craft-Programs/blob/master/lua/rom/apis/Textfield
301	https://github.com/mad1231999/Computer-Craft-Programs/blob/master/lua/rom/programs/ovarexample1
301	https://github.com/MewesK/Midcraft-Commander/zipball/master
301	https://github.com/MysticT/MysticOS
301	https://github.com/parkerkane/cpapi/tree/release
301	https://github.com/parkerkane/cpapi/zipball/release
301	https://github.com/P-T-/HTTPNet
301	https://github.com/P-T-/vfs
301	https://github.com/SeaLife/sg-control
301	https://github.com/SeaLife/sg-control/blob/master/bios.lua
301	https://github.com/SeaLife/sg-control/blob/master/sg_hooks.lua
301	https://github.com/SeaLife/sg-control/blob/master/sg.lua
301	https://github.com/SeaLife/sg-control/blob/master/sg_mod.lua
301	https://github.com/SeaLife/sg-control/blob/master/sg_screens.lua
301	https://github.com/Selim042/CC-Backports
301	https://github.com/Sir-Mr-Bman/IndustrialSecurity
301	https://github.com/Team-CC-Corp/Cinnamon
301	https://github.com/Team-CC-Corp/ClamShell
301	https://github.com/Team-CC-Corp/LuaLua
301	https://github.com/theoriginalbit/CCTube/issues
301	https://github.com/theoriginalbit/CCTube/tree/lighttube
301	https://github.com/tmerr/computercraftIRC/releases
301	https://github.com/tomass1996/Redworks-Floppy-Installer-CS/zipball/master
301	https://github.com/tomass1996/Redworks-Floppy-Installer-Java/zipball/master
301	https://github.com/TomCompiles/ComputerCraft/tree/master/GmailViewer
301	https://gitter.im/ccTCP/contribute
301	https://gitter.im/ccTCP/discussion
301	https://gitter.im/TARDIX/Dev
200	http://sitenil.comli.com/CC/API/essentials.lua
200	http://sitenil.comli.com/CC/API/json.lua
200	http://sitenil.comli.com/CC/Script/chatMonitor.lua
200	http://slides.com/jaredallard/swarm#/
404	https://mega.nz/#F&#33;T4pEBAhb
200	http://smiley43210.eu5.org/applications/download.php?appID=0
301	https://raw.github.com/infinikiller64/ComputerCraft/master/diskplayer
301	https://raw.github.com/theoriginalbit/CCTube/develop/CCTube
301	https://raw.githubusercontent.com/ChenThread/ctif/master/viewers/ctif-cc.lua
302	https://sites.google.com/site/ccserver12/
301	https://skydrive.live.com/redir?resid=EC6AEAB5E1C10775&#33;452
302	https://www.cubby.com/p/bc9962c5451d4d2386836a4b9518f4be/Minecraft
302	https://www.cubby.com/p/e3d917d2c1304fb78e04d7a36bc88d18/CC_BIOS
301	https://www.dropbox.com/gallery/52210120/1/fileMan?h=996eae
301	https://www.dropbox.com/s/5v4wkv0p2rv1vyr/emeter.zip
301	https://www.dropbox.com/s/kkfw8homrdon7c9/Base%20Defense.zip
301	https://www.dropbox.com/s/mz9fxj3h3uqhyyl/pic.png
301	https://www.dropbox.com/s/rktrz4b2t9bmbu8/Arduino.zip?dl=0
301	https://www.dropbox.com/s/xlj4n9gt0nwqy03/time
301	https://www.dropbox.com/s/z6t70syvvtte88f/KeyKard.zip
301	https://www.github.com/ccTCP
200	https://www.google.co.uk/search?q=16+segment+LED+displays&amp;amp;tbm=isch
200	https://www.mediafire.com/?uhe47lt2slyt5ws
301	https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&amp;hosted_button_id=XGB6KV8TM4NPJ
host not found	https://www.rapidshare.com/#&#33;download|689|928898173|ccelevator.rar|6
302	https://www.virustotal.com/en/file/45c07b17d8eeebc8ea9438b9e38a05ffbbbbeba35e3ac79cfaa77fe1cb8a421c/analysis/1415767582/
301	https://www.youtube.com/watch?v=2uhadL37svs
301	https://www.youtube.com/watch?v=CnAcEO1175c
301	https://www.youtube.com/watch?v=gp-nY_ATlbs
301	https://www.youtube.com/watch?v=Rp2mjNnsa9s
301	https://www.youtube.com/watch?v=WtI3Wl0tsFs
host not found	http://t3kbau5.tk/
200	http://techzunecc.operontech.com/programs-list.html
200	http://techzunecc.operontech.com/turtle-api.html
200	http://techzunecc.operontech.com/turtle-api.html
200	http://techzunecc.operontech.com/turtle-api.html
200	http://tesla1889.site90.com/examples/example.forth
200	http://tesla1889.site90.com/src/lib/ported_languages/forth.lua
410	http://thegreatstudio.webs.com
200	http://theory.stanford.edu/~amitp/GameProgramming/
200	http://theraceprokid.1talk.net/t6-computercraft-programs
200	http://turtlescripts.com/project/gjdh20-Act
200	http://turtlescripts.com/project/gjdhi8-Mass-Copy
301	http://twitter.com/stroughtonsmith/status/233296347409838081
404	http://users.aber.ac.uk/tis4/webmessage.zip
200	http://validator.w3.org/
200	http://webchat.esper.net/?channels=#computercraft
301	http://wiki.creativecommons.org/Frequently_Asked_Questions#Do_Creative_Commons_licenses_affect_fair_use.2C_fair_dealing_or_other_exceptions_to_copyright.3F
301	http://wiki.creativecommons.org/Frequently_Asked_Questions#I_don.E2.80.99t_like_the_way_a_person_has_used_my_work_in_a_derivative_work_or_included_it_in_a_collective_work.3B_what_can_I_do.3F
301	http://wiki.creativecommons.org/Frequently_Asked_Questions#When_are_publicity_rights_relevant.3F
301	http://wiki.creativecommons.org/Public_domain
200	http://wiki.darkserver.co.uk/Openperipheral_mech
200	http://www.4shared.com/file/6yPVWk1A/Wander_AI.html
200	http://www.4shared.com/file/IheV9dfU/Pathfinder_AI.html
200	http://www.4shared.com/file/Qo6VxkAt/screensaver.html
200	http://www.4shared.com/file/xkG34xPS/Pathfinder_II_2.html
404	http://www.bluetideos.weebly.com
host not found	http://www.eclipse-project.de/forum/viewtopic.php?f=19&amp;t=329
200	http://www.failreactor.com/images/2012-03-16_11.56_.57_.png
200	http://www.failreactor.com/images/2012-03-16_11.58_.21_.png
200	http://www.failreactor.com/images/2012-03-16_11.59_.20_.png
200	http://www.failreactor.com/images/2012-03-16_12.04_.36_.png
200	http://www.failreactor.com/set/88
200	http://www.freebsddiary.org/screen.php
301	http://www.github.com/mad1231999/Computer-Craft-Programs
host not found	http://www.jesusthekiller.com/?p=24
200	http://www.lozengia.com/tmas/Downloads/Minecraft/test/httpremote_server.rar
200	http://www.lozengia.com/tmas/Downloads/Minecraft/test/httpremote_server_v0.2.rar
200	http://www.lozengia.com/tmas/Downloads/Minecraft/test/input.php
200	http://www.lua.org/pil/
301	http://www.luarocks.org/en/Download
200	http://www.mediafire.com/?16l5ucd4c7arq4e
200	http://www.mediafire.com/?1ndm5rxdrkb0ccw
200	http://www.mediafire.com/?2k62u4i7i4wfk97
200	http://www.mediafire.com/?31s04tb9523etrl
200	http://www.mediafire.com/?55k3g6n89dr3v5k
200	http://www.mediafire.com/?5jf8kb9wqdxqq1v
301	http://www.mediafire.com/?79pf7prkyc9hi9j
200	http://www.mediafire.com/?7k1g4vi25psc5gq
200	http://www.mediafire.com/?81w2iqhddddnr45
200	http://www.mediafire.com/?8cj39zyt1b9jrhg
200	http://www.mediafire.com/?8ignmc6lg7w6cla
200	http://www.mediafire.com/?8yw9kslcoiodu9s
200	http://www.mediafire.com/?9t9kbtt9z220jgb
200	http://www.mediafire.com/?aoy2ljmw5kls0yu
200	http://www.mediafire.com/?aza20c7t31bg1mf
200	http://www.mediafire.com/?cgq6n42zrfr2yni
200	http://www.mediafire.com/download/1g4d4oon6zaf6po/CC+Pastebin+Fix.zip
200	http://www.mediafire.com/download/9x9vnzwuv9hlzo3/SiKeDTechLogin.zip
200	http://www.mediafire.com/download/bwl6p3siad60q1p/CCResourcePackExample2.zip
200	http://www.mediafire.com/download.php?2wbvv762soq0wbh
200	http://www.mediafire.com/download.php?66mh4q9t054t8hc
200	http://www.mediafire.com/download.php?77rsbg1k1g8psrh
200	http://www.mediafire.com/download.php?8z866q4dyezd49b
200	http://www.mediafire.com/download.php?a7we0qyh05me2fm
200	http://www.mediafire.com/download.php?bkxymubnulmkk5q
200	http://www.mediafire.com/download.php?ca65jotbt257ocg
200	http://www.mediafire.com/download.php?i5510u1v3xncsau
200	http://www.mediafire.com/download.php?kcxpvovo6l94hjv
200	http://www.mediafire.com/download.php?w7yfraba2vskqvp
200	http://www.mediafire.com/download.php?y6oviis3bmxjx0w
301	http://www.mediafire.com/download.php?zqv1fx1r3jy9442
200	http://www.mediafire.com/download/qr57m1aa20dakcd/CC_Channel_Lock.zip
200	http://www.mediafire.com/download/v1kmv1wu8oqc6i2/CCResourcePackExample1.zip
200	http://www.mediafire.com/download/z5q52mj0j2h7okr/Dynet%201.0.zip
200	http://www.mediafire.com/?g6h0dso3flac95w
200	http://www.mediafire.com/?gbppjttghp1vlhr
200	http://www.mediafire.com/?gg9uzq3h01rccbf
200	http://www.mediafire.com/?gtud4o89onompse
200	http://www.mediafire.com/?hama624ao9x13nl
200	http://www.mediafire.com/?hn3hli4t32ihpp4
200	http://www.mediafire.com/?iswtm251h05mg2j
301	http://www.mediafire.com/?kp5wf54pvc73zmb
200	http://www.mediafire.com/?l18xan2iq81gzyq
200	http://www.mediafire.com/?ldl5ra4lce9pdh0
200	http://www.mediafire.com/?mehe6u9ge82h9c7
200	http://www.mediafire.com/?mjs6puui1ta1kk9
200	http://www.mediafire.com/?nj349rtk2jlsigb
200	http://www.mediafire.com/?o2yh6k2x8ldb1m2
200	http://www.mediafire.com/?od43tscvocspqvc
200	http://www.mediafire.com/?p3cir62jt17e2q1
200	http://www.mediafire.com/?r4wr3el15srszqh
200	http://www.mediafire.com/?t0lpb3y9x173vkm
301	http://www.mediafire.com/?t2wwe7zffcgk9ua
200	http://www.mediafire.com/?th58eoc48lfi8fh
200	http://www.mediafire.com/?uiftfas325n0rtm
200	http://www.mediafire.com/?ukemmaoanboww1i
301	http://www.mediafire.com/?uqhdqb1l01j35at
200	http://www.mediafire.com/?uswkldd30ndhlp1
200	http://www.mediafire.com/view/?t5tdjdqtycy2y3m
200	http://www.mediafire.com/?x4w8nv1a3nmyy42
200	http://www.mediafire.com/?y204ri534yzqv2i
200	http://www.mediafire.com/?ywyveithe4gsf28
200	http://www.minecraftforum.net/topic/365357-123-eloraams-mods-redpower-2-prerelease-4e/
400	http://www.minecraftforum.net/topic/727142-fallout-3-map/#entry16283786
400	http://www.minecraftforum.net/topic/892282-11-computercraft-121/page__pid__12672007__st__4440#entry12672007
302	http://www.raywenderlich.com/4946/introduction-to-a-pathfinding
301	http://www.reddit.com/r/feedthebeast/comments/1ebmjx/i_figured_out_a_way_to_import_images_onto/
301	http://www.reddit.com/r/Minecraft/comments/r8c16/i_made_a_tool_to_locate_a_stronghold_using_only/
301	http://www.reddit.com/r/Minecraft/comments/r8c16/i_made_a_tool_to_locate_a_stronghold_using_only/
301	http://www.reddit.com/user/ItsMartin
host not found	http://www.RedworksOS.com
200	http://www.siz.co.il/
200	http://www.siz.co.il/
200	http://www.siz.co.il/
200	http://www.tekkitbyfifty.com/forum/m/8102934/viewthread/10058561-we-need-in-game-cash-to-start-awesome-ftb-internet-company-called-anvela/post/last#last
host not found	http://www.tlcairriders.com/Treebane/
400	http://www.twitter.com/https://twitter.com/#&#33;/elfcor
301	http://www.youtube.com/user/baseball4355
301	http://www.youtube.com/user/SAXGUY1999
301	http://www.youtube.com/watch?v=cD-EmbhNCEQ&amp;list=PLh7QW1CBWNGJR1ijZpEiz42axB_pVGLSR
301	http://www.youtube.com/watch?v=JewovkmYawY
301	http://www.youtube.com/watch?v=qzW3pxwW_iM&amp;feature=share&amp;list=PL69E38751E0C625FE
301	http://www.youtube.com/watch?v=tV-YOnuHDXY&amp;
200	http://xkcd.com/1179/
301	http://youtu.be/4R_qf89vxJ8
301	http://youtube.com/mrZitrone77
301	http://youtu.be/rPpxcnRGCOY?t=19m5s

All the hosts and how many times they show up:

Spoiler


62	http://www.mediafire.com
36	http://dl.dropbox.com
19	http://adf.ly
11	http://creativecommons.org
11	http://i.imgur.com
10	http://cc-get.djranger.com
8	http://computercraft.info
6	http://www.youtube.com
5	http://kapowcreations.com
5	http://www.failreactor.com
5	http://aafs01.funpic.de
5	http://imgur.com
4	http://en.wikipedia.org
4	http://s1147.photobucket.com
4	http://techzunecc.operontech.com
4	http://www.4shared.com
4	http://www.reddit.com
4	http://wiki.creativecommons.org
4	http://cur.lv
4	http://foxdev.co.uk
3	http://i1161.photobucket.com
3	http://ajworld.net
3	http://www.lozengia.com
3	http://sitenil.comli.com
3	http://mcgrapeseed.com
3	http://www.siz.co.il
3	http://host
3	http://bit.ly
3	http://g7.byethost.com
3	http://mad-os.comyr.com
3	http://mirror.openshell.no
3	http://mccraftcpl.proboards.com
3	http://www.minecraftforum.net
2	http://github.com
2	http://mieper.de
2	http://tesla1889.site90.com
2	http://hyperq.web44.net
2	http://mduk.pw
2	http://puu.sh
2	http://pts.failreactor.com
2	http://turtlescripts.com
2	http://db.tt
2	http://prntscr.com
2	http://youtu.be
1	http://validator.w3.org
1	http://download1583.mediafire.com
1	http://www.tekkitbyfifty.com
1	http://www.raywenderlich.com
1	http://ardera.funpic.de
1	http://www.tlcairriders.com
1	http://www.RedworksOS.com
1	http://m.youtube.com
1	http://minecraft.gamepedia.com
1	http://205.196.120.125
1	http://slides.com
1	http://controlguru.com
1	http://theory.stanford.edu
1	http://www.freebsddiary.org
1	http://bible.janvanrosmalen.nl
1	http://dc236.4shared.com
1	http://wiki.darkserver.co.uk
1	http://jeremy.vyska.info
1	http://hastebin.com
1	http://smiley43210.eu5.org
1	http://mitchfizz05.net
1	http://pixeltoast.x64.me
1	http://xkcd.com
1	http://thegreatstudio.webs.com
1	http://erayarslan.com
1	http://rosettacode.org
1	http://regex.info
1	http://www.luarocks.org
1	http://lua-users.org
1	http://craftnanny.org
1	http://sdrv.ms
1	http://sealife.top-web.info
1	http://www.jesusthekiller.com
1	http://www.github.com
1	http://www.eclipse-project.de
1	http://img600.imageshack.us
1	http://twitter.com
1	http://ac-get.darkdna.net
1	http://theraceprokid.1talk.net
1	http://backspace.cf
1	http://pkpoison378.wix.com
1	http://t3kbau5.tk
1	http://www.lua.org
1	http://sam.zoy.org
1	http://youtube.com
1	http://forestry.sengir.net
1	http://www.twitter.com
1	http://git.grimpen.net
1	http://paste42.de
1	http://openmods.info
1	http://master.mygame-community.de
1	http://lua.gts-stolberg.de
1	http://www.bluetideos.weebly.com
1	http://goo.gl
1	http://gyazo.com
1	http://users.aber.ac.uk
1	http://programcrafinc.webs.com
1	http://forums.technicpack.net
1	http://epicnessisbrettsmith21.enjin.com
1	http://downloads.xyzzy.cc
1	http://webchat.esper.net
1	http://login.minecraft.net
1	http://cdn.afterlifelochie.net
1	http://199.91.152.245
1	http://justynthecoder.tumblr.com

Edited on 23 June 2016 - 03:10 PM