[CC 1.1 - 1.481] Httptest v1.4

What does it do?
It makes use of the HTTP API to download a web-page of your choosing to a file on your ComputerCraft Terminal.

Why did I make this?
To give someone an idea for how to use HTTP API or to just download the internet, page for page. ;)/>

What are the prerequisites?

Spoiler

You have to enable the HTTP API first.
To do that, open your ComputerCraft config file, it can be found here: .minecraft\config\ComputerCraft.cfg
Look for …

enableAPI_http

… and change its setting from false to true
Then save the file and restart Minecraft.
Note: This file will be generated automatically the first time you successfully start Minecraft after installing ComputerCraft correctly.

How do I use it?

Spoiler

To get help on how to use it, just type:

httptest --help

For your convenience I'll post it here as well, though:

Spoiler

Usage: httptest <url> <output-filename>

Example:
httptest http://www.google.com/ webout
The response body will then be output into the file 'webout', line for line.

Where's the code?
Either here: http://pastebin.com/Kva1wLTe
Or here…

Spoiler

--[[
HttpTest by Espen v1.4
An example code to show how to use the ComputerCraft HTTP functions.

Thanks to Advert for making me aware of the redundant call to http.get after http.request!
--]]

--[[ Setting up variables ]]
local fileWriteOK
local url
local filename
local tArgs = { ... }


local function getHttpBody( url )
	http.request( url )
	
	while true do
		local event, url, hBody = os.pullEvent()
		if event == "http_success" then
			print( "HTTP SUCCESSnURL = "..url )
			return hBody
		elseif event == "http_failure" then
			error( "HTTP FAILUREnURL = "..url )	-- If the error is not catched, this will exit the program.
			return nil	-- In case this function is called via pcall.
		end
	end
end

local function processHttpBody( hBody, filename )
	local hFile
	
	if hBody then
			local body = hBody.readAll()	-- Read the whole body.
			hFile = io.open( filename, "w" )	-- Open the provided filename for writing.
			hFile:write( body )	 -- Write the body to the file.
			hFile:close()   -- Save the changes and close the file.
	else
		print( "Sorry, no body to process." )
	end
	
	hBody.close()   -- Do not know for sure if this is really necessary, but just in case.
end

local function checkOverwrite( filename )
	term.setCursorBlink( false )
	print("nWarning: File already exists. Overwrite? (Y/N)")
	
	while true do
			event, choice = os.pullEvent( "char" )  -- Only listen for "char" events.
			if string.lower( choice ) == "y" then return true end
			if string.lower( choice ) == "n" then return false end
	end
end

local function printUsage()
	print("Usage:")
	print("httptest <url> <output-filename>")
	print("Example:")
	print("httptest http://www.google.com/ webout")
	print("nThe response body will then be output into the file 'webout', line for line.")
end


--[[ ===== Execution Entry ===== ]]

--[[ Processing arguments ]]
if tArgs[1] then url = tArgs[1] end
if tArgs[2] then filename = tArgs[2] end

--[[ Making sure all necessary arguments were provided by the user ]]
if not url or not filename then
	printUsage()
elseif not fs.exists( filename ) or checkOverwrite( filename ) then
	processHttpBody( getHttpBody( url ), filename )	--If getHttpBody successfully returns a body, we continue with processing the body and saving it to a file.
end

Feel free to use, expand and improve on this code in any way you like. Also you don't have to mention me.
Only, if you share your modified code somewhere then it'd be nice if you could tell me about it, so that I can learn from your modifications, too. :)/>
K, hope this is of help to someone, have fun coding!

Cheers

EDIT #7: Updated thread title to CC v1.1 - v1.481+ and updated the prerequisite-instructions.
EDIT #6: Updated thread title to CC v1.47 and changed the prerequisite-instructions.
EDIT #5: Forgot a little "then" in the latest update (after checking for the "http_failure" condition). Java habits.^^

Very clean, and easy-to-read code.

One question: Aren't you requesting the website twice? (once, with http.request, once more with http.get)

From 'help http': "A period of time after a http.request() call is made, … Arguments are the url and a file handle if successful."

Advert, on 13 February 2012 - 02:44 PM said:

Very clean, and easy-to-read code.

One question: Aren't you requesting the website twice? (once, with http.request, once more with http.get)

From 'help http': "A period of time after a http.request() call is made, … Arguments are the url and a file handle if successful."

You're right, I must've totally overlooked that. *headdesk*
I'm changing the code as we speak and will update the post accordingly.

Thanks for the hint, much appreciated! ;)/>/>

---

Alright, just updated the first post. I thought I tried getting the body directly at first, but I think I might've tried this…

status, url = http.request( url )

… which didn't return anything, so I thought the code changed and the help-info just wasn't up to date.
But it didn't come to my mind that the body is returned along with the event instead.
Makes much more sense now.^^
So thx again for making me aware of this!

Not like Espen to derp mwahaha :(/>/> - sexy code tho ;)/>/>

If a server owner enabled HTTP API (E.g. for CC-Get) everyone could just use this code to download a virus or a trojaner to control the server. :unsure:/>/>
Same for clients, just downloading a file from the wrong site and your system is infected.

Leo Verto, on 06 March 2012 - 02:07 PM said:

If a server owner enabled HTTP API (E.g. for CC-Get) everyone could just use this code to download a virus or a trojaner to control the server. :unsure:/>/>
Same for clients, just downloading a file from the wrong site and your system is infected.

That's not really a danger, Leo Verto.
Because you can only download the file into the ComputerCraft folder and you can't execute .exe files with ComputerCraft.
You could write a Lua Virus, but the capabilites of that are pretty limited with Lua in CC and they don't have any effect on the real computer itself.
But even if it had, then that'd be a problem with ComputerCraft itself and not with the HTTP API per se.

Therefore the HTTP API does not endanger the computer any more than ComputerCraft itself does. B)/>/>

Should be moved to Tutorials?

FuzzyPurp, on 07 March 2012 - 05:55 AM said:

Should be moved to Tutorials?

What are you talking about? It's already in the tutorial section.

As of this post.

Espen, on 06 March 2012 - 02:30 PM said:

That's not really a danger, Leo Verto.
Because you can only download the file into the ComputerCraft folder and you can't execute .exe files with ComputerCraft.
You could write a Lua Virus, but the capabilites of that are pretty limited with Lua in CC and they don't have any effect on the real computer itself.
But even if it had, then that'd be a problem with ComputerCraft itself and not with the HTTP API per se.

Therefore the HTTP API does not endanger the computer any more than ComputerCraft itself does. B)/>/>

Yes it's actually a ComputerCraft problem, but I think your API makes it more easy to download files.
Another problem is you could spam the server with random files. :unsure:/>/>

I think it would be good to add a limit for downloaded files per computer (you could still use many computers) but a general problem is that there is no way to protect the code from being edited or to prevent the execution of specific commands.

Oh wait, is there any way of putting master programs in the rom folder of servers?
Like 'firewalls' or control programs to prevent abuse. B)/>/>

Leo Verto, on 07 March 2012 - 12:22 PM said:

[…] I think your API makes it more easy to download files.

It's not an API, but rather a tutorial for how to make use of the HTTP API.

Leo Verto, on 07 March 2012 - 12:22 PM said:

Another problem is you could spam the server with random files. :mellow:/>/>

I'm sorry but this is not an explicit problem of HTTP API.
Even if there was no HTTP API you could still spam a CC computer with random files by creating a program that does it.
Downloading webcontent via HTTP API is just another way to create files. It doesn't create them any faster or in higher numbers than letting a program randomly create files.

Leo Verto, on 07 March 2012 - 12:22 PM said:

Oh wait, is there any way of putting master programs in the rom folder of servers?
Like 'firewalls' or control programs to prevent abuse. ;)/>/>

Yes, if you're the admin / have access to the /rom folder then you can place programs there which can't be manipulated by non-privilidged users.
You can then make sure via the /rom/startup (or bios.lua) that your security programs are called on every computer start. Totally possible. :)/>/>

Please help! Every time I start minecraft the config file gets reset so I can't change font color, width and height or enable the http.

Solved! :)/>/>

So, I can use this to update ShadOS with a bit of coding and would it work with dropbox? Like… a pure text file?
EDIT:
I'll test the dropbox bit.
EDIT (2):
I am now coding a way to use this to auto-update ShadOS. (Dropbox) :P/>/> P.S. I will give you credit for your HTTP skills (Or maybe my HTTP Non-skills) Anyway, Asta la Vista! I'm off to make an updater!

How do you bypass the http url only thing, cause the one i want to get is https, so it wont let me.

coolblockj, on 16 May 2012 - 04:06 AM said:

How do you bypass the http url only thing, cause the one i want to get is https, so it wont let me.

Just use http://, no web server REQUIRES https:// to be on the URL.

thesbros, on 16 May 2012 - 07:38 PM said:

coolblockj, on 16 May 2012 - 04:06 AM said:

How do you bypass the http url only thing, cause the one i want to get is https, so it wont let me.

Just use http://, no web server REQUIRES https:// to be on the URL.

Actually, github raw does REQUIRE https.

Regardless, https support is in the next version.

Cloudy, on 16 May 2012 - 08:28 PM said:

thesbros, on 16 May 2012 - 07:38 PM said:

coolblockj, on 16 May 2012 - 04:06 AM said:

How do you bypass the http url only thing, cause the one i want to get is https, so it wont let me.

Just use http://, no web server REQUIRES https:// to be on the URL.

Actually, github raw does REQUIRE https.

Regardless, https support is in the next version.

Doesn't require it for me. Anyway, it's pretty simple to make something that will display the text of GitHub Raw on a regular http:// link.

thesbros, on 17 May 2012 - 01:50 AM said:

Cloudy, on 16 May 2012 - 08:28 PM said:

thesbros, on 16 May 2012 - 07:38 PM said:

coolblockj, on 16 May 2012 - 04:06 AM said:

How do you bypass the http url only thing, cause the one i want to get is https, so it wont let me.

Just use http://, no web server REQUIRES https:// to be on the URL.

Actually, github raw does REQUIRE https.

Regardless, https support is in the next version.

Doesn't require it for me. Anyway, it's pretty simple to make something that will display the text of GitHub Raw on a regular http:// link.

Im using dropbox, which does require https.

How would I simplify the code so it just downloads a chosen code to a chosen file without using events?

Advert, on 13 February 2012 - 02:44 PM said:

Very clean, and easy-to-read code.

Couldn't agree more!

I'm probably doing something obvious wrong, but it doesn't work for me. I changed the 0 into a 1 in the mod_computercraft file, opened minecraft, typed httptest into the console, and I got No such program. Double-checked if the 1 was saved, it was. Restarted minecraft, and it still didn't work. Any help would be appreciated.

You installed he program first, right?

… That's what I meant with obvious. I thought it was something of the HTTP api >_< What I'm now wondering about, what is an easy way to copy the file into a minecraft computer?

There is a real, actual linux program called "W3M" that gives you the ability to internet browse from terminal. I use it all the time when working with command line linux. One could navigate around the text cursor to press enter on things to open links and activate text boxes and little ASCII art things are supposed to be checkboxes and radio buttons. It would be cool if someone could develop that for computercraft, seeing all the other stuff like OSes people have developed for it.

In the getHttpBody function, you use a while true loop, but both paths of the if statement lead to returns, so this loop will never repeat and can be safely removed. You will want the loop and more robust handling of the pullEvent results if this code is to be used in an environment that could be generating other event types, like having it attached to bundled cable that's sending signals. For just a simple one-shot program to fetch a file, as this is, the existing code (minus the extraneous loop) should work fine.

In the specific case where there wouldn't occur any other events than HTTP events, the loop wouldn't be necessary, that's true.

But you have to keep in mind that even a key-press is an event. And with advanced computers on the horizon (beta) you never know if there will be even more events in the future which don't necessarily need one to hook something up to the computer. Like mouse events for example.

So to make it safe from any such accidental events I loop until one of the expected events occurs.
Also, if you were to ignore that and do it without the loop then you would definitely loose certainty over the event.
That is, you could get the message that there was an HTTP failure and you would have no way of knowing with certainty if that was because of an actual HTTP failure or another unintended event.

From a pure technical standpoint you are correct, though. It won't necessarily loop.
But the "necessarily" is the reason I have to loop in order to make sure I catch the right event.

At least that's my best-practice for playing it safe. :P/>/>

EDIT: Just saw that I wasn't explicitly checking for "http_failure" and as such what I was saying earlier about the random events wasn't actually prevented.^^
Fixed that now, but the while-loop is still needed for the reason I mentioned, i.e. the function shouldn't react to any event except "http_success" or "http_failure".
Personally, I'd add a timeout-functionality to prevent waiting endlessly for an HTTP answer, but I'll leave that as an exercise to the reader.
If someone really can't make heads-or-tails with that though, then just write me a PM. :)/>/>

coolblockj, on 17 May 2012 - 05:53 PM said:

thesbros, on 17 May 2012 - 01:50 AM said:

Cloudy, on 16 May 2012 - 08:28 PM said:

thesbros, on 16 May 2012 - 07:38 PM said:

coolblockj, on 16 May 2012 - 04:06 AM said:

How do you bypass the http url only thing, cause the one i want to get is https, so it wont let me.

Just use http://, no web server REQUIRES https:// to be on the URL.

Actually, github raw does REQUIRE https.

Regardless, https support is in the next version.

Doesn't require it for me. Anyway, it's pretty simple to make something that will display the text of GitHub Raw on a regular http:// link.

Im using dropbox, which does require https.

I use dropbox, it doesn't require it if it is in the public folder, at least for me. Just get the public link from the site, and not off of your computer (If you installed dropbox.)

Lolgast, on 31 May 2012 - 06:09 PM said:

… That's what I meant with obvious. I thought it was something of the HTTP api >_< What I'm now wondering about, what is an easy way to copy the file into a minecraft computer?

On a windows computer all of your ComputerCraft computers are in this folder:

%appdata%.minecraftsavesYOURWORLDSAVEcomputer

There will be numbered folders in there, starting from 0.
Every number-folder is a computer, with the number representing the computer ID.

Now, to copy a program into such a computer, just create a file in one of these folders (without a .lua extension!).
You can do this while Minecraft is running and the changes you make with a texteditor on a file in that folder will be immediately available in-game.
That way you can write code using a text-editor of your choice, while testing the code immediately in-game.

Hope this helps :P/>/>