http not working for google search

http.request("http://www.google.com/search?q=anything")

Kingdaro, on 20 November 2012 - 01:39 PM said:

Try https? (if cc can even use https, that is)

Lyqyd, on 20 November 2012 - 02:00 PM said:

Are you using http.request correctly? You know it throws an event with the results rather than returning them, correct?

http.request("http://www.google.com/search?q=define+greece")
local response
while true do
local event,url,sourceText=os.pullEvent()
if event == "http_success" then
  response=sourceText.readAll()
  break
elseif event == "http_failure" then
  error("http_failure")
end
end
local t,a=string.find(response,'<td valign="top" style="padding-bottom:5px;padding-top:5px"><table class="ts"><tr><td>')
if not t then
print("Definition not found.")
else
local b,t=string.find(response,'</td>',a)
print(string.sub(response,a+1,b-1))
end

http.request("http://www.google.com/")

dissy, on 20 November 2012 - 06:14 PM said:

Does anyone happen to know what HTTP_USER_AGENT is sent by the HTTP API?
If it contains Java or Lua, it is quite possible google is blocking search requests from it. They do that for a lot of scripting languages.

I've run into that problem with both Java and TCL, when using the default user agent. I had to change the user agent to match FireFox (Which unfortunately is against the Google terms of service) to get non-error result back.

Espen, on 21 November 2012 - 12:39 AM said:

It sends a Java UserAgent information.
You can find out the exact string by using the HTTP API on, e.g. http://whatsmyuseragent.com/ and then looking for "Your User Agent" in the response body.

http.request("http://whatsmyuseragent.com/")
local response
while true do
local event,url,sourceText=os.pullEvent()
if event == "http_success" then
  response=sourceText.readAll()
  break
elseif event == "http_failure" then
  error("http_failure")
end
end
local t,a=string.find(response,'<strong>Your User Agent:</strong>')
if not t then
print("Tag not found.")
else
local b,t=string.find(response,'<br /><br />',t)
print(string.sub(response,a+1,b-1))
end

$ wget -U "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0" http://www.google.com/search?q=kittens
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `search?q=kittens'

$ wget -U "Java/1.7.0_07" http://www.google.com/search?q=kittens
HTTP request sent, awaiting response... 403 Forbidden
2012-11-20 18:25:14 ERROR 403: Forbidden.

dissy, on 21 November 2012 - 12:30 PM said:

Sorry it's taken so long for me to verify this, but Google search does indeed block the Java user agent.
I've confirmed this using wget and the -U option (to specify a user agent)

This is with a FireFox user agent:

$ wget -U "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0" http://www.google.com/search?q=kittens
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `search?q=kittens'

This is the exact same query made 5 seconds later with the Java agent:

$ wget -U "Java/1.7.0_07" http://www.google.com/search?q=kittens
HTTP request sent, awaiting response... 403 Forbidden
2012-11-20 18:25:14 ERROR 403: Forbidden.

Unfortunately there doesn't appear to be any way to "fix" this directly in Lua, short of using a different search engine.
This might help: http://en.wikipedia...._search_engines

https:// www.googleapis.com/customsearch/v1?key={INSERT-YOUR-KEY}&cx=017576662512468239146:omuauf_lfve&q={SEARCH-TERM}