This is a read-only snapshot of the ComputerCraft forums, taken in April 2020.
CraftedCart's profile picture

[SOLVED!] HTTP Basic Authentication (Or OAuth) / Rate Limiting with GitHub's API

Started by CraftedCart, 03 October 2015 - 01:47 PM
CraftedCart #1
Posted 03 October 2015 - 03:47 PM
TL;DR: GitHub's rate limit is too small, how do I authenticate to increase it, or make my program use less requests?

I'm trying to make a GitHub client/repo downloader. Everything's going well so far, except that I seem to hit the GitHub rate limit. If I authorise, I can increase the rate limit from 60 to 5000, however I can't seem to figure out how to do that.

I've tried:So how would I authenticate with GitHub (With Basic Auth or OAuth)? It would be even better if it was secure

EG: I tried using my program to download Twitter Bootstrap. Before it had even finished finding all of the files, it hit the rate limit.
The program makes a request for every folder it finds in the repo
EDIT: Twbs/Bootstrap is an inappropriate repo (I really shouldn't use this as an example). Still, authentication methods would be nice

If authentication cannot be done, is there any way I can make my code use less api requests?
Note: jsonParser.lua converts the json from the API to a Lua table, and downloader.lua takes a table and actually downloads the files (Although this never happens with twbs/bootstrap as there are too many folders)
If it helps, jsonParser and downloader can be found here: https://github.com/C...CraftedCart/api
EDIT: JsonParser & Downloader has been relocated to https://github.com/CraftedCart/CC-Common/tree/master/api

Spoiler

repoName = "GitByte"

json = (loadfile "CraftedCart/api/jsonParser.lua")()

local files = {}
local dirs = {}

local function getDir(path)
  web = http.get("https://api.github.com/repos/twbs/bootstrap/contents" .. path)
  data = json:decode(web.readAll())

  for k, v in pairs(data) do
	if v["type"] == "dir" then
	  table.insert(dirs, v["path"])
	  print("Found Dir: " .. v["path"]) --TESTING
	  getDir("/" .. v["path"])
	elseif v["type"] == "file" then

	  table.insert(files, {v["name"], v["download_url"], "Downloads/" .. repoName .. "/" .. v["path"]})
	  print("Found File: " .. v["path"]) --TESTING
	end
  end
end

getDir("")

for k, v in pairs(dirs) do
  fs.makeDir(v)
end

os.loadAPI("CraftedCart/api/downloader.lua")
downloader = _G["downloader.lua"]
downloader.getAndInstall(files, 4)
Edited on 06 October 2015 - 06:36 PM
Bomb Bloke #2
Posted 04 October 2015 - 01:00 AM
That repo's a lot bigger than ComputerCraft's default drive limit (which is just under a megabyte!), and because the mod insists on using text-mode for all online read requests, some of the files can't be downloaded correctly anyway (not that ComputerCraft would be able to do much with them even if they were!). I feel your rate issues may be more to do with you targeting an inappropriate repo, rather than anything else.

That said, although I'm no expert on GitHub's API, these threads may or may not be useful to you:

http://www.computercraft.info/forums2/index.php?/topic/24269-github-gist-api/

http://www.computercraft.info/forums2/index.php?/topic/10966-github/

http://www.computercraft.info/forums2/index.php?/topic/22883-http-enhancements/
CraftedCart #3
Posted 04 October 2015 - 09:03 AM
That repo's a lot bigger than ComputerCraft's default drive limit (which is just under a megabyte!), and because the mod insists on using text-mode for all online read requests, some of the files can't be downloaded correctly anyway (not that ComputerCraft would be able to do much with them even if they were!). I feel your rate issues may be more to do with you targeting an inappropriate repo, rather than anything else.

That said, although I'm no expert on GitHub's API, these threads may or may not be useful to you:

http://www.computerc...ithub-gist-api/

http://www.computerc...c/10966-github/

http://www.computerc...p-enhancements/

Ah, maybe I shouldn't be targeting Twbs/Bootstrap…
I tried downloading some CC repos. Here are the results:
LuaIDE takes up 6 tokens, which is acceptable
FireWolf uses 15, which is OK.

Seeing as this is only designed to run on CC computers, I guess that would be alright, but authentication methods would still be nice
(I never checked how big bootstrap was before testing… :/)
SquidDev #4
Posted 04 October 2015 - 09:09 AM
Instead of using https://api.github.com/repos/{user}/{repo}/contents/{file}, use https://raw.githubusercontent.com/{user}/{repo}/master/{file} to download the file and https://api.github.com/repos/{user}/{repo}/git/trees/master?recursive=1 to get the tree, then you won't hit the rate limit.

You can see a pretty standard GitHub downloader here.
CraftedCart #5
Posted 04 October 2015 - 09:14 AM
Instead of using https://api.github.c...contents/{file}, use https://raw.githubus...}/master/{file} to download the file and https://api.github.c...ter?recursive=1 to get the tree, then you won't hit the rate limit.

You can see a pretty standard GitHub downloader here.

I did not know that I could get the whole tree, thanks :)/>

EDIT: Got my test download script implemented, now to implement it into my actual project
EDIT2: Done!
Edited on 04 October 2015 - 08:45 AM
minizbot2012 #6
Posted 04 October 2015 - 02:28 PM
Okay as seeing I've implemented it before I might be of some help. (the Gist API)

Basic authorization is basically taking a username and password, adding a colon (format "user:pass") between them and then using base64, then you add the string "Basic " before the encoded string, and yes that space is important. Github's API takes that string generated by the header named "Authorization". Be warned that when doing this security is important for the generated strings, as there is nothing stopping you from decoding the user+password from the string token. I did it as a mere experiment.

The reference I used in creating the gist API:

https://en.wikipedia..._authentication

Also note that to use Github's API is must use https otherwise it won't work.

A base 64 implementation (encoding only):


local function enc(data)
	return ((data:gsub('.', function(x)
		local r,b='',x:byte()
		for i=8,1,-1 do r=r..(b%2^i-b%2^(i-1)>0 and '1' or '0') end
		return r;
	end)..'0000'):gsub('%d%d%d?%d?%d?%d?', function(x)
		if (#x < 6) then return '' end
		local c=0
		for i=1,6 do c=c+(x:sub(i,i)=='1' and 2^(6-i) or 0) end
		return b:sub(c+1,c+1)
	end)..({ '', '==', '=' })[#data%3+1])
end

generating a token:


local function genToken(user, pass)
	local userp = user..":"..pass
	return "Basic "..enc(userp)
end
Edited on 04 October 2015 - 12:42 PM
CraftedCart #7
Posted 04 October 2015 - 03:01 PM
Spoiler
Okay as seeing I've implemented it before I might be of some help. (the Gist API)

Basic authorization is basically taking a username and password, adding a colon (format "user:pass") between them and then using base64, then you add the string "Basic " before the encoded string, and yes that space is important. Github's API takes that string generated by the header named "Authorization". Be warned that when doing this security is important for the generated strings, as there is nothing stopping you from decoding the user+password from the string token. I did it as a mere experiment.

The reference I used in creating the gist API:

https://en.wikipedia..._authentication

Also note that to use Github's API is must use https otherwise it won't work.

A base 64 implementation (encoding only):


local function enc(data)
	return ((data:gsub('.', function(x)
		local r,b='',x:byte()
		for i=8,1,-1 do r=r..(b%2^i-b%2^(i-1)>0 and '1' or '0') end
		return r;
	end)..'0000'):gsub('%d%d%d?%d?%d?%d?', function(x)
		if (#x < 6) then return '' end
		local c=0
		for i=1,6 do c=c+(x:sub(i,i)=='1' and 2^(6-i) or 0) end
		return b:sub(c+1,c+1)
	end)..({ '', '==', '=' })[#data%3+1])
end

generating a token:


local function genToken(user, pass)
	local userp = user..":"..pass
	return "Basic "..enc(userp)
end

Thanks but unfortunately that doesn't work
The enc function errors on line 14 (attempt to index ? (a nil value))
Line 14 was
return b:sub(c+1,c+1)
I tried using someone else's Base64 API (The one here: http://www.computerc...__fromsearch__1 ) but then after going through the whole thing, Nil was returned by http.post(). I did check the other B64 api against an online B64 encoder and it matched

Here's the code:
Spoiler

os.loadAPI("b64")

args = {...}

local function genToken(user, pass)
	local userp = user..":"..pass
	return "Basic "..b64.encode(userp) --Modified to use someone else's base64
end

print(args[1])
print(args[2])

token = genToken(args[1], args[2])
print(token)

web = http.post("https://api.github.com/rate_limit", nil, {["Authorization"] = token})
print(tostring(web))

EDIT: Nevermind, I'm silly.
I thought HTTP.get couldn't accept headers in CC, (so I assumed HTTP.post) It probably was an update that happened ages ago..
Edited on 04 October 2015 - 01:21 PM