This is a read-only snapshot of the ComputerCraft forums, taken in April 2020.
Wilma456's profile picture

Download Folder from GitHub

Started by Wilma456, 04 September 2017 - 11:48 AM
Wilma456 #1
Posted 04 September 2017 - 01:48 PM
What's the best way, to download a full folder from GitHub without writing all raw URLs in the code?
Edited on 04 September 2017 - 11:49 AM
SquidDev #2
Posted 04 September 2017 - 02:05 PM
You can get the full tree of a repository by using the trees API endpoint. This returns a JSON object with a property "tree", which can be iterated over. You can then fetch from https://raw.githubusercontent.com/ to download the actual contents.

Here's an example to download all files in the SquidDev-CC/Howl repository. I'm presuming you have a JSON API loaded already:

-- Fetch the file listing
local request = http.get("https://api.github.com/repos/SquidDev-CC/Howl/git/trees/master?recursive=1")
local contents = json.parse(request.readAll())
request.close()

for _, file in ipairs(contents.tree) do
  -- Fetch the actual contents. You can save these files somewhere, put them in an array, etc...
  local request = http.get("https://raw.githubusercontent.com/SquidDev-CC/Howl/master/" .. file.path)
  local handle = fs.open(file.path, "w")
  handle.write(request.readAll()
  handle.close()
  request.close()
end
Wilma456 #3
Posted 04 September 2017 - 02:19 PM
What json API do you use in the example?
SquidDev #4
Posted 04 September 2017 - 02:29 PM
What json API do you use in the example?
In that case, a purely hypothetical one. You'll probably want ElvishJerricco's API, which uses json.decode instead.
Wilma456 #5
Posted 04 September 2017 - 04:40 PM
This API has not the function "parse"
SquidDev #6
Posted 04 September 2017 - 04:56 PM
which uses json.decode instead.
Wilma456 #7
Posted 05 September 2017 - 06:23 PM
I had tried this code with the API that you linked:

os.loadAPI("/json")
local request = http.get("https://api.github.com/repos/SquidDev-CC/Howl/git/trees/master?recursive=1")
local contents = json.decode(request.readAll())
request.close()

for _, file in ipairs(contents.tree) do
  local request = http.get("https://raw.githubusercontent.com/SquidDev-CC/Howl/master/" .. file.path)
  local handle = fs.open(file.path, "w")
  handle.write(request.readAll())
  handle.close()
  request.close()
end
But I got the error 9: attempt to index ? (a nil value)
SquidDev #8
Posted 05 September 2017 - 06:33 PM
But I got the error 9: attempt to index ? (a nil value)
So the recursive file listing also includes directories, so you'll need to check that file.type == "blob" before downloading each file.
Edited on 05 September 2017 - 04:33 PM
Wilma456 #9
Posted 05 September 2017 - 06:46 PM
Same error, now with line 10

os.loadAPI("/json")
local request = http.get("https://api.github.com/repos/SquidDev-CC/Howl/git/trees/master?recursive=1")
local contents = json.decode(request.readAll())
request.close()

for _, file in ipairs(contents.tree) do
  if file.type == "blob" then
    local request = http.get("https://raw.githubusercontent.com/SquidDev-CC/Howl/master/" .. file.path)
    local handle = fs.open(file.path, "w")
    handle.write(request.readAll())
    handle.close()
    request.close()
  end
end
SquidDev #10
Posted 05 September 2017 - 07:03 PM
Same error, now with line 10
I'd presume a HTTP request is failing. Maybe try adding some print statements to determine which file is failing, print the error message from the HTTP response, etc… C'mon, I've seen your programs - I know you're capable of basic debugging skills.
Lupus590 #11
Posted 05 September 2017 - 10:38 PM
I know you're capable of basic debugging skills.

[OffTopic] Do we have a tutorial on debugging on the forums yet?
apemanzilla #12
Posted 05 September 2017 - 10:40 PM
You can use gitget. /shameless plug
Wilma456 #13
Posted 06 September 2017 - 05:10 PM
You can use gitget. /shameless plug
It looks like that gitget can only download a full repository and not only a folder. In my case I want to download to rom folder of CC from here or older Versions from here (different Tags on GitHub). Only the rom folder. Nothing else (e.g. bios).
Lupus590 #14
Posted 06 September 2017 - 09:13 PM
You could modify gitget.
apemanzilla #15
Posted 06 September 2017 - 10:22 PM
You can use gitget. /shameless plug
It looks like that gitget can only download a full repository and not only a folder. In my case I want to download to rom folder of CC from here or older Versions from here (different Tags on GitHub). Only the rom folder. Nothing else (e.g. bios).

Modify line 108 to add an additional check whether the file starts with a given path.
Wilma456 #16
Posted 07 September 2017 - 02:03 PM
With modifyed gitget, I would be able to download the rom folder from the offical Repo. But what's with older Versions. In the Repo of alekso56 they are sorted with Tags, but gitget only suports branches and not tags.
SquidDev #17
Posted 07 September 2017 - 02:19 PM
With modifyed gitget, I would be able to download the rom folder from the offical Repo. But what's with older Versions. In the Repo of alekso56 they are sorted with Tags, but gitget only suports branches and not tags.
Both should work fine. You can use branches, tags or specific commit hashes with git, and they'll all "just work".
Wilma456 #18
Posted 07 September 2017 - 02:42 PM
I had try "gitget alekso56 ComputercraftLua branch=1.79" but I get a "gitget:74 attempt to index ? (a nil value) "error
Bomb Bloke #19
Posted 08 September 2017 - 12:53 AM
The usage shown in the gitget script doesn't seem to match the usage shown in the gitget thread: try "1.79" instead of "branch=1.79". Failing that, rig the code to print the URL it's attempting to connect to so you can report it back here.
Wilma456 #20
Posted 11 September 2017 - 05:25 PM
I had now made a file with all files in /rom to solve the Problem for me. And with the list I don't have any dependencies to gitget or a json API. If you are intrested, you can find the file here