Grin - Github Release INstaller

Grin is a tool designed to make github based distribution simple. With grin, you can package a release together in a zip file and distribute that to CC users.

Github
Pastebin VuBNx3va

How it works
Unfortunately, there's a bug in LuaJ that prevents CC from getting binary data properly, so normal zips won't quite work. But we do want to use zip files! So to get around this, releases have to be zipped, then encoded with base64. On unix/linux/mac, use the base64 command line utility. On windows, use the certutil command line utility. Once you have your desired release packaged up, create a release on github, and attach the .zip.base64 file to the release.

That's it
That's all you need to do. Now any user can use grin -u YourUsername -r YourRepo dir to download, decode, and unarchive your release to the desired directory.

Tags
Grin takes an optional argument "-t tag_name" that specifies the tag of a release to download, if you don't want to download just the latest release.

Custom Installers
Of course, you can (and probably should) create your own installer that calls grin from pastebin with specified arguments. For example

shell.run("pastebin", "run", "VuBNx3va", "-u", "YourUsername", "-r", "YourRepo", "desiredDir")
print("Installed")

That's all it takes. Put your equivalent of that code on a pastebin and all your users have to do to install your project is pastebin run ID.

But there's now a new flag that lets your custom installers look how you like. The -emit-events (or -e) flag can be used to tell grin to emit events with progress messages instead of printing them. This allows you to do a custom view over Grin.

local ok, err
parallel.waitForAny(function()
	ok, err = pcall(shell.run, "grin", "-e", "-u", user, "-r", repo, dir)
end, function()
	while true do
		local e, s = os.pullEvent("grin_install_status")
		local x, y = term.getCursorPos()
		term.setCursorPos(1, y)
		term.clearLine()
		term.write(s)
	end
end)

assert(ok, err)
local x, y = term.getCursorPos()
term.setCursorPos(1, y)
term.clearLine()
print(user, "/", repo, " successfully downloaded")

A Note About Transparency

Grin's main feature is transparency. It's easy to install Grin packages, and users don't even have to know they're using Grin. It's really easy to make an installer that invokes Grin to install your package, and that installer can just be put on pastebin. Letting users do pastebin run CODE to get your program installed is a big plus. And when you have a big project, that's not always trivial to do. And all this comes with the added benefits of handling versioning seamlessly, and enabling binary downloads. I hope Grin will help us solve installations problem.

Is there interest in this growing into more of a package management tool? Installs could include a .grin directory with files that represent dependancies to download and install-time configurations to be made.

ElvishJerricco, on 01 October 2014 - 08:33 AM said:

Is there interest in this growing into more of a package management tool? Installs could include a .grin directory with files that represent dependancies to download and install-time configurations to be made.

You would create a DLL Hell in ComputerCraft by doing that but I think it's for the better. So go ahead! I'd love to see that feature.

Lignum, on 03 October 2014 - 03:30 PM said:

You would create a DLL Hell in ComputerCraft by doing that but I think it's for the better. So go ahead! I'd love to see that feature.

Well I ended up doing it (a bit differently) in a separate program based on this, Grin-Get. Uses Grin to perform the downloads while Grin-Get manages the packages and dependencies and whatnot. So Grin packages can be installed with Grin or Grin-Get, and packages can include Grin-Get specific features such as dependencies or use of the grin API.

ElvishJerricco, on 03 October 2014 - 05:04 PM said:

Well I ended up doing it (a bit differently) in a separate program based on this, Grin-Get. Uses Grin to perform the downloads while Grin-Get manages the packages and dependencies and whatnot. So Grin packages can be installed with Grin or Grin-Get, and packages can include Grin-Get specific features such as dependencies or use of the grin API.

Ah, yes, I've seen grin-get already earlier today. However I didn't notice that you had already implemented it. I'll be sure to try it out once I need it.

Just added a new feature. Installers can now create custom displays.

Update. Changes:

• Calls error() instead of just using printError and return.

Updated: Transferred ownership to the new team Yevano and I have started.

I have noticed that you are using the releases to download repositories. I would suggest being able to use the Github API to download repositories instead of limiting them to releases.
The Github API provides JSON-based responses to http GET queries. Say, to get the latest commit of repository ccLinux/kernel (click click) you'd GET https://api.github.com/repos/ccLinux/kernel/git/refs/heads/master and then get the value at response['object']['sha']. To get all files in that same repository in that commit, GET https://api.github.com/repos/ccLinux/kernel/git/trees/ee86ca54c547430915c93f6853b7b296432fc78b?recursive=1. You can test these in your browser, too.


TL;DR: Some people don't like releasing base64 encoded zipfiles, so, including a Github API-based option to download the whole repository in the latest commit would be nice. kthxbye.

Xepoof, on 03 December 2014 - 11:00 AM said:

I have noticed that you are using the releases to download repositories. I would suggest being able to use the Github API to download repositories instead of limiting them to releases.
The Github API provides JSON-based responses to http GET queries. Say, to get the latest commit of repository ccLinux/kernel (click click) you'd GET https://api.github.com/repos/ccLinux/kernel/git/refs/heads/master and then get the value at response['object']['sha']. To get all files in that same repository in that commit, GET https://api.github.com/repos/ccLinux/kernel/git/trees/ee86ca54c547430915c93f6853b7b296432fc78b?recursive=1. You can test these in your browser, too.


TL;DR: Some people don't like releasing base64 encoded zipfiles, so, including a Github API-based option to download the whole repository in the latest commit would be nice. kthxbye.

I know that you can download repos. That completely defeats the purpose of Grin. Grin has two main advantages. One is that the download system is efficient, and the other is that it differentiates the dev and user environments.

JVML for example, has to distribute a jar file. CC has a bug from LuaJ that makes downloading binaries impossible. So encoding to base64 is the only way to fix that. While we're at, we might as well support packaging the release nicely. That's why Grin uses .zip.base64. This way you can archive exactly what you need into the release and it works with everything.

As for how it differentiates the dev and user environments, if you have a large project, then there's a lot of files that you need in your source repo that you don't want users to have to have on their system. For example, if you need to do any kind of compiling, you don't want to be releasing the source and telling the user they have to do the compiling. You just want to release the compiled thing and only the compiled thing. You also likely don't want your test scripts to be in your release. So basically, a developer wants the ability to dictate exactly what makes it to an end users system, and archiving releases like this is the way to do that. Just as an example, JVML's released version is ⅓ the size of its source code version. That's huge.

Finally, I don't know about anyone else, but I don't like releasing unstable software to users. If they're pulling directly from the source code, they can catch a commit mid-development. That's not good. It's important to me that I can dictate exactly when and how a release is.

All the above things are the problems are the things Grin is here to solve. If you want to download a github repo, use the clone button. That's not a proper way to do releases.

Hello, I'm rather new to GitHub (in fact, I've started to use it due to this program). It would help me a lot if you could patiently explain to me how do I have to set the thing up in GitHub so the files I have in my computer can be translated to another computer without any kind of change. Up to this moment, all I have managed is a string:1626:malformed response error. Thanks a lot.
If it helps, the repository is at https://github.com/MarioBG/Olive_OS. Thanks again, and I hope we can fix this.

EDIT: I made some progress, and now, if I try with another repository I made (https://github.com/M...G/Mar-Mar-Olive), it shows this . Please help. Also, if you have a peek at the actual OS, I want to say this is NOT the final 1.2 version, just some kind of prerelease, but i didn't want to label it as such in fear it would not work.

Something is wrong with the way you're encoding the base64 file or the zip file. How exactly are you doing it?

ElvishJerricco, on 15 December 2014 - 08:27 PM said:

Something is wrong with the way you're encoding the base64 file or the zip file. How exactly are you doing it?

In the Olive zip, I had previously stored the content of my root folder from the computer in which I store the OS.

Oh wow. I'd never used certutil before. I'm a UNIX guy, and haven't even tested certutil. I opened your base64 file in a text editor and it has this header and footer and is separated into lines. That's not correct. It should just be a string of ascii characters, with no new lines and no header or footer. It would appear certutil is not the right way to encode base64 on windows.

Bits of me thing there should be a pure Lua way of zipping the file. I know the deflate library doesn't support it and I'm sure it would be very hard to write your own. I might do it if I have the time.

Alright, alright, we've made some progress. It now gets to download the file and recognise its content. It scans the zip and starts unarchiving, but at the very first file, each time I try… (note: I'm working on this repository)
In the middle one I tried again with certutil, but besides generating those certificate thingies, it also splits the file in lines, which makes it kind of impossible to understand for Grin, I suppose.

EDIT 2: Alright, THAT particular problem was because the computer had been set to have 0B HDD XD. Now I've somewhat progressed, and it does unpack some items, but when it gets to the .lua file extension data, it gives this error.

The contents of the file are as follow:

action_check
2048
L

I honestly don't get why that would pose a problem, but apparently it does. Thanks for your patience. Also, would there be any chance, slight as it may be, to rework one or two bits of this to make it compatible with CC 1.48? It would make a great difference to me, since the main feature I made this OS for was to have something for my 1.4.7 server (and then it got a bit out of control, I guess). Thanks a huge lot!

MarioBG, on 16 December 2014 - 06:20 PM said:

-snip-

Ok I've found the problem. I feel silly now =P I wrote the wrong variable name in the code that handles files that have no compression, and apparently your .lua had no compression. Fixed

EDIT: Also, what's not working in 1.48?

EDIT 2: Also I would like to know how you're encoding base64 now. I should updated my scripts to not use certutil.

The DLL would be very slow.

ElvishJerricco, on 17 December 2014 - 06:49 AM said:

MarioBG, on 16 December 2014 - 06:20 PM said:

-snip-

Ok I've found the problem. I feel silly now =P I wrote the wrong variable name in the code that handles files that have no compression, and apparently your .lua had no compression. Fixed

EDIT: Also, what's not working in 1.48?

EDIT 2: Also I would like to know how you're encoding base64 now. I should updated my scripts to not use certutil.

After the "Decoding Base64" message, it outputs grin:1376: too few arguments. I've had my issues of the sort in the OS itself. If it could be fixed though, it would be awesome :)/>.

I use an online tool now and paste the text onto the file. Not perfect, but it works. The tool is this one.

MarioBG, on 17 December 2014 - 03:01 PM said:

-snip-

That tool you're using prefixes the output with

data:;base64,

Make sure you're only copying what comes after that comma. I believe that is what's causing your error.

ByteMe, on 17 December 2014 - 08:02 AM said:

The DLL would be very slow.

And what are you talking about?

ElvishJerricco, on 17 December 2014 - 04:17 PM said:

MarioBG, on 17 December 2014 - 03:01 PM said:

-snip-

That tool you're using prefixes the output with

data:;base64,

Make sure you're only copying what comes after that comma. I believe that is what's causing your error.

I know that, and have only copied the string of chars from the comma onwards. I have successfully downloaded the thing in ccemuredux, which uses some 1.6 ComputerCraft, but not in the old CC emulator, which uses 1.45, so there must be some kind of issue there. Again, thanks for your patience :D/>

This program makes me grin

Geforce Fan, on 16 February 2015 - 12:03 AM said:

This program makes me grin

This post made me cringe.

Dr. Poof, on 16 February 2015 - 10:56 AM said:

Geforce Fan, on 16 February 2015 - 12:03 AM said:

This program makes me grin

This post made me cringe.

Haters gonna hate.

You could try to open files in luaJ in a binary format, by opening the files as txt files then convert every character to an integer value using string.byte(char), if this was the problem and there isn't a problem sending the binary files trough http.

Jarle212, on 19 February 2015 - 04:11 PM said:

You could try to open files in luaJ in a binary format, by opening the files as txt files then convert every character to an integer value using string.byte(char), if this was the problem and there isn't a problem sending the binary files trough http.

Why not just open the file in rb (read binary) mode?

KingofGamesYami, on 19 February 2015 - 05:16 PM said:

Jarle212, on 19 February 2015 - 04:11 PM said:

You could try to open files in luaJ in a binary format, by opening the files as txt files then convert every character to an integer value using string.byte(char), if this was the problem and there isn't a problem sending the binary files trough http.

Why not just open the file in rb (read binary) mode?

Because with binary mode you can't use readAll, but should still work. Not certain what hindrance the luaJ bug poses tho.

Jarle212, on 19 February 2015 - 04:11 PM said:

You could try to open files in luaJ in a binary format, by opening the files as txt files then convert every character to an integer value using string.byte(char), if this was the problem and there isn't a problem sending the binary files trough http.

Binary files ought to be dealt with with "rb" mode. That said, there is a bug with LuaJ that breaks any binary data being converted from java Strings to Lua strings. So an HTTP request has to go through that conversion, breaking any binary downloads.

ElvishJerricco, on 19 February 2015 - 07:23 PM said:

Jarle212, on 19 February 2015 - 04:11 PM said:

You could try to open files in luaJ in a binary format, by opening the files as txt files then convert every character to an integer value using string.byte(char), if this was the problem and there isn't a problem sending the binary files trough http.

Binary files ought to be dealt with with "rb" mode. That said, there is a bug with LuaJ that breaks any binary data being converted from java Strings to Lua strings. So an HTTP request has to go through that conversion, breaking any binary downloads.

Only breaks binary data where the character value is greater than 127(0x7F), at the place where the ascii table ends.
Edit: There seems to be a pattern, so it should be possible to fix the broken string.

Jarle212, on 19 February 2015 - 08:50 PM said:

ElvishJerricco, on 19 February 2015 - 07:23 PM said:

Jarle212, on 19 February 2015 - 04:11 PM said:

You could try to open files in luaJ in a binary format, by opening the files as txt files then convert every character to an integer value using string.byte(char), if this was the problem and there isn't a problem sending the binary files trough http.

Binary files ought to be dealt with with "rb" mode. That said, there is a bug with LuaJ that breaks any binary data being converted from java Strings to Lua strings. So an HTTP request has to go through that conversion, breaking any binary downloads.

Only breaks binary data where the character value is greater than 127(0x7F), at the place where the ascii table ends.
Edit: There seems to be a pattern, so it should be possible to fix the broken string.

Right and binary data quite often uses bytes over 127. And whatever pattern there may be can't be fixed because any byte that might be corrupted, also might not. There isn't any way to tell.

ElvishJerricco, on 19 February 2015 - 10:33 PM said:

Jarle212, on 19 February 2015 - 08:50 PM said:

ElvishJerricco, on 19 February 2015 - 07:23 PM said:

Jarle212, on 19 February 2015 - 04:11 PM said:

You could try to open files in luaJ in a binary format, by opening the files as txt files then convert every character to an integer value using string.byte(char), if this was the problem and there isn't a problem sending the binary files trough http.

Binary files ought to be dealt with with "rb" mode. That said, there is a bug with LuaJ that breaks any binary data being converted from java Strings to Lua strings. So an HTTP request has to go through that conversion, breaking any binary downloads.

Only breaks binary data where the character value is greater than 127(0x7F), at the place where the ascii table ends.
Edit: There seems to be a pattern, so it should be possible to fix the broken string.

Right and binary data quite often uses bytes over 127. And whatever pattern there may be can't be fixed because any byte that might be corrupted, also might not. There isn't any way to tell.

There is :P/>
if the char has a value of 194 or 195 the correct byte value will be between 0xA0 and 0xFF, by looking at the next character after that you can figure out what the original value was.
So let take: 0xF2 witch produces the two values 195 and 178
195 in binary: 1100 0011
178 in binary: 1011 [0010] <– this will be the lower 4 bits of the original so we got xxxx 0010
to get the 4 higher bits of the original byte we take the lower two bits of 195 like this -> 1100 00[11] put them in the 5th and 6th position so we got xx11 0010
to get the last two bits take the 5th and 6th bits of 178 like this -> 10[11] 0010 and put them in the 7th and 8th position and then we get 1111 0010 witch is the value of 0xF2.
This method works for bytes where the value is between 0xA0 and 0xFF. I haven't figured out the pattern for values between 0x80 and 0x9F yet.

Here is the code I got so far:

function fixString(brokenString)
local fixedString = "";
local bVal;
for i=1, brokenString:len() do
  bVal = string.byte(brokenString:sub(i,i))
  if bit.band(0x80,bVal) == 0 then
   --Nothing to fix
   fixedString = fixedString .. brokenString:sub(i,i);
  else
   --Fix broken string
   if (bVal == 195) or (bVal == 194) then
	--Value between 0xA0 and 0xFF
	i = i + 1;
	local bVal2 = string.byte(brokenString:sub(i,i));
	local hi = bit.bor(bit.blshift(bit.band(bVal,3),6),bit.band(bVal2,48));
	local lo = bit.band(bVal2,15);
	local fixed = bit.bor(hi,lo);
	fixedString = fixedString .. string.char(fixed);
   else
	--Value is 0x8X or 0x9X
   end
  end
end
return fixedString;
end

Jarle212, on 19 February 2015 - 10:52 PM said:

- snip -

Oh wow I never really looked all that deeply into this. It would appear that this is a Unicode issue. I don't know much about Unicode, but it seems that when making a String from a byte[] in Java, it tries to see the values >127 as unicode values, and messes with the string entirely, leaving behind a trailing byte or two from which you can extrapolate the original data.

So while now it appears we have a way to fix this on Lua side, I would guess that HTTP requests in CC itself can be changed from reading the response as a String, to reading the response as a byte[]. Then it can construct its own char[], one char for each byte, and produce a correct String from that char[].

Of course without CC being open source, we can't know if CC already does this or not, and we can't submit a pull request fixing the bug…

EDIT: Then again, the issue exists when calling Java code from Lua, passing a string. I would have to look into this, but I'd guess a Lua string is stored in a byte[], and LuaJ is using the direct byte[] → String method of converting a Lua string to a Java String. Of course this is basically speculation, but maybe I'll look into this…

ElvishJerricco, on 20 February 2015 - 01:24 AM said:

Jarle212, on 19 February 2015 - 10:52 PM said:

- snip -

Oh wow I never really looked all that deeply into this. It would appear that this is a Unicode issue. I don't know much about Unicode, but it seems that when making a String from a byte[] in Java, it tries to see the values >127 as unicode values, and messes with the string entirely, leaving behind a trailing byte or two from which you can extrapolate the original data.

So while now it appears we have a way to fix this on Lua side, I would guess that HTTP requests in CC itself can be changed from reading the response as a String, to reading the response as a byte[]. Then it can construct its own char[], one char for each byte, and produce a correct String from that char[].

Of course without CC being open source, we can't know if CC already does this or not, and we can't submit a pull request fixing the bug…

EDIT: Then again, the issue exists when calling Java code from Lua, passing a string. I would have to look into this, but I'd guess a Lua string is stored in a byte[], and LuaJ is using the direct byte[] → String method of converting a Lua string to a Java String. Of course this is basically speculation, but maybe I'll look into this…

Cool :>

Ok so looking at LuaString.java, we find these methods.

	public static LuaString valueOf(String string) {
		LuaString s = index_get( index_java, string );
		if ( s != null ) return s;
		char[] c = string.toCharArray();
		byte[] b = new byte[lengthAsUtf8(c)];
		encodeToUtf8(c, b, 0);
		s = valueOf(b, 0, b.length);
		index_set( index_java, string, s );
		return s;
	}

	public String tojstring() {
		return decodeAsUtf8(m_bytes, m_offset, m_length);
	}

	public static String decodeAsUtf8(byte[] bytes, int offset, int length) {
		int i,j,n,b;
		for ( i=offset,j=offset+length,n=0; i<j; ++n ) {
			switch ( 0xE0 &amp; bytes[i++] ) {
			case 0xE0: ++i;
			case 0xC0: ++i;
			}
		}
		char[] chars=new char[n];
		for ( i=offset,j=offset+length,n=0; i< j; ) {
			chars[n++] = (char) (
				((b=bytes[i++])>=0||i>=j)? b:
				(b<-32||i+1>=j)? (((b&amp;0x3f) << 6) | (bytes[i++]&amp;0x3f)):
					(((b&amp;0xf) << 12) | ((bytes[i++]&amp;0x3f)<<6) | (bytes[i++]&amp;0x3f)));
		}
		return new String(chars);
	}
	public static void encodeToUtf8(char[] chars, byte[] bytes, int off) {
		final int n = chars.length;
		char c;
		for ( int i=0, j=off; i< n; i++ ) {
			if ( (c = chars[i]) < 0x80 ) {
				bytes[j++] = (byte) c;
			} else if ( c < 0x800 ) {
				bytes[j++] = (byte) (0xC0 | ((c>>6)  &amp; 0x1f));
				bytes[j++] = (byte) (0x80 | ( c      &amp; 0x3f));				
			} else {
				bytes[j++] = (byte) (0xE0 | ((c>>12) &amp; 0x0f));
				bytes[j++] = (byte) (0x80 | ((c>>6)  &amp; 0x3f));
				bytes[j++] = (byte) (0x80 | ( c      &amp; 0x3f));				
			}
		}
	}

It would appear I'm correct. But there's also this method

	public static LuaString valueOf(byte[] bytes) {
		return valueOf(bytes, 0, bytes.length);
	}

So CC can encode and decode to and from byte arrays just fine. Might make for some tedious code but it'd fix the bug.

EDIT: Also, Jarle212 you can probably use the knowledge of how these methods work to your advantage in figure out how to fix it Lua side.

ElvishJerricco, on 20 February 2015 - 01:36 AM said:

-snip-

Yeah, I'm looking into it. Might post this as an "api" too, because it is the same problem when you are using rednet :)/>
Edit: need some sleep first tho.
Edit2: Lookin at the decodeAsUtf8 method you could probably just convert that to lua script and it should work. And the offset part seems to be correct too. Characters between 0xA0 and 0xFF starts with 195 or 194 and both of them can be written as: 0xCX while most of the characters between 0x80 and 0x9F got 0xEX at the beginning of the first character.
Edit3: Nvm, ^ didn't work like that :/

Edit4: Hit a problem, 0x90 and 0x81 is conflicting, so think this has to be solved in Java

Hi,

when I type grin-get install myusername/repo, it returns grin-install.lua: 30: no releases found.

Can you help me out because I don't understand how it works.

~Creator

Creator, on 27 February 2015 - 07:14 PM said:

Hi,

when I type grin-get install myusername/repo, it returns grin-install.lua: 30: no releases found.

Can you help me out because I don't understand how it works.

~Creator

Files need to be provided at GitHub releases. So If you look at ElvishJerricco/ClamShell you can see there is a clamshell.zip.base64 release. This is a zip of all the files base 64 encoded. You can automate this with a programming language though in the case of ClamShell it uses an Ant buildscript.

1. Zip your code
2. Base 64 encode it. You can run base64 –wrap=0 build.zip > build.zip.base64 on Linux.
3. On GitHub, go to releases, add a new one and add your build.zip.base64 file.

SquidDev, on 28 February 2015 - 09:24 AM said:

- snip -

Thanks for explaining this for everyone =) It should be noted that the if_windows task in that ant script doesn't work as I expected it to, and should probably just be removed.