Compile And Run Lua - ComputerCraft Forums (archive)

Pharap #1

Pharap's profile picture

839 posts

Location England

Posted 19 May 2013 - 04:28 AM

Intro:
Many people do not realise Lua is actually a compiled language because of ComputerCraft's open source ethic.
As I am big supporter of the ability to have closed source and compile programs, have spent the last 30 minutes writing a very basic compiler for lua code. As the default shell cannot outright run compiled lua because of the way it's set up, I also created a basic way to run the compiled code.
Really they are both more like wrappers for existing functions, but nonetheless these are the first compile-run programs for CC that I am aware of. Anyway, for those interested, I am submitting these programs (and their included functions) for public use.

Limits:

You may use the code provided as is or you can extract the individual 'compile' and 'run' functions for use, or even rewrite them to your own suiting, but you should give credit by providing the header provided in both files when you use the code in other files or projects.
It would also be appreciated if any inspired works mention this post or link to it or the files.

The Files:
Compile.lua: http://pastebin.com/S520NYMV
Run.lua: http://pastebin.com/xjBgXmfy

Usage:
Compile.lua inFile outFIle
Run outFile

Afternote:
Feel free too integrate the system into the shell, providing you leave credit..
I recommend making the shell test a file for the 'LuaQ' header and making it choose which loadfile function to use based on that.

Versions:
Version 1.0.1 - First working versions
Version 1.0.2 - Bug Fix.Added sleep to fix yielding. Credits to pingoleon60.
Version 1.0.3 - Speed Enhancement. Replaced sleep with queue event. Credits to theoriginalbit
Version 1.0.4 - Speed Enhancement. Added if statement to only yield when required.

Spoiler

PS-Considering what I'm normally like, I'm being very generous with this license. Seriously - I was going to provide a whole 12 paragraph EULA but instead decided to put faith in the programmers here and just leave a general 'give credit' clause.

Spoiler

PPS-"Under the circumstances I've been shockingly Nice~"

Edited on 21 May 2013 - 08:22 PM

superaxander #2

620 posts

Location Holland

Posted 19 May 2013 - 04:33 AM

I was planning on something like this but didn't know how I would do it but you did it so I don't need to anymore! Now to make it run without a run program…

M4sh3dP0t4t03 #3

570 posts

Location Germany

Posted 19 May 2013 - 04:38 AM

Well, closed source just doesn't fit into computercraft. There are two advantages of closed source: the first is that people can't steal your work and the second is that you can't sell open-source programs. To the first: in computercraft every program is in this forum, which is good moderated, so clones of programs get deleted very fast. And about the second one no one would pay for a simple cc program
Edit: and of course it's possible to decompile compiled programs

JustPingo #4

116 posts

Location France

Posted 19 May 2013 - 06:57 AM

Great work man ! (Under the circumstances you've been shockingly nice, but we didn't want our freedom ^^)

KingOfNoobs, on 19 May 2013 - 04:38 AM said:
And about the second one no one would pay for a simple cc program

And if you want to sell your programs ingame, versus diamonds or something else ?

M4sh3dP0t4t03 #5

570 posts

Location Germany

Posted 19 May 2013 - 06:59 AM

pingoleon60, on 19 May 2013 - 06:57 AM said:
Great work man !

KingOfNoobs, on 19 May 2013 - 04:38 AM said:
And about the second one no one would pay for a simple cc program

And if you want to sell your programs ingame, versus diamonds or something else ?

But that would mean that you're only publishing your program on one single server

JustPingo #6

116 posts

Location France

Posted 19 May 2013 - 07:00 AM

Why ? Oh I see.
You're right, but it's not very serious. If you want to publish your programs only on one server, you can.
And if you play on more than one server…

Oh, and I've an error during compile. Add a sleep(0) in the for loop, it's slower but it doesn't crash, I think.

Pharap #7

Pharap's profile picture

839 posts

Location England

Posted 19 May 2013 - 08:46 AM

KingOfNoobs, on 19 May 2013 - 04:38 AM said:
Well, closed source just doesn't fit into computercraft. There are two advantages of closed source: the first is that people can't steal your work and the second is that you can't sell open-source programs. To the first: in computercraft every program is in this forum, which is good moderated, so clones of programs get deleted very fast. And about the second one no one would pay for a simple cc program
Edit: and of course it's possible to decompile compiled programs

This is an opinion, not a fact: "closed source just doesn't fit into computercraft"
You forgot to mention compiling speeds up loading time because it means the Lua VM doesn't have to compile the code itself because it's loading it pre-compiled.
It also stops people pointing out annoying flaws in your code or panicking because your code looks horrible.

superaxander, on 19 May 2013 - 04:33 AM said:
I was planning on something like this but didn't know how I would do it but you did it so I don't need to anymore! Now to make it run without a run program…

To make your job easier: Run.lua will run non-compiled programs. Now all you have to do is dive into the shell and swap out/rework a couple functions.
Glad my 'compiler' is appreciated.

pingoleon60, on 19 May 2013 - 06:57 AM said:
Great work man ! (Under the circumstances you've been shockingly nice, but we didn't want our freedom ^^)

And if you want to sell your programs ingame, versus diamonds or something else ?

yay, people read random postscript spoilers.
Also, that second point is completely valid, I approve of that idea :D/>

pingoleon60, on 19 May 2013 - 07:00 AM said:
Why ? Oh I see.
You're right, but it's not very serious. If you want to publish your programs only on one server, you can.
And if you play on more than one server…

Oh, and I've an error during compile. Add a sleep(0) in the for loop, it's slower but it doesn't crash, I think.

Oversight on my part. I didn't think anyone would compile anything large enough to take 5 seconds lol
I'll add that in with a comment and credit :D/>

theoriginalbit #8

7508 posts

Location Australia

Posted 19 May 2013 - 08:50 AM

Firstly nice script :)/> its simple, yet elegant ;)/>

Pharap, on 19 May 2013 - 08:46 AM said:
pingoleon60, on 19 May 2013 - 07:00 AM said:
Why ? Oh I see.
You're right, but it's not very serious. If you want to publish your programs only on one server, you can.
And if you play on more than one server…

Oh, and I've an error during compile. Add a sleep(0) in the for loop, it's slower but it doesn't crash, I think.

Oversight on my part. I didn't think anyone would compile anything large enough to take 5 seconds lol
I'll add that in with a comment and credit :D/>

Secondly if speed is what you're after, and you don't mind about the program being un-terminatable while its running, I would suggest


os.queueEvent('bogus')
coroutine.yield()

its much faster than sleep(0)

JustPingo #9

116 posts

Location France

Posted 19 May 2013 - 08:51 AM

For the crash without sleep(0), my program was over 600 lines, so it's maybe for that.
But, if you have to compile a program, it's often a big program.

EDIT : TOB you're a ninja xD
Yeah, it's better than sleep(0), I forgot this.

And I think it's not useful with the runner, because I change it with the compiler, but I hadn't to do this with the runner because it work.

theoriginalbit #10

7508 posts

Location Australia

Posted 19 May 2013 - 08:59 AM

Oh, also just curious… in Compile, why do you have the level on your errors as 2? because that will throw it back to the bios. Did you mean to have 0 so that no file or line number is shown?

JustPingo #11

116 posts

Location France

Posted 19 May 2013 - 09:02 AM

Pharap, on 19 May 2013 - 08:46 AM said:
You forgot to mention compiling speeds up loading time because it means the Lua VM doesn't have to compile the code itself because it's loading it pre-compiled.

And you also forgot that a program (in this case, a program with 522 lines) who is 19 Kbytes weight become 12 Kbytes weight. Yeah, it's little, but if you use the HTTP Api to send the program, it's better for the server connexion.

EDIT : Oh, and I'd like to create a GUI for the compiler (I like create GUIs for command-types already made programs). Can I ?

Pharap #12

Pharap's profile picture

839 posts

Location England

Posted 19 May 2013 - 09:27 AM

theoriginalbit, on 19 May 2013 - 08:50 AM said:
Firstly nice script :)/> its simple, yet elegant ;)/>

Secondly if speed is what you're after, and you don't mind about the program being un-terminatable while its running, I would suggest
os.queueEvent('bogus')
coroutine.yield()
its much faster than sleep(0)

Thanks :D/>
I may consider rewriting it to do that later.
I'm not fussed about the compiler speed yet since technically you don't have to compile a program to run it, the compilation is more of a post-creation packaging step. I will get round to it at some point though.

pingoleon60, on 19 May 2013 - 08:51 AM said:
And I think it's not useful with the runner, because I change it with the compiler, but I hadn't to do this with the runner because it work.

Future Proofing~ :D/>

theoriginalbit, on 19 May 2013 - 08:59 AM said:
Oh, also just curious… in Compile, why do you have the level on your errors as 2? because that will throw it back to the bios. Did you mean to have 0 so that no file or line number is shown?

Force of habit lol

pingoleon60, on 19 May 2013 - 09:02 AM said:
And you also forgot that a program (in this case, a program with 522 lines) who is 19 Kbytes weight become 12 Kbytes weight. Yeah, it's little, but if you use the HTTP Api to send the program, it's better for the server connexion.

Haven't tried the http thing, but it should work. Something worth noting, for some reason if you try to write dumped functions as strings the string formatting messes them up and makes them not runnable. I think it's because the formatting deletes carriage returns for OS compatibility or something like that. It's worth knowing if you plan to write something that takes the sent program/function and stores it.

The size thing on the other hand is also very true, especially if the Lua VM does optimisations like Tail Calls. Not to mention complicated maths like 8 * (10 + 5), which since lua is stack based will be shrunk like mad. And all those many many lost whitespace chars and comments that become irrelevant.

I have sometimes wished Javascript and/or HTML were compiled. It would reduce site bandwidth, speed up browser loading time, speed up browser execution time and free up a load of space on thousands of servers. Web developers would complain (especially the ones I know), but I'm going to be harsh and say it will be a good change for them to experience what it's like being a traditional no-internets-attached programmer where libraries are all compiled, and you can't just rip apart programs without permission. A lot of developers obfuscate their code anyway to prevent copying (this site is definitely no exception: http://www.computerc....css?ipbv=32006) and compiled Javascript libraries would speed up library load times. My webdev friend is library obsessed, he never makes his own, always using about 10-15 libraries written by other people: bootstrap, YUI, googlemaps API, node.js, Sinatra (I know it's ruby, but I'm making a point) and countless others. Yet he has never made his own library. Think of how much faster that would all load if it was compiled.

I'll stop going on now, This is turning into a wall of text.

Sammich Lord #13

1214 posts

Location The Sammich Kingdom

Posted 19 May 2013 - 12:40 PM

This is pretty cool. I have seen some dude(Can't remember his name, it starts with a 'U' IIRC) back on CCNet create one of these. But you can just easily decompile it with a tool such as this http://sourceforge.n...rojects/unluac/

JustPingo #14

116 posts

Location France

Posted 19 May 2013 - 03:18 PM

Sammich Lord, on 19 May 2013 - 12:40 PM said:
But you can just easily decompile it with a tool such as this http://sourceforge.n...rojects/unluac/

I have tried, but I haven't succeeded yet. I saw on internet that if you put a special part of bytecode on the top of the file, it's automatically crash the decompiler. It's what happened to me. Anyone knows things about this ?

jesusthekiller #15

587 posts

Location Wrocław, Poland

Posted 19 May 2013 - 05:10 PM

"Supporter of Closed Source"? *pukes*

Nice and compact code tho…

Nina #16

Nina's profile picture

19 posts

Posted 19 May 2013 - 05:52 PM

There are quite some problems with this…

Closed-source/proprietary software (well except ComputerCraft (why is it closed-source anyway?)) really has no place here. Isn't this a community after all?
The sleep(0.1) trick is bad. Take any large program (I tried with LuaIDE) and attempt to compile and run it. Running LuaIDE like this would take 5954.9 seconds. That's around 1 hour and 40 minutes!
It is impossible to pass any arguments to programs executed with the supplied "Run" program.
Having a "Run" program in the first place is stupid – you make programs compiled with this dependent on your "Run" program, while they might as well have the code that does this by themselves.

JustPingo #17

116 posts

Location France

Posted 19 May 2013 - 05:58 PM

Nina, on 19 May 2013 - 05:52 PM said:
There are quite some problems with this…
Closed-source/proprietary software (well except ComputerCraft (why is it closed-source anyway?)) really has no place here. Isn't this a community after all?
The sleep(0.1) trick is bad. Take any large program (I tried with LuaIDE) and attempt to compile and run it. Running LuaIDE like this would take 5954.9 seconds. That's around 1 hour and 40 minutes!
It is impossible to pass any arguments to programs executed with the supplied "Run" program.
Having a "Run" program in the first place is stupid – you make programs compiled with this dependent on your "Run" program, while they might as well have the code that does this by themselves.

Nina, on 19 May 2013 - 05:52 PM said:
Closed-source/proprietary software (well except ComputerCraft (why is it closed-source anyway?)) really has no place here. Isn't this a community after all?

You haven't to use it specially for here, but on a server where you sell your programs in example.

Nina, on 19 May 2013 - 05:52 PM said:
The sleep(0.1) trick is bad. Take any large program (I tried with LuaIDE) and attempt to compile and run it. Running LuaIDE like this would take 5954.9 seconds. That's around 1 hour and 40 minutes!

Thanks to read the topic.

Nina, on 19 May 2013 - 05:52 PM said:
It is impossible to pass any arguments to programs executed with the supplied "Run" program.

True.

Nina, on 19 May 2013 - 05:52 PM said:
Having a "Run" program in the first place is stupid – you make programs compiled with this dependent on your "Run" program, while they might as well have the code that does this by themselves.

Easy to do yourself, it's just a basic program to introduce an idea. And see the main post.

jesusthekiller #18

587 posts

Location Wrocław, Poland

Posted 19 May 2013 - 06:24 PM

Instead of

sleep(0.1)

use

sleep(0.05)

or

os.queueEvent("derp")
os.pullEvent("derp")

JustPingo #19

116 posts

Location France

Posted 19 May 2013 - 08:50 PM

theoriginalbit, on 19 May 2013 - 08:50 AM said:
os.queueEvent('bogus')
coroutine.yield()
its much faster than sleep(0)

theoriginalbit #20

7508 posts

Location Australia

Posted 19 May 2013 - 09:46 PM

pingoleon60, on 19 May 2013 - 08:50 PM said:
theoriginalbit, on 19 May 2013 - 08:50 AM said:
os.queueEvent('bogus')
coroutine.yield()
its much faster than sleep(0)

It's also much faster than

jesusthekiller, on 19 May 2013 - 06:24 PM said:
or
os.queueEvent("derp")
os.pullEvent("derp")

MudkipTheEpic #21

892 posts

Location Where you'd least expect it.

Posted 19 May 2013 - 10:03 PM

theoriginalbit, on 19 May 2013 - 09:46 PM said:
pingoleon60, on 19 May 2013 - 08:50 PM said:
theoriginalbit, on 19 May 2013 - 08:50 AM said:
os.queueEvent('bogus')
coroutine.yield()
its much faster than sleep(0)
It's also much faster than
jesusthekiller, on 19 May 2013 - 06:24 PM said:
or
os.queueEvent("derp")
os.pullEvent("derp")

Wouldn't queuing an empty string, "", be faster by any terms? (Computational, maybe?) Because in Java, a string is an array of chars, which in LuaJ (i think) get reconstructed into a lua string. So would the length matter?

theoriginalbit #22

7508 posts

Location Australia

Posted 20 May 2013 - 12:07 AM

MudkipTheEpic, on 19 May 2013 - 10:03 PM said:
Wouldn't queuing an empty string, "", be faster by any terms? (Computational, maybe?) Because in Java, a string is an array of chars, which in LuaJ (i think) get reconstructed into a lua string. So would the length matter?

Its not actually the string that I'm saying is faster. Yes there may be slightly less overhead by using empty strings, but using coroutine.yield instead of sleep or os.pullEvent is where the real speed comes into effect. If you look at sleep, it calls os.pullEvent and keeps taking an event off the event queue waiting until it finds its timer. Then if you look at os.pullEvent it calls os.pullEventRaw into a table and then checks if it was a termination event and if it wasn't then returns the event table unpacked. Now if you look at os.pullEventRaw it quite simply returns the result of coroutine.yield, so in this case, why not just skip all the middle men and just call coroutine.yield, esp since we are only using it to prevent the yielding error.

Pharap #23

Pharap's profile picture

839 posts

Location England

Posted 20 May 2013 - 09:28 AM

Sammich Lord, on 19 May 2013 - 12:40 PM said:
This is pretty cool. I have seen Xuma, IIRC, back on CCNet create one of these. But you can just easily decompile it with a tool such as this http://sourceforge.n...rojects/unluac/

True, but compiling achieves more than just obfuscating the code, it makes the code faster to load and generally smaller.

jesusthekiller, on 19 May 2013 - 05:10 PM said:
"Supporter of Closed Source"? *pukes*

I'm not here to argue over open source and closed source, that's not the point of compiling.
If you don't like it, you can bog off to another thread.

Nina, on 19 May 2013 - 05:52 PM said:
There are quite some problems with this…
Closed-source/proprietary software (well except ComputerCraft (why is it closed-source anyway?)) really has no place here. Isn't this a community after all?
The sleep(0.1) trick is bad. Take any large program (I tried with LuaIDE) and attempt to compile and run it. Running LuaIDE like this would take 5954.9 seconds. That's around 1 hour and 40 minutes!
It is impossible to pass any arguments to programs executed with the supplied "Run" program.
Having a "Run" program in the first place is stupid – you make programs compiled with this dependent on your "Run" program, while they might as well have the code that does this by themselves.

Did you actually bother to read any of the posts?

Firstly, people should have the right to have their programs compiled. Aside from allowing the ability to have closed source it introduces all the other benefits that were mentioned such as reducing code size and improving program loading times.
Secondly, yes, we've covered that. I did say I only threw these together in a few minutes and I would fix all issues later. Same goes for accepting extra arguments. Both of these things you could fix yourself by the way, it would take less time than complaining about it.
Lastly, the reason it needs run is because of the way the CraftOS shell runs programs, which isn't my fault. I believe we also mentioned how it's possible to swap out some of the shell code in exchange for the loadfile function provided by run as it will run non-compiled programs as well as compiled ones.

jesusthekiller #24

587 posts

Location Wrocław, Poland

Posted 20 May 2013 - 02:14 PM

theoriginalbit, on 20 May 2013 - 12:07 AM said:
Its not actually the string that I'm saying is faster. Yes there may be slightly less overhead by using empty strings, but using coroutine.yield instead of sleep or os.pullEvent is where the real speed comes into effect. If you look at sleep, it calls os.pullEvent and keeps taking an event off the event queue waiting until it finds its timer. Then if you look at os.pullEvent it calls os.pullEventRaw into a table and then checks if it was a termination event and if it wasn't then returns the event table unpacked. Now if you look at os.pullEventRaw it quite simply returns the result of coroutine.yield, so in this case, why not just skip all the middle men and just call coroutine.yield, esp since we are only using it to prevent the yielding error.

Well, still got to learn about corountines.

Pharap, on 20 May 2013 - 09:28 AM said:
jesusthekiller, on 19 May 2013 - 05:10 PM said:
"Supporter of Closed Source"? *pukes*
I'm not here to argue over open source and closed source, that's not the point of compiling.
If you don't like it, you can bog off to another thread.

I think you missed second half of my post…

Lyqyd #25

Lyqyd's profile picture

8543 posts

Posted 20 May 2013 - 02:39 PM

This is an interesting tool, but all programs for ComputerCraft posted to the forums need to be in plain Lua due to the rules regarding malicious code.

nutcase84 #26

770 posts

Location In My Lonely Little Computer Corner

Posted 20 May 2013 - 03:32 PM

Lyqyd, on 20 May 2013 - 02:39 PM said:
This is an interesting tool, but all programs for ComputerCraft posted to the forums need to be in plain Lua due to the rules regarding malicious code.

Why not compiled?

Sammich Lord #27

1214 posts

Location The Sammich Kingdom

Posted 20 May 2013 - 03:52 PM

nutcase84, on 20 May 2013 - 03:32 PM said:
Lyqyd, on 20 May 2013 - 02:39 PM said:
This is an interesting tool, but all programs for ComputerCraft posted to the forums need to be in plain Lua due to the rules regarding malicious code.

Why not compiled?

Because you can't tell if it is malicious or not. You give me a piece of code and I can pretty much tell what it's supposed to do. If you have compiled code you have no idea what it does until you run it.

nutcase84 #28

770 posts

Location In My Lonely Little Computer Corner

Posted 20 May 2013 - 03:53 PM

Sammich Lord, on 20 May 2013 - 03:52 PM said:
nutcase84, on 20 May 2013 - 03:32 PM said:
Lyqyd, on 20 May 2013 - 02:39 PM said:
This is an interesting tool, but all programs for ComputerCraft posted to the forums need to be in plain Lua due to the rules regarding malicious code.

Why not compiled?
Because you can't tell if it is malicious or not. You give me a piece of code and I can pretty much tell what it's supposed to do. If you have compiled code you have no idea what it does until you run it.

*DERP* Lol.

M4sh3dP0t4t03 #29

570 posts

Location Germany

Posted 20 May 2013 - 04:01 PM

Sammich Lord, on 20 May 2013 - 03:52 PM said:
nutcase84, on 20 May 2013 - 03:32 PM said:
Lyqyd, on 20 May 2013 - 02:39 PM said:
This is an interesting tool, but all programs for ComputerCraft posted to the forums need to be in plain Lua due to the rules regarding malicious code.

Why not compiled?
Because you can't tell if it is malicious or not. You give me a piece of code and I can pretty much tell what it's supposed to do. If you have compiled code you have no idea what it does until you run it.

Except if you decompile it.

Sammich Lord #30

1214 posts

Location The Sammich Kingdom

Posted 20 May 2013 - 04:13 PM

KingOfNoobs, on 20 May 2013 - 04:01 PM said:
Sammich Lord, on 20 May 2013 - 03:52 PM said:
nutcase84, on 20 May 2013 - 03:32 PM said:
Lyqyd, on 20 May 2013 - 02:39 PM said:
This is an interesting tool, but all programs for ComputerCraft posted to the forums need to be in plain Lua due to the rules regarding malicious code.

Why not compiled?
Because you can't tell if it is malicious or not. You give me a piece of code and I can pretty much tell what it's supposed to do. If you have compiled code you have no idea what it does until you run it.
Except if you decompile it.

Which of course doesn't always work. Then let's say you have 50 different files and the main script pieces all 50 scripts together to form the script. That would make it very hard to check if it's malicious or not. Meaning, more people will get infected then if a moderator looks over the code and sees that it's malicious.

Pharap #31

Pharap's profile picture

839 posts

Location England

Posted 20 May 2013 - 10:23 PM

jesusthekiller, on 20 May 2013 - 02:14 PM said:
I think you missed second half of my post…

Complementing me doesn't exactly withdraw the former complaint.

Lyqyd, on 20 May 2013 - 02:39 PM said:
This is an interesting tool, but all programs for ComputerCraft posted to the forums need to be in plain Lua due to the rules regarding malicious code.

You wouldn't be able to put it on the forum anyway, the text formatting would screw things up.
Unless you upload it as a file attached to a thread or something, which people never do with programs anyway.

Personally I'm still hunting for an alternative to dropbox to host anything in binary format.
I would use github, but I don't like github, don't like the clienty programs and you have to pay to have things private.

There's only two lua-based things that I wouldn't want open sourced anyway, which means that if the forum's not going to accept them, I'll find a way to discuss it with interested parties off the forums.

ElvishJerricco #32

808 posts

Posted 21 May 2013 - 12:36 AM

Just FYI, when you loadstring() in Run.lua, you need to pass a second argument of fs.getName(inFile) so that error messages know the file name. Also, you can replace _G.loadfile with this version of loadfile and it works perfectly with both compiled code and uncompiled code. (yes I tested). The best part of that is that anything anywhere that loads code via os.run, shell.run, or anything else based on loadfile will run the compiled code just fine, so it even works from the shell without modifying the shell.

And finally, your os.queueEvent("") coroutine.yield() lines negate all the performance you're trying to save by running compiled code. Those two lines make the loading process very, very slow (again, tested). I had a three line program that when compiled and run with this, took like a half second to load (and that's not counting the compile time). I took those lines out and it was instant. You're probably not going to run into files so large that they reach that 5 second limit, so I'd say take the lines out.

Grim Reaper #33

504 posts

Location Seattle, WA

Posted 21 May 2013 - 02:48 AM

Nina, on 19 May 2013 - 05:52 PM said:
It is impossible to pass any arguments to programs executed with the supplied "Run" program.

I'm pretty sure his code is already passing any additional arguments received from the shell off to the loaded function which is returned by loadfile().


loadfile(args[1])(select(2,unpack(args)))

However, you can simply tweak the run program yourself to allow for passing arguments to a compiled program. Considering that loadstring() is simply taking your string data which is a dump of actual Lua code, you can pass augments to the function returned.

Like the following:


local compiledProgram = loadfile (args[1])
table.remove (args, 1) -- Remove file path from arguments.

-- Run with arguments.
compiledProgram (unpack (args))

Grim Reaper #34

504 posts

Location Seattle, WA

Posted 21 May 2013 - 02:49 AM

ElvishJerricco, on 21 May 2013 - 12:36 AM said:
You're probably not going to run into files so large that they reach that 5 second limit, so I'd say take the lines out.

He's right there, but if you still wanted to implement this safety feature, you might have a flag which is set only if the file size is large enough to exceed the five second limit.

MudkipTheEpic #35

892 posts

Location Where you'd least expect it.

Posted 21 May 2013 - 10:00 AM

Maybe if it is still slow, only yield when i%50==0 in the loop.

Edit: Also for getting better file-size, use this function written by Espen.

Spoiler




--[[ Helper Function for getSize( _file ):			  ]]
--[[   Returns the number of bytes for the given file.  ]]
--[[  Read the end for the actual function. ]]--
local function getNumBytes( _file )
  local hFile
  local byte
  local byteCount = 0

  -- Open file
  hFile = fs.open( _file, "rb" )

  -- Count bytes
  repeat
		byte = hFile.read()
		if byte ~= nil then byteCount = byteCount + 1 end
  until byte == nil

  -- Close file handle
  hFile.close()

  -- Return size
  return byteCount
end

--[[ Returns the exact size of files ]]
local function getSize( _file )
  -- Get filename-part of _file path.
  local name = string.match( _file, "[^%s/\.]+$" )
  -- Kindly ask fs.getSize for its opinion about the supposed file-size ^^
  local fileSize = fs.getSize( _file )

  if fileSize > 512 then
		return fileSize - #name   -- Minus the filename, as fs.getSize() counts that with large files as well.
  else
		-- Need to count the bytes due to behaviour of fs.getSize()
		return getNumBytes( _file )
  end
end

Edited on 21 May 2013 - 08:07 AM

Pharap #36

Pharap's profile picture

839 posts

Location England

Posted 21 May 2013 - 03:25 PM

ElvishJerricco, on 21 May 2013 - 12:36 AM said:
Just FYI, when you loadstring() in Run.lua, you need to pass a second argument of fs.getName(inFile) so that error messages know the file name. Also, you can replace _G.loadfile with this version of loadfile and it works perfectly with both compiled code and uncompiled code. (yes I tested). The best part of that is that anything anywhere that loads code via os.run, shell.run, or anything else based on loadfile will run the compiled code just fine, so it even works from the shell without modifying the shell.

And finally, your os.queueEvent("") coroutine.yield() lines negate all the performance you're trying to save by running compiled code. Those two lines make the loading process very, very slow (again, tested). I had a three line program that when compiled and run with this, took like a half second to load (and that's not counting the compile time). I took those lines out and it was instant. You're probably not going to run into files so large that they reach that 5 second limit, so I'd say take the lines out.

Run.lua will only run one file at a time, if someone doesn't know what file they're trying to run when they've just typed it into the shell, I don't hold up much hope for their skills as a programmer. As for the function, yep, absolutely no error checking, but pcall can be used to handle that for now. It's early days yet, so a working function is all I can promise for now.

I would certainly expect swapping the loadfile functions to work. (The fact I named the function loadfile is no mere coincidence.) Up until now though I hadn't got round to testing, but as there is now confirmation, I can amend the documentation.

I could develop an option for size and all sorts of other enhancements, but frankly this code is barely 2 or 3 days old, so I'm not worried about optimising it yet. If you want to optimise it, feel free to, but otherwise I will optimise and improve it when I have the time.
(as it happens I do have a solution that will work without the need for any special command line options or testing the size of the file, but I will sort that when I get time)

Grim Reaper, on 21 May 2013 - 02:49 AM said:
ElvishJerricco, on 21 May 2013 - 12:36 AM said:
You're probably not going to run into files so large that they reach that 5 second limit, so I'd say take the lines out.

He's right there, but if you still wanted to implement this safety feature, you might have a flag which is set only if the file size is large enough to exceed the five second limit.

It's not quite as simple as testing file size because some things take longer for the lexical parser to handle than others, plus a file could be absolutely crammed full of whitespace, which shouldn't really effect the compile time massively.

Grim Reaper, on 21 May 2013 - 02:48 AM said:
Nina, on 19 May 2013 - 05:52 PM said:
It is impossible to pass any arguments to programs executed with the supplied "Run" program.

I'm pretty sure his code is already passing any additional arguments received from the shell off to the loaded function which is returned by loadfile().
loadfile(args[1])(select(2,unpack(args)))
However, you can simply tweak the run program yourself to allow for passing arguments to a compiled program. Considering that loadstring() is simply taking your string data which is a dump of actual Lua code, you can pass augments to the function returned.

Like the following:
local compiledProgram = loadfile (args[1])
table.remove (args, 1) -- Remove file path from arguments.

-- Run with arguments.
compiledProgram (unpack (args))

I won't deny here, Nina's comment was made before I made that change.
I considered the table remove thing, but I thought the solution I used looked neater and fit on fewer lines.
Many people forget about the select function, I thought it deserved some attention.

ElvishJerricco #37

808 posts

Posted 21 May 2013 - 03:43 PM

Pharap, on 21 May 2013 - 03:25 PM said:
…snip…

I've actually been working on a major overhaul of CraftOS lately, and I added your function (with credit given in the source, credits, and helpfiles) to _G and it works flawlessly. From my custom shell, or from the default shell, you can run a compiled program with no extra program. As for optimization, compiling can sometimes take too long, and so I do recommend using the coroutine.yield stuff for the compiler. But running is almost instant. I've compiled and run some pretty big files and they ran pretty much instantly.

Pharap #38

Pharap's profile picture

839 posts

Location England

Posted 21 May 2013 - 10:12 PM

ElvishJerricco, on 21 May 2013 - 03:43 PM said:
I've actually been working on a major overhaul of CraftOS lately, and I added your function (with credit given in the source, credits, and helpfiles) to _G and it works flawlessly. From my custom shell, or from the default shell, you can run a compiled program with no extra program. As for optimization, compiling can sometimes take too long, and so I do recommend using the coroutine.yield stuff for the compiler. But running is almost instant. I've compiled and run some pretty big files and they ran pretty much instantly.

I'm currently working on my own OS as well, but it's much closer to Windows than CraftOS.
Not sure any more if I'm going to make it available to the forums for many reasons.

You'll be happy to know I just submitted a new version that might be faster, assuming an if statement and os.clock are faster than queue event and coroutine yielding.

theoriginalbit #39

7508 posts

Location Australia

Posted 22 May 2013 - 12:37 AM

Pharap, on 21 May 2013 - 10:12 PM said:
You'll be happy to know I just submitted a new version that might be faster, assuming an if statement and os.clock are faster than queue event and coroutine yielding.

Bug in the new version. The if statements are missing the () on os.clock.

Grim Reaper #40

504 posts

Location Seattle, WA

Posted 22 May 2013 - 02:30 AM

On your newest version, although I may be mistaken, you never check to see if the path provided is a directory. Nothing wrong with your compiling code itself, but it could crash the program if someone messed up or was deliberately trying to break your code.

Just a suggestion, though.

Pharap #41

Pharap's profile picture

839 posts

Location England

Posted 22 May 2013 - 03:08 AM

theoriginalbit, on 22 May 2013 - 12:37 AM said:
Pharap, on 21 May 2013 - 10:12 PM said:
You'll be happy to know I just submitted a new version that might be faster, assuming an if statement and os.clock are faster than queue event and coroutine yielding.
Bug in the new version. The if statements are missing the () on os.clock.

*sigh* fixed (this is why I don't like rushing out edits every time someone points something out)

Grim Reaper, on 22 May 2013 - 02:30 AM said:
On your newest version, although I may be mistaken, you never check to see if the path provided is a directory. Nothing wrong with your compiling code itself, but it could crash the program if someone messed up or was deliberately trying to break your code.

Just a suggestion, though.

I'll leave that for people to implement themselves. I can guarantee working code, I'm not going to guarantee error checking for many reasons including performance. Besides, the run program itself implements the minimal error checking required. The messages aren't very informative, but there's only two of them so it's not that bad.

wilcomega #42

wilcomega's profile picture

453 posts

Location Holland

Posted 28 May 2013 - 03:37 PM

i have looked into the code but it seems like it just decompiles it and then running it?
if thats the case you just made a program to make every program open source again

Pinkishu #43

Pinkishu's profile picture

445 posts

Posted 28 May 2013 - 08:22 PM

wilcomega, on 28 May 2013 - 03:37 PM said:
i have looked into the code but it seems like it just decompiles it and then running it?
if thats the case you just made a program to make every program open source again

imo it just loadstring's the bytecode
that loads all the function again but you dont get plaintext lua back

if someone knows their way around in lua bytecode compiling probably doesnt help much though

Fenthis #44

Fenthis's profile picture

12 posts

Posted 06 June 2013 - 04:18 PM

Well… this is bad. And I don't mean your code. Lua bytecode is NOT safe to load. I would consider this a bug in ComputerCraft, a server could easily be owned this way.

Random reference on this subject: http://lua-users.org/lists/lua-l/2010-08/msg00487.html

Symmetryc #45

Symmetryc's profile picture

500 posts

Posted 06 June 2013 - 04:38 PM

Fenthis, on 06 June 2013 - 04:18 PM said:
Well… this is bad. And I don't mean your code. Lua bytecode is NOT safe to load. I would consider this a bug in ComputerCraft, a server could easily be owned this way.

Random reference on this subject: http://lua-users.org...8/msg00487.html

I don't think that Computercraft uses Lua 5.2 (I'm not entirely sure).

Jarle212 #46

Jarle212's profile picture

196 posts

Location Norway

Posted 06 June 2013 - 04:53 PM

So native lua/java functions can not be dumped(fs, term.native, etc)

Fenthis #47

Fenthis's profile picture

12 posts

Posted 06 June 2013 - 06:51 PM

Symmetryc, on 06 June 2013 - 04:38 PM said:
I don't think that Computercraft uses Lua 5.2 (I'm not entirely sure).

AFAIK it uses LuaJ, a java implementation of Lua. However, I can't find any indication that it's bytecode is any safer than the stock Lua. (It's not just 5.2 that has problems, all versions of the stock Lua does, there is just a proof of concept for 5.2).

Symmetryc #48

Symmetryc's profile picture

500 posts

Posted 06 June 2013 - 08:38 PM

Fenthis, on 06 June 2013 - 06:51 PM said:
Symmetryc, on 06 June 2013 - 04:38 PM said:
I don't think that Computercraft uses Lua 5.2 (I'm not entirely sure).

AFAIK it uses LuaJ, a java implementation of Lua. However, I can't find any indication that it's bytecode is any safer than the stock Lua. (It's not just 5.2 that has problems, all versions of the stock Lua does, there is just a proof of concept for 5.2).

Oh, sorry, the link sort of implied that it was only in 5.2.

Engineer #49

Engineer's profile picture

1522 posts

Location The Netherlands

Posted 07 June 2013 - 01:52 AM

However this tool is awesome, it is useless. :/

Lyqyd, on 20 May 2013 - 02:39 PM said:
This is an interesting tool, but all programs for ComputerCraft posted to the forums need to be in plain Lua due to the rules regarding malicious code.

Lyqyd #50

Lyqyd's profile picture

8543 posts

Posted 07 June 2013 - 01:15 PM

Fenthis, on 06 June 2013 - 04:18 PM said:
Well… this is bad. And I don't mean your code. Lua bytecode is NOT safe to load. I would consider this a bug in ComputerCraft, a server could easily be owned this way.

Random reference on this subject: http://lua-users.org/lists/lua-l/2010-08/msg00487.html

That reference suggests that it wouldn't be a problem on ComputerCraft. If you can get an exploit to work, please message the developers, though! I'm sure they'd love to hear about it.

ElvishJerricco #51

808 posts

Posted 07 June 2013 - 11:32 PM

Fenthis, on 06 June 2013 - 04:18 PM said:
Well… this is bad. And I don't mean your code. Lua bytecode is NOT safe to load. I would consider this a bug in ComputerCraft, a server could easily be owned this way.

Random reference on this subject: http://lua-users.org/lists/lua-l/2010-08/msg00487.html

I've been studying lua bytecode extensively for the past week to make an assembly language and rest assured, bytecode is not dangerous. What that page is referring to is some kind of code signing or verification, which default lua doesn't do, and CC certainly doesn't do. The bytecode language is actually very similar to lua. It's really just a reorganization of the language that's harder to code and difficult to manage (although the management part is fixed by using an assembly language with higher level features). There is no risk to running bytecode. It only has access to the same stuff a normal lua program has access to.

Pharap #52

Pharap's profile picture

839 posts

Location England

Posted 20 June 2013 - 05:44 AM

Ok, just checking up on this topic and I would like to make a few things clear.
Firstly, the purpose of compiling to bytecode in this case is not the same as with most compiled languages.
The idea is firstly to save memory (servers should be thankful for this, transferring bytecode instead of the sourcecode will provide all sorts of benefits) and secondly so that the Lua VM does not have to compile the program every time it runs. Compile once, run many more times as opposed to compile every time.

Elvish Jerricco speaks quite a bit of sense here.
I also have looked into Lua Bytecode, and I would like to debunk the idea that Lua bytecode is in anyway dangerous.
In itself, Lua bytecode is completely harmless. Any possible harm that could arise will be from any native calls made.
Even with that in mind, the worst someone could do is delete all files or swap out the start up file.
The former isn't rectifiable but the latter is. Beyond that, any computercraft lua code is completely harmless.

Jarle212, on 06 June 2013 - 04:53 PM said:
So native lua/java functions can not be dumped(fs, term.native, etc)

No, because dumping a lua file only dumps lua bytecode, it can't dump Java bytecode. Lua runs in a virtual machine that runs on top of the Java VM. Native calls are just ways of interfacing with the Java functions that can provide functionality that Lua itself does not contain natively such as bit manipulation and file IO.

Pinkishu, on 28 May 2013 - 08:22 PM said:
if someone knows their way around in lua bytecode compiling probably doesnt help much though

The purpose of compiling is not obfuscation, that's just a side effect.

wilcomega, on 28 May 2013 - 03:37 PM said:
i have looked into the code but it seems like it just decompiles it and then running it?
if thats the case you just made a program to make every program open source again

Wrong way around. This is a compiler and a runner. The compiler compiles source code to byte code. The runner runs the bytecode.
These things occur naturally when you use computercraft anyway, but by using these two programs you can reduce the size of programs because bytecode is smaller than sourcecode and you can cut out the stage before running a program where the VM has to compile a program every time before it's run.