JVML-JIT - Java Virtual Machine in Lua, with JIT Compiler.

A long long time ago, in a forum post far away, a user made a cool program, that could run Java from Lua. He called it JVML. I was immediately interested. As soon as he put his work on GitHub, I cloned the code and got to work. It was amazing that this was possible, but there was lots to do. Classes with native methods had to be built from the ground up in Lua, exceptions didn't work, interfaces didn't work. I could tell how difficult this was going to be.

Then one day someone called Yevano took interest. He wanted to boost the speed of JVML by turning it into a JIT, which compiles the Java bytecode into Lua bytecode for better performance. ds84182 had been basically absent from JVML since his initial work, so we decided to work on this exclusively in Yevano's repo. Since then, it has undergone some amazing work that has turned it into a very capable tool. I'm proud to announce JVML-JIT.


Install: pastebin run NzNwtBQy
Or: grin-get install Team-CC-Corp/JVML-JIT

Github: Team-CC-Corp/JVML-JIT

Usage

If you didn't use grin-get for installation, make sure you add jvml/bin/ to your shell's path at startup.

jvml MyMainClass
jvml -jar MyExecutableJarFile

JVML-JIT uses the notion of the classpath. By default, the cc_rt.jar file is included, and the current directory the shell is at is included. The -cp argument can be used to add to this.

Building for JVML-JIT

Although in many cases, it will work out to just compile your java code normally and run those class files, it runs some risks. Our runtime library is incomplete and the compiler might use methods and classes we don't have implemented. To be safe about this and avoid that problem, use the bootclasspath compiler option.

javac -bootclasspath jvml/CCLib/jar/cc_rt.jar MySource.java

How it Works

JVML-JIT works by loading class files, and compiling their methods into Lua. In an attempt to be more efficient, no method is compiled until the first time it's called. These compiled methods are now just ordinary Lua functions.

FAQ

• Can I run Minecraft in Minecraft?
Sorry, but probably not. Minecraft makes use of extensive APIs that would take years to implement. That's just not going to happen.
• Can I run ______ in Minecraft?
Sure, if you can get all the classes and natives it depends on ported over. Any additions to the runtime library would be appreciated in a pull request! =)
• Why is it so slow? I thought the JIT was supposed to speed it up!
JVML-JIT will appear slow for small projects or projects that do many different things one time each. This is because compiling the methods to Lua is a bit of a slow process. Compiling roughly 60k lines of Lua assembly (which is a lot) takes the JIT about 15 seconds. Once a method is compiled, it's very, very quick. But if your program isn't using the same methods multiple times, you'll notice that you spend a lot of time compiling the variety of methods you use. Point is, JVML-JIT is better suited for larger projects that are meant to do long running, repetitive tasks. Perfect for a server!
• How can I contribute to help advance the runtime library?
If there's any classes or methods you'd like to see implemented in the runtime, anyone is free to fork the repo, add the features in, and send us a pull request. As long as your change works, we'll probably take all the help we can get.
• Where do I report bugs and issues?
Please direct any and all bugs and issues to JVML-JIT's issues page.

Special thanks to ds84182! Took all three of us to get this where it is today.

I've had a couple of people ask me how to let other people use the Java programs they've made, since you can't exactly pastebin a binary file. Really the only decent option out there right now is Grin or Grin-Get. But you should be using Git for your projects anyway ;)/>

The hope is that one day we'll be able to run javac inside JVML, so you can edit and compile your Java classes all inside ComputerCraft.

Yevano, on 10 October 2014 - 09:22 PM said:

The hope is that one day we'll be able to run javac inside JVML, so you can edit and compile your Java classes all inside ComputerCraft.

Isn't what that actually do ?
Never mind.

That's awesome though, but I wonder if that's pratically usable…

pingoleon60, on 10 October 2014 - 09:38 PM said:

Yevano, on 10 October 2014 - 09:22 PM said:

The hope is that one day we'll be able to run javac inside JVML, so you can edit and compile your Java classes all inside ComputerCraft.

Isn't what that actually do ?
Never mind.

That's awesome though, but I wonder if that's pratically usable…

Getting javac is definitely a long term goal. It's a big thing. But it's definitely practical.

ElvishJerricco, on 10 October 2014 - 09:59 PM said:

Getting javac is definitely a long term goal. It's a big thing. But it's definitely practical.

I meant :

JVML-JIT is awesome though, but I wonder if JVML-JIT is pratically usable…

pingoleon60, on 10 October 2014 - 10:11 PM said:

JVML-JIT is awesome though, but I wonder if JVML-JIT is pratically usable…

Ah. Gotcha. It's still got a ways to go to be where we want it, but for now it's definitely usable. It's got peripheral, terminal, redstone, and event support among many other things. It can do a lot.

I'll test it and give you feedback, but if that works as good as you say it does, I will surely spotlight it in my french ComputerCraft tutorial serie, because that is completly awesome.

I disappear way too much… It isn't good for my health… Anyways, this is really cool, I didn't know a JIT would be possible :o/>!

ds84182, on 11 October 2014 - 01:34 AM said:

I disappear way too much… It isn't good for my health… Anyways, this is really cool, I didn't know a JIT would be possible :o/>!

Thanks =) Yea it's come a long way.

Just omg!

Why didnt I find this sooner? I knew there was something like this out there written by ds, but it didnt seem practical at that point. You really got me motivated writing more programs, since I fell in love with Java for its OOP and classes implementation thing.. Just wow guys, keep it up! :D/>

You got yourself a bookmark to your repo, so I can check things while Im writing it :)/>

After creeping around for a while in the repo, it seems that one can only use the classes defined in JVML-JIT/CCLib/src/java. I get that everything should be added in some way in lua, but I don't get how.
If I can get some explanation on that subject, that would be very appreciated.
In return for that I will contribute by adding classes which I'm going to need for myself (im actually planning to use this for programs).

I get that you use native a lot, and you add your method to the natives table to make it function. You also have to pass arguments there as java bytecode string (if you follow me) and do some fancy stuff with toJString and all.
I really like this project, and I wish I had found it sooner than this. Thank you guys for creating it

Engineer, on 20 October 2014 - 01:30 AM said:

it seems that one can only use the classes defined in JVML-JIT/CCLib/src/java.

What do you mean by this? I have no troubles using classes in the cc package or from external classpaths.

ElvishJerricco, on 20 October 2014 - 05:59 AM said:

What do you mean by this? I have no troubles using classes in the cc package or from external classpaths.

I was speculating, and I meant all default java packages

Edit: after setting up a proper environment in my IDE, my speculation is correct. To explain myself more transparant with my previous post, let's say I want to use the FileSystem. Obviously, we want that to be through a java.io.File Object, however that class does not exist yet in JVML-JIT, we'd have to add it.

Obviously, I dont want to be a pain in the *** and ask you guys to implement (in this case) the File class. If I could do it myself, it will be easier for all of us, more classes get added and I can work through Java on cc projects! :D/>

Engineer, on 20 October 2014 - 01:30 AM said:

After creeping around for a while in the repo, it seems that one can only use the classes defined in JVML-JIT/CCLib/src/java. I get that everything should be added in some way in lua, but I don't get how.
If I can get some explanation on that subject, that would be very appreciated.
In return for that I will contribute by adding classes which I'm going to need for myself (im actually planning to use this for programs).

I get that you use native a lot, and you add your method to the natives table to make it function. You also have to pass arguments there as java bytecode string (if you follow me) and do some fancy stuff with toJString and all.
I really like this project, and I wish I had found it sooner than this. Thank you guys for creating it

Right now natives loading is a bit wonky, but I'll explain as best I can.

First, you'll want to create your class with native methods. Let's call it NativeTest and place it in the package mypackage. We'll make a native method in it called repeat, which will repeat a string a certain number of times.

package mypackage.NativeTest;

public class NativeTest {
	public static native String repeat(String str, int rep);
}

Now, here's the wonky bit. Right now all runtime natives are loaded automatically from the java/lang/native folder. We just need to create a file called NativeTest.lua in there. This is just a convention though. Technically you can name the file anything you want, or even place your methods in other files.

--# This just creates the natives table for this class if it hasn't been created yet.
--# Note that the native Lua files share the environment with the whole JVML runtime.
natives["mypackage.NativeTest"] = natives["mypackage.NativeTest"] or { }

--# Here we add our method definition. If you've not done too much with JNI before, the string here is a method signature.
--# The method signature describes the name, return type, and argument types of the method.
--# In this case, it translates to String repeat(String, int)
natives["mypackage.NativeTest"]["repeat(Ljava/lang/String;I)Ljava/lang/String;"] = function(str, rep)
	--# Java String requires conversion to a Lua string.
	local luaStr = toLString(str)

	--# ints are already represented as normal Lua numbers. (Note that longs are represented as bigints.)
	luaStr = luaStr:rep(rep)

	--# Now convert back to the Java String object and return.
	return toJString(luaStr)
end

Now we can call that native like this.

import mypackage.NativeTest;

public class Tester {
	public static void main(String[] args) {
		System.out.println(NativeTest.repeat("lol", 20));
	}
}

Phew… At least it's easier than normal JNI. :P/>

A few tips:

• Make sure to run ant even after just editing the Lua files. They don't need to be compiled, but doing this will make ant add them to the runtime jar.
• The conversions between Lua and Java types are as follows
• boolean, byte, short, int, and double are all represented as Lua numbers. boolean is either 0 or 1, not false or true.
• long is represented as a bigint from the bigint.lua library. You shouldn't try converting them to normal Lua numbers unless you know what you're doing.
• All Java objects (including strings) are represented as tables. These tables are numerically indexed to prevent doing hashtable lookups when possible. You can manipulate these objects in the following way:
• getObjectField(obj, name) returns the stored value in the named field. *If speed is important, you can use the fieldIndexByName table field within the object's class to get the field index ahead of time, and then get the field using obj[2][fieldIndex].
• setObjectField(obj, name, value) sets the named field. The above description of performance speedup applies here, too.
• newInstance(class) returns a new instance of a Java class without calling any constructor. You need to remember to call the constructor on the object explicitly.
• findMethod(class, name) searched and returns a method reference from the given class. I emphasize reference, because it is a table which contains the Lua function for calling the method, not the Lua functions itself. To call the method, do local ret = method[1](…).
• JVML doesn't try to do conversion on tables passed into the runtime from natives or anything like that. Passing a raw table to a method will most likely crash the VM.

* This is done like this:

local fieldTable = obj[2]
local fieldIndex = cl.fieldIndexByName["myField"]
--# intensive loop
for i = 1, 1000 do
	fieldTable[fieldIndex] = someMethodA[1]()
	someMethodB[1](fieldTable[fieldIndex])
end

Hope this helped.

Yevano, on 20 October 2014 - 03:58 PM said:

Now, here's the wonky bit. Right now all runtime natives are loaded automatically from the java/lang/native folder.

Actually, you can load natives from anywhere on the classpath. The ones in java/lang/native are loaded automatically, but there's some classes in the cc package that show that you can load natives at any place.

package mypackage;
public class NativeTest {
    public static native String rep(String s);

    static {
        System.load("path/in/classpath/to/my/natives.lua");
    }
}

Updated. Changes:

• System.in
• Turtle API

Updated: Bugfixes for Turtle and arithmetic

Updated: Change in ownership to the new team Yevano and I have started.

Upcoming: New JIT backend. Should be significantly faster. Also that new backend will be put into its own framework for other people to use as JIT backends.

From Java to Lua to Java again…
…is what this is doing

Geforce Fan, on 13 December 2014 - 08:24 PM said:

From Java to Lua to Java again…
…is what this is doing

And is that bad? Some people prefer the rules that java requires you to use.

Dragon53535, on 13 December 2014 - 08:28 PM said:

Geforce Fan, on 13 December 2014 - 08:24 PM said:

From Java to Lua to Java again…
…is what this is doing

And is that bad? Some people prefer the rules that java requires you to use.

Yet it has to be admitted that it is inefficient :P/>

Engineer, on 13 December 2014 - 08:31 PM said:

Dragon53535, on 13 December 2014 - 08:28 PM said:

Geforce Fan, on 13 December 2014 - 08:24 PM said:

From Java to Lua to Java again…
…is what this is doing

And is that bad? Some people prefer the rules that java requires you to use.

Yet it has to be admitted that it is inefficient :P/>

It's fun to make it the most efficient it can be though. I have to wonder, if LuaJ JIT'd to JVM, and JVM JIT'd to machine, how inefficient would the resulting code be?

ElvishJerricco, on 13 December 2014 - 08:37 PM said:

It's fun to make it the most efficient it can be though. I have to wonder, if LuaJ JIT'd to JVM, and JVM JIT'd to machine, how inefficient would the resulting code be?

Something i just thought of: Why didn't you name it JLua since LuaJ is Lua in Java, JLua would be Java in Lua :P/> (Or JavaL)

Dragon53535, on 13 December 2014 - 08:42 PM said:

ElvishJerricco, on 13 December 2014 - 08:37 PM said:

It's fun to make it the most efficient it can be though. I have to wonder, if LuaJ JIT'd to JVM, and JVM JIT'd to machine, how inefficient would the resulting code be?

Something i just thought of: Why didn't you name it JLua since LuaJ is Lua in Java, JLua would be Java in Lua :P/> (Or JavaL)

There's already another library called JLua

sci4me, on 13 December 2014 - 08:46 PM said:

There's already another library called JLua

I didn't realize there was a 2002 incomplete project called JLua. Huh…

I wasn't trying to say it was bad, I was just saying it's kinda funny it does that

Dragon53535, on 13 December 2014 - 08:42 PM said:

ElvishJerricco, on 13 December 2014 - 08:37 PM said:

It's fun to make it the most efficient it can be though. I have to wonder, if LuaJ JIT'd to JVM, and JVM JIT'd to machine, how inefficient would the resulting code be?

Something i just thought of: Why didn't you name it JLua since LuaJ is Lua in Java, JLua would be Java in Lua :P/> (Or JavaL)

I didn't name it =P

ElvishJerricco, on 13 December 2014 - 09:38 PM said:

I didn't name it =P

I know, you used the already made JVML from DS(whatevernumbershere) and made it into a JIT and are still updating it :P/>. I just thought it was a funny thought.

Besides the obvious inefficiency of going back and forth between VMs, since this is a JIT we can in theory make this just as fast as Lua, and even faster than Lua with heavy use of OOP constructs. The reason for this is that we optimize the object fields and methods to be indexed in a numerical array instead of a hashmap, and we don't need to use metatables to do class hierarchy. The only much slower thing would usually be startup time.

EDIT: This is not to say the VM is super efficient right now. Currently there's tons of work we need to do to optimize the bytecode, and we're currently working on a new JIT which should compile methods faster. Still, it's way faster than an interpreter at least.

Yevano, on 13 December 2014 - 10:05 PM said:

EDIT: This is not to say the VM is super efficient right now. Currently there's tons of work we need to do to optimize the bytecode, and we're currently working on a new JIT which should compile methods faster. Still, it's way faster than an interpreter at least.

Oh lord don't remind me of the old interpreter… That thing was a glorious mess =P

Updated. Changes:

• New emitter. Much faster and simpler to work with
• Based on the new asm.lua framework

Geforce Fan, on 13 December 2014 - 08:24 PM said:

From Java to Lua to Java again…
…is what this is doing

Then port Luaj to this and Java to Lua to Java to Lua to Java

JavaLuaCeption!

Just so you are aware, I am working on a project called CCGradle, it makes it easy to create programs for JVML-JIT using Gradle.

jamierocks, on 01 April 2015 - 10:12 AM said:

Just so you are aware, I am working on a project called CCGradle, it makes it easy to create programs for JVML-JIT using Gradle.

Sweet! Just FYI, I wouldn't recommend cloning the repo for the dependency. We keep the release and the dev environment separate. It'd be better if you could pull the asset from the latest release and decode that.

So I theory…if you use one of the peripheral addons that add full resolution displays an managed to get everything ported in… We could play minecraft in minecraft for real….if anyone could stand that <1 fps frame rate enough to play it. (Obviously I know this still isn't PRACTICAL nor ever will be)

ElvishJerricco, on 01 April 2015 - 10:55 PM said:

jamierocks, on 01 April 2015 - 10:12 AM said:

Just so you are aware, I am working on a project called CCGradle, it makes it easy to create programs for JVML-JIT using Gradle.

Sweet! Just FYI, I wouldn't recommend cloning the repo for the dependency. We keep the release and the dev environment separate. It'd be better if you could pull the asset from the latest release and decode that.

Do you have a download link like example.com/jvml-jit/1.0.zip? So users of CCGradle could specify the version?

The latest release is always here. You can look at Grin's download code to see how you can get the latest version for a Repo. You want to be looking for a file that looks like *.base64. Groovy has pretty decent JSON support.

SquidDev, on 02 April 2015 - 08:11 PM said:

The latest release is always here. You can look at Grin's download code to see how you can get the latest version for a Repo. You want to be looking for a file that looks like *.base64. Groovy has pretty decent JSON support.

Ok, I'll make sure to do that for the 1.0 release. I have just added setting the bootClasspath, for builds.

Also why would i want the .base64 file?

jamierocks, on 02 April 2015 - 09:53 PM said:

Also why would i want the .base64 file?

Grin downloads the .base64 file because it doesn't contain chars with value > 127 and (strings with) chars with value > 127 can cause problems in LuaJ.

Is this still being worked on? Or is this not actively developed anymore?

ardera, on 28 June 2015 - 03:47 PM said:

Is this still being worked on? Or is this not actively developed anymore?

Last commit was 3 months ago, so I suppose you could say the project is inactive at the moment. However, I still check the repo every now and then for pull requests (probably ElvishJerricco does too), so it's still possible to push to the project if you're interested. I think it was a really fun project to work on, but I lost interest in it. Maybe some day I'll get back to work on it, but for now I probably won't be contributing much.

Yevano, on 28 June 2015 - 06:25 PM said:

ardera, on 28 June 2015 - 03:47 PM said:

Is this still being worked on? Or is this not actively developed anymore?

Last commit was 3 months ago, so I suppose you could say the project is inactive at the moment. However, I still check the repo every now and then for pull requests (probably ElvishJerricco does too), so it's still possible to push to the project if you're interested. I think it was a really fun project to work on, but I lost interest in it. Maybe some day I'll get back to work on it, but for now I probably won't be contributing much.

The issue to me is the enormity of the task at hand. Java is a huge platform, with a lot of interworking parts. It's just not realistic to think we could build something that compares. So my interests have moved to building my own language. But that takes very careful design so I'm still in the theory stage at the moment.

0.5.3 released. Should resolve the issue where nothing would run on CC 1.74.