ComputerCraft Benchmarking (Emulators and Minecraft)

Hey all,

1lann wrote a quick, cheap, little script to benchmark ComputerCraft (as in, output a number grading it on how well it performs).

Please note the results will vary depending on your computer.

The results are extremely interesting. For the top numbers above the line, the lower the number is, the better it performed. For the total score at the bottom, the higher the number, the better it performed.

First up, ComputerCraft 1.58 running in Minecraft:



This is the most interesting one. ComputerCraft Beta 1.6pr0 running in Minecraft. Looks like the new windowing API slows it down dramatically. The negative score at the bottom basically means it took longer than a minute to complete.



Next up is my own ComputerCraft emulator Mimic. This is the old version in C++ running ComputerCraft 1.53:



Here is the new version of Mimic (running ComputerCraft 1.58) written in Javascript:



And here is CCEmu (can't believe I still have a version) running I have no idea what version of ComputerCraft:



If you've got a version of ComputerCraft, or want to test another emulator (I can't find a proper version of Love to run Sorroko's emulator), just run the script linked above and please post the results!

If you are intersected here is results from my testing. Out of curiosity what stats is your PC ?

Spoiler

PC: AMD x64 A10-6700 quad 3.70 GHz RAM 16GB DDR3 WIN 8.1 JAVA 7 update 51 MC 1.6.4-Forge9.11.1.96 CC 1.58

Engineer_Nalfar, on 02 February 2014 - 09:40 AM said:

If you are intersected here is results from my testing. Out of curiosity what stats is your PC ?

I'm using a Intel Core i7 2.7GHz, 16GB 1600MHz DDR3, Intel HD Graphics 4000 1GB, Mac OSX, Java 6, Minecraft 1.6.4.

Why Java 6?

Engineer_Nalfar, on 02 February 2014 - 09:40 AM said:

If you are intersected here is results from my testing.

No CC version…?

Here is CCLite (Sorroko's version) 2.10 (CC 1.6 pr0 beta version of it):

PC (Just my laptop that I use for light internet surfing):
Intel(R) Core(TM) i5-3350P CPU @ 3.10GHz

8.0 GB DDR3 Memory

Windows 8.1

Lotsa images - Tests done on an AMD Phenom II X2 545 @ 3.51GHz, Windows 7 32-bit

Spoiler

Ran it in my emulator (a fork of Sorroko's older code)
1.58 - non threaded renderering:


1.58 - threaded rendering (default settings):


1.58 - threaded rendering (Increased uplink lock):


1.6 - Sorroko's emulator:


1.58 - Sorroko's emulator:

Neat utility.

My new MacBook Pro gets the following:

Text Rendering: 1.5
Color Rendering: 1.7
Advanced Rendering 3.0
Computing Power: 2.5

Total (out of 50): 46.3

You should post this in utilities.

This looks really interesting. Nice to see that the latest Mimic is pretty much the fastest of them all. Out of curiosity, why didn't 1lann post it if he made it?

nutcase84, on 02 February 2014 - 06:52 PM said:

Why Java 6?

OS X and Java have a strange relationship. You generally can't use the version you want, it's sh•t or nothing.

GNU/Linux Ubuntu 13.10
Oracle Java 7
i5 @ 3.00Ghz
8Gb Ram [ 2gb Java ]
NVidia GeForce GTS 450
Direwolf20 Modpack for 1.64 CC 1.57 / OP snap 112 / OCCS 1.6.4.1b

i also have my terminal size set to 29x81
[attachment=1462:Screenshot from 2014-02-03 07:25:24.png]

- i note that on a 3x3 colour monitor attached and running the benchmark on that, the score is much better… a 4x5 monitor, much worse… running a max size monitor on top i get a total of 27.1

on my 2007 macbook pro [ core 2 duo, 4gb, nvidia something, java 6 ] i get an overall score of -9.8 … mind you, the same modpack runs at about 20FPS on it, so…

oeed, on 03 February 2014 - 02:15 AM said:

Out of curiosity, why didn't 1lann post it if he made it?

He didn't think anyone would be interested, so I said I'd do it for him if he didn't want to :P/>

GravityScore, on 03 February 2014 - 03:19 AM said:

oeed, on 03 February 2014 - 02:15 AM said:

Out of curiosity, why didn't 1lann post it if he made it?

He didn't think anyone would be interested, so I said I'd do it for him if he didn't want to :P/>

Ah I see :P/>

Bomb Bloke, on 02 February 2014 - 09:54 PM said:

Engineer_Nalfar, on 02 February 2014 - 09:40 AM said:

If you are intersected here is results from my testing.

No CC version…?

fixed

GravityScore, on 02 February 2014 - 09:18 AM said:

This is the most interesting one. ComputerCraft Beta 1.6pr0 running in Minecraft. Looks like the new windowing API slows it down dramatically. The negative score at the bottom basically means it took longer than a minute to complete.

Now I think of it, you might want to mention this in the bug reports post.

oeed, on 03 February 2014 - 05:04 AM said:

Spoiler

GravityScore, on 02 February 2014 - 09:18 AM said:

This is the most interesting one. ComputerCraft Beta 1.6pr0 running in Minecraft. Looks like the new windowing API slows it down dramatically. The negative score at the bottom basically means it took longer than a minute to complete.

Now I think of it, you might want to mention this in the bug reports post.

It could be effect of all the debug code that probably got left inside this version. I don't think any changes could make this kind of difference.

wojbie, on 03 February 2014 - 05:27 AM said:

It could be effect of all the debug code that probably got left inside this version. I don't think any changes could make this kind of difference.

I was thinking it was due to the window API that all drawing in advanced computers is now wrapped in. Because this is Lua side it would likely be much slower than Java side, probably causing the terrible slowness.

I think I just broke the program xDD


#EDIT:

[namedspoiler="Specs]
[CODE]
00:00.0 RAM memory: NVIDIA Corporation MCP61 Host Bridge (rev a1)
00:01.0 ISA bridge: NVIDIA Corporation MCP61 LPC Bridge (rev a2)
00:01.1 SMBus: NVIDIA Corporation MCP61 SMBus (rev a2)
00:01.2 RAM memory: NVIDIA Corporation MCP61 Memory Controller (rev a2)
00:02.0 USB controller: NVIDIA Corporation MCP61 USB 1.1 Controller (rev a3)
00:02.1 USB controller: NVIDIA Corporation MCP61 USB 2.0 Controller (rev a3)
00:04.0 PCI bridge: NVIDIA Corporation MCP61 PCI bridge (rev a1)
00:05.0 Audio device: NVIDIA Corporation MCP61 High Definition Audio (rev a2)
00:06.0 IDE interface: NVIDIA Corporation MCP61 IDE (rev a2)
00:07.0 Bridge: NVIDIA Corporation MCP61 Ethernet (rev a2)
00:08.0 IDE interface: NVIDIA Corporation MCP61 SATA Controller (rev a2)
00:08.1 IDE interface: NVIDIA Corporation MCP61 SATA Controller (rev a2)
00:09.0 PCI bridge: NVIDIA Corporation MCP61 PCI Express bridge (rev a2)
00:0b.0 PCI bridge: NVIDIA Corporation MCP61 PCI Express bridge (rev a2)
00:0c.0 PCI bridge: NVIDIA Corporation MCP61 PCI Express bridge (rev a2)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Address Map
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Link Control
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series]
[/CODE]
[/namedspoiler]

This is the output of "lspci" on my Ubuntu 13.10 :)/>

#EDIT2:

Tested on
gamax92's emulator. Wonderful program btw :D/>

oeed, on 03 February 2014 - 02:15 AM said:

This looks really interesting. Nice to see that the latest Mimic is pretty much the fastest of them all. Out of curiosity, why didn't 1lann post it if he made it?

nutcase84, on 02 February 2014 - 06:52 PM said:

Why Java 6?

OS X and Java have a strange relationship. You generally can't use the version you want, it's sh•t or nothing.

Hmm… my old macbook worked fine with 7… but I can relate because my PowerBook hated openJRE or whatever it's called. XD

Well, I don't have a screenshot because it never actually finished. But I tried running it in my OS…

Let's just say 30 minutes in it was still doing the colour test. I think I might need to look in to some optimisation.

nutcase84, on 03 February 2014 - 08:25 AM said:

oeed, on 03 February 2014 - 02:15 AM said:

This looks really interesting. Nice to see that the latest Mimic is pretty much the fastest of them all. Out of curiosity, why didn't 1lann post it if he made it?

nutcase84, on 02 February 2014 - 06:52 PM said:

Why Java 6?

OS X and Java have a strange relationship. You generally can't use the version you want, it's sh•t or nothing.

Hmm… my old macbook worked fine with 7… but I can relate because my PowerBook hated openJRE or whatever it's called. XD

both is true.
apple used to write their own JRE until after 1.6 when they handed over responsibility to Oracle/ Users to install their own choice - mainly because the support of the JRE was annoying them as it was allowing a malware infection vector on OSX… so up until OS X 10.7.3 you had no choice and had to use the Apple one … after that, you can use the Oracle one here… i've not upgraded tho as it was such a pain to get the RXTX serial library working that i daren't touch it :)/>


back on track: i ran the benchmark on my home server instance, it crashed the server… :(/>
- out of memory
so i adjusted the java params and tried again… 9.5 :(/>
mind you, it was raining in game at the time… but i could hear the other computers fans speed up as soon as i hit enter on the benchmark :)/>

edit: interesting: 23 when it's not raining.

this server is only a core2 duo 2.something ghz with 4gb - 1.5 for java - but it's running a headless linux install [no window system] to keep it's overheads down…

i'm tempted to install MineOS or similar [anyone any experience with it, is it worth it?] - to really take it to a bare bones install, if i do, i'll do the same test and post here…

GravityScore, on 03 February 2014 - 07:53 AM said:

wojbie, on 03 February 2014 - 05:27 AM said:

It could be effect of all the debug code that probably got left inside this version. I don't think any changes could make this kind of difference.

I was thinking it was due to the window API that all drawing in advanced computers is now wrapped in. Because this is Lua side it would likely be much slower than Java side, probably causing the terrible slowness.

You are right - did 2 test on pr0 with windowing api(multishell) and without it(tlco-ed out to bios and ran normal shell to test that)
Both tests done on Portable Computer cause it was handy that way

This is with windowing api working:

And this is with it gone


Soo if you are displaying a lot of stuff fast get rid of Windows :D/>
CC more like real world every day.

If the new terminal emulation layer was java side not lua it might run smother?

Examle
local current screen = getTerminalWindow() – gets current terminal object.
local newScreen = getNewTerminalWindow()
newScreen:write("Hello")
newScreen:setCursorPos(1,1)
newScreen:display() – activates the screen to be displayed.


That way the heavy lifting can be done by java witch does a quicker job of buffering screen content.

it's probably a balance though isn't it… most CC computers aren't used for high-speed graphics… so a graphical performance test is interesting, but perhaps not applicable in practice. To place the code more tightly coupled into the java would mean that the heap always needs that code even if you're not doing graphical output… with it being a lua side api, then you're only increasing the chunk-space in lua if required.

- though i say this from vaguely educated postulation, not any knowledge of either dan's reasoning or methodology, or indeed lua chunks :)/>
… but it seems plausible as a reason, it's at least an excuse, so hey i'm saying it :)/>

nutcase84, on 02 February 2014 - 06:52 PM said:

Why Java 6?

Oracle hasn't released Java 7 to OS X yet — it's still dev beta — 'cause they're inferior to Sun. Either Oracle — or Apple — is still rolling out updates for Java 6 though with security fixes and such.
If an OS X user wants Java 7 its a manual download and install, and then most of the time an environment fix 'cause it Java programs stop working, especially LWJGL it really doesn't like Java 7, I had to go and download the latest LWJGL to put into Minecraft so that I could play it.

Because oeed tested this on his OS, here is a test on mine:

I ran this on the computer, wirelessly connected through to the monitor through GeniusStream. I opened a full screen (as you can press F3 to see a list of all open things in a scrollable list) CraftOS prompt on the remote monitor, then I just did pastebin run Ef4SdA3k. With CC 1.6 pr0, here is the result (note that the program runs on the sending computer (surrounded in peridot) and every monitor call is sent over to the remote computer. This still needs optimization.)


EDIT: Damn ninja. And this was on the aforementioned laptop, mc 1.6.4

wojbie, on 03 February 2014 - 04:59 PM said:

You are right - did 2 test on pr0 with windowing api(multishell) and without it(tlco-ed out to bios and ran normal shell to test that)
Both tests done on Portable Computer cause it was handy that way

This is with windowing api working:
[snip]

And this is with it gone
[snip]

Soo if you are displaying a lot of stuff fast get rid of Windows :D/>
CC more like real world every day.

Wow that's an incredible difference! I feel like someone should put this on the bug report thread, but I don't see much that can be done about it (apart from maybe moving it to Java, but I don't think dan will do that).

GravityScore, on 04 February 2014 - 02:48 AM said:

wojbie, on 03 February 2014 - 04:59 PM said:

You are right - did 2 test on pr0 with windowing api(multishell) and without it(tlco-ed out to bios and ran normal shell to test that)
Both tests done on Portable Computer cause it was handy that way

This is with windowing api working:
[snip]

And this is with it gone
[snip]

Soo if you are displaying a lot of stuff fast get rid of Windows :D/>
CC more like real world every day.

Wow that's an incredible difference! I feel like someone should put this on the bug report thread, but I don't see much that can be done about it (apart from maybe moving it to Java, but I don't think dan will do that).

Done.

http://www.computercraft.info/forums2/index.php?/topic/16882-computercraft-16-beta-bug-reports/page__view__findpost__p__163468

I think that based on awsmazinggenius' post there is probably quite a bit that could be changed, clearly there are ways to use a buffer without shocking speed (not that I can really talk). I think the main problem with OneOS is the fact that it uses a buffer and only draws on yield. I'll be looking in to it.

Freack100, on 03 February 2014 - 08:00 AM said:

I think I just broke the program xDD
*snip*

That's like impossible… like literally impossible xD. The version gravs used and that is posted (which he is aware is out-of-date but was too lazy to update his results with) should have all of the results subtracted by 1 (and the total subtracted by 4), because in the test are sleeps totalling up to 1 second per test. So it's impossible with the version gravs is using to score anything below 1 second, so idk how you accomplished that xD.

1lann, on 04 February 2014 - 04:44 AM said:

Freack100, on 03 February 2014 - 08:00 AM said:

I think I just broke the program xDD
*snip*

That's like impossible… like literally impossible xD. The version gravs used and that is posted (which he is aware is out-of-date but was too lazy to update his results with) should have all of the results subtracted by 1 (and the total subtracted by 4), because in the test are sleeps totalling up to 1 second per test. So it's impossible with the version gravs is using to score anything below 1 second, so idk how you accomplished that xD.

Well, can't argue with that.

Also, I see you've left South Korea now :P/>

oeed, on 04 February 2014 - 05:34 AM said:

Also, I see you've left South Korea now :P/>

*cough* stalker. :P/>

Uhm… Maybe its time to write a fix? If you can…

YoYoYonnY, on 01 May 2014 - 02:06 PM said:

-snip-
Uhm… Maybe its time to write a fix? If you can…

Either your computer or the new window API is too slow to run this.