A Smaller Top Level Coroutine Override

We all know NeverCast's Top Level Coroutine Override. It's pretty nice and very useful. But I've figured out a way to do the same job with much simpler code!

os.queueEvent("modem_message")
local p = _G.printError
function _G.printError()
        _G.printError = p
        os.run({}, "n")
end

As you can see it's extremely simple. bios.lua runs two things in its waitForAny call. One is the root shell, and the other is the rednet coroutine. The first thing this program does is queue a fake modem_message event. The lack of parameters, and the lack of sanity checks in the rednet.run function, makes this crash the rednet loop. The next thing bios.lua calls is printError, so that's what gets overridden with a function that resets to the original, and starts an arbitrary program (in this case, /n). This has one technical advantage over the NeverCast's original program. It works no matter how many nested shells you're in. So if you launch "shell" from the root shell, and then run this override, it still works.

pastebin get L9Se9RVR override

Testing…

ZudoHackz, on 02 September 2013 - 03:37 AM said:

Testing…

There's an issue with it. This particular one doesn't work in a startup file because the shell runs startup before rednet gets the chance to launch. The following one takes advantage of that fact and ONLY works in a startup file

local r = rednet.run
function rednet.run()
	rednet.run = r
	error("",0)
end

local p = printError
function _G.printError()
	_G.printError = p
	local ok, err = pcall(function() assert(loadfile("n"))() end) -- mildly complex series of calls to present errors rather than actually error here
	if not ok then printError(err) end
end

So a hybrid between the two would be

os.queueEvent("modem_message")
local r = rednet.run
function rednet.run()
	error("", 0)
end

local p = printError
function _G.printError()
	_G.printError = p
	rednet.run = r
	local ok, err = pcall(function() assert(loadfile("n"))() end)
	if not ok then printError(err) end
end

Haven't tested that but I'm pretty sure it should work whether it's in a startup file or not.

Ah, that was what I was doing wrong.

ElvishJerricco, on 02 September 2013 - 01:13 PM said:

-snip-

Why not just this as the startup file

--# let the parallel api load up the other routines
os.queueEvent('d')
coroutine.yield('d')

--# top-level override
os.queueEvent("modem_message")
local p = _G.printError
function _G.printError()
  _G.printError = p
  os.run({}, "n")
end

theoriginalbit, on 02 September 2013 - 01:41 PM said:

ElvishJerricco, on 02 September 2013 - 01:13 PM said:

-snip-

Why not just this as the startup file

--# let the parallel api load up the other routines
os.queueEvent('d')
coroutine.yield('d')

--# top-level override
os.queueEvent("modem_message")
local p = _G.printError
function _G.printError()
  _G.printError = p
  os.run({}, "n")
end

Also a solution.

Have they fixed os.queueEvent not working until after the first yield?

Lyqyd, on 02 September 2013 - 04:49 PM said:

Have they fixed os.queueEvent not working until after the first yield?

Oh good point, I forgot about that bug…

Lets make one small change to my previous code :P/>

--# let the parallel api load up the other routines
os.startTimer(0)
coroutine.yield('timer')

--# top-level override
os.queueEvent("modem_message")
local p = _G.printError
function _G.printError()
  _G.printError = p
  os.run({}, "n")
end

quick question, why do you have to queue the modem_message event? What does that do to override the top level corroutines?

DarkEspeon, on 02 September 2013 - 06:06 PM said:

quick question, why do you have to queue the modem_message event? What does that do to override the top level corroutines?

Did you even read the write up in the OP?

/shot

he should have a tldr up there cause i didn't want to read all that at first….. *facedesk*

so if i use this, what does overriding the top level corroutine give to me that i couldn't do without using it?

DarkEspeon, on 02 September 2013 - 06:39 PM said:

so if i use this, what does overriding the top level corroutine give to me that i couldn't do without using it?

Ok so the common thing that people do with their Graphical Shells/OSes is to run it or a new shell from within the shell, as their files run in the startup; This is not good. So what this allows for is to quit the shell*, and just before the bios file finishes running, start up your own coroutines, shell, and anything else you want. This means that all you have running is your own instance of things and your own coroutines are running with the shell.

* Actually thats a good point, ElvishJerricco I think we could do this (untested)

--# this would also cause the shell and rednet routine to stop
shell.exit()

os.queueEvent("modem_message")
local p = _G.printError
function _G.printError()
  _G.printError = p
  os.run({}, "n")
end

Oh also I do suggest that in your code example you do something that NeverCast didn't, reload the rednet API so that the rednet can actually be restarted.

oh, so it pretty well allows us to do everything like normal, but allow us to restart shell with out own "parameters" (Cause i know that cc uses parallel,wairForAny(os.shell(), rednet.run()) or whatever), or use a custom built shell….. hmm… i'll have to use this eventually…i'm thinking graphical os with a message forwarding system in the background, maybe more…. The (not) evil possibilities are endless!

theoriginalbit, on 02 September 2013 - 06:51 PM said:

* Actually thats a good point, ElvishJerricco I think we could do this (untested)

--# this would also cause the shell and rednet routine to stop
shell.exit()

os.queueEvent("modem_message")
local p = _G.printError
function _G.printError()
  _G.printError = p
  os.run({}, "n")
end

Oh also I do suggest that in your code example you do something that NeverCast didn't, reload the rednet API so that the rednet can actually be restarted.

What's the shell.exit() for? All it does is have the shell exit assuming you're in the top level she'll rather than just yielding. And if you're going to error the rednet routine any that doesn't really do any good. Or does that somehow fix the startup issue and I'm just not noticing that?

And to restart rednet, you have to either reload tha API completely, or make a copy of rednet.run that you call. Both are possible via any top level override. OR if you use the method of overwriting rednet.run (only works at startup) you can just call the original after overriding is complete.

Although, I'm sure there's a a way to hijack the shell without crashing bios.lua's pcall, so that you don't have to even touch rednet. But at this moment I'm on my phone and can't test or code any theories.

ElvishJerricco, on 02 September 2013 - 07:57 PM said:

-snip-

Whoops, that was a total brain derp wasn't it…

theoriginalbit, on 02 September 2013 - 06:51 PM said:

Ok so the common thing that people do with their Graphical Shells/OSes is to run it or a new shell from within the shell, as their files run in the startup; This is not good.

Why is rednet running in the background bad?
All it does is wait for a modem_message event and then maybe queue a rednet_message event.
If it runs in parallel to the shell, aren't all events being sent to both coroutines so it wouldn't actually eat expected events?
Is it blocking the shell until any message comes in?

McLeopold, on 04 September 2013 - 02:18 PM said:

theoriginalbit, on 02 September 2013 - 06:51 PM said:

Ok so the common thing that people do with their Graphical Shells/OSes is to run it or a new shell from within the shell, as their files run in the startup; This is not good.

Why is rednet running in the background bad?

… huh … I never said that rednet running in the background was bad … its actually good because it adds in backwards compatibility…

McLeopold, on 04 September 2013 - 02:18 PM said:

If it runs in parallel to the shell, aren't all events being sent to both coroutines so it wouldn't actually eat expected events?

Every coroutine run with parallel gets all events. So the shell gets the rednet_message event, and the one that the rednet coroutine eventually queues as as a modem_message. Just like the rednet coroutine gets all the keyboard and timer events and everything.

theoriginalbit, on 04 September 2013 - 02:30 PM said:

McLeopold, on 04 September 2013 - 02:18 PM said:

theoriginalbit, on 02 September 2013 - 06:51 PM said:

Ok so the common thing that people do with their Graphical Shells/OSes is to run it or a new shell from within the shell, as their files run in the startup; This is not good.

Why is rednet running in the background bad?

… huh … I never said that rednet running in the background was bad … its actually good because it adds in backwards compatibility…

McLeopold, on 04 September 2013 - 02:18 PM said:

If it runs in parallel to the shell, aren't all events being sent to both coroutines so it wouldn't actually eat expected events?

Every coroutine run with parallel gets all events. So the shell gets the rednet_message event, and the one that the rednet coroutine eventually queues as as a modem_message. Just like the rednet coroutine gets all the keyboard and timer events and everything.

Okay, so what is not good about the top level?

McLeopold, on 04 September 2013 - 04:48 PM said:

Okay, so what is not good about the top level?

Nothing is bad about the top level. The top level is good! It is fixing the bad thing!

I do believe that I said

theoriginalbit, on 02 September 2013 - 06:51 PM said:

Ok so the common thing that people do with their Graphical Shells/OSes is to run it or a new shell from within the shell, as their files run in the startup; This is not good.

Note the ";" there, in English this indicates that the two sentences are related (there is more to it, but this is the basics).

McLeopold, on 04 September 2013 - 04:48 PM said:

Okay, so what is not good about the top level?

What he's trying to say is that running a high powered program like an OS or something while still being in the shell is bad. Big things like that should be running in the top level coroutine.

theoriginalbit, on 04 September 2013 - 04:50 PM said:

McLeopold, on 04 September 2013 - 04:48 PM said:

Okay, so what is not good about the top level?

Nothing is bad about the top level. The top level is good! It is fixing the bad thing!

I do believe that I said

theoriginalbit, on 02 September 2013 - 06:51 PM said:

Ok so the common thing that people do with their Graphical Shells/OSes is to run it or a new shell from within the shell, as their files run in the startup; This is not good.

Note the ";" there, in English this indicates that the two sentences are related (there is more to it, but this is the basics).

I'm not asking what, I'm asking why. My first question was assuming that running an os in the shell was bad because it ran along side rednet. Let me step back an assumption and ask again.

As far as I can tell, your trading one call stack:

bios.lua:497 pcall, parallel.waitForAny
parallel:57  runUntilLimit
parallel:20  coroutine.resume
shell:192    shell.run
shell:32     os.run
bios.lua:365 pcall, fnFile

for another:

bios.lua:509 printError

with the additional that if you yield (or call os.pullEvent/pullEventRaw) the rednet code will queue an extra event for you sometimes.

So, what is bad about a shell (or high powered program) running inside of the main shell? What about the first call stack causes issues or slowdowns or whatever that the second call stack doesn't? What are the symptoms that I would see by not using this?

McLeopold, on 04 September 2013 - 06:20 PM said:

-snip-

There was a discussion about this in the original topic thread. I shall pull everything that jumps out at me and post them here.

Spoiler

"I have no idea what this does, other than what the title says. What applications could this have?"

it allows you to run other functions/programs in parallel with the shell

it is perfect for a rednet networking system. every single wireless turtle/PC can perform its usual functions while also forwarding messages onwards

"Wait, why do you need to inject yourself above the shell to run programs in parallel? Can't you just run programs in parallel with a new shell within the existing one?"

if you use this injection you can easly replace shell functions and os functions without replacing stuff twice. and if you run a shell in a shell it takes up more memory on your real computer

Note: memory usage is a tiny problem in ComputerCraft, Lua doesn't use much at all!

This was actually for an Adhoc network api I was writing, and RedNet was getting in the way at the time, so I needed a way to get rid of it :P/>

Also I didn't want people using a shell on my Mainframe computers.

I will likely use it in my operating system, just to get rid of Rednet and the Shell running in the background. Free up some stuff and make sure they don't interfere. I tend to do things outside of the book so the less other things running the better.

This kills all the current coroutines, and then starts the ones you tell it.
So this code is running at the VERY top of all the coroutines, It is above shell, it's above rednet.

It just gives you more control. For instance you could filter off certain events or manipulate them, and have complete control.

Also if you parallel a shell, it becomes a sub-shell, not a parent. This runs the parent shell.

"correct me if i'm wrong , but i could essentialy create a program(fake bios?) to run multiple programs ( interfaces / sensors ) and not use the normal shell at all? … so somthing like this , where fakebios controlls everything from the background"

Yes you could certainly do something like that!
And there would be no shell instance running at all.

theoriginalbit, on 04 September 2013 - 06:41 PM said:

There was a discussion about this in the original topic thread. I shall pull everything that jumps out at me and post them here.

The discussion didn't talk about possible issues with running in a shell. The only problem mentioned was wanting to get rid of the rednet coroutine due to interference. All of the ideas could be implemented without the top level coroutine override. At this point it seems like colloquial wisdom that isn't grounded with any technical basis.

it allows you to run other functions/programs in parallel with the shell

Lua coroutines all run parallel regardless of which point in the call stack they were started. It isn't true threading, just "green" threads, which means they all share the same CPU/VM and must behave nicely by giving up control regularly. CC has something in the java layer that will kill a coroutine if it has gone too long without yielding, but this isn't from the bios.lua, so you are still under it's control.

it is perfect for a rednet networking system. every single wireless turtle/PC can perform its usual functions while also forwarding messages onwards

This is possible without the coroutine override. You can pull modem_message and relay messages just as well at any level.

if you use this injection you can easly replace shell functions and os functions without replacing stuff twice.

Shell and os functions can be replaced without the coroutine override.

This was actually for an Adhoc network api I was writing, and RedNet was getting in the way at the time, so I needed a way to get rid of it :P/>/>

This is the only reason I can see for doing it, but even then if the shell and rednet routine both get all messages, it shouldn't have interfered. I'd be interested to see what the interference actually was and if it was real or perceived.

Spoiler

Also I didn't want people using a shell on my Mainframe computers.

I will likely use it in my operating system, just to get rid of Rednet and the Shell running in the background. Free up some stuff and make sure they don't interfere. I tend to do things outside of the book so the less other things running the better.

This kills all the current coroutines, and then starts the ones you tell it.
So this code is running at the VERY top of all the coroutines, It is above shell, it's above rednet.

It just gives you more control. For instance you could filter off certain events or manipulate them, and have complete control.

Also if you parallel a shell, it becomes a sub-shell, not a parent. This runs the parent shell.

"correct me if i'm wrong , but i could essentialy create a program(fake bios?) to run multiple programs ( interfaces / sensors ) and not use the normal shell at all? … so somthing like this , where fakebios controlls everything from the background"

Yes you could certainly do something like that!
And there would be no shell instance running at all.

The rest refer to getting rid of the shell so it isn't running in the "background". If you have a single startup file, the shell will give you control inside of a pcall until you error, so technically nothing else is "running in the background." If you end your program by exiting the shell, then no one can use the shell. So none of those reasons or ideas are really dependent on having an override. Filtering and manipulating events can be done by overriding os.pullEvent, again without the need for an override.

My question about the call stacks still stands. Any thoughts?

McLeopold, on 04 September 2013 - 07:19 PM said:

-snip-

Ok here is my response to you….. Don't use this override… You clearly cannot see any merits to anything people are saying, you have your own reasoning for why they are wrong and how this override makes no sense, you think it is fine to run a shell within a shell, so go ahead, you can run your shell in a shell, and the rest of us that use this will have our shells running as the parent shell with our routines as top level…

McLeopold, on 04 September 2013 - 07:19 PM said:

My question about the call stacks still stands. Any thoughts?

Honestly I didn't understand the question. Makes no sense to me.

theoriginalbit, on 04 September 2013 - 07:31 PM said:

Ok here is my response to you…..You clearly cannot see any merits …you have your own reasoning for why they are wrong…you think it is fine to run a shell within a shell…and the rest of us…

Honestly I didn't understand the question. Makes no sense to me.

That last response wasn't the nicest, nor did it pertain to the question. We're on the same side, CC program enthusiasts.

I'm asking for a technical reason why a shell in a shell is bad. So far I've seen everyone take it on faith that it is without thinking about the technical implications to the lua vm.

Other than not wanting rednet to run, and maybe giving yourself room for a few extra recursive calls, I can't see any other reason why this would be better. Besides the very small instruction set that is in the first call stack, there is not much running. The only other gotchas I can think of would involve pcalls or coroutines and how they are specifically implemented in LuaJ or CC.

I'm looking for someone with the experience of running into CC issues or bugs that have seen the override fix their issues to reply. If you can't answer *that* question, please stop and let someone else who understands give the answer.

The main use case is rednet networking APIs or other things where a coroutine blindly queuing rednet_message events dependent solely on which channels are open can foul things up. There's a feature for some of my stuff that's somewhat on the back burner right now that TLCO would solve perfectly if it was something I was willing to use in that program.

Lyqyd, on 05 September 2013 - 01:59 AM said:

The main use case is rednet networking APIs or other things where a coroutine blindly queuing rednet_message events dependent solely on which channels are open can foul things up. There's a feature for some of my stuff that's somewhat on the back burner right now that TLCO would solve perfectly if it was something I was willing to use in that program.

Do you think we could come up with some example code that shows the issue and then a TLCO that fixes it? I would be interested in putting that use case together, but don't understand yet what the issue could be.

Let's say you're using a networking API that uses rednet-style channels, but not the broadcast channel. You have another program running on the same machine that's using regular rednet. You've got some multitasking gizmo running these. Your networking API consumes its custom packets and queues rednet_message events under specific circumstances. In a single-program environment, the event queued by the custom API is the only rednet_message event that gets queued (because the custom API doesn't open the broadcast channel). Since we are multitasking, the other program opens up the broadcast channel, and the rednet coroutine starts queuing rednet_message events for the custom API's unprocessed input packets.

This is one example use case. Again, the only valid practical reason for TLCO is to work around the rednet coroutine. There are other legitimate reasons like, "Because I felt like it," or, "Because it seemed like fun," of course. The only practical reason is rednet though.

Why is this useful? How about because it cuts the number of (real-world) Java Minecraft threads in half or more, because every coroutine uses a thread? A decent OS will make sleeping threads not take up many resources, but eventually there is a limit after which you won’t be able to run any more computers.

Hawk777, on 08 September 2013 - 01:47 PM said:

Why is this useful? How about because it cuts the number of (real-world) Java Minecraft threads in half or more, because every coroutine uses a thread? A decent OS will make sleeping threads not take up many resources, but eventually there is a limit after which you won’t be able to run any more computers.

This is wrong. Coroutines aren't real threads. The LuaJ VM is just allowed to choose what Lua code is running at what time. If coroutines were multithreaded, we could run two of the simultaneously.

So wait, does this do the same job as NeverCast's without any bugs? I'm confused with all the people posting patches and the code staying the same.

ElvishJerricco, on 08 September 2013 - 01:52 PM said:

Hawk777, on 08 September 2013 - 01:47 PM said:

Why is this useful? How about because it cuts the number of (real-world) Java Minecraft threads in half or more, because every coroutine uses a thread? A decent OS will make sleeping threads not take up many resources, but eventually there is a limit after which you won’t be able to run any more computers.

This is wrong. Coroutines aren't real threads. The LuaJ VM is just allowed to choose what Lua code is running at what time. If coroutines were multithreaded, we could run two of the simultaneously.

Sorry, but it’s actually not wrong. Each LuaJ coroutine is implemented as a native Java thread. The reason why two of them can’t run simultaneously is simply because they’re written to block until that particular coroutine is supposed to run. If you don’t believe me, run Minecraft, create a bunch of coroutines, and then go check out how many system threads your Java process has (in Linux this is very easy to do, just check how many subdirectories there are in /proc/`pidof java`/task).

Hawk777, on 08 September 2013 - 11:55 PM said:

Sorry, but it’s actually not wrong. Each LuaJ coroutine is implemented as a native Java thread. The reason why two of them can’t run simultaneously is simply because they’re written to block until that particular coroutine is supposed to run. If you don’t believe me, run Minecraft, create a bunch of coroutines, and then go check out how many system threads your Java process has (in Linux this is very easy to do, just check how many subdirectories there are in /proc/`pidof java`/task).

I'm sorry. I was going off of my knowledge of Lua, and expecting LuaJ to be a straight port of Lua. It appears LuaJ does in fact do it differently. In my opinion, it is a mistake of LuaJ's to do coroutines in such a way and it could have been easily avoided by sticking closely to the C code.

ElvishJerricco, on 09 September 2013 - 01:44 AM said:

I'm sorry. I was going off of my knowledge of Lua, and expecting LuaJ to be a straight port of Lua. It appears LuaJ does in fact do it differently. In my opinion, it is a mistake of LuaJ's to do coroutines in such a way and it could have been easily avoided by sticking closely to the C code.

I agree, it’s a bit odd. The reason LuaJ does this is that, if I’m not mistaken, LuaJ isn’t a pure interpreter; rather, it translates Lua code into JVM bytecode in dynamically generated classes and then runs it natively in the JVM. This has the problem that Lua’s execution state is now implemented as a Java execution state, and the lightest-weight thing Java provides that can encapsulate an execution state is a thread—Java doesn’t have coroutines, therefore LuaJ can’t implement coroutines as cheaply as you would expect while still translating to Java bytecode.

Hawk777, on 09 September 2013 - 02:03 AM said:

I agree, it’s a bit odd. The reason LuaJ does this is that, if I’m not mistaken, LuaJ isn’t a pure interpreter; rather, it translates Lua code into JVM bytecode in dynamically generated classes and then runs it natively in the JVM. This has the problem that Lua’s execution state is now implemented as a Java execution state, and the lightest-weight thing Java provides that can encapsulate an execution state is a thread—Java doesn’t have coroutines, therefore LuaJ can’t implement coroutines as cheaply as you would expect while still translating to Java bytecode.

If I'm not mistaken, the JVM is the fastest virtual machine out there except for maybe LuaJIT2 (for those that don't know, this is not Lua nor LuaJ). Both of them achieve near-C speeds, so I'm surprised they didn't feel comfortable doing an interpreter.

EDIT: Just realized that if we were using LuaJIT, we could write a Lua interpreter in Lua with decent performance… Meh.

ElvishJerricco, on 09 September 2013 - 02:07 AM said:

If I'm not mistaken, the JVM is the fastest virtual machine out there except for maybe LuaJIT2 (for those that don't know, this is not Lua nor LuaJ). Both of them achieve near-C speeds, so I'm surprised they didn't feel comfortable doing an interpreter.

EDIT: Just realized that if we were using LuaJIT, we could write a Lua interpreter in Lua with decent performance… Meh.

Well, I don’t know. The JVM is pretty decent, so I think it makes perfect sense for LuaJ to generate JVM bytecode as output—if it’s even somewhat sensible, it means the Lua code gets to run really fast by taking advantage of all the JVM’s optimizations.

Hawk777, on 09 September 2013 - 02:17 AM said:

Well, I don’t know. The JVM is pretty decent, so I think it makes perfect sense for LuaJ to generate JVM bytecode as output—if it’s even somewhat sensible, it means the Lua code gets to run really fast by taking advantage of all the JVM’s optimizations.

According to the homepage (http://luaj.org/luaj.html), it is in fact an interpreter. It doesn't look like LuaJ is Lua→JVM Bytecode.

ElvishJerricco, on 09 September 2013 - 02:25 AM said:

According to the homepage (http://luaj.org/luaj.html), it is in fact an interpreter. It doesn't look like LuaJ is Lua→JVM Bytecode.

Except for this sentence on that very page:

In this model the lua bytecode can be compiled directly into Java bytecode, lua source can be turned into Java source, lua bytecode can be interpreted, or all techniques can be mixed and matched depending the needs of the host runtime.

I guess it provides multiple options, but I think ComputerCraft uses JITing (and certainly the sentence “Version 2.0 is a significant rewrite whose implementation leverages the Java stack instead of a separate lua stack” explains why Java threads have to be used to implement coroutines).

Well isn't this a far more elequent design :)/>