This is a read-only snapshot of the ComputerCraft forums, taken in April 2020.
NeverCast's profile picture

Top Level Coroutine Override

Started by NeverCast, 02 November 2012 - 02:02 PM
NeverCast #1
Posted 02 November 2012 - 03:02 PM
Hey Guys,

This has to be like my first topic or something, maybe even my first post, I'm not sure.
Anyway recently I wanted to design a networking API, but the problem was I couldn't run a top level coroutine, after talking to people and disputing with them on IRC for the last few days and being told it was very difficult or impossible I devised a way to launch functions at top level, by basically messing around with the shell and bios, getting them to quit and then launch my own code.

There is a lot of possibilities with this, and the full power could probably only be realised in the hands of an Advanced/Pro Lua Coder, but here is my attempt at some community contribution. I only started learning Lua a few weeks to a month ago. So I welcome any advice or input.

I've not made any test code for this, but it should all work how I've described it. It should let you spawn whatever top level programs you want to.

The example I have loads the shell per normal, but does not start rednet for redstone input. Just to show that it works.

You're welcome to use the code how you like, but I ask that you keep my credit on it.
You should use this as your startup file, it might work not as startup file, but I never tested it that way.

http://pastebin.com/2SQAzrWi

Enjoy, and please let me know what you think.

Edit: Re-uploaded to Pastebin via my account so I can start pushing updates and fixes.
Cloudy #2
Posted 06 November 2012 - 08:31 AM
Cool hack!
Sammich Lord #3
Posted 06 November 2012 - 11:50 AM
Cool hack!
Defiantly!
wraithbone #4
Posted 06 November 2012 - 10:17 PM
This is really great, earlier this year I was trying to run a coroutine that did things while the shell ran the front end. But kept things going in the BG.
This is pretty much what I wanted. Awesome.
Cranium #5
Posted 07 November 2012 - 06:21 AM
I have no idea what this does, other than what the title says. What applications could this have?
KaoS #6
Posted 07 November 2012 - 07:39 AM
it allows you to run other functions/programs in parallel with the shell
Cranium #7
Posted 08 November 2012 - 06:23 AM
Well, color me noob, but I still don't know how I can use this :P/>/>
Leo Verto #8
Posted 08 November 2012 - 06:35 AM
Awesome, now to find a tekkit server to test it :P/>/>
wilcomega #9
Posted 08 November 2012 - 07:10 AM
hey, this is really really cool man. how can i print my own stuff instead of changing the default messages?
KaoS #10
Posted 08 November 2012 - 07:32 AM
it is perfect for a rednet networking system. every single wireless turtle/PC can perform its usual functions while also forwarding messages onwards
kazagistar #11
Posted 08 November 2012 - 08:48 AM
Wait, why do you need to inject yourself above the shell to run programs in parallel? Can't you just run programs in parallel with a new shell within the existing one?
wilcomega #12
Posted 08 November 2012 - 08:59 AM
if you use this injection you can easly replace shell functions and os functions without replacing stuff twice. and if you run a shell in a shell it takes up more memory on your real computer
PixelToast #13
Posted 08 November 2012 - 09:02 AM
if you use this injection you can easly replace shell functions and os functions without replacing stuff twice. and if you run a shell in a shell it takes up more memory on your real computer
you mean the couple KB of lua bytecode and function environments? pah.
kazagistar #14
Posted 08 November 2012 - 09:07 AM
I mean, this seems like optimizing for a non-issue (since each shell will use up maybe a millionth of your memory), but it is still pretty neat.
PixelToast #15
Posted 08 November 2012 - 09:12 AM
I mean, this seems like optimizing for a non-issue (since each shell will use up maybe a millionth of your memory), but it is still pretty neat.
i usually only optimize cpu and not memory >_>
kazagistar #16
Posted 08 November 2012 - 09:17 AM
i usually only optimize cpu and not memory >_>

I optimize CPU bound code for CPU and memory bound code for memory (which almost never happens in CC, it is all interrupt/hardware bound stuff in general) and the rest for understandability and functionality.
NeverCast #17
Posted 07 January 2013 - 07:48 PM
This was actually for an Adhoc network api I was writing, and RedNet was getting in the way at the time, so I needed a way to get rid of it :P/>

Also I didn't want people using a shell on my Mainframe computers.

Wow it's so good to see all the great replies, Thanks guys :)/>
bbqroast #18
Posted 15 January 2013 - 08:56 AM
The amount of control we get over the upper level stuff in CC has always saddened me. I can't work it out, but does it require a change to any of the ROM files?
KaoS #19
Posted 15 January 2013 - 06:28 PM
man I took a look through the code and I gotta say I'm super confused, on line 36 you make the table prevState and nowhere else in the program do you add any values to it… how do you then call functions and values from that table throughout the program?? :huh:/>

I must admit the getParentShell is quite smart though. I have tried replacing the printError function in my programs but it refuses to work on ccEmu, not sure about real CC :(/>

the below code still uses the normal printError

local pE=printError
rawset(_G,printError,function(...)
print("functional")
sleep(5)
printError=pE
return printError(...)
end)
printError('me')

EDIT, derpity derp derp…. no quotes -_-/> works if you use

local pE=printError
local tO={}
for i=1,5 do
if rawget(getfenv(i),"printError")~=nil and not tO[getfenv(i)] then
  tO[getfenv(i)]=true
  rawset(getfenv(i),"printError",function(...)
   print("functional")
   sleep(5)
   printError=pE
   return printError(...)
  end)
end
end

you can then replace the print and sleep with whatever you need
NeverCast #20
Posted 15 January 2013 - 09:12 PM
The amount of control we get over the upper level stuff in CC has always saddened me. I can't work it out, but does it require a change to any of the ROM files?

Nope that's the point, you don't have to change ROM files, I've recently been looking in to something more advanced than this for overriding the bios.

man I took a look through the code and I gotta say I'm super confused, on line 36 you make the table prevState and nowhere else in the program do you add any values to it… how do you then call functions and values from that table throughout the program?? :huh:/>

I must admit the getParentShell is quite smart though. I have tried replacing the printError function in my programs but it refuses to work on ccEmu, not sure about real CC :(/>

the below code still uses the normal printError

local pE=printError
rawset(_G,printError,function(...)
print("functional")
sleep(5)
printError=pE
return printError(...)
end)
printError('me')

EDIT, derpity derp derp…. no quotes -_-/> works if you use

local pE=printError
local tO={}
for i=1,5 do
if rawget(getfenv(i),"printError")~=nil and not tO[getfenv(i)] then
  tO[getfenv(i)]=true
  rawset(getfenv(i),"printError",function(...)
   print("functional")
   sleep(5)
   printError=pE
   return printError(...)
  end)
end
end

you can then replace the print and sleep with whatever you need

I'm sorry you don't understand it :P/>
I wish I'd know about rawset back when I wrote this code, fortunately there was no metatables that are using __newindex so it didn't make much difference.
I'm glad people are still looking at this post. I look forward to it one day being useful.

I will likely use it in my operating system, just to get rid of Rednet and the Shell running in the background. Free up some stuff and make sure they don't interfere. I tend to do things outside of the book so the less other things running the better.
KaoS #21
Posted 15 January 2013 - 10:25 PM
Could you possibly explain what you are doing with the prevState table?
NeverCast #22
Posted 16 January 2013 - 07:59 AM
In the inject function I'm assigning the original functions in to the prevState table, and then in the recover function I put them back :)/>
KaoS #23
Posted 16 January 2013 - 08:15 PM
there they are lol. makes perfect sense… I didn't see where you assigned them a value :D/> thanks
NeverCast #24
Posted 16 January 2013 - 08:28 PM
You're very welcome :D/>
RunasSudo-AWOLindefinitely #25
Posted 16 January 2013 - 09:20 PM
Wow. That's… Incredibly virus-like. It's just the way you say "injection" makes it sound like a virus. No doubt this could have malicious applications…

Really neat hack! Might be really useful for something like an OS…
NeverCast #26
Posted 29 January 2013 - 11:14 AM
It can very much have malicious applications, Perfect keyloggers can be made with this. A shell with no parent shell could be running, making it look like the user just has a normal computer. Mean while in the background, an injected function, keeping quiet. Logs everything you do!
Pinkishu #27
Posted 29 January 2013 - 01:37 PM
I still don't really get how the whole thing is different from just using parallel to run a shell and a background-task :/

@Keylogger: uh well.. won't you still see the code that starts and injects anyway? or if you hide that, then you could have still just done it without the injecting stuff, or do i get something wrong
I mean its a nice hack, I just don't really get its use :D/>
NeverCast #28
Posted 29 January 2013 - 04:24 PM
This kills all the current coroutines, and then starts the ones you tell it.
So this code is running at the VERY top of all the coroutines, It is above shell, it's above rednet.

It just gives you more control. For instance you could filter off certain events or manipulate them, and have complete control.

Also if you parallel a shell, it becomes a sub-shell, not a parent. This runs the parent shell.

Maybe someone else can explain it better, I'm not so good. I just type code :P/>
ElvishJerricco #29
Posted 30 January 2013 - 01:16 AM
Wow. I'm gonna have to study how you did this and see if it's applicable to my current not-an-OS project. Really great.
Cloudy #30
Posted 30 January 2013 - 01:40 AM
Well the rednet coroutine shall shortly be dying, so this is short lived ;)/>
ElvishJerricco #31
Posted 30 January 2013 - 06:05 AM
Well the rednet coroutine shall shortly be dying, so this is short lived ;)/>/>

First of all, thank you. This implies cables an I like that =)

Second of all, my plans for this are actually to build a custom shell, rather than anything with rednet. So this shan't be short lived for me.
NeverCast #32
Posted 30 January 2013 - 08:47 AM
Well the rednet coroutine shall shortly be dying, so this is short lived ;)/>

Awh cloudy, It'll still have it's uses :P/>
I just no longer will need to worry about the rednet coroutine getting in the way of my strict timing redstone :D/>

I'm really excited for the next version of CC, Because every version is great. Unfortunately I'm very much in the dark about what to expect. I don't like suspense! :(/>
etopsirhc #33
Posted 30 January 2013 - 09:55 AM
this looks extreamly interesting, and if i'm reading it properly would pr perfect for os's

correct me if i'm wrong , but i could essentialy create a program(fake bios?) to run multiple programs ( interfaces / sensors ) and not use the normal shell at all?

so somthing like this , where fakebios controlls everything from the background

fakeBios
  screen
  general input
  redstone output
NeverCast #34
Posted 30 January 2013 - 10:06 AM
this looks extreamly interesting, and if i'm reading it properly would pr perfect for os's

correct me if i'm wrong , but i could essentialy create a program(fake bios?) to run multiple programs ( interfaces / sensors ) and not use the normal shell at all?
so somthing like this , where fakebios controlls everything from the background

...

Yes you could certainly do something like that!
And there would be no shell instance running at all.
etopsirhc #35
Posted 30 January 2013 - 10:10 AM
this looks extreamly interesting, and if i'm reading it properly would pr perfect for os's

correct me if i'm wrong , but i could essentialy create a program(fake bios?) to run multiple programs ( interfaces / sensors ) and not use the normal shell at all?
so somthing like this , where fakebios controlls everything from the background

...

Yes you could certainly do something like that!
And there would be no shell instance running at all.

sweet! gonna start work on that now then =D
*been putting it off way too long*
NeverCast #36
Posted 30 January 2013 - 10:13 AM
I look forward to seeing it!
TheVarmari #37
Posted 31 January 2013 - 05:40 AM
How would I use this?
NeverCast #38
Posted 31 January 2013 - 11:39 AM
That would depend on what you wanted to use it for.
TheVarmari #39
Posted 01 February 2013 - 03:10 AM
That would depend on what you wanted to use it for.

Say… Could I make this override shell.run or just the os.run?
I tried it with shell, but it looks like it would run after my line, negating the effect…
NeverCast #40
Posted 01 February 2013 - 08:54 AM
You can already override shell.run and os.run without anything fancy. Just re-declare them as a new function.

This removes shell, allowing your code to run INSTEAD of shell, instead of on top of the shell.

Alternatively it can run next to the shell like a Hardware Driver, instead of again, as a program under shell.
FuuuAInfiniteLoop(F.A.I.L) #41
Posted 15 April 2013 - 11:50 AM
I have no idea what this does, other than what the title says. What applications could this have?
You can create a real OS, cause it dont run with craftOS since you terminated
Espen #42
Posted 15 April 2013 - 02:30 PM
That isn't entirely correct, it still runs on CraftOS.
It just kills the parent shell and allows one to let other programs run at the top level of CraftOS.

At least I wouldn't say the shell is CraftOS.
The shell is used to display that, but IMO CraftOS really is the custom Lua environment within which "bios.lua" and "shell" are executed.
ElvishJerricco #43
Posted 15 April 2013 - 02:43 PM
That isn't entirely correct, it still runs on CraftOS.
It just kills the parent shell and allows one to let other programs run at the top level of CraftOS.

At least I wouldn't say the shell is CraftOS.
The shell is used to display that, but IMO CraftOS really is the custom Lua environment within which "bios.lua" and "shell" are executed.

Actually, the way this works, bios.lua gets to its last line. And its last function call (os.shutdown) calls the custom code. So it's still under CraftOS. But only just barely.
Espen #44
Posted 15 April 2013 - 11:09 PM
Actually, the way this works, bios.lua gets to its last line. And its last function call (os.shutdown) calls the custom code. So it's still under CraftOS. But only just barely.
Yes, it might've executed its last call, but as long as the custom code that was called for that hasn't terminated, the bios.lua will technically still run.
After the custom code has terminated without calling the (proper) os.shutdown(), execution would continue in bios.lua, only to find there is nothing more to execute and then bios.lua would terminate.

I haven't traced program execution yet though, I've only surmised this execution flow from the code itself.
So take it with a grain of salt and please correct me if I'm wrong.

But apart from that I mentioned that I considered CraftOS not to be bios.lua, but the Lua environment in which both bios.lua and the shell run in.
The LuaJ backbone, so to speak. And under that definition it's irrelevant if the bios.lua still runs or not when trying to determine what custom Lua code does or doesn't run under CraftOS. It is the environment that makes running any CC programs, bios.lua included, possible at all.

You could perhaps say that bios.lua and the shell are part of CraftOS by setting up CraftOS in such a way that user input is possible.
But they alone aren't CraftOS.
Just trying to clear up any misunderstanding, so no offense. :)/>
If you disagree with my definition of what constitutes CraftOS, then I'd like to hear your arguments for that.
FuuuAInfiniteLoop(F.A.I.L) #45
Posted 17 April 2013 - 03:20 PM
Actually, the way this works, bios.lua gets to its last line. And its last function call (os.shutdown) calls the custom code. So it's still under CraftOS. But only just barely.
Yes, it might've executed its last call, but as long as the custom code that was called for that hasn't terminated, the bios.lua will technically still run.
After the custom code has terminated without calling the (proper) os.shutdown(), execution would continue in bios.lua, only to find there is nothing more to execute and then bios.lua would terminate.

I haven't traced program execution yet though, I've only surmised this execution flow from the code itself.
So take it with a grain of salt and please correct me if I'm wrong.

But apart from that I mentioned that I considered CraftOS not to be bios.lua, but the Lua environment in which both bios.lua and the shell run in.
The LuaJ backbone, so to speak. And under that definition it's irrelevant if the bios.lua still runs or not when trying to determine what custom Lua code does or doesn't run under CraftOS. It is the environment that makes running any CC programs, bios.lua included, possible at all.

You could perhaps say that bios.lua and the shell are part of CraftOS by setting up CraftOS in such a way that user input is possible.
But they alone aren't CraftOS.
Just trying to clear up any misunderstanding, so no offense. :)/>
If you disagree with my definition of what constitutes CraftOS, then I'd like to hear your arguments for that.
I have understood that CraftOS is the bios.lua and the java part cause the shell, is only a program to interact with it, and for the java part, only the apis like fs, term, etc can be consireded part of CraftOS cause the other part are the blocks and the luaJ part
TheOddByte #46
Posted 18 April 2013 - 12:00 PM
Awesome.. Still trying to figure out how you did this :P/>
theoriginalbit #47
Posted 16 June 2013 - 05:24 AM
Ok so I've been using this lately for one of my projects and I've noticed 3 things:
  1. There are some times where the function getParentShell() does not actually find the parent shell. I'm unable to find the cause, but it has occurred a few times now;
  2. The rednet coroutine is killed, like it should be. But attempting to restart it will cause an error saying "rednet:87:rednet is already running" because of the local variable bRunning;
  3. This one is a note to all: There is an entire function in the code that is not used and therefore could be removed. `_replSleep`;
Anyone have any ideas for #1 and #2?
KaoS #48
Posted 16 June 2013 - 05:33 AM
1: you are using next(env) to check what was first put into the table. This is not a completely reliable method but in this case it will work fine. I think the issue is the for loop only goes from 1 to 5


local function getParentShell()
  local at=0
  while true do
    at=at+1
    local ok,env = pcall(function() return getfenv(at) end)
    if not ok then break end
    if table.size(env) == 1 then
	  local i,v = next(env) -- Grab first
	  if i == "shell" then
	    return v
	  end
    end   
  end
  return nil
end
should hopefully work

2: you have to unload and reload the rednet API, I ran into this exact issue too
theoriginalbit #49
Posted 16 June 2013 - 05:36 AM
1: Ahh, I thought it may have been something along those lines, but not being 100% confident in environments I wasn't too sure.
2: oh of course. derp. there is another one to add to the list of KaoS dealing with my derps…

Thanks KaoS :)/>
jesusthekiller #50
Posted 16 June 2013 - 08:34 AM
Made an test OS using it. Pretty nice!
NeverCast #51
Posted 21 July 2013 - 11:16 PM
It's awesome to see people using this!
I'll be using it shortly as well and combining it with my mesh network api. Which I'll also be posting to the forums.

Hey the last post on this was my birthday, yay!
theoriginalbit #52
Posted 22 July 2013 - 12:48 AM
I'll be using it shortly as well
Maybe you should make the improvements and bug fixes that have been suggested and update it too xD


for example:
— fix `getParentShell` to KaoS' solution
— unload the rednet api and reload it ( so that others don't have to xP )
— remove the `_replSleep` function
— I'm sure there is more you can fix
Edited on 21 July 2013 - 10:51 PM
Lyqyd #53
Posted 22 July 2013 - 01:02 AM
Heh, actually, it seems like half the point of using something like this would be to load up your own rednet API replacement.
Dejected #54
Posted 22 July 2013 - 01:37 AM
Cool Hack
KaoS #55
Posted 22 July 2013 - 08:19 AM
Heh, actually, it seems like half the point of using something like this would be to load up your own rednet API replacement.

well I load up default rednet and an additional addon. no reason not to include what works fine :)/>

I would also advise you to permit users to add/remove/edit coroutines at the top level after the override as this is incredibly useful. This will require you to use the coroutine API directly but it also allows you to make events queued for specific coroutines etc
theoriginalbit #56
Posted 22 July 2013 - 12:03 PM
Heh, actually, it seems like half the point of using something like this would be to load up your own rednet API replacement.
True…. boolean flag! xD

I would also advise you to permit users to add/remove/edit coroutines at the top level after the override as this is incredibly useful. This will require you to use the coroutine API directly but it also allows you to make events queued for specific coroutines etc
Or the user could could just do what I did with that program I sent you a while ago, and when they use it remove the parallel.waitForAny and put in their own call….. of course the other option, passing in a function pointer, and if it's not there make it parallel.waitForAny, and if it's there use it instead :D/>
KaoS #57
Posted 23 July 2013 - 07:54 AM
Or the user could could just do what I did with that program I sent you a while ago, and when they use it remove the parallel.waitForAny and put in their own call….. of course the other option, passing in a function pointer, and if it's not there make it parallel.waitForAny, and if it's there use it instead :D/>
Ah well you essentially did the same thing except customized so every parallel call could edit itself after launch. A very effective system too
Geforce Fan #58
Posted 12 August 2013 - 06:37 PM
Sorry to gravedig this, but when I use this I get an error on line 158 of bios and can't type anything or use control + r, control + s, or even control + t. I'm running the program unedited. Help.
edit: when I run it plain in a folder(not on startup) it gives me an error on 166 too.

edit: re-reading the post, it's meant to be at startup, and it works fine.
Zudo #59
Posted 02 September 2013 - 03:58 AM
This is really useful, thanks.

Hey the last post on this was my birthday, yay!

Which is the day after my birthday!
theeboris #60
Posted 12 March 2014 - 09:58 PM
Maybe an retarded question, what can I do with this?
theoriginalbit #61
Posted 12 March 2014 - 10:16 PM
Maybe an retarded question, what can I do with this?
read through the last 3 pages of replies, there is no need for us to explain it again.
lukasdragon #62
Posted 09 April 2014 - 04:31 AM
I am trying to create a computer remote control system using the new portable computers, How ever I am trying to secretly inject the client into computer alongside the shell, However the computer just wont boot up? is this method outdated?
Lyqyd #63
Posted 09 April 2014 - 03:25 PM
That sounds a bit… malicious.
Piorjade #64
Posted 16 September 2016 - 08:30 AM
Sadly this doesn't work for me (could be outdated :P/>)
Sewbacca #65
Posted 17 September 2016 - 05:11 PM
What? What does getfenv(numberValue)?
And work this hack without getfenv?
I think not, so it isn't usefull, when we update to Lua 5.2 :(/>
apemanzilla #66
Posted 22 September 2016 - 11:51 PM
Here's a super simple and up-to-date version that doesn't use getfenv.

local _pe = _G.printError
function _G.printError()
  _G.printError = _pe
  --# do top level stuff, i.e. start new shell coroutine
end
os.queueEvent("terminate")
Edited on 22 September 2016 - 09:58 PM
Sewbacca #67
Posted 23 September 2016 - 03:47 PM
Here's a super simple and up-to-date version that doesn't use getfenv.

local _pe = _G.printError
function _G.printError()
  _G.printError = _pe
  --# do top level stuff, i.e. start new shell coroutine
end
os.queueEvent("terminate")
Nice idea =)
Piorjade #68
Posted 23 September 2016 - 05:23 PM
Here's a super simple and up-to-date version that doesn't use getfenv.

local _pe = _G.printError
function _G.printError()
  _G.printError = _pe
  --# do top level stuff, i.e. start new shell coroutine
end
os.queueEvent("terminate")
Nice idea =)

using os.queueEvent("rednet_message", 0) works too
apemanzilla #69
Posted 23 September 2016 - 08:31 PM
Here's a super simple and up-to-date version that doesn't use getfenv.

local _pe = _G.printError
function _G.printError()
  _G.printError = _pe
  --# do top level stuff, i.e. start new shell coroutine
end
os.queueEvent("terminate")
Nice idea =)

using os.queueEvent("rednet_message", 0) works too

More characters :)/>
Anavrins #70
Posted 23 September 2016 - 11:29 PM
Depending on the use case, let's say, somehow you're running a shell inside a shell, simply doing queueEvent("terminate") will not cause a TLCO, instead it will terminate the coroutine of the nested shell.
Using queueEvent("modem_message", 0) will ensure that you're terminating the top-level shell, no matter how many layer of shells you're in.
Edited on 23 September 2016 - 09:30 PM
apemanzilla #71
Posted 24 September 2016 - 04:03 PM
Depending on the use case, let's say, somehow you're running a shell inside a shell, simply doing queueEvent("terminate") will not cause a TLCO, instead it will terminate the coroutine of the nested shell.
Using queueEvent("modem_message", 0) will ensure that you're terminating the top-level shell, no matter how many layer of shells you're in.

If you're running it first thing in startup, which is the main place you would use a TLCO, then it will work fine.
Anavrins #72
Posted 24 September 2016 - 07:44 PM
I know, what I'm saying is that using the modem event will work 100% of the time, no matter what kind of shenanigans they do before running it.
Running a shell in a shell is just one example in which case using the terminate event can fail.
Edited on 24 September 2016 - 07:10 PM