LoadAPI with active code which runs in parallel with the system/code which is loading the API

The title pretty much says it but I've also taken the time to make a very clean and logical layout of the process which I'm envisioning. However, to no avail I've been trying to make an API which will create and init a coroutine. The coroutine is effectively a loop which will yield after each iteration, which returns control to the API. My issue is that I need to return control to the program/OS which is LOADING the API and run that program/OS in a multi-task fashion with the COROUTINE. So far the closest I've come to making this work without having the programmer have to manually manage the coroutine in their programs is either creating a startup file which runs the OS in 'parallel' with loading the API which defeats the purpose of it being an API AND the OS will never yield which means that the coroutine will stay suspended forever therefore effectively killing it.

Any and all help with this is immensely appreciated, I've also included the process diagram below.


http://s24.postimg.org/oac2n66tx/Perfect_API_Page_1.jpg

That won't work in CraftOS. You'd need to create your own OS to add this functionality.

Er, could you not override os.pullEvent() so that it acts as a coroutine manager, in addition to its normal functionality?

Lyqyd, on 13 July 2015 - 06:56 AM said:

That won't work in CraftOS. You'd need to create your own OS to add this functionality.

Yea that was my conclusion also, the problem is, I simply am only trying to make an API to do this. Ugh the things you just sometimes cannot do.

Bomb Bloke, on 13 July 2015 - 07:16 AM said:

Er, could you not override os.pullEvent() so that it acts as a coroutine manager, in addition to its normal functionality?

That's a fantastic idea, which I had not thought of nor yet tried so I will test that tomorrow and get back to you! Although I guess I'd have to learn how to force CraftOS to yield….. : /

——

Thanks for the quick replies guys!

jacky500, on 13 July 2015 - 07:21 AM said:

Although I guess I'd have to learn how to force CraftOS to yield….. : /

You don't have to - it already does it all the time! :)/>

If this is the case why is this code dysfunctional?

--[[file as coroutine]]
while true do 
  print('coroutine is live')
  coroutine.yield()
end

>lua
lua>parallel.waitForAny(os.loadAPI('file'),os.run({},"/rom/programs/shell"))
coroutine live
CraftOS 1.7
>[Enter]
>[Enter]
>exit
coroutine live --Presses [Enter]
coroutine live --Presses [Enter]
coroutine live --Presses [Enter]
coroutine live --Presses [Enter]
coroutine live --Presses [Enter]

Shouldn't CraftOS technically be yielding when I press [Enter] since it is always using pullEvent() and pullEvent() uses pullEventRaw() which in turn uses coroutine.yield() or is this somehow handled by CraftOS internally (possibly spawns its own coroutine to handle the event system?) ((sorry I only am doing any code for computercraft because the community is large and is a good way for me to help and get help and get to know ppl + I get to learn awesome stuff all the time lol))

parallel.waitForAny() expects you to pass it at least two function pointers. If you try to execute the functions yourself, than you'll instead end up passing the parallel API whatever those functions return. Your code isn't even managing to start parallel.waitForAny() - it's instead busy trying to resolve the arguments you wish to pass to it.

Now if you wrapped those calls inside function declarations, like so, then it'd do something closer to what you're wanting:

parallel.waitForAny(function() os.loadAPI('file') end, function() os.run({},"/rom/programs/shell") end)

It's worth pointing out that the technique I initially suggested isn't simple coding. You won't be able to use the parallel API to pull it off; you'd need to understand how that API works so that you could devise your own coroutine manager.

If you're happy running a second shell alongside your code (which is what the above line does), then by all means stick with that.

Bomb Bloke, on 13 July 2015 - 07:16 AM said:

Er, could you not override os.pullEvent() so that it acts as a coroutine manager, in addition to its normal functionality?

Perhaps, but you'd need an awfully hacky system to prevent sending duplicate events to your coroutine.

Bomb Bloke, on 13 July 2015 - 08:00 AM said:

parallel.waitForAny() expects you to pass it at least two function pointers. If you try to execute the functions yourself, than you'll instead end up passing the parallel API whatever those functions return. Your code isn't even managing to start parallel.waitForAny() - it's instead busy trying to resolve the arguments you wish to pass to it.

Now if you wrapped those calls inside function declarations, like so, then it'd do something closer to what you're wanting:

parallel.waitForAny(function() os.loadAPI('file') end, function() os.run({},"/rom/programs/shell") end)

It's worth pointing out that the technique I initially suggested isn't simple coding. You won't be able to use the parallel API to pull it off; you'd need to understand how that API works so that you could devise your own coroutine manager.

If you're happy running a second shell alongside your code (which is what the above line does), then by all means stick with that.

Ideally no, I would not want to execute a second shell but that was just to get an idea of what you meant by CraftOS always yielding. I also am not wanting to use the parallel API at all because I do not like it and it seems a bit un-needed since you could still do it yourself. So with that said I will make an attempt at a coroutine manager by overriding and see how it plays out. Thanks again :)/>

Bomb Bloke, on 13 July 2015 - 08:00 AM said:

Ok, I think I'm on the right track now, but to avoid a more pressuring headache until I find a more concise workaround, is it possible to '*copy*' the data/env from one shell to another and then terminate the pre-existing shell? (Like having two cmd.exe open and killing one based on PID) (btw, That should be a suggestion, having process ID's, I'm not sure if CC already has this or not though)

Well, I reckon you can, but the process is arguably messier than that suggested by your first question.

For giggles, I've written out the sort of code I originally referred to; or at least, I believe this will do the trick. I'm honestly quite curious as to what you'll do with it.

Spoiler

-- The function we're going to run alongside everything else:
-- (Just an example of what you might use)
local function activeCode()
	local counted = 0
	
	repeat
		local x, y = term.getCursorPos()
		local xSize, ySize = term.getSize()
		
		term.setCursorPos(1, ySize)
		term.write("Events counted: "..counted.." (F12 to kill)")
		term.setCursorPos(x, y)
		
		local event, par = os.pullEvent()
		counted = counted + 1
	until event == "key" and par == keys.f12
end

local myCoroutine, oldCoroutineYield, hackySystem, myCoroutineFilter = coroutine.create(activeCode), coroutine.yield, false

coroutine.yield = function(filter)
	if hackySystem then
		return oldCoroutineYield(filter)
	else
		local myEvent, ok
		hackySystem = true
		
		repeat
			myEvent = {oldCoroutineYield()}

			if coroutine.status(myCoroutine) ~= "dead" and (myEvent[1] == myCoroutineFilter or myEvent[1] == "terminate" or not myCoroutineFilter) then
				ok, myCoroutineFilter = coroutine.resume(myCoroutine, unpack(myEvent))
			end
		until myEvent[1] == filter or myEvent[1] == "terminate" or not filter
		
		hackySystem = false
		if coroutine.status(myCoroutine) == "dead" then coroutine.yield = oldCoroutineYield end
		return unpack(myEvent)
	end
end

It can be loaded as an API, but if you don't have any actual API functions to add to it, then you may as well just run it as a regular script.

And please don't let me catch you calling coroutine.yield() directly if you don't need to - bear in mind it's called for you most any time you call a function that pauses execution for any reason. os.pullEvent() is one of the most blatant examples, but there's also sleep(), turtle.forward(), read(), etc…

Bomb Bloke, on 13 July 2015 - 11:24 AM said:

Thanks for the code above, I read a post by KingofGamesYami where he's doing something VERY similar and from that stems this code.
OP: http://www.computercraft.info/forums2/index.php?/topic/19908-run-code-in-background/page__fromsearch__1

function activeCode()
  msgBuffer = {}
  while true do
    local e = {os.pullEvent()}
    if e and e[2] == "modem_message" then
      msgBuffer[#msgBuffer+1] = {e[6],e[3]}
    end
    os.pullEventRaw()
  end
end

--Backup pullEventRaw and create our internal coroutine
local pullEventRaw_Backup = os.pullEventRaw
local coActive = coroutine.create(activeCode)

--Override os.pullEventRaw()
function os.pullEventRaw(sFilter)
  while true do
    local event = {pullEventRaw_Backup()}
    --Define internal coroutines to check for
    if coroutine.status(coActive) == "suspended" then
      coroutine.resume(coActive, unpack(event))
    end
    --Return any events
    if sFilter == event[1] or not sFilter then
      return unpack(event)
    end
  end
end

I haven't tested out your example yet but just looking at it, it seems rather similar except you're directly overriding coroutine.yield() and giving a bool as to whether to use the 'hacky' version of it and you're restoring the original coroutine.yield() when that coroutine dies which is a really nice idea btw.

Bomb Bloke, on 13 July 2015 - 11:24 AM said:

Lyqyd, on 13 July 2015 - 08:02 AM said:

Thanks guys! It's all working now! This project is going to be so great when its finished! Again thanks SOO much!

Looks like you're gonna get duplicate events that way–one event per each other coroutine on the system. Unless I'm missing something, Bomb Bloke?

Using the code jacky quoted from Yami's tutorial, you'd indeed have that problem. I see it's also failing to handle the filter for "coActive" (or "coFoo", whatever you want to call it).

If you're talking about my code, the mechanism for preventing duplicates should be quite obvious to you. At least, I can't think of a weak point.

My code doesn't actually fail to handle the filter - it just doesn't need to save it. See if you can follow my logic here:

*My os.pullEventRaw pulls events without the filter and then waits until it finds an appropriate one
*When calling coroutine.resume, the stuff I pass is going to be passed to another "instance" of the same function, which, as I said, doesn't pass anything back anyway.

My code does duplicate events, if you have multiple coroutines - there's a reason I only show a single coroutine running. Otherwise, it'll just be a gigantic mess.

The only thing jacky is misusing is the os.pullEventRaw on line 8, which is unnecessary and probably eats events he might want to be checking.

KingofGamesYami, on 14 July 2015 - 03:53 AM said:

My code doesn't actually fail to handle the filter - it just doesn't need to save it. See if you can follow my logic here:

*My os.pullEventRaw pulls events without the filter and then waits until it finds an appropriate one
*When calling coroutine.resume, the stuff I pass is going to be passed to another "instance" of the same function, which, as I said, doesn't pass anything back anyway.

Er, yes, it does need to save it. See, when the function you build your coFoo() coroutine out of - foo() - yields, it passes back whatever parameter was handed to coroutine.yield() (in your case, the string "modem_message"). It expects this to mean it'll only be resumed when there's an event of that type at the front of the queue.

So what happens if you resume it with a different event type? Well, if the rest of the code in foo() is rigged to assume it'll only be resumed with "modem_message" events (which would be quite reasonable), odds are it'll crash messily, yes?

Bomb Bloke, on 14 July 2015 - 03:27 AM said:

If you're talking about my code, the mechanism for preventing duplicates should be quite obvious to you. At least, I can't think of a weak point.

Your code does prevent duplication, but doesn't handle terminate events correctly–these should bypass the filtering.

Separately, I've no idea what I was on about when I made my initial comments about this being something you'd need to control the whole coroutine stack for. Whoops.

Lyqyd, on 14 July 2015 - 05:26 AM said:

Your code does prevent duplication, but doesn't handle terminate events correctly–these should bypass the filtering.

Fixed, thanks. :)/>

Bomb Bloke, on 14 July 2015 - 04:33 AM said:

KingofGamesYami, on 14 July 2015 - 03:53 AM said:

My code doesn't actually fail to handle the filter - it just doesn't need to save it. See if you can follow my logic here:

*My os.pullEventRaw pulls events without the filter and then waits until it finds an appropriate one
*When calling coroutine.resume, the stuff I pass is going to be passed to another "instance" of the same function, which, as I said, doesn't pass anything back anyway.

Er, yes, it does need to save it. See, when the function you build your coFoo() coroutine out of - foo() - yields, it passes back whatever parameter was handed to coroutine.yield() (in your case, the string "modem_message"). It expects this to mean it'll only be resumed when there's an event of that type at the front of the queue.

So what happens if you resume it with a different event type? Well, if the rest of the code in foo() is rigged to assume it'll only be resumed with "modem_message" events (which would be quite reasonable), odds are it'll crash messily, yes?

The thing is, when the coroutine "ccFoo" calls, for example,

os.pullEvent( "message" )

It actually calls my function,

function os.pullEventRaw(sFilter)
  while true do
    local event = {pullEventRaw_Backup()}
    --Define internal coroutines to check for
    if coroutine.status(coActive) == "suspended" then
      coroutine.resume(coActive, unpack(event))
    end
    --Return any events
    if sFilter == event[1] or not sFilter then
      return unpack(event)
    end
  end
end

…which calls the old os.pullEventRaw without any parameters. Therefor, nothing will ever be passed by coroutine.yield, short of the user calling coroutine.yield directly themselves.

My function does work as a filter - on the other side of the coroutine.yield.

I'm not able to fix the duplicating events - any sort of coroutine manager (even parallel) running somewhere will screw it up - because os.pullEventRaw will be called multiple times per event. I really don't know how I'd figure out if the function is being called in some sort of coroutine manager, if it's possible at all.

KingofGamesYami, on 14 July 2015 - 01:43 PM said:

…which calls the old os.pullEventRaw without any parameters. Therefor, nothing will ever be passed by coroutine.yield, short of the user calling coroutine.yield directly themselves.

Ok, I see it now. You're quite right - making the assumption that the user won't call coroutine.yield() with a filter request, os.pullEventRaw() will indeed only return the requested event type back when called, regardless as to whether it's called by the coroutine or otherwise. My bad. (Though as Lyqyd pointed out, os.pullEventRaw() should also be returning "terminate" events regardless of the filter!)

KingofGamesYami, on 14 July 2015 - 01:43 PM said:

I'm not able to fix the duplicating events - any sort of coroutine manager (even parallel) running somewhere will screw it up - because os.pullEventRaw will be called multiple times per event. I really don't know how I'd figure out if the function is being called in some sort of coroutine manager, if it's possible at all.

The code I posted above handles it.

And it's fairly easy to do; simply set a boolean when the doctored coroutine.yield() is called, and unset it immediately before returning. If the doctored function is called again before the boolean unsets, pass through to the regular coroutine.yield().

In this manner, the coroutine will never attempt to resume itself, and if a number of functions are running together through eg the parallel API, then only the first of those will ever attempt to resume it.

Yeah, that's my bad on the terminate event. Though it's not hard to add, I didn't add it in the first place (*smacks self*).

I don't quite understand the boolean thing, but the more I think about it the more I can see it working.