[1.75] OS dropping timers

I'm experiencing a problem in all versions of ComputerCraft I've used.

Versions: CC 1.75 and 1.80pr0 and 1.80pr11. (I understand the new devs builds are unstable, however, the bug wasn't version dependent, I believe it exists within the OS VM itself.)

Description: Any code that takes longer than 1 tick (0.05 seconds) to execute will cause all other timers to drop. This includes if the timer was scheduled to execute after the 1 tick delay. (e.g. a 1 tick computer action will cause timers 1 full second after the completed action finished to drop). I also have the timers recurse, however, only 1 timer will

Expected Result: The computer to not drop timers.

Reproduction Instructions: To demonstrate the circumstances I had, I wrote a simple proof of concept. It's available here, https://pastebin.com/TrZhvhBn. I create a list of timers and insert them into a map storing the timer ID as the key and the ordinal number of the timer as the value. Running the program allows you to reproduce the problem I had.

I've included controls to see the results in real time. If you enable the delay, you'll notice an immediate drop off of all but one timer. If you disable the delay, the lone timer will still be the only one. All timers will return once the delay is disabled and the map is rebuilt.

I'm not very good at Java but I will also take a look at the code and see if I can identify the potential issue in the mod. Thank you for your time.

Looking at your example code, I see you're calling os.sleep(0.5) at line 53. Here's the source for that function:

function sleep( nTime )
    if nTime ~= nil and type( nTime ) ~= "number" then
        error( "bad argument #1 (expected number, got " .. type( nTime ) .. ")", 2 ) 
    end
    local timer = os.startTimer( nTime or 0 )
    repeat
        local sEvent, param = os.pullEvent( "timer" )
    until param == timer
end

As you can see, running this loop after pulling your first timer will merrily eat up the other three you set before allowing your script to continue further. Note also that new timers have a minimum expiry time of "the next server tick", so even requesting a sleep delay of 0 would be enough to ensure that that all your other timers would be pulled here first.

In fact, any time you call any function that "pauses" (eg rednet.receive(), turtle.forward(), sleep(), various functions offered by third-party peripherals…), that function will be pulling events to do it, and odds are it'll be pulling and then simply discarding other events that you might've wanted to pull elsewhere in your script during the process.

The solution is to either not call those functions (manage the events they use within your own code instead), or to call them from within a different coroutine (eg via the parallel API).

Bomb Bloke, on 13 July 2017 - 01:40 AM said:

The solution is to either not call those functions (manage the events they use within your own code instead), or to call them from within a different coroutine (eg via the parallel API).

while running do
	parallel.waitForAny(handleTimer, humanInteraction)
end

I already am using the parallel API in the sample I had posted. I thought the sleep function might get a lot of attention since it calls an event itself. Here's the original sample I was using where I discovered this problem. Sleep is NOT called. The computer is busy using open peripherals to move items and the same problem exists.

https://github.com/j.../autobee_cc.lua

For reference:
Timer handler: https://github.com/jetpack-maniac/autobee/blob/4270603dbd5a7890c644fe9371e7a7687039c973/autobee_cc.lua#L106
Function ran when timer triggers: https://github.com/jetpack-maniac/autobee/blob/4270603dbd5a7890c644fe9371e7a7687039c973/autobeeCore.lua#L239

I've since changed the original map scheme. What I wanted to do was a staggered check apiaries. The OS dropping all timers made that not possible to implement as is though. I don't mind creating my own code for things but if I notice a shortcoming, I'm going to bring it forward so we can fix it. I use the original scheme for the Open Computers version and it does not have any issues.

Bomb Bloke, on 13 July 2017 - 01:40 AM said:

In fact, any time you call any function that "pauses" (eg rednet.receive(), turtle.forward(), sleep(), various functions offered by third-party peripherals…), that function will be pulling events to do it, and odds are it'll be pulling and then simply discarding other events that you might've wanted to pull elsewhere in your script during the process.

In this case, checkApiary calls checkOutput which in turn calls getItemMeta or getStackInSlot. Both these functions will run code on the main server thread, causing the computer thread to yield and gobble all events on the queue. You can see this in a bit more detail by adding a logging system to the parallel loop.

parallel.waitForAny(
  function()
	while running do
		  parallel.waitForAny(handleTimer, handlePeripheralAttach, handlePeripheralDetach, humanInteraction)
	end
  end,
  function()
	while true do
		  print(os.pullEvent())
	end
  end
)

You should see lots of timer and task_complete events, the latter coming from peripherals.

I'm not as familiar with OpenComputer's internals as much as I am with CC's, but I believe their peripherals do not depend on the event queue (I think they just block the computer from executing). Consequently sleep events are not discarded.

Jetpack_Maniac, on 13 July 2017 - 03:51 AM said:

I already am using the parallel API in the sample I had posted.

Sure you are, but you're not using it to bundle off your sleep() call into a different coroutine. You're calling sleep() from within the same coroutine where you're also wanting to monitor other events, and hence you're going to get interference.

The idea is that every coroutine parallel manages for you gets passed a copy of every event your main script is getting resumed with (bearing in mind that that's running inside a coroutine too - in ComputerCraft, all your code is). The code within each coroutine can pull and discard as many events as it likes without affecting the code within the other coroutines.

Jetpack_Maniac, on 13 July 2017 - 03:51 AM said:

The OS dropping all timers made that not possible to implement as is though.

I want to stress that the timers do expire correctly and their events do go into the end of the event queue correctly - but you then cause those timer events to be removed from the queue by calling a function that removes them.

Granted, you might not expect certain functions to do that (and good luck finding any documentation telling you which ones do!), but again: you're looking for pauses. You've probably noticed getStackInSlot isn't all that fast, for example. Most functions that "interact with the world" are yielding in the same way as sleep() does - they initiate an action that will trigger an event when completed, and then they simply pull and discard events from the front of the queue over and over until they find the one that signals their job to be done.

As a general rule of thumb you can assume pretty much every function from the OpenPeripherals mod will do this.

I can prove that simply blocking the computer's execution will not drop the events.

Try this code:

local id = os.startTimer(0.05) --# set a timer to go of in 1/20th of a second.
print( "Started timer " .. id )
local startTime = os.time()
while os.time() - startTime < 0.1 do end --# purposely block for 1/10th of a second.
while true do
  local e, i = os.pullEvent( "timer" )
  print( "Found timer " i )
  if i == id then
    break
  end
end

Bomb Bloke, on 13 July 2017 - 02:34 PM said:

The idea is that every coroutine parallel manages for you gets passed a copy of every event your main script is getting resumed with (bearing in mind that that's running inside a coroutine too - in ComputerCraft, all your code is). The code within each coroutine can pull and discard as many events as it likes without affecting the code within the other coroutines.

Alright, I've got a much better understanding of what you mean now. The state of one chunk is being affected by another chunk. I'm going to investigate the issue further.

Thanks to all who contributed to the discussion. Though I'm actually disturbed this was moved from bugs to ask a pro despite being a known issue with ComputerCraft.

Jetpack_Maniac, on 13 July 2017 - 05:02 PM said:

Bomb Bloke, on 13 July 2017 - 02:34 PM said:

The idea is that every coroutine parallel manages for you gets passed a copy of every event your main script is getting resumed with (bearing in mind that that's running inside a coroutine too - in ComputerCraft, all your code is). The code within each coroutine can pull and discard as many events as it likes without affecting the code within the other coroutines.

Alright, I've got a much better understanding of what you mean now. The state of one chunk is being affected by another chunk. I'm going to investigate the issue further.

Thanks to all who contributed to the discussion. Though I'm actually disturbed this was moved from bugs to ask a pro despite being a known issue with ComputerCraft.

It's not an issue with ComputerCraft though, it's just the way OpenPeripherals chooses to handle events.

apemanzilla, on 05 August 2017 - 09:38 PM said:

It's not an issue with ComputerCraft though, it's just the way OpenPeripherals chooses to handle events.

No, it's an issue not exclusive to OpenPeripherals; it extends to a great many of ComputerCraft's own functions and features.

Bomb Bloke, on 13 July 2017 - 01:40 AM said:

In fact, any time you call any function that "pauses" (eg rednet.receive(), turtle.forward(), sleep(), various functions offered by third-party peripherals…), that function will be pulling events to do it, and odds are it'll be pulling and then simply discarding other events that you might've wanted to pull elsewhere in your script during the process.

The reason I moved this thread out of bugs is because you haven't so much "reported an issue" as you have "discovered how ComputerCraft works" (and even then your initial understanding was heavily flawed, and I'm not entirely certain that you entirely get it now). It's like reporting that your car's effective engine power drops when entering overdrive. Yeah a lot of people seem to expect the opposite, but it's still intended product behaviour, and it works that way for a reason.

ComputerCraft handles nearly all world interactions via events. By doing so, whenever one script is waiting for something, another script has the opportunity to execute code - ComputerCraft only ever actively executes one co-routine at a time, no matter how many computers are in your world (they all run through the one VM!). Most scripts spend most of their time yielding, waiting for an event with which to be resumed. If you don't directly call os.pullEvent() yourself, but instead call functions that do it for you, then the fact remains that you're pulling events.

If you need certain events to remain in your queue until a certain point within a given coroutine, then it's up to you not to call anything that's going to remove them until then. Yes, there are times when you'll want to perform multiple tasks that rely upon different event types, and that can make things complex, but that's not some sort of "oversight". It's why the mod offers helper APIs such as parallel.

Bomb Bloke, on 11 August 2017 - 02:22 AM said:

The reason I moved this thread out of bugs is because you haven't so much "reported an issue" as you have "discovered how ComputerCraft works" (and even then your initial understanding was heavily flawed, and I'm not entirely certain that you entirely get it now). It's like reporting that your car's effective engine power drops when entering overdrive. Yeah a lot of people seem to expect the opposite, but it's still intended product behaviour, and it works that way for a reason.

Yes, I expected something different. I understand what you've said previously. It's contrary to standard industry behavior. My expectations were very different on how my events should be handled. No other event handler behaves like ComputerCraft. Those solutions also don't cause the problems of which I encountered and posted in my OP. Apache Kafka, a real world event handler used in many million dollar software applications, does not consume my timers. It can handle millions of events in its queue and does not interfere with it. Even the other Minecraft computer mod, OpenComputers, does not behave like this.

ComputerCraft handles nearly all world interactions via events. By doing so, whenever one script is waiting for something, another script has the opportunity to execute code - ComputerCraft only ever actively executes one co-routine at a time, no matter how many computers are in your world (they all run through the one VM!). Most scripts spend most of their time yielding, waiting for an event with which to be resumed. If you don't directly call os.pullEvent() yourself, but instead call functions that do it for you, then the fact remains that you're pulling events.

Again, let's go back to Apache Kafka. When Kafka handles events across threads it does not pull and discard the event when it views it. It checks to see if the event it pulled is the one it wanted, if it's not it does not remove it from the queue. That is the behavior I was expecting originally. As also stated previously this is unique to ComputerCraft, OpenComputers do not behave like this.

If you need certain events to remain in your queue until a certain point within a given coroutine, then it's up to you not to call anything that's going to remove them until then. Yes, there are times when you'll want to perform multiple tasks that rely upon different event types, and that can make things complex, but that's not some sort of "oversight". It's why the mod offers helper APIs such as parallel.

Oversights do exist, separate calls will consume each other's events; prominent examples being RedNet and Turtles. Solutions like Kafka has tens of thousands of man hours of developers and architects alike in creating the most robust system possible in the competitive tech market. We have opportunities to learn from that in order to give people here the best experience possible plus transferable skills to the outside world.

If you don't agree that the label of the original post should be "bug", I'm more than happy to change it to be a potential enhancement.

Lastly, in between several points you made your phrasing and choice of words were very stingy on a personal level. I come and post here because I want to better the mod and community. I'm not attacking you or anyone else here.

In ComputerCraft, under typical coroutine management systems, each coroutine gets to see each event exactly once. Events are destructively pulled from the actual queue only one time, when the computer's coroutine "tree" is resumed. That coroutine then resumes each of its children with that event so that they can use it if desired. That continues down the tree until each coroutine in the computer has been resumed with the event, at which point, the computer yields until it is resumed with the next event. The individual coroutines never destroy events; the only destructive portion of the process is done before any thread in the computer receives the event. ComputerCraft does offer several convenience functions (like sleep, rednet.receive, etc.) that allow users to abstract away the event model when they only care about one kind of event. All of those functions (with the more-recent exception of the turtle functions) can be copied and slightly modified to work as one component of an event handling loop, if the user cares about other events as well. These functions are only there to help users, and are not the only way to accomplish the things they do.

So, that is the designed/desired behavior for events within the computers. As you have made clear, you would like to see a different behavior. What should the coroutine handlers in each computer do instead? How do they know whether the child coroutine wants to preserve and re-queue the event it was resumed with, versus it being a "desired" event, versus it being neither desired nor requiring re-queueing? What should the handler do when attempting to resume coroutines with events that do not match their currently-set filter? What benefit to the users does this change provide, considering that ComputerCraft is not intended to be used in multi-million-dollar applications, and many users never dig into the guts of the event system?

I'm interested in furthering the discussion. The change(s) you are suggesting are not minor, but it is an interesting perspective.

This discussion reminded me of this API I wrote a while back. It sounds like it mimics the behavior you were expecting, though in a roundabout fashion.

</shameless self-promotion>