Questions about os.pullEvent()-handling and event cues

Atm I try to fix a bug, where a timer seems to miss sometimes an event and is then dead.
As I also want to make the program fit for multi player (even more events) I searched
the forum and wiki for details / examples about how event queues are handled, bur was not successful.
Before I do more tests (and may insert other bugs) I'd like to get some expert advice.

Listening to mutliple event types and process the event:

myTimer = os.startTimer(1)

while true do
  -- event, modemSide, senderChannel, replyChannel, message, senderDistance = os.pullEvent("modem_message")
  event, par1, par2, par3, par4, par5 = os.pullEvent()

  -- event processing
  if event=="timer" and par1==myTimer then
	-- handle timer event
	myTime=myTime+1
	myTimer = os.startTimer(1)

  elseif event=="modem_message" then
	if par2==44 then
	  -- do stuff for channel 44
	elseif par2==33 then
	  -- do stuff for channel 33
	else
	  -- not interested in this message
	end
	
  else
	-- not interested in this event
  end
end	

Question 1 (just to make sure)
While the program executes code e.g. stuff for channel 44, incoming messages are lost, as there is no os.pullEvent listening to them?

Parallel listening and processing (pseudo code):

local "manually managed queue"

local function pullEvents()
  event, par1, par2, par3, par4, par5 = os.pullEvent()

  -- filtering of interesting events
  -- if interesting then put event, par1, par2, par3, par4, par5 in a manually managed queue
end

while true do
  parallel.waitForAny(
	   pullEvents,
					  
	   function()
		  -- event processing
		  -- get somehow FIFO event, par1, par2, par3, par4, par5 from manually managed queue

		  if event=="timer" and par1==myTimer then
			-- handle timer event
			myTime=myTime+1
			myTimer = os.startTimer(1)
		
		 elseif event=="modem_message" then
		  if par2==44 then
			-- do stuff for channel 44
		  elseif par2==33 then
			-- do stuff for channel 33
		  else
			-- not interested in this message
		  end

		else
		  -- not interested in this event
		end
		-- make sure to remove the event from manually managed queue
	  end
	  )				  
end	

Question 2:
Does the above pseudo code help to reduce missed event? I think it still may happen?

Question 3:
I probably should not use any os.sleep() commands in the event processing part, as the internally generated events will mess everything up?

Question 4:
I'm thinking about using a dedicated timer computer, which only sends time information on request, nothing more.
Then it's probably tricky to get this information into the modem_message processing.
Maybe s.th. like a ) request time information for processing an event b ) put the not fully solved event back on queuec ) resolve this event after time info has arrived?

This stuff is interesting :)/> but not really easy! :(/>

Edit: queue

Events should not be missed. If they are being missed, something you are calling is eating events (sleep, turtle.forward, etc.)

Question 1: incoming messages cannot be lost, as the java part of CC is always storing events. Events are then given to the computer when it yields. It's literally a queue, imagine a table of events that acts something like this:

local events = {}
function addEvent( ... )
  events[ #events + 1 ] = { ... }
end
function pullEvent()
  return table.remove( events, 1 )
end

The actual handling is a bit different, given it waits until an event is in queue, but this'll give you a general idea of how events are handled.

Question 2: events cannot be missed, however they can be eaten. The pullEvent running in parallel would help if, for example, you called an event - eating function (sleep, turtle.forward, etc.) within your other function.

Question 3: Yes, anything that internally pulls events ends up 'discarding' events from the queue, something you may want to avoid. sleep, read, turtle movements, anything that 'pauses' your code while it is running will do this.

Question 4: A separate computer would not help, as you would still have to listen for an event. If you can catch the 'modem_message' event, why not catch the 'timer' event in the first place?

Ups, a lot of more search results using the correct word queue instead of cue… :unsure:/>

KingofGamesYami, on 17 May 2015 - 08:47 PM said:

…

Question 1-3:
Oh, your answer really help me to understand what is going on.
Does this mean when I have two computers: The first I keep really busy for a while (without eating up events). The second spams messages via modem.
When the first computer finishes the boring stuff and pulls events, it can still read each message sent by the second computer?

Is commands.exec also such an event-eater?

Question 4:
Think I got this as well.
I found some (hopefully all) of the event-eaters in my code. The command computer displays some messages to the players. It has to sleep in between to give enough time to read the text…
Atm I outsource the commands.say and commands.tell to a second command computer.

Kaikaku, on 17 May 2015 - 10:04 PM said:

Does this mean when I have two computers: The first I keep really busy for a while (without eating up events). The second spams messages via modem.
When the first computer finishes the boring stuff and pulls events, it can still read each message sent by the second computer?

Not exactly. Only one computer is ever running at a given time - that's one of the reasons programs are terminated for 'too long without yielding'. Assuming the other computer sent multiple messages before it yielded, the behavior you describe would be correct. If I go too much more into this I'm going to end up explaining coroutines… Perhaps you'd like to look into a coroutine tutorial. It may clear things up when you consider each computer to be a separate coroutine running in different environments.

Kaikaku, on 17 May 2015 - 10:04 PM said:

Is commands.exec also such an event-eater?

Yes

ComputerCraft Wiki said:

Available only to Command Computers, executes the specified MineCraft command, yields until the result is determined, then returns it. If command executes successfully then it returns true + the output of the command as a numerically-indexed table, otherwise returns false + an error message as as a numerically-indexed table. Compare commands.execAsync, which ignores the result and returns immediately, without yielding.

Kaikaku, on 17 May 2015 - 10:04 PM said:

Question 4:
Think I got this as well.
I found some (hopefully all) of the event-eaters in my code. The command computer displays some messages to the players. It has to sleep in between to give enough time to read the text…
Atm I outsource the commands.say and commands.tell to a second command computer.

You can always use os.startTimer to delay things. Many programs use a single os.pullEvent, followed by many if/elseif statements and executing different code depending on the results.

commands.say / commands.tell can always be executed without eating events by use of execAsync, as can any other command except those specified in the commands api, which I believe do not require events at all.

Peripheral calls are another type of "event eater" - many yield in order to go off and get data from your MineCraft world, then resume when that data comes back to them by way of an event.

Truth be told, commands.execAsync() is a really weird exception to the rule (at least, since it was introduced up to and including now, where we have CC 1.74pr20); although it doesn't yield (at least, I'm 97% sure of that; it acts in a manner that makes me think Dan performed voodoo on it), while commands it performs are executing, events have a random chance of not making it into the event queue (as if it were yielding). You can, for example, completely miss a timer event. This bug is impossible to code around (at least, not 100% - WorldPorter can survive about a couple of hundred such glitches per build, for eg) and I could write a whole essay on the problems it brings up, but I can tell you that the less commands are running at a time the more stable it is. It's never *stable*, though.

But so long as you aren't using that one function (and perhaps commands.exec(), which presumably just runs commands.execAsync(), waits for the associated event, then returns it), all events can be caught if you code things correctly.

A quick way to tell if a given function yields is to run this code snippet, inserting your mystery function into myFunc(). As-is, the example points out that "sleep()" is yielding in order to get "timer" events. If you called commands.exec(), it'd tell you about the "task_complete" events that pulls; turtle functions will reveal yields for "turtle_response" events, and so on. Peripheral calls (to eg OpenPeripheral devices) can return all sorts of weird stuff.

To be clear, events are always pulled from the front of the queue, and new events that occur are always added to the end. Say you have a timer event at the end of the queue, and you do os.pullEvent("timer"). This will pull the first event in the queue, see it's not a timer, and discard it. Then it'll pull the second, and likewise throw it away, and so on until it gets to the end of the queue, finds the event it wants, and returns that. Hence sleep(1) will in all likelihood empty your event queue out completely. Your use of timers is definitely the better option.

On the other hand, if you aren't pulling events, then they just sit in the queue. The queue never magically empties unless you do something to pull the events out of it.

When you *must* use functions that pull events (a peripheral call for eg) alongside your own event-dependent code, that's when you bundle them away into a separate coroutine. A basic coroutine manager can pull all events, and pass copies to all coroutines it's handling - if one's running an "event eater" function, then it still can't eat the copies that's being passed to the other coroutines! The parallel API trivialises this process by handing you such a coroutine manager on a platter.

This won't help you with the broken commands.execAsync() thing, but yeah.

KingofGamesYami and Bomb Bloke, first of all thank you guys! :)/>

KingofGamesYami, on 18 May 2015 - 12:10 AM said:

…

ComputerCraft Wiki said:

Available only to Command Computers, executes the specified MineCraft command, yields until the result is determined, then returns it. If command executes successfully then it returns true + the output of the command as a numerically-indexed table, otherwise returns false + an error message as as a numerically-indexed table. Compare commands.execAsync, which ignores the result and returns immediately, without yielding.

…

I think I understand now the concept of yielding. I wasn't sure before.

KingofGamesYami, on 18 May 2015 - 12:10 AM said:

…
commands.say / commands.tell can always be executed without eating events by use of execAsync, as can any other command except those specified in the commands api, which I believe do not require events at all.

One thing I will do, is to get rid of as many event eaters in my main program as possible.

Bomb Bloke, on 18 May 2015 - 01:25 AM said:

…
Truth be told, commands.execAsync() is a really weird exception to the rule (at least, since it was introduced up to and including now, where we have CC 1.74pr20); although it doesn't yield (at least, I'm 97% sure of that; it acts in a manner that makes me think Dan performed voodoo on it), while commands it performs are executing, events have a random chance of not making it into the event queue (as if it were yielding). You can, for example, completely miss a timer event. This bug is impossible to code around (at least, not 100% - WorldPorter can survive about a couple of hundred such glitches per build, for eg) and I could write a whole essay on the problems it brings up, but I can tell you that the less commands are running at a time the more stable it is. It's never *stable*, though.
…

Your WorldPorter already helped me a lot, as my builder program for the pyramids now runs smoothly - well, let's say reliably - all 175,000 command block commands.
However, I will let it trigger at the end of execution a restart of the timer in the calling command computer. Just to be on the save side. I still will loose some time during generation, but I can and will live with that. (Probably I could do all in one computer to ease that issue, but for now it's more important to me to keep things a bit separated and manageable for me.)

Bomb Bloke, on 18 May 2015 - 01:25 AM said:

…
A quick way to tell if a given function yields is to run this code snippet, inserting your mystery function into myFunc(). As-is, the example points out that "sleep()" is yielding in order to get "timer" events. …

That's a nice tool I will try out later or in case of emergency ;)/>

Bomb Bloke, on 18 May 2015 - 01:25 AM said:

…
When you *must* use functions that pull events (a peripheral call for eg) alongside your own event-dependent code, that's when you bundle them away into a separate coroutine. A basic coroutine manager can pull all events, and pass copies to all coroutines it's handling - if one's running an "event eater" function, then it still can't eat the copies that's being passed to the other coroutines! The parallel API trivialises this process by handing you such a coroutine manager on a platter.
…

Yes, I have some must sleep situations. Nothing fancy, just delaying display of text messages. I've a quick and dirty solution in mind for now, that hopefully will do the trick.

Hmm - this is distressing to me. Let me make sure I understand - if I have a loop pulling events, and reacting to them, there are commands I can call, as simple as turtle movement, that will cause me to miss events that trigger during their execution? But this is on a per-CC-computer basis - I can have 20 turtles dancing around, and if I have *one* sitting there doing nothing but processing events (and presumably adding them to a durable queue or such) - that one won't miss any (or at least any it should be receiving)?

It smells like we need a proper pub/sub library. Hmmm.

My use-case is a remote procedure execution thing, where missing a beat could cause execution to go wildly off course; I wonder if what this really means is that events are not a great mechanism for transmitting that sort of data. Probably. :-)

Good info, thank you all.

bortels, on 28 May 2015 - 04:19 PM said:

Hmm - this is distressing to me. Let me make sure I understand - if I have a loop pulling events, and reacting to them, there are commands I can call, as simple as turtle movement, that will cause me to miss events that trigger during their execution? But this is on a per-CC-computer basis - I can have 20 turtles dancing around, and if I have *one* sitting there doing nothing but processing events (and presumably adding them to a durable queue or such) - that one won't miss any (or at least any it should be receiving)?

It smells like we need a proper pub/sub library. Hmmm.

My use-case is a remote procedure execution thing, where missing a beat could cause execution to go wildly off course; I wonder if what this really means is that events are not a great mechanism for transmitting that sort of data. Probably. :-)

Good info, thank you all.

Events are per-computer/turtle/etc. For example, if you queue an event on one computer, it will not be shown on another.

There are certain functions that may cause you to miss events, because they rely on events themselves. turtle.forward, for instance, doesn't know if it went forward or not until it pulls a relevant event. Until that event is pulled, it's going to discard any events it receives.

To avoid this, have a 'worker' function running in parallel with a 'receiving' function, where the receiving function stores messages in a table for the worker function to execute.

Events are, in CC, the only way of transmitting data. rednet relies on modem_message events, http.get (and post) rely on their own events (IIRC http_success and http_failure).

I hope I've helped you :)/>