Events across world reload?

Hi, hello and good day. I apologize for my first post being a question, but I'm also offering a workaround (for my particular requirements) which might help others, so I hope it's excusable. Short disclaimer: I did search the forums using the in-built search, google and browsing manually, but did not find any topics addressing my problem explicitly, so I'm pretty confident this is not a duplicate of some EAQ list entry. If it is, please, show me the way :)/>

To the point: I've recently been getting into the perilous topic that is persistency in Lua in general, and in ComputerCraft in particular. In my research, I've come across some posts indicating that events should actually "survive" across saving/quitting a world and reloading it (or chunk unloading/reloading). The posts in question are:
- immibis on state saving
- Cloudy commenting on the event queue

Since one of the two is from a ComputerCraft developer, I was quite hopeful. But after some unsuccessful experimentation I decided to have a peek at the Java side of things (I hope this is not considered rude, it was done truly only with the goal of understanding event persistency in mind). From what I saw, the event queue is stored without the expected response IDs for the individual "commands". This results in them having a response ID of -1 after reloading the world, and thus no event will be dispatched on the Lua side.

Therefore my question: is this intended behavior, and if so, for what reason? Storage size/bandwidth? It seems to me that saving the response IDs would not be that bad of an overhead, plus it could even be patched in without breaking backwards compatibility (if it's not present in the save, just fall back to -1). If it is not intended behavior, how do the chances stand of this being fixed anytime soon?

Finally, my workaround: I only needed this in the context of persistently storing a Turtle's position. Somewhat along the lines of what immibis wrote, although a little more simplistic (native call, save id, wait for event; if set when loading wait for event on startup). Seeing as this won't work, here's what I've decided to do instead: check the fuel. This obviously only works if the turtles use fuel (but seriously, why wouldn't they). The basic idea is: save the fuel state before issuing the move command, if there's a reboot while moving wait at the startup for a bit, then check if the fuel level changed. Since only successful moves consume (exactly one) fuel, this is a pretty good indicator as to whether the last movement command was successful or not. It works really well for me, but at the same time is hardly satisfactory, given its hacky nature.

Anyway, I'd be happy to get some official feedback on the not-dispatching-events-for-pending-commands-after-reloads thing. Thanks!

PS: I'm using the latest version at this date, 1.53. Also, though I don't think it's really necessary, here's the (simplified) code I've tested with.

Event driven and broken:

Spoiler

local x,y,z,facing,moving,moveId=0,0,0,0,false,-1
function save() --[[ saves all state variables ]] end
function load() --[[ loads all state variables ]] end
-- moveFunction is a native (forward, back, up, down) and direction is for
-- keeping track where we are going to update our state after the move.
function beginMove(moveFunction,direction)
	moveId = moveFunction()
	if moveId == -1 then return false end
	moving = direction
	save()
	return finishMove()
end
function finishMove()
	local event, responseID, success
	while event ~= "turtle_response" or responseID ~= moveId do
		event, responseID, success = os.pullEvent("turtle_response")
	end
	-- Updates x,y,z based on movement type and facing.
	if success then updateState() end
	moving = false
	moveId = -1
	save()
	return success
end
load()
if moving then finishMove() end
-- Imagine some code for continually moving the turtle in a circle.
moveInCircle()

Workaround:

Spoiler

local x,y,z,facing,moving,preMoveFuel=0,0,0,0,false,0
function save() --[[ saves all state variables ]] end
function load() --[[ loads all state variables ]] end
function beginMove(moveFunction,direction)
	preMoveFuel = turtle.getFuelLevel()
	local moveId = moveFunction()
	if moveId == -1 then return false end
	moving = direction
	save()
	return finishMove(moveId)
end
function finishMove(moveId)
	local success
	if moveId >= 0 then
		local event, responseID
		while event ~= "turtle_response" or responseID ~= moveId do
			event, responseID, success = os.pullEvent("turtle_response")
		end
	else
		-- Wait until we are sure we could have finished an active move.
		os.sleep(1)
		if preMoveFuel == turtle.getFuelLevel() then
			success = false
		else
			-- Possibly some more checks here (that we did not move more than once or refueled).
			success = true
		end
	end
	-- Updates x,y,z based on movement type and facing.
	if success then updateState() end
	moving = false
	save()
	return success
end
load()
if moving then finishMove(-1) end
-- Imagine some code for continually moving the turtle in a circle.
moveInCircle()

Edit: fixed a derp in the code samples (missing success check around updateState()).

Edit 2: I built a small API based on the mentioned workaround.

Split into new topic.

The fuel-checking is a novel approach! It's something I haven't seen before, and quite a good workaround. Full state persistency is planned, but it's a long-term planned feature. Fuel checking is probably the best thing one can do until then. For the events, you might wish to point Cloudy toward this topic if you see him on IRC.

Lyqyd, on 01 July 2013 - 02:34 PM said:

The fuel-checking is a novel approach! It's something I haven't seen before, and quite a good workaround. Full state persistency is planned, but it's a long-term planned feature. Fuel checking is probably the best thing one can do until then. For the events, you might wish to point Cloudy toward this topic if you see him on IRC.

Thanks, for the quick reply! I was hoping it was, since I had not seen it elsewhere :)/>

I'm really looking forward to the full state persistence. I can only imagine how painful that must be to implement, in particular in Java, without custom memory allocation and such.

As for the IRC, I'll check it out, thanks.

The use of the fuel counter is a great idea - I hope you don't mind if i steal it ;)/>.

getting scripts to endure through reboot is is a huge problem with computercraft especially on servers. Because most people use the file system which saves the files "instantly" while the server's world state is saved periodically this causes a loss of sync. if the turtle relies on files it will usually think it's ahead.

Because of this fact you have to use indicators that are saved with the world save and only use the file system to save initial states. The only things that are saved with the world are the turtle positions, its inventory and fuel (as you just pointed out).

Using the fuel method to track movement position and then adding in 2 slots in the inventory that move sticks around according to the direction the turtle is facing you can get a pretty reliable system of figuring out where the turtle is.

I found out about the limitation of session persistence the hard way - I converted my entire fir cutting program into a state machine . and the the program rarely endured server crashes.(the state machine conversion wasn't as bad as it looks).

I'm currently working on a mining well program where I want to add server crash /chunk unload recovery and the fuel will probably be the key to identifying the turtle's exact location the problem is always the fact that the turtle can refuel itself. The current way I thought about handling it was saving the inventory stock and then looking at the number of energy pipes. The issue is that the pipes are used to both build the rows and connect them to the power source when there is a new row. the formula is total = inventory_pipes + x +y so I thought about using total-inventory=x+y and use the constraints (x<lx and y<ly) to find a valid location . The problem is that there are multiple solution to this formula so i thought about maybe keeping track of the row number using the inventory or something (keep track of the row number=y for example). .

Using the fuel as a tracking method will solve my problem because my program moves are extremely predictable (one case): back,up ,up,sleep,down down and repeat . at the end of the row it goes forward x times then back 3 times (in this case) . this means that the total cycle is (tried to be somewhat accurate to the code add 2 for another case) (lx*5 +1 +lx+1 +3) if lx== 10 then the fuel cost will be 50+1+10+1+3= 65. to calculate current state and position. The row number is math.floor((use start_fuel - turtle.getFuelLevel())/65) +1 . To find the turtle's current state I can (use start_fuel - turtle.getFuelLevel()%65 . if i get a number less than 50 I know it's in the first loop and taking %5 of that I can get the turtle's exact y position and also know if the turtle was in a placement stage or a cleanup stage. at over 50 I know it's cleaning up and starting the next row at around 62.

Moving sticks around is unnecessary, as turtle turns cannot fail. If you write the new direction to file, then turn, you will always resume with the correct orientation.

Interesting, I did something like that a while ago to keep track of turtle position through restarts accurately.
Though I just saved the fuel counter before move and direction the turtle moves (using direction the turtle faces) so forward while facing norht would be moving north..
Then on startup (cause thats what would be called on restart i assume) I checked if the fuel left differs from the last saved fuel and if it did i modified the last saved position (since it would be saved again once the move is complete) to match the actual position again

So kind of what you do there i guess xD My code was messier though <.<

Lyqyd, on 03 July 2013 - 10:48 AM said:

Moving sticks around is unnecessary, as turtle turns cannot fail. If you write the new direction to file, then turn, you will always resume with the correct orientation.

From my limited experience of turtles on a server , after the server crash and reload the turtle sees file from a future time that was rolled back while it's current position is the rolled back position . This is why I don't trust files too much .

Niseg, on 03 July 2013 - 11:14 AM said:

Lyqyd, on 03 July 2013 - 10:48 AM said:

Moving sticks around is unnecessary, as turtle turns cannot fail. If you write the new direction to file, then turn, you will always resume with the correct orientation.

From my limited experience of turtles on a server , after the server crash and reload the turtle sees file from a future time that was rolled back while it's current position is the rolled back position . This is why I don't trust files too much .

Hmm interesting, should remember that for when I try something like that again
Did you actually close the file after writing to it?

Server crashes don't count, as all bets are off regardless. Some nasty server crashes can even result in the turtle and its tile entity being separated. Even the sticks trick wouldn't work in event of a crash, so why waste the slots?

Niseg, on 03 July 2013 - 05:54 AM said:

The use of the fuel counter is a great idea - I hope you don't mind if i steal it ;)/>.

Thanks, and please do, I'm happy if it helps!

Pinkishu, on 03 July 2013 - 10:54 AM said:

Interesting, I did something like that a while ago to keep track of turtle position through restarts accurately.

It's one of these ideas that seem pretty obvious in hindsight, right?

As for the turning, I just realized this might actually be somewhat of a problem when waiting on startup. Imagine you queue 99 turn commands (via the native functions) and one move. Then the turtle gets unloaded. Now you'd have to wait quite long before the turtle would actually move… I'll have to do some testing, but am hoping that something inconsequential like turtle.detect() will do the trick (after a normal os.sleep() to make sure the queue isn't full and the turtle.detect() returns instantly because of that). Simply blocking until a turn is complete pretty much solves this.

Edit: just had a quick chat with Cloudy. He said he might (no promises) look into serializing the event/command response IDs, which should make the events be forwarded to Lua even across turtles being unloaded and loaded again. Just for anyone else interested.

Lyqyd, on 03 July 2013 - 01:18 PM said:

Server crashes don't count, as all bets are off regardless. Some nasty server crashes can even result in the turtle and its tile entity being separated. Even the sticks trick wouldn't work in event of a crash, so why waste the slots?

I know server crashes can introduce all sorts of catastrophes but it's not always the case . Because of server crash prevalence and it's common side effect is a rollback it might be a good idea to invest some time into recovering from it.

I agree that moving sticks around is somewhat of a crazy idea. I think that it's better to just minimize the turns. If you minimize the turns in your script you can cover 80-90% of the cases . Exploiting the fuel level to find the state you can sometimes figure out exactly if there was a turn in the recovered state and then find a way to deal with it. The best way to deal with a "confusing" state is to just crash, because sometimes continuing a script that doesn't know where it is would cause more damage than good.