local variables

I'm creating a save/load api for Advance Turtle Operating Environment, and I really can't see a way to get local variables. If I were to replace them, I would basically reconstruct the variables outside the environment, in a table. However, I can't figure out what the pattern would be to find & replace the local definitions anyway, and the calls to them.


This is annoyingly hard to find:

local annoyingtable, troll = {
 variable = "troll",
 annoying = "whatever",
 howto = "dothis",
}, "awesome"

and then they call it something like this

if annoyingtable.howto == "dothis" then
 annoyingtable["annoying"] = true
end

I thought about using "local%a+%^\n+%a" but realized this would not work on tables declared this way.

If I could replicate debug.getlocal() that would work, but we don't have debug :(/>

please explain a little further or better the overall goal of doing this. Also providing a useful example of your API code, and their interaction with it, and then whatever you want to do with that, may be useful… I'm struggling to understand exactly what it is you're doing here, as far as I can tell you want to turn all their local variables into global variables before running it so that you can modify them (?) which really begs the question as to why are you trying to do that, seems like a bit of a design flaw if you need to (or want to) do that…

theoriginalbit, on 24 May 2014 - 02:54 PM said:

please explain a little further or better the overall goal of doing this. Also providing a useful example of your API code, and their interaction with it, and then whatever you want to do with that, may be useful… I'm struggling to understand exactly what it is you're doing here, as far as I can tell you want to turn all their local variables into global variables before running it so that you can modify them (?) which really begs the question as to why are you trying to do that, seems like a bit of a design flaw if you need to (or want to) do that…

To clarify, I'm trying to save the variables before closing the program. The api is far from finished, but right now it looks for the call to it.

the code is here: https://github.com/KingofGamesYami/Advance-Turtle-Operating-Environment/blob/master/.apis/progress.lua

the only way you can access local variables is with the debug library. no amount of patterns will solve this at all. you can get the variable name if you wish by using a pattern on the file, but having the variables name will give you no advantage as you'll still not be able to access it 'cause its, you know, local…

Which is why I'm trying to replace

local variable = value

with

declare( variable, value )

which would pass the variable to my program. It would still be buggy though.

You can use this pattern:

local%s+([_%w]+)%s-=%s-(.+)

and replace it with

declare%( %1, %2 %)

KingofGamesYami, on 24 May 2014 - 04:08 PM said:

Which is why I'm trying to replace

local variable = value

with

declare( variable, value )

which would pass the variable to my program. It would still be buggy though.

In Lua, local variables are stored in internal VM variables called registers. These registers are unique to every function, making them similar to a stack frame. If you want to set the local variables before running the function, you'd need to rewrite your own bytecode into the function so that the variables get set. I don't see how this would be beneficial though, as you don't have a way of saving the IP (instruction pointer), which points to the next instruction to be executed. Even if you're successful somehow in restoring all local variables, the function would need to be restarted from the top. You can certainly inject bytecode to force a jump at the start, but there's no way of knowing where to jump.
tl;dr: Even with fancy bytecode magic, it's not possible.

Of course, it seems all you're doing is slicing off plaintext and then loading the butchered program. Maybe it would be a better idea to separate the resumable turtle programs into a list of functions which define each resumable step. The API could save which function was last being run, and run that one on the next program start. For state saving, you could save relevant variables using the API (something like progress.saveVar(name, value)). I'm guessing your approach is to try and make the progress saving seem as natural as possible, but without the debug library or even an ability to save a program's state to resume at a later time, I feel this probably isn't too bad of a compromise.

EDIT: I can provide an example of what I'm talking about if you want.

Would this be of any help ? Of course, using this method, you'd have to define a function for each local variable you want to retrieve but there's probably another way around.

local someLocalTable = {
   ["child"] = "value"
}

function getSomeLocalTable()
  return someLocalTable
end

--# You can retrive someLocalTable or any child of
--# someLocalTable like so:
local table = getSomeLocalTable()
local child = table["child"]

Yevano, on 24 May 2014 - 08:38 PM said:

KingofGamesYami, on 24 May 2014 - 04:08 PM said:

Which is why I'm trying to replace

local variable = value

with

declare( variable, value )

which would pass the variable to my program. It would still be buggy though.

In Lua, local variables are stored in internal VM variables called registers. These registers are unique to every function, making them similar to a stack frame. If you want to set the local variables before running the function, you'd need to rewrite your own bytecode into the function so that the variables get set. I don't see how this would be beneficial though, as you don't have a way of saving the IP (instruction pointer), which points to the next instruction to be executed. Even if you're successful somehow in restoring all local variables, the function would need to be restarted from the top. You can certainly inject bytecode to force a jump at the start, but there's no way of knowing where to jump.
tl;dr: Even with fancy bytecode magic, it's not possible.

Of course, it seems all you're doing is slicing off plaintext and then loading the butchered program. Maybe it would be a better idea to separate the resumable turtle programs into a list of functions which define each resumable step. The API could save which function was last being run, and run that one on the next program start. For state saving, you could save relevant variables using the API (something like progress.saveVar(name, value)). I'm guessing your approach is to try and make the progress saving seem as natural as possible, but without the debug library or even an ability to save a program's state to resume at a later time, I feel this probably isn't too bad of a compromise.

EDIT: I can provide an example of what I'm talking about if you want.

uhh… wat :wacko:/>
I kind of get what you mean, you want to create a "function manager" sort of thing,
so

completed = {func1, func2, func2}
inprogress = func4
inqueue = {func5, func6, func7}

which would be a valid option. However, I have no idea how to write that myself…

have you even considered this problem

local function foo()
  turtle.forward()
  turtle.turnLeft()
  turtle.forward()
  turtle.turnLeft()
  local moved = turtle.forward() --# program ends here
  if moved then
    print("yeah!")
  end
end

foo()

how will you make sure that when resuming the program you resume it at the correct place? there is no way from Lua to get any information about the stack, heap, or current line the program is up to.

I actually feel like suggesting a partial (not the methods that would allow breaking out of the sandbox (as they claim)) debug library because of this issue. If someone wants to go do that, be my guest.

awsmazinggenius, on 25 May 2014 - 01:00 AM said:

I actually feel like suggesting a partial (not the methods that would allow breaking out of the sandbox (as they claim)) debug library because of this issue. If someone wants to go do that, be my guest.

I'm not actually sure which method allows you to break sandbox, but I'd say if its any method it'd be the getlocal method, maybe getupvalue

theoriginalbit, on 25 May 2014 - 12:38 AM said:

have you even considered this problem

local function foo()
  turtle.forward()
  turtle.turnLeft()
  turtle.forward()
  turtle.turnLeft()
  local moved = turtle.forward() --# program ends here
  if moved then
    print("yeah!")
  end
end

foo()

how will you make sure that when resuming the program you resume it at the correct place? there is no way from Lua to get any information about the stack, heap, or current line the program is up to.

I considered it, and I would probably add this to counter it:

api[func_name] = function() 
  --define rest of function
end

basically forcing the user to declare their functions to my program. IDK if saving states is actually a viable option. I'm leaning towards making the user actually know what they are doing. (similar to what Yevano suggested)

Yevano, if you made that api, and created a pull request, I would put you on the list of contributors to my project and would be extremely thankful. I realize that I can't force you to do anything, but again it would help a lot. Even if you made just some of the api, it would be great to have something to debug, instead of what I have now (which doesn't work).

KingofGamesYami, on 25 May 2014 - 04:08 AM said:

Yevano, if you made that api, and created a pull request, I would put you on the list of contributors to my project and would be extremely thankful. I realize that I can't force you to do anything, but again it would help a lot. Even if you made just some of the api, it would be great to have something to debug, instead of what I have now (which doesn't work).

It's late for me now, but rather than make an API I'll give you some example code on what I'm talking about tomorrow and maybe I can help you from there.

Yevano, on 25 May 2014 - 05:16 AM said:

KingofGamesYami, on 25 May 2014 - 04:08 AM said:

Yevano, if you made that api, and created a pull request, I would put you on the list of contributors to my project and would be extremely thankful. I realize that I can't force you to do anything, but again it would help a lot. Even if you made just some of the api, it would be great to have something to debug, instead of what I have now (which doesn't work).

It's late for me now, but rather than make an API I'll give you some example code on what I'm talking about tomorrow and maybe I can help you from there.

Ok, thanks :)/>

Here's some user code I whipped up.

local function step1()
	io.write("Press 1-9: ")

	while true do
		local e, k = os.pullEvent("key")
		if k >= 49 and k <= 58 then
			print(string.char(k))
			local tiles = k - 48
			progress.vars.tiles = tiles	 --# progress.vars has a metatable attached, and it will
											--# save this tiles variable for us to a file.
			return false					--# Don't repeat.
		end
	end
end

local function step2()
	local tiles = progress.vars.tiles	   --# Load the tiles variable. If the program ever failed,
											--# the variable we be loaded from file. Else, it gets
											--# loaded from memory.
	if tiles > 0 then
		turtle.forward()
		progress.vars.tiles = tiles - 1	 --# Decrement.
		return true						 --# Repeat this function, we're not done yet.
	end
	return false							--# Done, don't repeat.
end

local function step3()
	print("Done!")
	return false
end

progress.addSteps(
	step1,
	step2,
	step3,
)

progress.start()

So what should happen, is you enter a number 1-9, the turtle moves forward that number of times, and then prints "Done!". If the program exits prematurely, it should be resumable from a generated runnable file which simply loads the saved variables and step functions and resumes on the one where it left off. It should also be noted that no steps should ever have side effects if they fail. Basically, variable changes should be tied to the function which made them, and if the step function did not finish, those changes should be thrown away. This way, you never have variables made which shouldn't have been made yet when you resume the program later.

EDIT: Thanks tobit, neat trick.

Yevano, on 25 May 2014 - 04:59 PM said:

Paste, because it looks terrible on the forum http://pastebin.com/23THRv2n.

if you use

--# these types of comments, it doesn't turn anything green 'cause of suspected strings

Now, don't get mad at me for necroing this topic… but I have started on an api similar to what Yevano suggested above. I've run into an issue that I would like to solve here. (hey, coping from github keeps the syntax highlighting… didn't know that before :D/> too bad it doesn't keep the indentation :(/>

current code

local tasks = {}
local executing = 0

function addTask( task, num )
  r = num or 1
  for i = 1, r do
   table.insert( tasks, {task = task, args = {} } )
  end
end

function clearTasks()
  do
    table.remove( tasks )
  until #tasks = 0
end

local function saveTasks()
  local file = fs.open( ".turtle/prog", "w" )
  file.writeLine( textutils.serialize( tasks ) )
  file.close()
end

function startTasks()
  saveTasks()
  for i = 1, #tasks do
    executing = i
    tasks[i].task( unpack(tasks[i].args) )
    local file = fs.open( ".turtle/task", "w" )
    file.writeLine( i )
    file.close()
  end
end

function addArgs( ... )
  args = {...}
  for i = 1, #args do
    table.insert( tasks[executing + 1].args, args[i] )
  end
  local file = fs.open(".turtle/args", "a")
  file.writeLine( textutils.serialize( args ) )
  file.close()
end

function resume()
  local file = fs.open( ".turtle/task", "r" )
  executing = file.readAll()
  file.close()
  file = fs.open( ".turtle/prog", "r" )
  tasks = textutils.unserialize( file.readAll() )
  file.close()
  file = fs.open( ".turtle/args", "r" )
  tasks[executing + 1].args = textutils.unserialize( file.readAll() )
  file.close()
  for i = executing, #tasks do
    executing = i
    tasks[i].task( unpack(tasks[i].args) )
    local file = fs.open( ".turtle/task", "w" )
    file.writeLine( i )
    file.close()
  end
end

Anyway, the problem I want to address is a pause function. Basically, I want the user to be able to add progress.pause( […] )to their task function, which will return to their main program, where they could use if then statements to figure out what they want to do, possibly adding a different task depending. They could then call progress.resume( […] ) to continue the program. (which I will probably have to edit later, it's not really done yet)

Edit: screwed up the code… fixing

It sounds a like you're wanting to familiarise yourself with the source of a the parallel API, then write a modified version of that. I forget if you already have an understanding of how it works.

Bomb Bloke is indeed correct, it seems like a modified parallel API would do the trick. I have written an API myself that achieves this task plus others.

KingofGamesYami, on 03 June 2014 - 02:43 AM said:

Now, don't get mad at me for necroing this topic…

the thread creators are allowed to resurrect their topics in order to ask questions related to their script. its actually preferred as it reduces AaP threads.

I'm pretty sure he's not trying to emulate parallel. IIRC, he wanted a persistent system which could be resumed over things like server restarts or arbitrary program terminations.

Yevano, on 03 June 2014 - 01:31 PM said:

I'm pretty sure he's not trying to emulate parallel. IIRC, he wanted a persistent system which could be resumed over things like server restarts or arbitrary program terminations.

Exactly. Parallel would be useless since I want the program to run in the correct order. If you meant coroutines, that would be somewhat closer to what I'm trying to do here. In fact, one of the ideas I recently came up with combined OO and coroutines to create an object that can be paused and resumed.

Edit: another problem I ran into: cannot serialize type function. The only way around this (that I can see) is to run the program again, but when it calls startTasks() it would resume from the point I had it at. This produces some problems with what I want to do with the pause/resume, because the logic will be all messed up.

As it happens, yes, the parallel API is a co-routine manager - that was indeed why I mentioned it. In case I (somehow) wasn't clear before, I was not suggesting that you throw it into your script verbatim, I was suggesting you use it as a guide (or even a template) to create what you seem to be talking about. The difference seems to be only in the order and conditions under which you resume execution, something trivial to alter.

If I've got you wrong, and you're actually wanting a way of specifically pausing a function when a server starts to shut down or something, then that leaves me scratching my head as to how you'd detect such an event in the first place.

Bomb Bloke, on 03 June 2014 - 02:35 PM said:

As it happens, yes, the parallel API is a co-routine manager - that was indeed why I mentioned it. In case I (somehow) wasn't clear before, I was not suggesting that you throw it into your script verbatim, I was suggesting you use it as a guide (or even a template) to create what you seem to be talking about. The difference seems to be only in the order and conditions under which you resume execution, something trivial to alter.

If I've got you wrong, and you're actually wanting a way of specifically pausing a function when a server starts to shut down or something, then that leaves me scratching my head as to how you'd detect such an event in the first place.

Uh, no… The way it works is like this:
1) The program loads the api (duh)
2) The program defines functions
3) The functions are put in a table within the api
4) The program tells the api to start the programs
5) The api runs the functions, passing on important variables if necessary

I want the functions being run by the api to be able to return to the program.

KingofGamesYami, on 03 June 2014 - 02:50 PM said:

Uh, no… The way it works is like this:
1) The program loads the api (duh)
2) The program defines functions
3) The functions are put in a table within the api
4) The program tells the api to start the programs
5) The api runs the functions, passing on important variables if necessary

I want the functions being run by the api to be able to return to the program.

I still feel it necessary to bring this back up

theoriginalbit, on 25 May 2014 - 12:38 AM said:

have you even considered this problem

local function foo()
  turtle.forward()
  turtle.turnLeft()
  turtle.forward()
  turtle.turnLeft()
  local moved = turtle.forward() --# program ends here
  if moved then
	print("yeah!")
  end
end

foo()

how will you make sure that when resuming the program you resume it at the correct place? there is no way from Lua to get any information about the stack, heap, or current line the program is up to.

unless you plan on forcing them to wrap each statement in a function there is no way around this!

theoriginalbit, on 03 June 2014 - 02:59 PM said:

-snip-

Uhm, yes that is what I'm doing… they define a function (or use a predefined one like turtle.forward) then pass it to my api. To make the turtle go forward 5 times:

progress.addTask( turtle.forward, 5 )
progress.startTasks()

KingofGamesYami, on 03 June 2014 - 03:04 PM said:

Uhm, yes that is what I'm doing… they define a function (or use a predefined one like turtle.forward) then pass it to my api. To make the turtle go forward 5 times:

progress.addTask( turtle.forward, 5 )
progress.startTasks()

So an example of the way they'd need to program the code I provided would be this?

local function foo()
  progress.addTask(turtle.forward)
  progress.addTask(turtle.turnLeft)
  progress.addTask(turtle.forward)
  progress.addTask(turtle.turnLeft)
  local moved = progress.addTask(turtle.forward)
  progress.addTask(function() if moved then progress.addTask(print, "yeah!") end end)
end

progress.addTask(foo)
progress.startTasks()

theoriginalbit, on 03 June 2014 - 03:09 PM said:

KingofGamesYami, on 03 June 2014 - 03:04 PM said:

Uhm, yes that is what I'm doing… they define a function (or use a predefined one like turtle.forward) then pass it to my api. To make the turtle go forward 5 times:

progress.addTask( turtle.forward, 5 )
progress.startTasks()

So an example of the way they'd need to program the code I provided would be this?

local function foo()
  progress.addTask(turtle.forward)
  progress.addTask(turtle.turnLeft)
  progress.addTask(turtle.forward)
  progress.addTask(turtle.turnLeft)
  local moved = progress.addTask(turtle.forward)
  progress.addTask(function() if moved then progress.addTask(print, "yeah!") end end)
end

progress.addTask(foo)
progress.startTasks()

Uh, no. More like:

local function foo()
  progress.addTask( turtle.forward )
  progress.addTask( turtle.turnLeft )
  progress.addTask( turtle.forward )
  progress.addTask( function() progress.addArgs( turtle.turnLeft() ) end )
  progress.addTask( function( moved ) if moved then print( "yeah!" ) end end )
end

foo()
progress.startTasks()

Edit: more efficient

Sounds like you'd be better off defining groups of functions, rather then trying to split one all-inclusive list at set points with a "return" code. Either way, if you pass control back to the parent script, then suddenly you've lost your server-crash protection… is that right, or am I still off-base?

Anyway, in regards to your "addTask" system, why not pass the functions as the names of their table keys, along with any arguments they need? Not sure how you're going to record them otherwise (it's not like you can store the pointer!). For eg, say you do:

progress.addTask( {"turtle","select"}, {1} )

In the API, you'd then do something like:

function addTask(functionTable, argsTable)
  table.insert(taskTable,{["functionTable"]=functionTable,["argsTable"]=argsTable})
end

function runTask(task)
  local myFunction = _G
  for i=1,#taskTable[task].functionTable do myFunction = myFunction[taskTable[task].functionTable[i]] end
  myFunction(unpack(taskTable[task].argsTable))
end

function startTasks()
  for i=1,#taskTable do runTask(i) end
end

… plus whatever "saving" or other organisational work you want to perform. I suspect this'd also make it easier for you to implement your "returning" thing, whatever the problem with that is.

By the by, I quite liked this solution to the resuming thing, in case you've not seen it. It's not exactly finished (nor would it even be practical to deal with all possible scenarios - the results of file reads for eg), but the basic idea behind it is a very simple one which struck me as very clever.

Just want to note that all you need to do is run the script once to get the functions back, then call the index of the one you were on before the program last exited. If you do this, you'll always be loading the function anyway, so using the function pointer is okay. Refer to my post here.

Ah, so the pointers aren't saved at all. That indeed makes more sense.

I'll look into the state restoring post you mentioned, even if it isn't finished, I could possibly modify it to my purposes. I'll still be using the task manager thing, but I might add that as possible alternative. One of the ideas I had was to overwrite all the turtle movement stuff, save the results, and then make the movement not move, but supply it with the appropriate response. It'd be harder to do then what I'm doing now though.