Suggestion: Persitant Storage

I wanted to post this to the suggestions section, but I'm not allowed to create a topic there.

I have been digging through this forum for a while searching for a good solution to the "turtle loses state on chunk unload / server restart" problem. While saving the state to a file after every state change works, this puts a very high load on the server. Especially if multiple turtles are doing this.

Now what I suggest is a simple persistent storage api: (ps)

ps.store(name, table)
This function will remember the given table and store all its contents on a forced shutdown. E.g. server shutdown / chunk unloading

ps.load(name)
This function can be used to load a previously stored table. It returns nil if there is no table with such a name.

The idea is that you have one big table in your program which holds all important state. Then at the start of the program you register this table with the persistent storage api.
The persitent storage api then garantuees that it will save the contents of this table on server shutdown / chunk unload. When your program then comes back up, it can check if a stored table exists and restore its state and continue with its operation. This would only require a file write on server shutdown / chunk unload and not a file write after every state update and would heavily decrease server load.

I hope you like my idea.

Kind Regards
Ingrater

This suggestion can be done with tools we already have, and it doesn't even fix the problem of persistence with Lua scripts.

I think I misread a little bit, and this might work.

Even if they were to implement this, it would still create a high server load if the API is being used a lot. Cookiebal created something sort of like this, (here) but again, it could create high server load from all the file writes.

There'd be no reason to use any other kind of storage. I don't see this happening.

Apemanzilla, on 26 November 2013 - 12:50 PM said:

Even if they were to implement this, it would still create a high server load if the API is being used a lot. Cookiebal created something sort of like this, (here) but again, it could create high server load from all the file writes.

Why would this create high load? The only time the server actually has to do something is upon shutdown. Are you sure you understood the concept correctly?

Lyqyd, on 26 November 2013 - 12:52 PM said:

There'd be no reason to use any other kind of storage. I don't see this happening.

This is not another kind of storage. This is an api to workaround the losing state issue of turtles and computers. While a real fix will certanly be hard to implement (because the entire state of the LUA virtual machine would have to be saved), this would be quite easy to implement. Given the computercraft sourcecode I would be able to implement this myself in a day or two.

Of course it's another kind of storage. You put data into it, you get data out of it. The advantage to this is that it would always be perfectly saved in its current state on chunk unload. Given that, you'd have no reason to use the file system at all, since you could just use the persistent store to hold the data you needed and never worry about it.

Not really. The idea would be that only the program that stored the data can retrieve it. So other programs won't be able to retrieve the data. You would still need the file system for sharing or storing data.
As a callback on shutdown is not wanted, this is the only other solution I can think of to fix the biggest issue in computer craft. The currently used solution to the problem (saving to a file after every state change) is really bad. As it causes massive I/O on the server and may cause the sever hard drive to die earlier just because computercraft has a basic design issue.

To make the save to file less of a issue, computercraft could use a virtual file system. Not writing files to the disk, instead keeping them in memory until the server shuts down. But as it is currently implemented writes always go straight to the actual server disk.

You could textutils.serialize the table and then encrypt it (ComputerCraftCrypto is an API that adds 256-Bit AES encryption - search for it) and save it to a file.

All this would require on the java side is either a way to register a function that gets called on shutdown or a shutdown event, but both of those things have been denied before.

My goal (when my OS is done) is to file.write(textutils.serialize(cfg)) the persistent storage table right before every top-level coroutine.yield() if anything in the persistent storage table changed (detected with metatables). I don't think this method of doing it in lua is as bad as everyone (including myself) thinks because all modern operating systems for real computers have filesystem caches and textutils.serialize is relatively fast (unless you're me and stupidly store all your everything in persistent storage).

awsmazinggenius, on 26 November 2013 - 10:01 PM said:

You could textutils.serialize the table and then encrypt it (ComputerCraftCrypto is an API that adds 256-Bit AES encryption - search for it) and save it to a file.

That's not the point. Using that method, you'd need to save your state ridiculously often since you can't "catch" shutdown events, especially forced Ctrl+S shutdowns and chunk unloads.

What he's suggesting is the ability to keep a table in your code, which can be changed and will be saved once the computer is unloaded or shut down by force, reducing load on the hard drive and making sure your program state is always saved.

How about adding a limit on size for the table and how many tables you can store? I think this would add an interesting aspect to mod.

What if this were some sort of database peripheral? (Not the one that currently exists; I believe that was meant for connecting to real online databases.) You would place it next to the computer, wrap it, and store a (limited?) number of databases in it. Updating the table would update the information in the database peripheral itself. Though I feel like that would create the same amount of server load, seems unnecessary, and wouldn't be easy to implement.

Though I do feel that it would make sense for the data stored in the persistence API itself should have some sort of sensible home; from a player's point of view, implementing it as an API would sort of just have this random data magically accessible by the computer.

Ability to change own NBT? At least a special part.

Or special RAM storage, cleared at comp shutdown, not cleared at reset or unload.

This gets into the category of shutdown routines and allowing the computer to do things on a forced shutdown. They've been suggested but ultimately ctrl+s needs to somehow guarantee a shutdown. I've suggested it before with no luck, but I still think the best solution is to allow exactly one coroutine cycle to run after ctrl+s. CC already terminates a coroutine that runs too long so it's no problem if something takes too long, and it makes it easy to just wait for a shutdown event in a program to do a quick no-user-input-required save.

ElvishJerricco, on 27 November 2013 - 03:01 AM said:

This gets into the category of shutdown routines and allowing the computer to do things on a forced shutdown. They've been suggested but ultimately ctrl+s needs to somehow guarantee a shutdown. I've suggested it before with no luck, but I still think the best solution is to allow exactly one coroutine cycle to run after ctrl+s. CC already terminates a coroutine that runs too long so it's no problem if something takes too long, and it makes it easy to just wait for a shutdown event in a program to do a quick no-user-input-required save.

The problem is though that almost nothing yields, except for os.pullEvent or peripheralMethods, so that is probably why they didn't include it. You can do too much in one coroutine then desired.

That's also why I really want just this storage type, but that it is only sandboxed to allow file editing, and then for one file. But that's most likely never going to happen.

Thats why I suggested a automatic save of a file instead of a callback on shutdown. You could do about anything in a callback which could cause big issues. But saving a table is a very limited task which can be carried out on the java side of things. No unpredictable behaviour included.

Engineer, on 27 November 2013 - 10:42 AM said:

The problem is though that almost nothing yields, except for os.pullEvent or peripheralMethods, so that is probably why they didn't include it. You can do too much in one coroutine then desired.

Ingrater, on 27 November 2013 - 02:10 PM said:

Thats why I suggested a automatic save of a file instead of a callback on shutdown. You could do about anything in a callback which could cause big issues. But saving a table is a very limited task which can be carried out on the java side of things. No unpredictable behaviour included.

I don't think having the ability to do anything in the callback is a bad thing. I think it's actually a good thing as long as there's no way for the callback to postpone shutdown for more than a couple of seconds. And if only one coroutine cycle is run, there isn't any way.

ElvishJerricco, on 27 November 2013 - 03:15 PM said:

I don't think having the ability to do anything in the callback is a bad thing. I think it's actually a good thing as long as there's no way for the callback to postpone shutdown for more than a couple of seconds. And if only one coroutine cycle is run, there isn't any way.

But then, what if I call os.reboot() in this shutdown fase? I mean really, you shut it down for a reason, not to reboot it again or something silly like that. If you really need to save something, write to a file when move or something else, there are solutions for the problem.

Sure, its irreliable, but thats the same like you are working on your project and you didnt save anything. Then you lose power and your project didnt save - it are the same conditions, you didnt save it so you lost it.

Engineer, on 27 November 2013 - 03:39 PM said:

ElvishJerricco, on 27 November 2013 - 03:15 PM said:

I don't think having the ability to do anything in the callback is a bad thing. I think it's actually a good thing as long as there's no way for the callback to postpone shutdown for more than a couple of seconds. And if only one coroutine cycle is run, there isn't any way.

But then, what if I call os.reboot() in this shutdown fase? I mean really, you shut it down for a reason, not to reboot it again or something silly like that. If you really need to save something, write to a file when move or something else, there are solutions for the problem.

Sure, its irreliable, but thats the same like you are working on your project and you didnt save anything. Then you lose power and your project didnt save - it are the same conditions, you didnt save it so you lost it.

Except you didn't lose power. The universe magically decided to suspend your location from existence. Then when the universe restores your location to existence, everything is exactly how it was before and nothing and nobody cared or noticed. Except your computer, which in the blink of an eye is suddenly at a clean boot state again.