Save Large Tables To File Better Than With Textutils.serialize()

Hello everybody and welcome to a small brainteaser!

At the moment I am playing with Immibis's "Adventure Map Peripheral" and I am able to save a building(block by block) into a file using the textutils.serialize() on the table that keeps all the block information for that building.
The problem I have is that I need to also save the nbt data for all those blocks(currently only saving x, y, z, block id and metadata).
Before I start doing this and find out that I make huge files I wonder if there is a better way(a more space efficient way) of storing a table into a file.

- why save a building on file?
Because I need to make it appear and disappear at will for an adventure map
- why save nbt data?
Because chests and modded blocks
- how do you construct the table?
With table.insert()
- have you searched for any database API?
Yes but they are more effort than worth it and they still work with tables… also the nbt data is not all the same(structure/length) so databases are out of the equation.

I hope you all had a pleasant stay!
Sincerely,
sky

Split into new topic.

There are undoubtedly more compact ways to store the table, but they will be slower.

Out of curiosity, just how large are your files getting? These must be rather sizable buildings if you're having issues like this.

But anyway, I would do something like this:

Here is an example storage file. We'll call it "totem.txt"

return {
  0,0,0,5,1,
  0,1,0,5,2,
  0,2,0,5,1,
  0,3,0,5,2
  0,4,0,54,{NBT_FORMAT_HERE}
}

The format is as thus:
- The first 3 numbers are the x,y, and z coords respectively
- The 4th number is the block ID
- The 5th number/table is either the metadata or NBT, respectively

I know nothing about the Adventure Map Peripheral, so you'll have to do some hard thinking about how to get the NBT into a good format. However, it should be fairly simple if all you need is items.

And to load this file:

  local totem = dofile("totem.txt") --#Now totem contains the table from totem.txt
  for i=1,#totem, 5 do
    local x,y,z = totem[i], totem[i+1], totem[i+2],
    local blockID = totem[i+3]
    local meta = totem[i+4]
    if type(meta) == "table" then
      --#Handle NBT stuff
    elseif type(meta) == "number" then
      --#Handle setting metadata
    end
  end

This may not technically be the "fastest" (or even the smallest) , but unless you plan on having the computer constantly shutting down/needing to reload the file it should be fine once loaded into memory.


I have another method of making the file size even smaller, but it's going to take me a bit to write up. If the above works for you great, if not hang on a few minutes while I get it together.

(Or, if you feel up to it, try looking up vectors. It may just be simpler for you to do so than have me explain it)

For the second format, I'm going to assume that you have some repeatable patterns in your buildings. For example, let's make a file that allows us to make a 10 high totem pole in only 2 lines.

"totem.txt" is as thus:

return {
  {0,0,0},{0,9,0},nil,5,2
  0,10,0,54,{NBT_FORMAT}
}

Now, if the first three items are numbers, then we know to go by the regular format (x,y,z,id,meta/nbt)

However, if the first two items are tables, then we know it is a vector and the format is:
1) The starting point
2) The end point
3) Nothing (because are using a for loop that jumps by 5)
4) ID
5) meta/nbt

To parse this is a bit more difficult. We will need to make a function that converts our points into a sort of equation describing the locations of the blocks.

! I will come back later and finish this post (or, hopefully, someone else can finish it for me) ! It's the end of my lunch break and I need to get back to work :)/>

Bubba, on 31 July 2013 - 02:29 PM said:

For the second format, I'm going to assume that you have some repeatable patterns in your buildings. For example, let's make a file that allows us to make a 10 high totem pole in only 2 lines.
[… additional awesomeness here … ]

First of all I want to thank you for such a fast response! Especially Bubba! :)/>
The second format is of course amazing but I am afraid it is way to complicated to implement, especially since there will be a lot of modded blocks…. so the same id and even metadata but a whole different nbt data set.
On the first format I have to say that you are giving me ideas and I think that will be the direction I'll take. But i come to you with a different problem for this solution.
The table.insert function coupled with the textutils.serialize give me tables that look like this
{[1]={[1]=0,[2]=0,[3]=0,[4]=134,[5]=0,},[2]={[1]=0,[2]=0,[3]=0,[4]=134,[5]=0,}}
So more than half of the file is occupied by keys which of course i don't need.
How can i turn that to this:
{{0,0,0,134,0,},{0,0,0,134,0,}}
Do i have to do my own serialize function?
I found something made by immibis but I'm not sure I understand it all( http://www.computercraft.info/forums2/index.php?/topic/302-api-proper-serialization/ ) and most of all not sure if it gives me smaller files…. will have to test once I get home from work.

Well you could do a basic serialize function if you want. However, serialization would only really be necessary if you are using a program to design these buildings. If you've been typing in coordinates manually, you might as well just type straight into the file.

Anyway, here's how you could do a simple serialization if you wanted to go with the first method:

local function serialize(tData, isChild) --#We need that isChild argument for the recursion later on in the function
  local final_string = isChild and "{" or "return {" --#If this is not the first iteration, the final string should not have a return statement in it
  for i=1, #tData do --#For every element in tData
    if type(tData[i]) == "number" then --#If the element is a number
      final_string = final_string..tostring(tData[i]).."," --#Just add it into the string table with a comma after it
    elseif type(tData[i]) == "string" then
      final_string = final_string.."[["..tData[i].."]]"..","
    else --#Otherwise if the element is a table (like for NBT), call this function with the table element as the argument
      final_string = final_string..serialize(tData[i],true)..","
    end
  end
  return final_string.."}" --#Finish off the final string table with a "}" and return the result
end

And an example…

local coords = {
  3,2,3,5,2,
  3,5,4,54,0,
  8,2,1,5,{52,225,"something"}
}

local serialized = serialize(coords)
print(serialized)

Output:

return {3,2,3,5,2,3,5,4,54,0,8,2,1,5,{52,225,[[something]],},}

Bubba, on 01 August 2013 - 06:32 PM said:

Well you could do a basic serialize function if you want. However, serialization would only really be necessary if you are using a program to design these buildings. If you've been typing in coordinates manually, you might as well just type straight into the file.
[… awesomeness …]

I am using a program to scan an already existing building and save all the data into a table. The table is then serialized and saved into the file.

Bubba, on 01 August 2013 - 06:32 PM said:

[…]
Anyway, here's how you could do a simple serialization if you wanted to go with the first method:
[… epic algorithm …]

I am humbled by your love of programming!
You sir have given me all that I needed! I am in you dept!