One cause might be because you're trying to put a pointer to the table in the table.
When a table is defined, your variable gets set up as a pointer to that table in memory (like a shortcut). If you make a copy the contents of that variable into another variable, what gets copied is the
pointer, not the table:
local myTable1 = {345,345,45,6754,35132}
local myTable2 = myTable1
myTable2[3] = "Test"
print(myTable1[3])
Running the above should print out "Test", as both myTable1 and myTable2 point to the same table.
Given this, you can see that it's possible for an element in a table to point to that same table:
local myTable = {65,423,4,346,34}
myTable[1] = myTable
print(myTable[1][2])
When printing, the above should get the first element from the myTable table - which leads back to the same table! - then get the second element from that (423). This is a form of recursion.
If you try to serialise such a table, textutils will check myTable[1] and see a pointer to a table, which it'll then try to serialise. So the next step in doing that is to check myTable[1][1], which happens to contain a pointer to a table, leading it to check myTable[1][1][1], and so on until you run out of RAM to track the infinite loop you've just jumped into.
Hence rather then allowing you to try this, textutils just errors out as soon as recursion is detected.
Post the script you're having trouble with if you can't see how any of this applies.
