CRYPTUM :: Password Based File Encrypter

Well, when it comes to encryption and you must write the code manually, I think this is the best possible solution. And the coolest fact about this is its extendability. Basically, you can insert any possible PRNG instead of math.random - and it will work. So, if you have something cryptographically secure within easy reach, just use it! And if you haven't, but still want to increase security, you can re-create the standart math.random (that is Linear congruential generator), but with your own coefficients.

Btw, interpreting a password like THAT is really a unique idea, but I believe it's insecure in some way. Just because the value of bseed is a bit predictable. I think you should do it in the honest way (that means, bseed should contain the same amount of information). To do this you should concatenate the numbers not in decimal digits, but in binary:

bseed = 0
for i=1,#pass do
   bseed = bseed*256+string.byte(pass:sub(i,i))
end

Also, instead of having a small pstr (you introduced this variable because of "invalid value" error, right?) at line 17 you should take a remainder of division by 256 before calling string.char:

endt = endt..string.char((string.byte(tte:sub(i,i))+(tArgs[3]=="d" and shift or -1*shift))%256)

So now pstr = 255 and everything is working.