[Discussion] bitwise operations in lua - file opreations?

NEW PROBLEM see last post

—————————————— Original Post ————————–
While working on GP-IS progect I started investigating encryption methods to increase network security in GP-IS after much googling I found that lua's lack of bitwise operations makes encryption Impossible as there is no way to manipulate data at a raw bit level. has anyone here had similar problems and if so did you find a solution.

I'm looking for a way to XOR a string with another string. this will be a simple form of encryption but also harder to break the larger the key.

links to information i found interesting.


http://lua-users.org...yptographyStuff
all the examples seam to use third party methods to encrypt.

http://www.google.co...6BHt642ZmyQsA-Q

http://eder.us/projects/lcrypt/

*cough* bit API *cough* :P/>/>
It lets you do any bitwise operation with numbers. My OS has md5 hashing for passwords, and it uses the API, so it's possible to do other types of encryption.

So I need to convert my string's to int's with string.byte and then bit.bxor(m, n) them so simple thanks. I never thought to look in the apis for a solution . There goes a few hours i will never get back researching lua encryption. trust MIT to have already implemented a solution. ow and thanks dan200 for including it in CC

i just sent a file through rednet unencrypted and it got corrupted. the file type was PNG.
is there a way to read a file as data not strings.I'm trying to make this encrypt files at a data level so that all file types not just text/lua files can be encrypted. But if lua cant event read a file in as data then this will not be possible.all the unknowns characters got converted into question marks. i guess lua is just not built to handle encryption. Made it into a gif so it is easer to spot the difference.

Relevant sections of my file FTP program.
how file is read

Spoiler

    if fs.exists("/FTP/"..string.sub(e2,9,string.len(e2))) then
	 file = fs.open("/FTP/"..string.sub(e2,9,string.len(e2)),"r")
	 local sholding = file.readAll()
	 file.close()
	 nodeSender(e1,"FILEDAT "..sholding)
	 print("file sent")
	 sholding = nil
    else
	 print("failed to send File not found")
    end

how file is saved

Spoiler

   if tagLine == "FILEDAT" then
    print("reciving file")
    file = fs.open("/"..info,"w")
    file.write(string.sub(e2,9,string.len(e2)))
    file.close()
    print("download Compleet")
   end

Spoiler

links to the images seprate.
http://i.imgur.com/3jAOZ.png
http://i.imgur.com/C8PaP.png

RedNet is not binary safe. You will need to use an encoding mechanism to ensure every byte is less than 128; BinHex or Base64 would probably be suitable.

Hawk777, on 08 April 2012 - 08:53 AM said:

RedNet is not binary safe. You will need to use an encoding mechanism to ensure every byte is less than 128; BinHex or Base64 would probably be suitable.

i haven't checked it the data is altered when read from the file yet. if it is not then this code i found will be perfect for binary file sending.

http://lua-users.org/wiki/BaseSixtyFour

I'm still not particularity happy about the way rednet works hopefully you bringing it to peoples attention will result in a fix for data compatibility in next ver.

so after incorporating the above code into my program first of all it is slow and has caused failure to yeild one time I used it. when it did work the same question marks appear. so I can conclude that the file when read with

Spoiler

	if fs.exists("/FTP/"..string.sub(e2,9,string.len(e2))) then
	 file = fs.open("/FTP/"..string.sub(e2,9,string.len(e2)),"r")
	 local sholding = file.readAll()
	 file.close()
	 sholding = enc(sholding)
	 nodeSender(e1,"FILEDAT "..sholding)
	 print(sholding)
	 print("file sent")
	 sholding = nil
	else
	 print("failed to send File not found")
	end

will always be corrupted unless it is a text/lua file. this is a bug ( almost borderline feature request ) as such i will prepare a proper bug report with a example program tomorrow as it is late. The file in question was 16.2 KB in size. and yes i did decode it properly and checked it with notepad++ as previous example above.

I tried the line local sholding = enc(file.readAll()) but that caused failure to yield so i split it up to slightly reduce the work load.

EDIT
i decided against posting it as a bug as lua was never designed for data manipulation it makes perfect sence that it would not have a read function for data. so i posted it in suggestions topic here
http://www.computerc...rror-corection/

I replied to your repost.

and I read your reply.

but the amount of CPU time spent reading in then encoding and then again decoding and saving is unacceptable as such I will no longer attempt a safe method for non character data transfer. considering that the XOR in bit api is a cpu hod by its self. it would not be possible for a series of routers to handle the excessive loads caused by the cumulative circumstances.

I thank you for your reply.

[RANT]
XOR is a basic function of a processor core to emulate it on a emulated platform in a emulated machine is just plan ludicrous.
WINDOWS / JAVA / LUA
it is just not a good idea. call me when you upgrade to 5.2 and XOR is properly incorporated.
[/RANT]
the above rant is aimed at dan200 not Hawk777

BigSHinyToys, on 09 April 2012 - 07:22 PM said:

and I read your reply.

but the amount of CPU time spent reading in then encoding and then again decoding and saving is unacceptable as such I will no longer attempt a safe method for non character data transfer. considering that the XOR in bit api is a cpu hod by its self. it would not be possible for a series of routers to handle the excessive loads caused by the cumulative circumstances.

I thank you for your reply.

[RANT]
XOR is a basic function of a processor core to emulate it on a emulated platform in a emulated machine is just plan ludicrous.
WINDOWS / JAVA / LUA
it is just not a good idea. call me when you upgrade to 5.2 and XOR is properly incorporated.
[/RANT]
the above rant is aimed at dan200 not Hawk777

Lua was never designed to be used for encryption. Using a C-library is much faster at that. On top of that, ComputerCraft uses LuaJ, a Java implementation of Lua.

Yes, it's unfortunate that the binary file API only provides the ability to read one byte at a time, necessitating the slow construction of a large table for concatenation into a string. However, this overhead only happens when reading or writing a file; there's no reason why your intermediate routers should ever do this (they could just take in a string and send it out again, which is very fast even for very large strings). Perhaps we should put in the Suggestions forum the ability to read large blocks of binary files directly into strings?

Of course, if you’re doing Base64 encoding anyway, you need each byte individually, so there’s no reason to put them together into a string first; it’s probably better to read one byte at a time from the file in groups of three, insert the corresponding four-character strings into a table, and then finish up with a call to “table.concat” to build the full Base64-encoded string (this should be much faster than concatenating each small string to a growing string, because doing many concatenations is slow).

Hawk777, on 09 April 2012 - 10:28 PM said:

Yes, it's unfortunate that the binary file API only provides the ability to read one byte at a time, necessitating the slow construction of a large table for concatenation into a string. However, this overhead only happens when reading or writing a file; there's no reason why your intermediate routers should ever do this (they could just take in a string and send it out again, which is very fast even for very large strings). Perhaps we should put in the Suggestions forum the ability to read large blocks of binary files directly into strings?

Of course, if you’re doing Base64 encoding anyway, you need each byte individually, so there’s no reason to put them together into a string first; it’s probably better to read one byte at a time from the file in groups of three, insert the corresponding four-character strings into a table, and then finish up with a call to “table.concat” to build the full Base64-encoded string (this should be much faster than concatenating each small string to a growing string, because doing many concatenations is slow).

The question is: why would you even want to do this in Lua? Lua is not designed to do any heavy lifting. You should do this in C or Java instead. And if you were to do it in Lua, you can do fh:read(1024) for 1024 bytes. I don't know if this works in CC, though.

On top of that, do you really need this grade of privacy in Minecraft? In my opinion, It'd be more fun to create my own encryption, rather than translate some other encryption into Lua.

Advert, on 09 April 2012 - 11:57 PM said:

The question is: why would you even want to do this in Lua? Lua is not designed to do any heavy lifting. You should do this in C or Java instead.

Because, for better or for worse, Lua is what we have. It takes a lot less skill and effort to write something in Lua than it does to write a Minecraft mod to plug into CC (even getting MCP set up is more effort than just pulling up a Lua interpreter in CC). And if you play on a server that you don’t run, or if you do run it but you want to seem fair to everyone else by not just installing all your own stuff, you simply can’t use anything but Lua.

Advert, on 09 April 2012 - 11:57 PM said:

And if you were to do it in Lua, you can do fh:read(1024) for 1024 bytes. I don't know if this works in CC, though.

This doesn’t work in CC. Whether you’re using a file handle from the “io” API or from the “fs” API, in binary mode, the “read” function takes zero parameters and returns one byte.

Advert, on 09 April 2012 - 11:57 PM said:

On top of that, do you really need this grade of privacy in Minecraft? In my opinion, It'd be more fun to create my own encryption, rather than translate some other encryption into Lua.

Personally, I think encryption is mostly pointless in CC altogether. If you play on a server where people destroy each other’s blocks, they could just go into your base, destroy any maximum-priority floppy drive you might have, and install their own code to sniff your password if they want to recover your encrypted data. If you play on a server where destroying each other’s stuff is forbidden, you don’t need encryption: just keep the data on a computer behind a locked door (if you need access to it at a public place via a password, use access control instead of encryption by having the public computer send the password to the secured one which checks it before returning the data).

That’s beside the point though. There are lots of applications for binary files other than encryption (and for that matter, encryption need not be used only on files). Maybe you want binary because you’re dealing with large volumes of data (unlikely in CC, but you never know). Maybe you want binary because you want to build an image decompresser that can display a PNG file on a monitor in ASCII art, or on a giant grid of Lumar lamps, or something. Maybe you want to implement a cryptographic hash function, like MD5, which you’ll use to verify the integrity of some files during a software auto-update protocol (I think I’m probably going to be doing this at some point), not because you’re worried about security but just because you want to make sure there wasn’t accidental corruption. There’s no reason we shouldn’t hope for a better way of manipulating binary data.

I agree with all of the above post.

I don't particular need to do encryption but it is just not good practice to make a FTP sever client pair that cant handle any file with random data patterns. Given the over heads I will now have to conciser whether i want to implement a fix or not. This is not a simple process i would have to alter the way the routers handle packets and give it some kind of header.
In terms of encryption i was going for something extremely simple just XOR the key and the data when you reach the end of the keys length start back at front. extremely simple. I think i will scrap this idea and just send them in base64 it should be as good as encrypted anyway. I dough most people would think to decipher it using base64.

Hawk777 maybe dan should add a way to read a file directly to base64 and a way to save that to file. would be extremely useful.
something like.
fs.open("/rom/proframs/redset","rb")
local val = file.readAll64()
rednet.send(val)