Encryption/Decryption?

I wanted to take my password door to the next level with encrypted files so that people can't hack the password.

Okay…

I recommend using hashing, specifically gravityscore's sha256 (it's on the forums somewhere). Encryption isn't suited to storing passwords.

short example of sha256:

local file = fs.open( "password", "r" ) --#open pass file
local hashpass = file.readAll()  --#read it
file.close()  --#close it
local password = read("*") --#get user input
if sha256( password ) == hashpass then
  --#congrats
else
  --#oops
end

Encryption isn't really suitable for this sort of thing. You'd generally use a hash.

The idea is that when you go to code in the correct password, you instead hash it and put that hash value into your script. When the user is prompted to type the password later, you hash their input and see if it matches the hash you made earlier. If they match, then hey presto! You let them in. If they manage to get access to the files stored on the computer, then they still don't see your password - they only see the hash you made from it.

Though if you have to worry about users sniffing your data, then you're already doing things wrong. Have a think about how users might go about breaking your system - odds are the solution isn't a coding one.

Anyways, What if I want a string of data encrypted with a userdefined key?
Like encrypting the transmission of rednet signals.

For password storage, you'll want hashing
I made a script in which you give it your password and it will create a password prompt script with your salted password hash.

For secure communication over rednet, you might want to take a look at KillaVanilla's Secure Tunnels, which uses Diffie-Hellman key exchange and AES-128