KillaVanilla's Various APIs

Hello all.
After my previous topics about my APIs mysteriously disappeared, I decided to settle on making one big topic to avoid cluttering the topic list.

Anyways, on to the APIs:
CRF ("Common" Routing Framework):
That's right, it's back! This API / program (can't tell what it is) implements the Internet! To be precise, it implements a framework to route packets from one computer to another. It acts much like the Internet Protocol (IP) does in real life.

Functions:

• run(debugFlags) – Run the CRF's main function. This is meant to be run in the background (except when running with debug flags; see below).
• send(to, data) – Send a string to the specified computer, assuming that there is a route listing for that computer.
• getRoutingTable() – Get a copy of the internal routing table used by the CRF; this table is a dictionary mapping hosts on the network to their next-hop interfaces.
• getRoute(to) – Get the full route to the specified host.
• compileDebugFlags(ev, tm, rg, sf, de, pe, le) – Compiles a set of "debug flags" into a single integer that can be passed to run().
• Debug flags:
• A set of debug flags is a 7-bit integer; each flag corresponds to a single bit. 0x7F corresponds to a set with all flags on.
• ev - "Events"; this corresponds to every debug output besides the ones relating to route generation.
• tm - "Timed" / "Timing"; this corresponds to all debug output regarding the internal timers used.
• rg - "routeGen"; this corresponds to all debug output regarding the route generation mechanism. This includes the BFS search algorithm used.
• de - "dataEvents"; this corresponds to all debug output regarding incoming packets.
• pe - "pingEvents"; this corresponds to all debug output regarding the Reachability Protocol used internally.
• le - "lsaEvents"; this corresponds to all debug output regarding sent and received link-state advertisements (LSAs).

Notes:

• I'm planning on implementing something similar to iptables and Netfilter, or possibly just a generalized extension system. Stay tuned.
• I'm also planning on writing some Transportation and Application layer protocols; these will go in the Programs section.
• As with Rednet Tunnels (see below), there's no way to ensure that nodes are who they say they are; authentication is left to the user.

Get it here. (Pastebin code: BhZT6Pt7)

AES (Advanced Encryption Standard):
This API implements the Advanced Encryption Standard (aka Rijndael) with a 128-bit (16 byte) block size. It also provides functions to encrypt and decrypt bytestreams (tables of bytes) and strings.
It also provides a function that implements the Davies-Meyer one-way compression function (basically, a hash function).

Functions:

• encrypt_block(data, key) – Encrypt a 16-byte block of data, using a variable-length key. Blocks (for both keys and plaintext) are stored as 16-element tables. For example:

  
  key = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}
  

• decrypt_block(data, key) – Decrypt a block of data.
• encrypt_bytestream(data, key, iv) – Encrypt a table of bytes. Padding consisting of 0's is added to the end of the bytestream if the length is not a multiple of 16. The IV (initialization vector) is in the same format as a block.
• decrypt_bytestream(data, key, iv) – Decrypt a table of bytes. Any padding added to the end of the bytestream is automatically removed.
• encrypt_bytestream_ecb(data, key) – Encrypt a table of bytes in Electronic Codebook mode. This method of encryption is not secure and is only kept for compatibility reasons.
• decrypt_bytestream_ecb(data, key) – Decrypt a table of bytes encrypted with ECB mode.
• encrypt_str(data, key, iv) – Encrypt a string. If an IV is not provided, the function defaults to ECB mode.
• decrypt_str(data, key, iv) – Decrypt a string.
• davies_meyer(data, h0) – Compress a block of data using the Davies-Meyer compression function. h0 (hash[0]) acts as the IV for this function.
• new_ctrMode(key, iv) – Create a new object describing AES in Counter (CTR) mode. This turns AES into a stream cipher.
• Counter Mode functions:
• set_key(self, key) – Set the key. <key> can be either a string that has 16 characters or more or a table with 16 entries or more.
• set_ctr(self, ctr) – Set the counter. Again, <ctr> can be either a string that has 16 characters or more or a table with 16 entries or more. In addition, <ctr> can also be a 32-bit number.
• generate(self, bytes) – Generate <bytes> bytes of keystream.
• The exact order of the keystream depends on the key; the counter (or IV) determines where you are in the keystream.
• All functions return the encrypted/decrypted data in the same format as the input. For example, encrypt_str() and decrypt_str() will return strings.
• Keys can be any bytestream that has at least 16 bytes. The functions automatically use the correct settings for each key length.
• For reference, the key lengths used are 128, 192, and 256 bits, or 16, 24, and 32 bytes respectively. Only the first 32 bytes of the key are used when the length of the key is greater than 32 bytes.

Find it here. (Pastebin code: rCYDnCxn)
Example Program
FileHider

Base64 Encoder / Decoder:
This API implements a Base64 encoder and decoder.

Functions:

• encode(data) – Encode a bytestream with Base64. Returns a string.
• decode(encoded_data) – Decode a Base64 encoded bytestream. Returns a table.

Find it here. (Pastebin code: pp3kpb19)

Secure Rednet Tunnels:
This API implements (mostly) secure end-to-end encrypted communications via Rednet (modems).
The function's used are reminiscent of the "old" Rednet API (i.e no channels, no one else sees your messages but the receiver, etc.), aside from the lack of a broadcast function (which is intentional; broadcast would probably never work with key exchange).

The API / protocol uses Diffie-Hellman key exchange with relatively small primes, and exchanges both a key and an IV.
The encryption algorithm used is AES-128 (see above).

Functions:

• connectionHandler(modem, debug) – This function handles incoming connection attempts and also handles processing of incoming data. You will need this function running in the background (in a coroutine) in order to be able to be connected to and to receive (but not send) data. If debug is set, then the function prints debug info (such as incoming data and the stages of key negotiation).
• openTunnel(modem, computerID, debug) – Open a tunnel to a specified computer. This function will take a while to run, and returns a "connection ID" that is needed to send data.
• sendTunnel(modem, connectionID, data) – Send a string with the specified connection.
• sendTunnel_raw(modem, connectionID, packetType, data1, data2, data3) – Send a raw "packet" (datagram).
• sendTunnel_withFreq(modem, connectionID, data, sendFrequency, replyFrequency) – Send a string with the specified connection. The receiver will emit a fake "modem_message" event with the specified frequencies and data, as well as the usual "secure_receive" event. Additionally, said fake event will appear to come from the left side.
• listenTunnel(connectionID, timeout) – Listen for incoming data on the specified connection, and return it if we get any data. If the timeout elapses without any data being received, then the function returns nil. Otherwise, it returns the sender's computer ID and the received and decrypted data.
• closeTunnel(modem, connectionID) – Closes the specified connection.
• switchConnectionID(modem, oldCID, newCID) – Switch connection IDs.
• switchKeys(modem, connectionID, key, iv) – Switch the specified connection's symmetric key and initalization vector.
• openConnectionRaw(connectionID, key, iv) – Open a connection without doing an exchange.
• installTunnelAsModem(realModem, connectionID, sideName) – Create a fake peripheral object that works with the peripheral API (i.e it will be listed by peripheral.getNames(), peripheral.isPresent(sideName) will return true, etc.) and has methods. This fake object will, for all intents and purposes, look like a wireless modem, complete with all methods. This function cannot overwrite real peripherals.
• getAllOpenConnections() – Returns a list of connection IDs in ascending order.

Events:

• secure_receive – We have received data on a secure connection. Other arguments: the sender's ID, the connection ID, the received and decrypted data.
• secure_connection_open – A secure connection has been opened. Other arguments: connector's computer ID, connection ID
• secure_connection_close – A secure connection has been closed. Other arguments: the closer's computer ID, connection ID
• secure_connection_newID – A secure connection has switched IDs. Other arguments: the connection's old ID, the new connection ID
• secure_connection_newKey – A secure connection has switched keys. Other arguments: The initator's computer ID, the connection ID
• secure_connection_ack / secure_connection_nak – Used internally, while switching IDs. Other arguments: The connection ID, the sender's computer ID

Packet Types (case sensitive):

• Message - A standard message. Parameters: data (string)
• fakeModemMsg - A message meant to be received as a faked modem message. Parameters: data (string), transmitFrequency, replyFrequency (both numbers)
• close - A message indicating that connection is to be closed. Parameters: none
• newKey - A message supplying a new key and initalization vector pair. Parameters: key, iv (both tables, see AES information above for more info)
• newID - A message indicating that the connection's ID is to be changed if possible. Parameters: new connection ID.
• ack_pos - A message indicating success. Currently only used when changing connection IDs (to indicate that the change was successful).
• ack_neg - A message indicating failure. Currently only used when changing connection IDs (to indicate that the change was not successful).

Notes:

• Due to the way the API works, no "broadcast" support will be implemented at all.
• Multi-user connection (and thus, broadcast / multicast) support is included. You will need to distribute the session keys yourself, however. One-to-one communications still work as normal.
• The events (aside from "secure_receive") fire regardless of whether the event was triggered locally (by the current computer) or by another party.
• Every party in a multi-party connection will need to know both the connection ID and symmetric key / initalization vector. openConnectionRaw() can help here.
• This protocol uses small primes in the key exchange, which lowers security slightly.
• Nevertheless, prime generation will still take several seconds (less than a minute on my testbed).
• Secure Rednet Tunneling's fake peripherals will conflict with other APIs and programs that modify any of the peripheral API's methods.
• The connection handler function is vulnerable to a Denial of Service attack similar to a SYN flood; if an attacker attempts to connect to a server but never goes past the first part of the handshake, then the server will wait until the timeout period (30 seconds by default) has elapsed.

Find it here. (Pastebin code: r8AFFkbV)

Cryptographically Secure Pseudo-random Number Generator:
This API implements two random number generators, the Mersenne Twister PRNG and the ISAAC CSPRNG, using the Mersenne Twister to seed the ISAAC algorithm.

Functions:

• initialize_mt_generator( seed ) – Seed the Mersenne Twister PRNG.
• extract_mt( min, max ) – Get a number from the Mersenne Twister.
• seed_from_mt(seed ) – Seed ISAAC using numbers from the Mersenne Twister, possibly seeding the MT beforehand (with the supplied seed).
• generate_isaac( entropy ) – Generate a new batch of numbers using the ISAAC algorithm, using user-defined entropy if supplied. The "entropy" argument is optional and is a 256-entry table of seed values.
• random( min, max ) – Get a number from the ISAAC PRNG.
• The min and max arguments default to 0 and (2^32)-1, respectively.

Find it here. (Pastebin code: D1th4Htw)

Arbitrary Precision Unsigned Integers:
This API implements arbitrary-precision arithmetic. Numbers are stored as tables, with each entry specifying one digit, in least-significant-digit first order.
More detailed information can be found on the old topic.

Functions:

Spoiler

• toBigInt( number or string, base ) – Convert a number or string to its big-integer representation. The base argument only applies if the number to be converted is of type string.
• toStr( bigInt ) – Convert a big-integer to its string representation.
• add( a, b ) – Calculate a+b, in big-integer representation.
• sub( a,b ) –Calculate a-b, again, in big-integer representation.
• mul( a,b ) – Calculate a*b.
• div( a,b ) – Calculate a/b.
• mod( a,b ) – Calculate a%b.
• exp( a,b ) – Calculate a^b.
• toBinary( a ) – Convert a number to its binary representation.
• fromBinary( a ) – Convert a number from a binary representation.
• bitwiseOR( a, b ) – Calculate a | b.
• bitwiseAND( a, b ) – Calculate a &amp; b.
• bitwiseXOR( a, b ) – Calculate a (XOR) b.
• bitwiseNOT( a ) – Calculate ~a (note: only complements up to the highest bit in a.)
• bitwiseLeftShift( a, b ) – Calculate a << b.
• bitwiseRightShift( a, b ) – Calculate a >> b.

Find it here. (Pastebin code: th3Vrppb)

SHA-256
I actually wrote this a few months ago but never got around to publishing it. It's an implementation of SHA-256.

Functions:

• digest(bytestream) – Produce a SHA256 digest of a bytestream.
• digestStr(string) – Produce a SHA256 digest of a string. Uses digest() internally.
• hmac(data, key) – Produce a MAC (message authentication code) using SHA256 and the HMAC construction. Both data and key should be bytestreams.
• As with the AES API, the functions return the types they accept as input.

Find it here. (Pastebin code: 9c1h7812)


Error Detection Codes:
This API implements 16 and 32 bit CRCs, as well as the Adler32 and Fletcher32 checksum algorithms.

Functions:

• adler32(input) – Calculate a Adler32 checksum.
• fletcher32(input) – Calculate a Fletcher32 checksum.
• crc16(input) – Calculate a 16-bit CRC checksum, using the CRC-16-CCITT polynomial (0x1021).
• crc32(input) – Calculate a 32-bit CRC checksum, using the CRC-32 polynomial (0x04C11DB7).
• All functions take bytestreams as input, and produce integers (type: "number").

Find it here. (Pastebin code: SimEHhCH)

Good idea :)/>/> you should copy the others in too ;)/>/> are you missing one of your latest?

Edit: hmm the forum time seems to be out by 6hrs for me :/

theoriginalbit, on 30 April 2013 - 08:01 PM said:

Good idea :)/>/> you should copy the others in too ;)/>/> are you missing one of your latest?

Edit: hmm the forum time seems to be out by 6hrs for me :/

I should, come to think of it.

Also, I have no idea what I'm missing, if any..

Well it just seems like one is missing, I feel like you made a 4th one.

theoriginalbit, on 30 April 2013 - 08:19 PM said:

Well it just seems like one is missing, I feel like you made a 4th one.

Oh well. Perhaps you're thinking of my FileHider program?

Anyhow, updated. I decided to release my SHA256 implementation.

Edit: Updated again. Secure Tunnels now has functions to allow developers to run (most) normal rednet protocols and programs "over" Secure Tunnels, using fake peripheral objects.

Hello, sorry to be finicky but is there any chance you could post a small example program for the AES-128. I'm a touch overwhelmed by it.
Cheers, Lord of Shadows

LDShadowLord, on 05 May 2013 - 08:40 AM said:

Hello, sorry to be finicky but is there any chance you could post a small example program for the AES-128. I'm a touch overwhelmed by it.
Cheers, Lord of Shadows

Here you go.

This program is a heavily commented file encryption program.
All it is supposed to do is read a file's bytes, encrypt them, and write them back (again, in binary mode). It also writes the randomly-generated key and IV to another file, in case anyone else wants to try and write a decryption program.
The decryption program shouldn't be too hard to write; it should also be good practice.

Excuse me, but I want to just encrypt something with a key supplied as a string. Is that possible? Also, what is the IV for?

prasselpikachu, on 07 May 2013 - 11:33 AM said:

Excuse me, but I want to just encrypt something with a key supplied as a string. Is that possible? Also, what is the IV for?

You can convert strings into blocks (whether that block is a key or plaintext is up to you).
Example:

local key = {}
local my_key_string = "This is a key" -- 13 characters long
for i=1, 16 do
	key[i] = string.byte(my_key_string, i, i) or 0
	-- This both converts the characters to bytes and adds padding.
end

-- You could also use strToBlocks():
key = AES.strToBlocks(my_key_string)[1] -- Replace "1" with whatever group of 16 you want to use.

As for your question on IVs:

The initialization vector (IV for short) is used as essentially another key.
The IV, when used in conjunction with one of many modes of operation, is used to ensure that the encrypted message "looks" random.
The simplest mode of operation (and the one most think of when they hear "encryption") is Electronic Codebook (ECB) mode.
ECB mode consists of breaking the message into blocks and encrypting each block separately.

The problem with doing this is that blocks that share the same plaintext will have the same ciphertext; the encryption does nothing to hide the fact that parts of the message are similar.

This is easily seen when looking at an image of Tux that was encrypted with ECB:


Using a mode that uses an IV (such as the CBC mode used in the API) ensures that the message (aka whatever you want to encrypt) actually looks random:


The above image could have been encrypted with CBC, CTR, or any of the other modes of operation.

In case anyone's wondering, I got the images from the article on "block cipher modes of operation" on Wikipedia.

Updated; I've released CRFv3. Details are in the main post.

Also, a snippet I decided not to add to the OP:
A simple way of running CRF alongside the default shell is:

if not _G["startedOnce"] then
    -- Any code in here will only execute the first time the startup file is run.
	_G["startedOnce"] = true
	os.loadAPI("CRFv3") -- Replace with whatever you saved CRF as
	term.clear()
	term.setCursorPos(1,1) -- Don't really *need* these two lines, they just make it look good
	parallel.waitForAll(function() CRFv3.run() end, function() shell.run("shell") end)
end
-- Any code out here will execute both times the startup file is run.

This is meant to be used in a startup file.

This snippet of code:

1. Starts when the computer boots,
2. Saves a flag to the global namespace saying that it's already running, and
3. Executes CRF's main function alongside another instance of the shell (and thus, itself).
4. The second instance of the startup file doesn't do anything thanks to the conditional.

Hooray for recursive startup files!

Your security scripts are awesome! Thanks!

Great Job will use it a lot in my mainframe program :]

Also a simple script utilizing the Secure Rednet tunnels would be nice.

There seem's to be a problem with the SHA implementation.

Comparing the hash with the same hash generated by Immibis's Cryptographic Accelerator and here they produce the same hash while this implementation doesn't.

Spoiler

I'm trying to create a mechanism where passwords are hashed by what's available, so if there's a crypto accelerator it uses that, otherwise it falls back to this implementation.

distantcam, on 20 November 2013 - 03:24 AM said:

There seem's to be a problem with the SHA implementation.

Comparing the hash with the same hash generated by Immibis's Cryptographic Accelerator and here they produce the same hash while this implementation doesn't.

Spoiler

I'm trying to create a mechanism where passwords are hashed by what's available, so if there's a crypto accelerator it uses that, otherwise it falls back to this implementation.

Have you tried GravityScore's SHA-256 implementation? Does it agree with the others?

Yes GravityScore's calculates the correct hash as the other implementations. Thanks for pointing me to it.

Ok, don't take this as reviving an old topic but…
I made Secure Rednet Tunneling a bit faster at connections, I haven't tweaked it to be any faster but it's alot faster than it was.
You can find it here: http://pastebin.com/4LGs2n6t
Basically, I added an event timeout, where it queues and pulls an event at certain intervals instead of all the time
It should be 50 times faster (since I only wait every 50 iterations)
I tested it and it is much faster than the real api :)/>

ds84182, on 02 February 2014 - 02:03 PM said:

Ok, don't take this as reviving an old topic but…
I made Secure Rednet Tunneling a bit faster at connections, I haven't tweaked it to be any faster but it's alot faster than it was.
You can find it here: http://pastebin.com/4LGs2n6t
Basically, I added an event timeout, where it queues and pulls an event at certain intervals instead of all the time
It should be 50 times faster (since I only wait every 50 iterations)
I tested it and it is much faster than the real api :)/>

I've actually tried something similar with this in the past before, only instead of using a counter, it saved calls to os.clock(), like so:

local lastPause = os.clock()
while true do
    -- ...processing...
   if os.clock() - lastPause >= 2.7 then -- adjust the 2.7 to however long you want to go between pauses; it has to be within 3 seconds though
	   os.queueEvent("")
	   os.pullEvent()
	   lastPause = os.clock()
    end
end

Love the AES library, although I have some issues using the CTR mode.
I get "aes:788: attempt to perform arithmetic __add on number and nil" upon calling ctr:generate(bytes).
It's trying get the "IncAmt" variable, which is undefined.

Anavrins, on 12 March 2014 - 04:39 AM said:

Love the AES library, although I have some issues using the CTR mode.
I get "aes:788: attempt to perform arithmetic __add on number and nil" upon calling ctr:generate(bytes).
It's trying get the "IncAmt" variable, which undefined.

…Are you using the latest version of the AES API?
The issue appears to be fixed in the version on Pastebin.

KillaVanilla, on 12 March 2014 - 08:45 AM said:

…Are you using the latest version of the AES API?
The issue appears to be fixed in the version on Pastebin.

Assuming you mean this pastebin: http://pastebin.com/rCYDnCxn
Then no, it still occurs for me somehow, and it still is that undefined "incAmt" variable in the increment_ctr() function.

Anavrins, on 12 March 2014 - 12:56 PM said:

KillaVanilla, on 12 March 2014 - 08:45 AM said:

…Are you using the latest version of the AES API?
The issue appears to be fixed in the version on Pastebin.

Assuming you mean this pastebin: http://pastebin.com/rCYDnCxn
Then no, it still occurs for me somehow, and it still is that undefined "incAmt" variable in the increment_ctr() function.

Oh, whoops. Forgot about that.
I'll get to fixing it.

So How do I get rid of the table stuff at the end of my SHA256 output?
How comes the table bit changes all the time even though its the same script run?

Spoiler

--SHAtest

os.loadAPI("sha256")

output = "This is my password123456"

print(sha256.digestStr(output))

edit:
solved I belive. If you see a mistake in how i have achived this please tell me guys.

os.loadAPI("sha256")
password = "This is my password123456"
text =sha256.digestStr(password)
print(text)

why does my script receive a java exception thrown error when i use xxx = "testing" base64.encode(xxx)

i tried moving th quotes but that just gives a table after decode

Rougeminner, on 01 December 2014 - 06:00 AM said:

why does my script receive a java exception thrown error when i use xxx = "testing" base64.encode(xxx)
i tried moving th quotes but that just gives a table after decode

That's the expected behavior, since base64 is used to encode binary into text, and once decoded you can't store binary into a string.
Each keys in the table contains the string.byte() of each characters, so "hello" would be {104, 101, 108, 108, 111}
You can turn it back into a string with this function

local function byteToString(bytes)
  return (("%c"):rep(#bytes)):format(unpack(bytes))
end

Um… you could also use this much simpler (and faster) way:

string.char(unpack(bytes))