Packet switched networking - cover big distances with wireless

Hi everybody,

I just finished the first (more or less) working version of 'the internet'.

Latest version: 0.02 (10-03-2012 11:15)

Spoiler

10-03-2012 0.02
Misc bugfixes
Code cleanup
Added impersonation protection from terminals

09-03-2012 0.01
Initial release

WHAT IS IT?
wireless signals can carry only 50 blocks far. Suppose two terminals, A and B are too far away. There is no way to communicate wirelessly. With wires, it would become difficult to communicate with multiple devices. What to do?
Consider the following setup:

A(id=1) ——— gateway(id=2) ———– B(id=3)

The gateway runs a special program, listening for incoming connections. If A wants internet access, it first registers using the REGISTER command. This broadcasts a request to which any gateway in range will reply. A will choose one of the gateways and send it a registration confirmation. This gateway now will be able to forward incoming packets for A to A. Suppose A and B register.

The message protocol is simple: a message has the following form: <dest_id>|<source_id>|<message>|<misc info>
Example: if A would want to send 'hi' to B, it sends the following to the gateway: 3|1|hi!|
The gateway knows how to find B and will deliver the message.

IT GETS BETTER THOUGH!

Imagine the following. All nodes are gateways, and each may be attached to one or more terminals. The lines mean that the gateways are in each others range.



Let's repeat our last example. If a terminal 1 would be connected to gateway 0, and he wants to send a packet to terminal 3, connected to 2? No problem. Same packet as before: 3|1|hi!|. The gateways will take care of the shortest path from 1 to 3!

There are 4 files, to be found as attachments to this message

The main api, called 'mnet'
The gateway program
The register program for terminals
And a small tool, NetCat. Use it like this: nc <id> (must be in range)
You can then type commands, and it will wait a few secs for an answer. useful for debugging.

REMOVE THE FILE EXTENSION from all 4 files. I wasn't allowed to upload it with another extension than .txt..

HOW TO USE

• Install: put the three programs in .minecraft/mods/ComputerCraft/lua/rom/programs, and the api, mnet, in ./minecraft/mods/ComputerCraft/lua/rom/apisM
• Make a few terminals
• Put wifi on top
• (optionally, put lamp block underneath, will light up while processing a packet.)
• run gateway or terminal

you can now send packets from registered terminals that are not in range, as long as you have a path of gateways that are in range between the two.

UNDERSTANDING THE SYSTEM

Spoiler

For debugging purposes, you can use the get_gwlist (get gateway list) that requests a list of the network. It can be obtained as follows: (0 is the gateway, 2 is the client)

nc 0
0|2|get_gwlist|

The server will reply with:

2|0|gateway_gwlist|0>3 0>1 3>4 :2>0 6>4
What does this mean? The part before the semicolon describes the gateways, and connections between gateways. Thus, there are 4 gateway: 0, 1, 3 and 4. 0 is connected to 3 and 1, and only 3 is connected to 4.
The part after the semicolon describes hosts, and a>b means "For a, send to gateway b". So, 2 is registered to gateway 0 and 6 is registered to gateway 4.

PLEASE
If anyone has suggestions of how to improve the code or functionality, please let me know!
Please test it, it's not completely done yet. It should work, as long as no gateways are removed from the system. In the future, and with enough enthousiasm from you guys, I might add support for this all.

Sounds cool, I'm afraid I can't test it at the moment, but this sounds really advanced, probably even better than Cookieball's Skynet. ;)/>/>
Can't you just let the program add the sender id? At the moment you could just impersonate as another computer. :mellow:/>/>

I think it might be cool to track a message, so you can see where it came from and which way it took.

I experienced the problem with uploading files without .txt
The solution is simple: .zip and you can also store them all in one .zip! :mellow:/>/>
Anyway back on topic:
Sorry if i sound like an idiot! But i don't know how to install the mnet API ;)/>/>

This is an important step, but what I admit I would love to see, is a ports/sockets system. The existing rednet is only capable of sending one message at a time. It looks as though your system is capable of sending messages sequentially to different hosts, which is great; but I'm still not really seeing how that is fully multiple message capable. Unfortunately, I don't think I'd have a prayer of coding something like this myself, but I really might have to try.

Ludburghmdm, on 09 March 2012 - 10:43 PM said:

I experienced the problem with uploading files without .txt
The solution is simple: .zip and you can also store them all in one .zip! :mellow:/>/>
Anyway back on topic:
Sorry if i sound like an idiot! But i don't know how to install the mnet API ;)/>/>

Hi, thanks for the tip, ill do that next time. You can install the api by adding the file to .minecraft/mods/computercraft/lua/apis, if I recall well. Ill add instructions in my first post tomorrow.

Ok thanks! (And your welcome)

Modems (as contrasting with bundled cable) can carry multiple message "simultaneously" (actually it's just that messages are delivered instantly), so that part isn't a problem. There's no port-number-like primitive, but you can always tack a short string on the front indicating what application or subsystem the message is for (for example, every RedNet message in my railway system starts with "RWY:").

petrus4, on 09 March 2012 - 10:46 PM said:

This is an important step, but what I admit I would love to see, is a ports/sockets system. The existing rednet is only capable of sending one message at a time. It looks as though your system is capable of sending messages sequentially to different hosts, which is great; but I'm still not really seeing how that is fully multiple message capable. Unfortunately, I don't think I'd have a prayer of coding something like this myself, but I really might have to try.

"ports" are supported, in some way. Recall the message format: <to_id>|<from_id>|<data>|<misc>
Internally, gateways use the <data> field to describe the 'type' of their message. However, it can very well be a port number, if you're systems are configured to use this. I just don't see yet why you'd use this :-) readable identifiers are more convenient - like Hawk777 says.

The system as it is right now is best compared to UDP, not TCP. I'm working on some sort of 'delivery failure notification' status codes.

Leo Verto, on 09 March 2012 - 10:40 PM said:

Can't you just let the program add the sender id? At the moment you could just impersonate as another computer. :mellow:/>/>
I think it might be cool to track a message, so you can see where it came from and which way it took.

That's a nice idea, indeed, the gateway could add the source id, in order to avoid terminals to spoof the source. I'll add this in the next update.

EDIT:
Implemented!


This is running on a terminal with id 3, registered to gateway 0. On the third line, I do a legitimate request to gateway 5, to get his known list of gateways. Notice 5|3, which means "To 5, from 3". On the 4th line, I receive the reply. But, on the fifth line, I try again, with a spoofed source id (4). The gateway notices this, and before forwarding the packet, an error is returned. I'll upload the new files in a sec

Another problem I'm having with rednet at the moment is persistence; in the sense that I can send a single command over it, but once that single command has finished executing, the receiver stops listening and I have to restart the program. Even if I could write a listen daemon, I'm not sure if that would work with CraftOS; although MysticOS mentioned multitasking. I might have to give that a try.

petrus4, on 10 March 2012 - 02:55 PM said:

Another problem I'm having with rednet at the moment is persistence; in the sense that I can send a single command over it, but once that single command has finished executing, the receiver stops listening and I have to restart the program. Even if I could write a listen daemon, I'm not sure if that would work with CraftOS; although MysticOS mentioned multitasking. I might have to give that a try.

You don't have to use rednet.receive() to listen for messages.
You can pull events via os.pullEvent() and run that inside a loop. This way after the receiver is done with executing your command, it will just loop back to the beginning and wait for another event.

So I'd suggest try looking into os.pullEvent(). There are a multitude of examples all over the forum.
Take a look at the tutorial section, the Ask a Pro section etc. and use the search with "os.pullEvent".
That should give you plenty of resources and help you out. :mellow:/>/>

Espen, on 10 March 2012 - 03:03 PM said:

petrus4, on 10 March 2012 - 02:55 PM said:

Another problem I'm having with rednet at the moment is persistence; in the sense that I can send a single command over it, but once that single command has finished executing, the receiver stops listening and I have to restart the program. Even if I could write a listen daemon, I'm not sure if that would work with CraftOS; although MysticOS mentioned multitasking. I might have to give that a try.

You don't have to use rednet.receive() to listen for messages.
You can pull events via os.pullEvent() and run that inside a loop. This way after the receiver is done with executing your command, it will just loop back to the beginning and wait for another event.

So I'd suggest try looking into os.pullEvent(). There are a multitude of examples all over the forum.
Take a look at the tutorial section, the Ask a Pro section etc. and use the search with "os.pullEvent".
That should give you plenty of resources and help you out. :mellow:/>/>

I think what your saying is something like this:

while true do
action, senderID, text = os.pullEvent()
if action == "rednet_message" then
print(senderID, " ", text )
end
end

Made the code just now. Dunno if it works

Thanks, Espen. :mellow:/>/>

And, does anyone have an idea for the following: an autonomous turtle, mining stuff, that still listens for commands from my central server? Only thing I came up with until now is to poll the server every few seconds in an "is there a command for me" fashion.. not exactly the ideal solution… Anyone experience with threading? If I could have a thread that saves received unprocessed packets, I could check every second if something has been received.

@Monochrome:
Yes, you can let it run multiple threads at the same time.
Take a look at the last example here: http://www.computercraft.info/forums2/index.php?/topic/144-useful-tutorials-resources/page__view__findpost__p__2241

To add a little more information:
parallel.waitForAny() will stop all co-routines as soon as the first co-routine has finished running.
parallel.waitForAll() waits until all co-routines are finished.

I managed to crash the gateway by sending a invalid packet:

|get_gwlist|

is there some fix or what? also I am going throu your code to add I.P. v2 support XX.XX with 11.XX for individual networks.

I made a diagram for my design.
{if cant post with image then post link to image end}

https://docs.google.com/open?id=0B7nap5k8HCLyYXF2YVZCQUJTcS1xZ000QVN1RGZRZw

all i need now is to figure out the base code race you to the end :)/>/> but first can i post the code if i edit in some more?

edit: i am not clueluss :D/>/>

In the mnet API, in parse(), it should be checking for sep1, sep2, and sep3 being nil, instead of always checking sep1.

sep1 = string.find(packet, "|")
if (sep1 == nil) then return nil, nil, packet, nil end

sep2 = string.find(packet, "|", sep1 + 1)
if (sep2 == nil) then return nil, nil, packet, nil end

sep3 = string.find(packet, "|", sep2 + 1)
if (sep3 == nil) then return nil, nil, packet, nil end

After I changed that I was unable to crash the gateway with malformed packets. However, to be safe, I also created a simple wrapper that runs gateway in a loop so that if it does crash it will restart.

I don't like how your system relies upon a single master server to keep track of everything. How will you route packets from distant gateways to your master server before they have IPs assigned to them? And what happens if this server becomes unavailable?

You haven't told us what routing protocol you use

That's a pretty important item in network design

Is there any possible way you could be a little clearer with your instructions? For example, How to send/receive/read messages. Post a tutorial video or something, this might as well be in chinese.

This network is really handy, ive got some nifty ideas im working on using this, Great job!