netOS development thread - first OS with "internet" build in.

grabie2 #1

grabie2's profile picture

78 posts

Location Poland

Posted 15 December 2012 - 07:40 AM

Hello everyone!

This is development thread - here we're discussing newest versions on project SVN, if you want to get current release see netOS main thread: http://www.computerc...uild-innot-yet/

I'm looking for someone to help me with that pretty difficult task, I will need:
- printer driver (printing queue, page management, paper/ink management, API for it, user application[console], future printing over network, mapping network printers)
- beta tester
- someone to help programming GUI
- printer application for GUI that will be created
- possibly more

I'm Bartek from Poland, 15 years old and I love computer networks. I'm creating my own implementation of Internet that will work on CC computers. The protocols i'm planning to have native support:
- ARP
- UDP
- TCP
- DHCP
- HTTP
- FTP
- DNS
- TELNET

Network will be made of layers - similar to real Internet:
1st - rednet
2nd - IP - thx to huettner94
3rd - UDP/TCP
4th - application protocols for example DNS, HTTP, FTP

I also plan(later) to implement AES cryptography, so there will be things like:
- SSH
- SSL
- VPN
BUT performance will be big issue, possibly I will make addon to CC(like peripherals) that will add special encrypting API.

There will be native dynamic routing protocols, I don't yet know which, or maybe my own - that's really far future.

For more see project page(below).

It's multitasking OS, so you'll be able to have multiple consoles at the same time, for example lua interpreter on local terminal, paint on left monitor, information about your turtles on top screen, star wars video on right and more!
Of course I'll make it so you can easily switch between terminals.

I've implemented VTYs - virtual terminals. Currently they aren't supporting colors, but will soon.
Also textutils API has disabled all printing features, cause they aren't supporting VTYs.
To switch between terminals use CTRL+<number>.

Here is project page: http://netos.robotronika.pl
If you found any bugs please post it on bugtracker: http://netos.robotronika.pl/newticket
Do download source code use SVN: http://svn.robotronika.pl/svn/netos/
user: netos
password: netos

Currently I have implemented about 60% of OS API and very little of networking. After I've done working OS I'll go back to networking stuff.

Changelog(SVN revisions):
23:

working IP layer
ifconfig program(for now) that shows all interfaces and their configuration

21:

moved ip manipulating functions to separate library
the same with interfaces
ARP is working
fixed few bugs
added "lock" mode ONLY for drivers, this locks execution of other tasks for some critical operations
probably other changes that I forgot about
everything form revisions 18-20, I forgot about this change log, sorry.

17:

initial version
added VTYs
locked printing functions of textutils API
first step in program privilege levels
some bug fixes

PS. Sorry if I made language mistake or something isn't understandable, I'm still learning ;)/> Feel free to point it out to me.</number>

netOS and all included programs made by grabie2 are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

Edited on 23 January 2013 - 06:23 AM

billysback #2

536 posts

Posted 15 December 2012 - 07:45 AM

This looks good but it should go in "General" as there isn't actually any code here, sorry :P/>

CoolisTheName007 #3

302 posts

Posted 15 December 2012 - 09:09 AM

You may be interested in checking out these posts if you don't feel like re-inventing the wheel (on your own):
http://www.computercraft.info/forums2/index.php?/topic/1708-lyqydnet-rednet-api/
http://www.computercraft.info/forums2/index.php?/topic/3178-rednet-frequencies/page__fromsearch__1
http://www.computercraft.info/forums2/index.php?/topic/6476-networking-towards-a-complete-solution/page__fromsearch__1

Ballistic Buddha #4

Ballistic Buddha's profile picture

15 posts

Posted 15 December 2012 - 09:23 AM

I'm super excited for this, I can't wait until this comes to fruition with a working Telnet/SSH daemon so I can finally just remote into a CC Computer and use my OS's native terminal, and bypass the in-game terminal completely.

grabie2 #5

grabie2's profile picture

78 posts

Location Poland

Posted 15 December 2012 - 10:20 AM

@billysback
Sorry for that, some code will be there(or link to that) in few days, so please don't mind me :D/>

@CoolisTheName007
I will write everything on my own, cause I like it ;)/> But still thanks for links I may get some ideas from that

@Ballistic Buddha
Keep in mind that SSH will be slow… and I think that i'll use instead of 1024 bit keys - 32/64/128 bit keys, beacause of lag.

Thanks for support!

mibac138 #6

128 posts

Location Poland

Posted 15 December 2012 - 10:44 AM

Siemka! ;)/>

Też pl jestem :D/>

Masz skype? XD

grabie2 #7

grabie2's profile picture

78 posts

Location Poland

Posted 15 December 2012 - 11:14 AM

Hey!

I've got skype. I wrote you a pm ;)/>

Expenox #8

Expenox's profile picture

142 posts

Location In the depths of my computer.

Posted 15 December 2012 - 11:20 AM

Nice!

grabie2 #9

grabie2's profile picture

78 posts

Location Poland

Posted 15 December 2012 - 01:01 PM

Project page was just created, still need some work, but i want to push that early code to you ;)/>
Unfortunantly it doesn't have any networking yet, as I said first I need to create the OS core
http://netos.robotronika.pl

huettner94 #10

huettner94's profile picture

27 posts

Posted 16 December 2012 - 06:18 AM

great Idea :D/>
It would probably be usefull if you put an IP-Layer above the rednet connection.
ICMP would be really usefull as well (ping, traceroute)

grabie2 #11

grabie2's profile picture

78 posts

Location Poland

Posted 16 December 2012 - 06:33 AM

OK, base multitasking is done!
It still needs API and documentation, but now I think i'll start making network stuff ;)/>

grabie2 #12

grabie2's profile picture

78 posts

Location Poland

Posted 16 December 2012 - 01:36 PM

I have big question to everyone interested in this OS:
Do you want me to modify "rom" files or I should make it "download one file by paste bin and it will do everything"? - answer in pool

AfterLifeLochie #13

423 posts

Location AfterLifeLochie's "Dungeon", Australia

Posted 16 December 2012 - 01:38 PM

Moved to General - the Programs section is for released software only.

grabie2 #14

grabie2's profile picture

78 posts

Location Poland

Posted 16 December 2012 - 01:43 PM

Ohh that's why I couldn't submit post edit :D/>
Thanks for moving and sorry for my mistake.

grabie2 #15

grabie2's profile picture

78 posts

Location Poland

Posted 16 December 2012 - 10:42 PM

What do you mean by not doing it right?

grabie2 #16

grabie2's profile picture

78 posts

Location Poland

Posted 17 December 2012 - 11:08 AM

OK, quick update for today:
- I've coded about 30% of layer 2
- Probably I've broken loading driver APIs, so system in current repository version won't boot

also I run public SVN for this project:
http://svn.robotronika.pl/svn/netos
user: netos
pass: netos
Sorry, but because of server configuration, I can't run SVN without authentication.
DNS is fresh setup so it can not work for about a hour

CoolisTheName007 #17

302 posts

Posted 17 December 2012 - 11:28 AM

Your multitasking code is inefficient in the sense that it forces CC to resume netOs each 0.1 s.

grabie2 #18

grabie2's profile picture

78 posts

Location Poland

Posted 18 December 2012 - 11:46 AM

yes, but without that, in this implementation, if task uses "addEventListener" it yields and possibly will resume after long period of time, in this case netOS will resume only if current executing task yields and passed 0.1 s, you can see the delay when you boot the system, when shell runs.

So it will allow to run all tasks in no more than 0.5 s after yield. Or maybe you have better suggestion, if so please let me know. I can change it to like 0.2 s and run all tasks after 3 ticks, more delay will be to noticeable for end user.

Maybe after I implement network in efficient way, I'll change multitasking system, but for now (if there are not major bugs) it will stay mainly like it is.
Maybe prioritization, so user programs run every 0.5 s and drivers/other after longer period of time. Or other event system.

I had idea to actually make "addEventListener" to add callback methods to table and call them when needed, but it was harder to implement and I've done it like it is.

Sorry, but today I had no time to write anything. Tomorrow will be probably the same, maybe Thursday/Friday I'll had time for netOS.

CoolisTheName007 #19

302 posts

Posted 18 December 2012 - 12:35 PM

grabie2, on 18 December 2012 - 11:46 AM said:
snip

No problem man. I do it so that I yield to CC always (putting os.startTimer(timeout) before) where timeout is 0 if some task wants to run, or the minimum number of seconds needed for firing the next task. The I os.pullEventRaw until someone is ready to run.

grabie2 #20

grabie2's profile picture

78 posts

Location Poland

Posted 18 December 2012 - 06:36 PM

What does os.startTimer(0), it immediately generates event or does nothing?

Orwell #21

1054 posts

Posted 18 December 2012 - 06:45 PM

Are you actually going to build the application layer on the 'TCP/ (maybe UDP)' layer? Cause then DNS without UDP would be kinda strange huh?

Edit: And yes, os.startTimer(0) should queue an event immediately, you could as well use 'os.queueEvent()'

CoolisTheName007 #22

302 posts

Posted 18 December 2012 - 10:17 PM

Orwell, on 18 December 2012 - 06:45 PM said:
Edit: And yes, os.startTimer(0) should queue an event immediately, you could as well use 'os.queueEvent()'

I wonder, if I just want to yield and get control back ASAP, which one is the better…probably os.queueEvent().

grabie2 #23

grabie2's profile picture

78 posts

Location Poland

Posted 19 December 2012 - 04:48 AM

Orwell, on 18 December 2012 - 06:45 PM said:
Are you actually going to build the application layer on the 'TCP/ (maybe UDP)' layer? Cause then DNS without UDP would be kinda strange huh?

Edit: And yes, os.startTimer(0) should queue an event immediately, you could as well use 'os.queueEvent()'

I could build DNS directly over IP layer, but i'm not sure for now how I want to implement it. Or DNS will be implemented much later, it's not so needed.

CoolisTheName007, on 18 December 2012 - 10:17 PM said:
Orwell, on 18 December 2012 - 06:45 PM said:
Edit: And yes, os.startTimer(0) should queue an event immediately, you could as well use 'os.queueEvent()'
I wonder, if I just want to yield and get control back ASAP, which one is the better…probably os.queueEvent().

Actually I didn't want to make main loop loop as fast as possible, to save CPU I wanted to loop every event or 0.1 sec depending which one was first. And just in case refresh all programs every 5 events.

CoolisTheName007 #24

302 posts

Posted 19 December 2012 - 05:00 AM

grabie2, on 19 December 2012 - 04:48 AM said:
Actually I didn't want to make main loop loop as fast as possible, to save CPU I wanted to loop every event or 0.1 sec depending which one was first. And just in case refresh all programs every 5 events.

I leave to the tasks to decide if they want to sleep for a while or not; if all tasks are waiting either for some time to pass or for an os event, then I sleep the minimum time until the next wake up time and in the process, stop sleeping if any of the tasks wakes up.

grabie2 #25

grabie2's profile picture

78 posts

Location Poland

Posted 19 December 2012 - 06:24 AM

Hmm maybe I'll implement few process queues and if the one with tasks waiting for time(send os call) will be emty then I'll sleep for an event

Thanks for support!

grabie2 #26

grabie2's profile picture

78 posts

Location Poland

Posted 19 December 2012 - 08:27 AM

OK, made it boot again, there were few bugs. And I think that I need to clean up my code before doing more networking stuff, so today/tomorrow I'll be cleaning and improving base system code and multitasking.

Shnupbups #27

671 posts

Location That place over there. Y'know. The one where I am.

Posted 19 December 2012 - 09:23 AM

Make it so you download an installer onto a CC PC automatically and it pulls all the needed files off of pastebin for them. No rom-editing and all files that are needed installed with one. Simple as that.

grabie2 #28

grabie2's profile picture

78 posts

Location Poland

Posted 19 December 2012 - 09:39 AM

yea, all releases will have installer, this is "in development"

grabie2 #29

grabie2's profile picture

78 posts

Location Poland

Posted 24 December 2012 - 02:51 PM

Merry Christmas!

EDIT: I just pushed new revision to SVN, so you guys can see what i've done. It's not completed yet, but I think, that you can see where it is going. If you can, please point to me weak points of my conception, from last revision you need to update boot and bin/netOS

I've been ill for few days and I didn't done anything with OS. After Christmas I will work on base OS code - maybe not complete, but rewrite of multitasking, API handling, driver system and few other things. When that will be done I'll go back to networking.

And for future, do you want repeaters in one network or only by routing between networks? Answer in pool

lieudusty #30

lieudusty's profile picture

475 posts

Posted 24 December 2012 - 07:38 PM

This looks great!

grabie2 #31

grabie2's profile picture

78 posts

Location Poland

Posted 26 December 2012 - 05:52 AM

New revision just pushed, OS now boots(but shell isn't yet starting - to make testing easier) correctly multitasking should be working.
Now I'm doing some tests on network and multitasking, after i'm sure that everything works i'll add shell back, now with header reading

And question to someone who is better in lua: I have that code in "bin/netOS", it's creating environment for task to run:


local baseAPI = {}  --no print, write functions - programs must use term.print/write instead
baseAPI.type = type
baseAPI.sleep = os.sleep
baseAPI.config = config
baseAPI.store = store
baseAPI.table = table
baseAPI.string = string
baseAPI.error = error
baseAPI.pairs = pairs
baseAPI.ipairs = ipairs
baseAPI.tostring = tostring
baseAPI.tonumber = tonumber

And how to make sure that there are all base lua functions?
Is there any other way than typing them by hand?
If you will be posting code, I have table.merge(tableA, tableB) function, that returns merged tableB into tableA, like this:


function table.merge(t1, t2)
  for k, v in pairs(t2) do
    if (type(v) == "table") and (type(t1[k] or false) == "table") then
	  table.merge(t1[k], t2[k])
    else
	  t1[k] = v
    end
  end
  return t1
end

Thanks for advises!

grabie2 #32

grabie2's profile picture

78 posts

Location Poland

Posted 26 December 2012 - 01:02 PM

OK, I made it load and run everything correctly, but I didn't push new release because I didn't completed tests on network(rednet layer) tonight. Unfortunately I have to go sleep it's 1 AM and i'm waking up at 6:30 ;)/> Tomorrow I plan close this rewrite and redesign of netOS and go back to work on networking, first ARP and then IP. After that i'll implement TCP and basic telnet, so there will be some use for my OS ;)/>
Then i'll make every protocol more optimized(no more serialized tables), write routing software and more!

EDIT:

Just pushed new version, working OS, and rednet layer. Unfortunately ARP crashes itself after packet arrival. Maybe today i'll have ARP working correctly, but I'm not promising anything ;)/>

EDIT #2:

Actually I'm making some changes to multitasking and making test shell for OS. So networking stuff later. Basically I need communication between tasks, but not through global event pool.

EDIT #3:

OK, little redesign of multitasking and we can send message to another task without other task knowingWOOT!
Change log:
- tasks now have PID(Process ID)
- now we have one queue of processes - main loop checks state of task
- by default task subscribes to "os_message" event, but can use "os.removeEventListener("os_message")" to filter this out
- for now as "shell" starts modified version of lua(because it receives parameter - process PID), only for testing purposes
- in main OS file I have some function documented

After IP layer is done I'll implement custom term API, to use escape sequences and streams - to make programs independent of platform(virtual terminal, local terminal, network terminal, etc.)
I should really start working on documentation on project page(http://netos.robotronika.pl)
If you found any bugs please post it on bugtracker: http://netos.robotronika.pl/newticket
Do download source code use SVN: http://svn.robotronika.pl/svn/netos/
user: netos
password: netos

Edited on 27 December 2012 - 09:33 AM

grabie2 #33

grabie2's profile picture

78 posts

Location Poland

Posted 28 December 2012 - 03:31 PM

I'm working hard on mutliconsole system, so you can have at least for now ran remote control program on one and work on second, we can call it indev release :D/>
BUT I went into few bugs:
http://www.computerc...th-environment/
http://www.computerc...allocation-bug/
and some that I resolved by myself or I'm not sure if they are real bugs

HotGirlEAN #34

93 posts

Posted 28 December 2012 - 04:51 PM

When is this going to be released?
It's been 13 days since this topic was created…

Orwell #35

1054 posts

Posted 28 December 2012 - 09:08 PM

HotGirlEAN, on 28 December 2012 - 04:51 PM said:
When is this going to be released?
It's been 13 days since this topic was created…

A link to the source code is right there in the OP..?

HotGirlEAN #36

93 posts

Posted 28 December 2012 - 09:21 PM

Not sure if this topic name has anything to do with it:
"In Development/Not released"

Orwell #37

1054 posts

Posted 28 December 2012 - 11:24 PM

So you're waiting for an actual release? I'm sure that he's still working on it, projects like this aren't created in 13 days. Until then, you can follow his progress in the source code and experiment with it.

grabie2 #38

grabie2's profile picture

78 posts

Location Poland

Posted 28 December 2012 - 11:58 PM

source code is open to you ;)/> It's on project SVN and can be looked up on project page:
SVN: http://svn.robotronika.pl/svn/netos
user: netos
pass: netos

project page: http://netos.robotronika.pl

HotGirlEAN, on 28 December 2012 - 04:51 PM said:
When is this going to be released?
It's been 13 days since this topic was created…

It's going to be released when it's ready ;)/> First public 'release' will be in 3-5 days, but without networking. It will only contain multitasking AND multiconsole, so you can have your remote control program on first console and local control program on second

* 'release' means it will have installer in one file

EDIT: As of a lot of trouble with multiple consoles I don't know when I will release first beta.

Edited on 31 December 2012 - 01:12 AM

grabie2 #39

grabie2's profile picture

78 posts

Location Poland

Posted 02 January 2013 - 02:20 AM

OK, finally tracked down problem with TTYs, but unfortunately I have to(again) rewrite some of multitasking code to fix this.
Also my school starts tomorrow, so I will have less tome to work on this. I really want to release it ASAP, but some bugs are just stopping me.

EDIT: OK, finally problem with messing up termAPI in VTYs resolved, now I have to fix writing functions, and I'll commit version to SVN and post download here, so you guys can see what I've done!
Hopefully that will happen tomorrow, but cause of school it can be delayed up to Saturday.

Edited on 03 January 2013 - 08:16 AM

grabie2 #40

grabie2's profile picture

78 posts

Location Poland

Posted 05 January 2013 - 12:34 PM

OK, I MADE IT!
However I need to rewrite some base CC stuff( apis ), cause some of them are broken. I think I'll be able to push beta download tomorrow. I have some other projects that have priority over netOS and I have to work on them first.
There are some netOS mechanics to change on order to make some APIs working(textutils). Currently everything that uses this API is broken, so I just unload it in boot, will be fixed until release. In just a few moments I'll push new version to SVN.

Just remember that first beta will not have ANY networking stuff. I'll try to document everything on wiki before release.

PixelToast #41

2217 posts

Location 3232235883

Posted 05 January 2013 - 12:44 PM

o.O
i might use some of this when i make the networking part of my OS / BIOS
im still trying to get my serial adapter to work
eventually ill make it boot from a bundled cable

grabie2 #42

grabie2's profile picture

78 posts

Location Poland

Posted 05 January 2013 - 12:56 PM

Don't steal my plans ;)/>
As I said in first post, I plan on simulating real Internet mechanics that includes also things like booting over network, so you have one file bootloader which will connect to specified server and download OS from TFTP to RAM and run it ;)/>
BUT that's really far away from what I have now.

grabie2 #43

grabie2's profile picture

78 posts

Location Poland

Posted 08 January 2013 - 05:30 AM

Looks like most of the stuff is working, still need to change the way APIs are handled, but I'm going to post it in programs section ;)/>

PixelToast #44

2217 posts

Location 3232235883

Posted 08 January 2013 - 05:37 AM

grabie2, on 05 January 2013 - 12:56 PM said:
Don't steal my plans ;)/>
As I said in first post, I plan on simulating real Internet mechanics that includes also things like booting over network, so you have one file bootloader which will connect to specified server and download OS from TFTP to RAM and run it ;)/>
BUT that's really far away from what I have now.

im not doing actuall networking here
im going to have LAN cables be like USB devices or peripheral cables
if a computer says its a bootable disk drive then you can essentially boot over lan

grabie2 #45

grabie2's profile picture

78 posts

Location Poland

Posted 08 January 2013 - 10:46 AM

Ohh OK, got you ;)/>
Your will be much faster, however mine will be more fun to watch. :D/>

In few minutes I'll submit color supporting VTYs.

BTW.
There is thread in programs section: http://www.computercraft.info/forums2/index.php?/topic/8366-netos-first-truly-multitasking-os-with-real-world-networking-build-innot-yet/ (for those ones that do not read first post ;)/> )

grabie2 #46

grabie2's profile picture

78 posts

Location Poland

Posted 11 January 2013 - 10:19 AM

I'm leaving for about a week for winter holidays, so development will be suspended for that time.

Currently I have few bugs with ARP, it's not receiving packets, or sending them incorrectly, or even doesn't have proper interface table
IP protocol is untouched for now.
so status for now is 40% IP/70% ARP

grabie2 #47

grabie2's profile picture

78 posts

Location Poland

Posted 20 January 2013 - 11:52 AM

I've got ARP working. I'll try to submit new SVN revision tonight. I think I'll be able to run basic IP tomorrow. Thanks for feedback and great support!

grabie2 #48

grabie2's profile picture

78 posts

Location Poland

Posted 23 January 2013 - 07:21 AM

IP layer is working!
It's on SVN right now, so you can check it out.
Now I'll document everything, correct some bugs and wrong conceptions and then I'll program UDP(will be pretty easy) and after that TCP

Thanks for support!

Also I'm looking for someone to help me with that pretty difficult task, I need:
- printer driver (printing queue, page management, paper/ink management, API for it, user application[console], future printing over network, mapping network printers)
- beta tester
- someone to help programming GUI
- printer application for GUI that will be created

grabie2 #49

grabie2's profile picture

78 posts

Location Poland

Posted 29 January 2013 - 05:56 AM

Currently I'm rewriting my multitasking and netOS file itself, I'm also planning to rewrite VTYs, so they'll behave a little bit batter

EDIT:
Currently I have working multitasking, from now eventListeners are gone! What will replace them? to os.pullEvent you could pass only one filter, however currently you're able to pass multiple filters!
I'm not pushing this to SVN right yet, cause OS itself isn't working, but probably I'll have it done on Monday.

Edited on 30 January 2013 - 08:04 AM

Lyqyd #50

Lyqyd's profile picture

8543 posts

Posted 01 March 2013 - 11:05 AM

Locked by request.