7 posts
Posted 02 January 2013 - 01:22 AM
For a few months now, I have been designing a whole OS system which would expand on the current core ComputerCraft OS. It is an overhaul that adds some advanced features common in today's OSes to the CC OS. But after the months, I have realized that I simply do not have the time, and probably also do not have the Lua knowledge, to pull this off in a feasible timeframe. Therefore I have come to these forums to present my ideas, and hopefully spur on development of this idea.
Background
It all started with a scheduler design - a OS level component that essentially adds in 'processes'. It worked to the extent of backgrounding processes from the shell with & worked, and you could 'kill' processes or use 'ps' to list them all, but it had significant drawbacks like background processes using write() or print() would still print to the terminal. Obviously this isn't how it works on most common OSes. This scheduler design seems to be similar to Cyclonit's CycloProcess.
Minor Edit: I guess I should post my code for the scheduler to prevent this thread from being bumped into General for lacking code… file attached as [attachment=850:bios.lua.gz]
Minor Edit 2: And here are the ps and kill utilities to go with that scheduler: [attachment=851:kill.gz] [attachment=852:ps.gz]
Design Draft
File Descriptors
To fix the problem of terminal output in background processes, I decided to follow how Linux does it, with file descriptors and redirection. When running from the shell, each process would get two file descriptors 'opened' by default - stdin and stdout. These would in turn point to the terminal, resulting in input from the terminal being sent to the process, and output from the process being sent to the terminal. But it should also allow for redirection using > and <, much like is sh/Bash, where output can be redirected to/from another device or to a file.
If you do a 'ps' in Linux ('ps aux' to see more) you also see a 'tty' field for each process. I figured that this handles direct terminal access - if a process in CC uses write() or print() it will be handled by the stdout file descriptor. On the other hand if the process uses direct term.*() functions, then the file descriptor layout will be overwritten and output will still go to the terminal. This is solved by the tty parameter - if set to null, then any term.*() functions operate on nothing, since the process has no terminal.
Events
This is an intertwined problem with file descriptors - since most input in CC is dealt by the event model, events would have to become private (per-process) to get the input redirection to work. The idea here would be for each process to have an event queue - the OS handles 'hardware-level' events, and passes them down to the processes that have registered interest in those events.
A future example would be a networking API - a process could bind to a port, signalling to the OS that it wants all traffic on [port] to come to it. Assuming the port isn't currently bound to a different process, it is allowed, and networking signals that pass through the networking API to arrive at [port] get sent to that process.
A slightly simpler example is the stdin file descriptor itself. Processes with 'term' open as an input file descriptor would get keypresses (that is, char and key events) sent to them.
BIOS
This is unrelated to the above, but still fits in with the mantra of an OS overhaul. The current bios.lua is the OS. Unfortunately this allows very little customization, already preloading all the APIs before handing control down to the user. My suggestion would be to turn bios.lua into a simple bootloader - it would read a configuration file to see what it should do (such as timeout options, default kernel, kernel list etc.) then present an kernel menu to the user, boot directly into the default kernel (if the timeout expires, or there is only 1 kernel), or go into an error console to allow manual entering of commands (aka when there is no configuration file).
I haven't looked at it in depth, but max96at's Bootloader could be a place to start. The bootloader would integrate heavily with the next section - filesystem layout.
Filesystem
The title says it all - a filesystem layout to provide consistent locations for CCSB compatible OSes. The following will be a bit of a text dump, since I am not going to re-write my notes on it:
Environment Variables
I haven't though about this feature much, but I suspect it will be necessary to implement some of the things above. I won't bore you with the details of environment variables, but they must fit the criteria of each process getting its own copy of environment variables (a snapshot, if you will) when it is created so that it can modify them without affecting other running processes. I am not sure exactly how exporting an environment variable will work however.
Compatibility
One of the main aims of CCSB is to maintain full compatibility with the current CC OS - this means leaving things that exist currently in the same places, and modifying the current APIs rather than creating new ones. It should be possible - write() and print() can be written to use the stdout file descriptor of the process, without the process needing to know anything about file descriptors. Obviously a program that reads global events and then assumes it can use them (like redstone or rednet events) will fail because of the private event model, but this could be solved with a decleration in the OS that enables a compatibility mode for that process, passing all hardware events to it (apart from char and key if its stdin is unbound or pointing somewhere other than the terminal).
Future Possibilities
Virtual Filesystems
I don't know exactly how this will work, or if it is even possible, but it would be really cool to have virtual filesystems. By this I mean an area of the filesystem that looks like it is part of the computer itself, but is actually either a virtual construct or is located elsewhere. For example, you could bind parts of the filesystem together - /mnt could point to /rom/programs, so accessing /mnt/ls would run /rom/programs/ls. This example is a simple example of a 'located elsewhere' filesystem, but more complex examples would be like a network filesystem, so /mnt could point to a directory on a completely different computer, perhaps even over a network - the real world example of this would be network shares, across the NFS, SMB or CIFS protocols.
Real virtual filesystems could include /tmp or /proc - when the system is shut down they cease to exist, leaving them fresh when the system is booted up again.
Permissions
Users and groups are a while off, but would give access control to the filesystem and programs. This would require a major rewrite of file handling code, so that each file and directory has an associated permission, owner and group, which defines how it can be used by different users.
Final Notes
I hope that certain parts of this design draft can be incorporated into core CC. Mainly the BIOS and filesystem structure, since they are the building blocks to allow greater modularity.
I will accept any feedback on the design draft, and with review may also change certain parts. This project is supposed to enhance the ComputerCraft experience, but this will not be possible without the suggestions and review of the real users and developers of ComputerCraft. Ideas on how to improve a component will be greatly appreciated, and suggestions for a new component of the Standards Base will be considered for inclusion.
Changelog
01/01/13 - First release
Background
It all started with a scheduler design - a OS level component that essentially adds in 'processes'. It worked to the extent of backgrounding processes from the shell with & worked, and you could 'kill' processes or use 'ps' to list them all, but it had significant drawbacks like background processes using write() or print() would still print to the terminal. Obviously this isn't how it works on most common OSes. This scheduler design seems to be similar to Cyclonit's CycloProcess.
Minor Edit: I guess I should post my code for the scheduler to prevent this thread from being bumped into General for lacking code… file attached as [attachment=850:bios.lua.gz]
Minor Edit 2: And here are the ps and kill utilities to go with that scheduler: [attachment=851:kill.gz] [attachment=852:ps.gz]
Design Draft
File Descriptors
To fix the problem of terminal output in background processes, I decided to follow how Linux does it, with file descriptors and redirection. When running from the shell, each process would get two file descriptors 'opened' by default - stdin and stdout. These would in turn point to the terminal, resulting in input from the terminal being sent to the process, and output from the process being sent to the terminal. But it should also allow for redirection using > and <, much like is sh/Bash, where output can be redirected to/from another device or to a file.
If you do a 'ps' in Linux ('ps aux' to see more) you also see a 'tty' field for each process. I figured that this handles direct terminal access - if a process in CC uses write() or print() it will be handled by the stdout file descriptor. On the other hand if the process uses direct term.*() functions, then the file descriptor layout will be overwritten and output will still go to the terminal. This is solved by the tty parameter - if set to null, then any term.*() functions operate on nothing, since the process has no terminal.
Events
This is an intertwined problem with file descriptors - since most input in CC is dealt by the event model, events would have to become private (per-process) to get the input redirection to work. The idea here would be for each process to have an event queue - the OS handles 'hardware-level' events, and passes them down to the processes that have registered interest in those events.
A future example would be a networking API - a process could bind to a port, signalling to the OS that it wants all traffic on [port] to come to it. Assuming the port isn't currently bound to a different process, it is allowed, and networking signals that pass through the networking API to arrive at [port] get sent to that process.
A slightly simpler example is the stdin file descriptor itself. Processes with 'term' open as an input file descriptor would get keypresses (that is, char and key events) sent to them.
BIOS
This is unrelated to the above, but still fits in with the mantra of an OS overhaul. The current bios.lua is the OS. Unfortunately this allows very little customization, already preloading all the APIs before handing control down to the user. My suggestion would be to turn bios.lua into a simple bootloader - it would read a configuration file to see what it should do (such as timeout options, default kernel, kernel list etc.) then present an kernel menu to the user, boot directly into the default kernel (if the timeout expires, or there is only 1 kernel), or go into an error console to allow manual entering of commands (aka when there is no configuration file).
I haven't looked at it in depth, but max96at's Bootloader could be a place to start. The bootloader would integrate heavily with the next section - filesystem layout.
Filesystem
The title says it all - a filesystem layout to provide consistent locations for CCSB compatible OSes. The following will be a bit of a text dump, since I am not going to re-write my notes on it:
/
apis/
turtle/
boot/
config/
dev/
help/
home/
programs/
computer/
turtle/
rom/
apis/
turtle/
boot/
kernel.lua
config/
help/
programs/
turtle/
computer/
http/
startup
tmp/
var/
startup
apis/ stores all APIs for the system. APIs are read from /apis first, followed by /rom/apis/. apis/turtle/ stores turtle specific APIs.
boot/ stores the kernels for the computer. The chosen kernel is read by bios.lua and executed. If none exists /rom/boot/kernel.lua is run instead.
config/ stores configuration data. Default configuration is provided in /rom/config/, while system-specific configuration is stored in /config/
/dev/ is an extension to the core design, giving 'devices' which could be used in the file descriptor context.
help/ stores help files, with the same delegation as the other directories
programs/ stores programs. Computer-specific programs are stored in programs/computer/, turtle-specific programs are stored in programs/turtle/ and HTTP programs are stored in programs/http/
startup is a script that runs on startup (assuming the default kernel).
/home/ stores user home areas in preparation for a multi-user environment
/rom/ stores read only files, as 'defaults' for the system. /rom/boot/kernel.lua is a default kernel for computers that lack their own kernel.
/tmp/ stores temporary files, and is wiped on shutdown and startup by bios.lua
/var/ stores program data (variable data).
Environment Variables
I haven't though about this feature much, but I suspect it will be necessary to implement some of the things above. I won't bore you with the details of environment variables, but they must fit the criteria of each process getting its own copy of environment variables (a snapshot, if you will) when it is created so that it can modify them without affecting other running processes. I am not sure exactly how exporting an environment variable will work however.
Compatibility
One of the main aims of CCSB is to maintain full compatibility with the current CC OS - this means leaving things that exist currently in the same places, and modifying the current APIs rather than creating new ones. It should be possible - write() and print() can be written to use the stdout file descriptor of the process, without the process needing to know anything about file descriptors. Obviously a program that reads global events and then assumes it can use them (like redstone or rednet events) will fail because of the private event model, but this could be solved with a decleration in the OS that enables a compatibility mode for that process, passing all hardware events to it (apart from char and key if its stdin is unbound or pointing somewhere other than the terminal).
Future Possibilities
Virtual Filesystems
I don't know exactly how this will work, or if it is even possible, but it would be really cool to have virtual filesystems. By this I mean an area of the filesystem that looks like it is part of the computer itself, but is actually either a virtual construct or is located elsewhere. For example, you could bind parts of the filesystem together - /mnt could point to /rom/programs, so accessing /mnt/ls would run /rom/programs/ls. This example is a simple example of a 'located elsewhere' filesystem, but more complex examples would be like a network filesystem, so /mnt could point to a directory on a completely different computer, perhaps even over a network - the real world example of this would be network shares, across the NFS, SMB or CIFS protocols.
Real virtual filesystems could include /tmp or /proc - when the system is shut down they cease to exist, leaving them fresh when the system is booted up again.
Permissions
Users and groups are a while off, but would give access control to the filesystem and programs. This would require a major rewrite of file handling code, so that each file and directory has an associated permission, owner and group, which defines how it can be used by different users.
Final Notes
I hope that certain parts of this design draft can be incorporated into core CC. Mainly the BIOS and filesystem structure, since they are the building blocks to allow greater modularity.
I will accept any feedback on the design draft, and with review may also change certain parts. This project is supposed to enhance the ComputerCraft experience, but this will not be possible without the suggestions and review of the real users and developers of ComputerCraft. Ideas on how to improve a component will be greatly appreciated, and suggestions for a new component of the Standards Base will be considered for inclusion.
Changelog
01/01/13 - First release
