808 posts
Posted 23 May 2013 - 02:28 AM
Project NewLife
Project NewLife is a suite of applications and APIs intended to make the CraftOS experience better. First, it should be known that Project NewLife is not meant to be an operating system. Development began on NewLife when capabilities were needed that CraftOS alone couldn't provide. Any program written for CraftOS will still work perfectly under NewLife. But NewLife adds many capabilities that were much more difficult before.
Overview
CraftOS brings to the table some useful APIs like textutils, the shell API, and even a rednet API. Project NewLife adds to this with these features and more.
- A new shell
- Multitasking
- A UNIX-like pipeline
- A lua to bytecode compiler
- Running bytecode compiled lua programs
- A command line tool for editing the path
- Shell scripts
- Argument processing API
- JSON API
- Extended string and table libraries
- File extension based custom file loading for APIs
- Virtual files
- Daemons
- A github repo downloader
- LASM - CC's first alternate programming language.
Karma.lua
Understanding how to take advantage of the NewLife system requires understanding of its core. This is called karma.lua. When NewLife loads, the file /NewLife/karma.lua is the first thing that's run after the CraftOS shell is killed. The purpose of karma.lua is to run all the karma extensions. Extension programs can be found in /NewLife/ext. Karma loads these files with no protections. They all read and write to the _G table as their environment.
Writing an extension
Spoiler
If you want to add primitive level APIs that are loaded at runtime and cannot be unloaded by a third party program, you will need to write a Karma extension.First create a file in /NewLife/ext. This file will house all of your code. Also note that any functions or variables declared globally will be accessible to any other code run at any time in the system. So if you want to keep something private to your file, make sure to use the local keyword. Also, add your extension to karma.lson.
There is an exception to that for the init function though. If you declare an init function as global, it does set _G["init"] to your function. But after your file is loaded, karma.lua takes that value and keeps it. Then it wipes _G["init"] so that it can get all the init functions from all the files. Then once all the extensions have been loaded, all of the init functions are called, with the config table from karma.lson in /NewLife/etc passed as an argument.
When to write an extension
Why write an unprotected extension when you could just write an API? There is no good reason unless you are modifying existing APIs or you're writing functions which are intended to be a primitive type of function. For example, the loadlibs.lua extension creates a function called loadAPI. It differs from os.loadAPI in that it returns the API instead of setting a global variable to the API. It's meant to be a primitive for getting code, and is low level. So it is an extension.
Object Oriented Programming
One of the Karma extensions is called NLClass.lua. This extension adds a system-wide OOP API. The following is an explanation on how to use it.
Spoiler
Classes can either be declared in functions or in files with a .nlclass extension which are then loaded via loadAPI.
function MyClass()
function test()
print('test')
end
end
local obj = new(MyClass).init()
obj.test()
This will print "test." Note that MyClass is not called directly. Any variables declared globally in a class will not become global. Instead, they will become publicly available in the object returned by new(). That's why the function 'test' is not made global, but is instead a method in the object. Also, every class implicitly has an init value, which is often overridden with new parameters for initialization. Init should always return the object (by using "return this"), so that you can call new().init() in one line.
function MyClass()
local a
function init(_a, _c)
super.init()
a = _a
c = _c
end
end
local obj = new(MyClass).init(3, 4)
print(obj.a)
print(obj.c)
The first print statement will not output anything. The second one will print 4. The "a" variable in obj is private because of the local keyword. But "b" is publicly available because it was defined in init as if it were a global.
Also note that no function has to be called for NLClass.lua to use MyClass as a class. But that's not true in the case of subclassing.
function MyClass()
function init()
print("MyClass")
end
end
function MySubClass()
function init()
super.init()
print("MySubClass")
end
end
class(MySubClass, MyClass)
local obj = new(MySubClass).init()
This will print
MyClass
MySubClass
The super instance variable in a class works as expected, with multiple inheritence supported.
And finally, setting up a class and its setting its superclass from within the class file:
function MyClass()
function __setupClass()
class(MyClass, SuperClass)
end
end
When an object of MyClass is created, if that class hasn't yet had its __setupClass() called, it will be called before creating the object. This allows you to set your superclass from within the class.
NEW!
There's a class built into the system called NLObject that is the superclass of any class instantiated, even when not defined as the superclass. Only built in methods are init(), which does nothing notable, and instanceof(class), which returns true if the object is an instance of class
Loading APIs and File Type Loaders
There's some new functionality alongside os.loadAPI now. Directories can be loaded recursively, APIs can be loaded and returned locally, and NLClasses can be kept in files instead of functions.
Spoiler
First, it's important to know how APIs handle their file extensions now. In /NewLife/ldr, you will find several file type loaders. When loading an API, if it has a file type n, and there exists a file /NewLife/ldr/n.lua, that loader code will be used to load the file, and the file extension will be stripped from the API name.
-- MyApi.lua
function test()
print('test')
end
-- MyProgram.lua
os.loadAPI('MyApi.lua')
MyApi.test()
Even though MyApi has the .lua extension, that is now stripped when I try to access the loaded API by name.
Any file without a known file extension will be loaded with the /NewLife/ldr/lua.lua loader. The lua loader takes all the code and runs it into an API table, as we are used to in standard CraftOS. But there is also a loader for NLClasses with the file type .nlclass.
-- MyClass.nlclass
function init()
print('init')
end
-- MyProgram.lua
os.loadAPI('MyClass.nlclass')
local obj = new(MyClass).init()
This will print "init" because MyClass is loaded as a class, and the .nlclass extension is stripped from the name.
The other built in types are .json and .txtutl. A .json file will be decoded as json text, and a .txtutl file will be decoded with textutils.unserialize().
Writing a loader is easy. Let's say we want to write a loader for the file type .image which treated each line as a line on the screen. Each character in the line is a hex value 0-15 representing the color to display. The loaded API would be a two dimensional table.
--/NewLife/ldr/image.lua
local path, api = ... -- path is the file we are being told to load
-- api is the table that we can either put our data in,
-- or use as the environment (for stuff like functions)
local file, err = fs.open(path, "r")
assert(file, err)
local count = 1
for line in file.readLine do
tLine = {}
for i=1, #line do
tLine[i] = 2 ^ tonumber(line:sub(i,i), 16) -- use hexadecimal
-- do 2 ^ char to get the colors api color
end
api[count] = tLine
count = count + 1
end
return api
You must return the value you want to be given as the API. The api table passed in is typically what you would use as either the place to load into, or the environment to set your functions to. Also note that you are allowed to error() and assert() in here. IT SHOULD BE EXPECTED THAT LOADING APIS WILL FAIL
APIs can be loaded locally now. If you don't want to load a file system wide, you can use the function loadAPI. It's not os.loadAPI. It's just loadAPI. A global function.
-- MyApi.lua
function test()
print('test')
end
-- MyProgram.lua
local api = loadAPI("MyApi.lua")
api.test()
This is valid. And MyApi will not be present in the global scope. The API was loaded locally. This is why it should be expected that loading an API will fail. Although os.loadAPI() checks for errors and returns false if there was one, loadAPI does not do this. It expects that you will surround the loadAPI call in a pcall() and check yourself. Similar to how java often requires you to surround things in a try-catch block.
And finally, directories can be loaded recursively. Take the following folder structure.
MyApi
|-First.lua
|-Second.nlclass
|-MoreApi
|-Third.lua
|-Last.nlclass
os.loadAPI('MyApi')
MyApi.First.test()
local sec = new(MyApi.Second).init()
MyApi.MoreApi.Third.func()
local las = new(MyApi.MoreApi.Last).init()
The loaded API is a table with all the APIs inside the directory loaded.
Multitasking
NewLife functions on processes. Theres an NLClass loaded at boot called Process which is used to create a process from either a function or a file.
Spoiler
local procEnv = setmetatable({}, {__index = getfenv()})
local proc = new(Process).init("filename.lua", procEnv, stdin, stdout, ... )
while proc.status() ~= "dead" do
os.pullEvent()
end
This loads a file "filename.lua" into a process and waits for that process to be complete. When a process dies, it create an event called "process_death" which is why we don't have to worry about os.pullEvent() stalling. stdin and stdout are meant to be the IO pipeline files.
The Pipeline
Every process has an input file and an output file. Most processes run with stdin (standard in) and stdout (standard out). The IO files don't have to be actual files on disk. The input file just needs to be a table with a "readLine" function, and the output file just needs to be a table with a "write" function. stdin points to the current process's input file. stdout points to the process's output file.
When a process iterates (goes one cycle until it yields the coroutine), the global functions read and write redirect to the IO files of the process. This means that when print is called, print calls write, which calls the output file of the process. So if you had the output file being an actual file on disk, then printing would just write to that file.
The custom shell in NewLife (called ClamShell), supports using these IO capabilities in a way similar to the UNIX pipeline.
ls > testFile
This command in ClamShell would take the output of the ls program and direct it to the file "testFile". But if testFile were actually a program, it would continually read from the ls output until there was nothing left to read from. So if you run ls > testFile once, and this creates a file with the output of ls, you cannot run the same command again. ClamShell would try to run the testFile from the previous running of the command as a program and redirect its input to the ls output. Obviously this wouldn't work.
The solution is that you can force ClamShell to treat testFile as a file instead of a command.
ls >: testFile
If you have a colon on either side of the pipe symbol ( > ), the command on the same side will be forced to be treated as a file.
There is a program built into NewLife called "type". Type takes input until read() returns nil, and prints all the read text. This is primarily used in the pipeline.
someFile :> type
That code will read someFile and print it to the screen. This allows some basic logging capabilities. For example,
ls >: somefile :> type
ls outputs to the file. The file outputs to type. Type outputs to the screen. So all the data is saved in somefile, and printed on the screen.
There's also a program called "ps" which lists running processes and lets you kill any of them.
And finally, the last program I'll talk about here is echo. Echo takes its arguments and simply prints them to the output. This is useful for simple stuff like "echo "testing!" > myfile" because you can save text to a file quite easily.
Multiple File Programming
In CraftOS, there's no good way to use multiple files in a program. Project NewLife allows the creation of processes from directories of code.
Spoiler
MyProgram
|-SomeClass.nlclass
|-main.coroutine
-- SomeClass.nlclass
function init()
print("MyProgram!")
end
-- main.coroutine
new(SomeClass).init()
In the shell, you could use the MyProgram directory as a valid program. The files are all actually loaded via the loadAPI() function, so they all get run. But once they're all loaded, the main.coroutine file, which was loaded as a coroutine, will be resumed with the arguments being the process arguments. In this case, main.coroutine just creates an instance of SomeClass, which prints "MyProgram!"
This capability allows for much more complex projects to be handled much more elegantly.
Daemons and Virtual Files
Spoiler
There is a subclass of Process called Daemon. Daemon's constructor simply requires the path to the file to run, and the path to the folder where the log file should be held. The Daemon class automatically handles setting the stdin and stdout files for the process. There is no standard input, and the output is the log file. At boot time, any programs stored in /NewLife/srv will be started as daemons.Along with Daemons are virtual files. The fs API has been modified to allow the creation of files that don't write to the disk, but instead talk to a process.
-- My daemons main file
function main()
fs.createVirtualFile("/NewLife/dev/test", { w=writeHandle, wb=writeBytesHandle,
a=appendHandle, ab=appendBytesHandle,
r=readHandle, rb=readBytesHandle})
end
Creating a virtual file expects you to give a handle for all the types of file modes that you expect to be used. These handles must provide all the functions used by typical file system handles. Everything else on the computer will see this virtual file as nothing more than a normal file. Therefore, it is imperative that your file handle functions do not crash. If they crash, the process trying to use the file crashes. The expected way to deal with this is that none of the logic occurs in the file handle functions. All the logic is handled by your process, data is shared between the handles and the process, and the handles alert your process that logic needs to occur via an os.queueEvent() call.
The purpose of /NewLife/dev is to be a folder for virtual files to exist. This is where it might be smart to keep your files.
Smaller Features
Spoiler
startup.luaWhen ClamShell starts up, it does what the CraftOS shell does in that it looks for a file to run. But instead of running startup (that would not go well), it runs startup.lua.
Compiling
For those who wish to compile their lua code, there is a program called luac which can take a file and compile it to lua bytecode. The function "loadfile" has been modified to read files by bytes and load the string as code, so anything that loads programs through loadfile can run both text based lua programs and bytecode ones.
Extended string and table libraries
Spoiler
The string library has a "split" function added to it.
function string.split(str, pat, maxNb)
@param str The string to split
@param pat The pattern to use as the delimiter
@param maxNb The max number of splits to make
@return A table of the split values
The table library has several functions for inverting, clearing, and copying tables.
function table.copy(t, recursiveMT)
@param t The table to copy
@param recursiveMT Whether the metatable should be copied recursively or not
@return A copy of t
function table.recursiveCopy(t, recursiveMT)
@param t The table to copy
@param recursiveMT Whether the metatable should be copied recursively or not
@return A copy of t, with all table values also copied
function table.clear(t)
@param t The table to clear
@purpose Empty all values out of a table
function table.invert(t)
@param t The table to invert
@return A table whose keys are t's values, and whose values are t's keys
function table.arrayCopy(src, pos1, dest, pos2, len)
@param src The source tables
@param pos1 The position in src to start copying from
@param dest The destination table
@param pos2 The position in dest to start copying to
@param len The number of entries to copy.
@purpose To copy a section of data from src to dest
function table.arrayCopyRecursive(src, pos1, dest, pos2, len)
@param src The source tables
@param pos1 The position in src to start copying from
@param dest The destination table
@param pos2 The position in dest to start copying to
@param len The number of entries to copy.
@purpose To copy a section of data from src to dest, and also recursively copy any table entries being copied
JSON API
A simple API for going between lua values and JSON text.
function json.decode(text)
@param text The text to turn into a lua value
@return The lua value
function json.encode(val)
@param val The value to turn into JSON
@return JSON
function json.encodePretty(val)
@param val The value to turn into JSON
@return JSON formatted to be pretty
A path editing program
NewLife adds a program called path which can display the current path, and edit it.
path add /someplace
path remove /someplace
A shell script program
cs is a program that takes a file and runs each line as a shell script. It's not complicated but much more is planned for it.
A github repo downloader
gh downloads github repos. Use the command like this:
gh -user {username} -repo {repo name} {directory to save}You can get Project NewLife by downloading it from its github repo. I do plan to have an automatic downloader and installer, as well as an updater, but for now this is fine.
As requested, a quick installation guide.
Download the entire repo as a zip, extract the zip, and go into the Project-NewLife folder. Put the two files and the folder in the root folder of your computer. /startup.lua is optional, since it's intended to be edited by the user.
Changelog:
Spoiler
v1.1- Added: LASM; An assembly language for Lua bytecode
- Added: NLObject. The superclass of EVERYTHING
- Changed: NLClass no longer calls init() automatically for you. This allows selected initializers.
- Changed: Reworked the JSON encoder. Much simpler.
- Changed: txtutl file extension changed to lson
v1.0.2
- Added: .coroutine extension. Loads the file as a coroutine.
- Changed: Multi-file programming no longer uses main.lua with a main function. Instead, use main.coroutine, which will be resumed on launch with the arguments.
v1.0.1
- Added: Virtual files
- Added: File extension loaders
- Added: Support for loading JSON and text serialized by textutils as APIs
- Added: Support for loading txt files as text in APIs
- Added: A daemon class
- Added: A github repo downloader
- Changed: stdin/stdout now point to the current process's in and out, rather than the keyboard and terminal.
- Changed: os.queueEvent now supports queueing event with tables as parameters
- Changed: The process.lua karma extension is now an API in /NewLife/lib. No more annoying global functions
- Changed: os.run now uses Processes
- Removed: FileStream.nlclass. Wasn't necessary.
v1.0
- Initial release
