RFC: TRoR (Terminal Redirection over Rednet)

This is a post seeking serious discussion of the creation of a standard of sorts for terminal redirection over rednet (TRoR hereafter).

Terminal Redirection over Rednet is a mostly unexplored area with a huge potential set of benefits. The creation of a standard system of interaction between sender and receiver will greatly simplify any efforts to create compatible programs using TRoR. To that end, I propose the following standard packet structure:

<two-character packet type code>:;<message contents>

Please note that any non-semicolon character can be placed between the colon and the semicolon by either end, which MAY be used for any purpose, but standard-compliant TRoR programs SHALL NOT malfunction if any characters are present there. This basically means that to get the message contents, a shell.match(message, ";(.*)") is suggested.

Here is a list of the two-character packet codes and their uses:

TW - carries written text to the client as the message contents.
TC - sets the cursor position on the client screen, format: "<x>,<y>".
TG - gets the cursor position from the client.
TD - gets the screen size from the client.
TI - used to return information to the server, such as screen size, cursor position, or color/non-color.
TE - used to clear the client screen.
TL - clears the current line of the client screen.
TS - scrolls the client screen.
TB - sets the cursor blink, using string literals "true" and "false"
TF - sets the foreground color.
TK - sets the background color.
TA - gets the value of term.isColor() from the client.

So, an example packet exchange might look like:

1: TD:;nil
2: TI:;51,19
1: TA:;nil
2: TI:;true
1: TC:;1,1
1: TF:;16
1: TK:;32768
1: TW:;CraftOS 1.4
1: TC:;1,2
1: TW:;>

Here is a packet type/name translation table, for more human-readable handling of these packets, which we will be using in our example implementation:

local packetConversion = {
    textWrite = "TW",
    textCursorPos = "TC",
    textGetCursorPos = "TG",
    textGetSize = "TD",
    textInfo = "TI",
    textClear = "TE",
    textClearLine = "TL",
    textScroll = "TS",
    textBlink = "TB",
    textColor = "TF",
    textBackground = "TK",
    textIsColor = "TA",
    TW = "textWrite",
    TC = "textCursorPos",
    TG = "textGetCursorPos",
    TD = "textGetSize",
    TI = "textInfo",
    TE = "textClear",
    TL = "textClearLine",
    TS = "textScroll",
    TB = "textBlink",
    TF = "textColor",
    TK = "textBackground",
    TA = "textIsColor",
}

So, that's basically the end of the proposed standard, on to the example implementation:

And here are a couple of utility functions that we'll use for easier handling of the packet format:

local function send(id, type, message)
    return rednet.send(id, packetConversion[type]..":;"..message)
end

local function awaitResponse(id, time)
    id = tonumber(id)
    local listenTimeOut = nil
    local messRecv = false
    if time then listenTimeOut = os.startTimer(time) end
    while not messRecv do
        local event, p1, p2 = os.pullEvent()
        if event == "timer" and p1 == listenTimeOut then
            return false
        elseif event == "rednet_message" then
            sender, message = p1, p2
            if id == sender and message then
                if packetConversion[string.sub(message, 1, 2)] then packetType = packetConversion[string.sub(message, 1, 2)] end
                message = string.match(message, ";(.*)")
                messRecv = true
            end
        end
    end
    return packetType, message
end

Now, to redirect terminal output to the client, we will need to construct a terminal redirect table with all of the normal term functions, except that it sends them to the client rather than outputting anything on the screen. Here is a function that constructs a new redirect table. The function is passed a computer ID and constructs a redirect target that sends all of the term commands to that computer ID. Note that it references the two functions above, so if using this table, be sure to include those functions above this one.

local function textRedirect (id)
    local textTable = {}
    textTable.id = id
    textTable.write = function(text)
        return send(textTable.id, "textWrite", text)
    end
    textTable.clear = function()
        return send(textTable.id, "textClear", "nil")
    end
    textTable.clearLine = function()
        return send(textTable.id, "textClearLine", "nil")
    end
    textTable.getCursorPos = function()
        if send(textTable.id, "textGetCursorPos", "nil") then
            local pType, message = awaitResponse(textTable.id, 2)
            if pType and pType == "textInfo" then
                local x, y = string.match(message, "(%d+),(%d+)")
                return tonumber(x), tonumber(y)
            end
        else return false end
    end
    textTable.setCursorPos = function(x, y)
        return send(textTable.id, "textCursorPos", math.floor(x)..","..math.floor(y))
    end
    textTable.setCursorBlink = function(B)/>
        if b then
            return send(textTable.id, "textBlink", "true")
        else
            return send(textTable.id, "textBlink", "false")
        end
    end
    textTable.getSize = function()
        if send(textTable.id, "textGetSize", "nil") then
            local pType, message = awaitResponse(textTable.id, 2)
            if pType and pType == "textInfo" then
                local x, y = string.match(message, "(%d+),(%d+)")
                return tonumber(x), tonumber(y)
            end
        else return false end
    end
    textTable.scroll = function(lines)
        return send(textTable.id, "textScroll", lines)
    end
    textTable.isColor = function()
        if send(textTable.id, "textIsColor", "nil") then
            local pType, message = awaitResponse(textTable.id, 2)
            if pType and pType == "textInfo" then
                if message == "true" then
                    return true
                end
            end
        end
        return false
    end
    textTable.isColour = textTable.isColor
    textTable.setTextColor = function(color)
        return send(textTable.id, "textColor", tostring(color))
    end
    textTable.setTextColour = textTable.setTextColor
    textTable.setBackgroundColor = function(color)
        return send(textTable.id, "textBackground", tostring(color))
    end
    textTable.setBackgroundColour = textTable.setBackgroundColor
    return textTable
end

So, to handle these packets on the client end, we have a simple function that takes slightly-processed information about these packets and responds appropriately. It takes the server computer ID, the human-readable packet type, and the value sent with it as parameters. Note that this also uses the send() helper function above.

local function processText(conn, pType, value)
    if not pType then return false end
    if pType == "textWrite" and value then
        term.write(value)
    elseif pType == "textClear" then
        term.clear()
    elseif pType == "textClearLine" then
        term.clearLine()
    elseif pType == "textGetCursorPos" then
        local x, y = term.getCursorPos()
        send(conn, "textInfo", math.floor(x)..","..math.floor(y))
    elseif pType == "textCursorPos" then
        local x, y = string.match(value, "(%d+),(%d+)")
        term.setCursorPos(tonumber(x), tonumber(y))
    elseif pType == "textBlink" then
        if value == "true" then
            term.setCursorBlink(true)
        else
            term.setCursorBlink(false)
        end
    elseif pType == "textGetSize" then
        x, y = term.getSize()
        send(conn, "textInfo", x..","..y)
    elseif pType == "textScroll" and value then
        term.scroll(tonumber(value))
    elseif pType == "textIsColor" then
        send(conn, "textInfo", tostring(term.isColor()))
    elseif pType == "textColor" and value then
        value = tonumber(value)
        if (value == 1 or value == 32768) or term.isColor() then
            term.setTextColor(value)
        end
    elseif pType == "textBackground" and value then
        value = tonumber(value)
        if (value == 1 or value == 32768) or term.isColor() then
            term.setBackgroundColor(value)
        end
    end
    return
end

Here's a stub of a processing loop for receiving those packets. Note that this references the packetConversion table above.

while true do
    event = {os.pullEvent()}
    if event[1] == "rednet_message" then
        if packetConversion[string.sub(event[3], 1, 2)] then
            packetType = packetConversion[string.sub(event[3], 1, 2)]
            message = string.match(event[3], ";(.*)")
            processText(serverNum, packetType, message)
        end
    end
end

So, that's the proposed TRoR standard and the example implementation, which are used in my net shell program (nsh), to great effect. I propose that this TRoR specification be adopted as a semi-formal standard, and would like to request serious commentary on the standard, but not so much the example implementation (as any standard-compliant program would be free to implement it in its own way).

It looks fine so far, but is there a packet type for user input?

Hmm, that's an interesting point. It is a little outside the scope of the TRoR standard, but it would probably be good to standardize as well. In nsh, I'm using packet type designations from LyqydNet and a custom event packing scheme. I think for the sake of standardization, it would be better to use a separate packet type and use textutils.serialize() to pack the event. So, using the same format:

EV:;<textutils serialized event data>

And for the translation table entries, I suggest:

  EV = "event"
  event = "EV"

This allows us to easily unserialize and unpack() the data on the receiving end. Perhaps a standard for connection establishment and other control packets would be handy as well. I'll write up the scheme nsh uses (adapted from LyqydNet) in a subsequent post this evening.

elseif pType == "event" and value then
	    value = textutils.unserialize(value)
	    os.queueEvent(unpack(value))
end

This would work for processing events.

To capture and transmit events on the client's side, then I think that you would need a separate thread.

KillaVanilla, on 27 November 2012 - 10:57 AM said:

elseif pType == "event" and value then
	    value = textutils.unserialize(value)
	    os.queueEvent(unpack(value))
end

This would work for processing events.

To capture and transmit events on the client's side, then I think that you would need a separate thread.

Of course, specific implementation of event handling (if any) would be up to the program(s) in question. For instance, my nsh program utilizing the TRoR manages different coroutines for each connected client session, so os.queueEvent() would be the wrong solution there. That is a useful example implementation, though, so thanks! You also don't end up needing a separate thread to capture events on the client side in nsh's case (since it handles incoming rednet messages and incoming keyboard/mouse events in the same loop). :)/>

I managed to do the same thing without packets. I haven't exactly tested it 100%, but it works for what it was designed for.
I adapted this from it working with LAN cables for CCLan. Also, I borrowed your design for running the shell in a thread and updating it using events, so credit for the core design goes to you, Lyqyd.

However, the status of the connection isn't managed by the software; the user is expected to be aware of this information.
Basically, the receiver can manipulate the machine of the transmitter while the transmitter can also use their machine, but the transmitter can see what
the receiver is doing.

So, the file system the shell will be manipulating will be on the transmitter and only on the transmitter.

local tArgs = { ... } -- Capture arguments inputted at the top level shell.
--[[
        LAN Cable Remote Shell          PaymentOption
                                        Black Wolf Server
                                        27 November 2012

        This program allows for the connection of one
        computer to another and redirecting terminal
        output to the receving machine. Both the receiving
        machine and transmitting machine will be able
        to provide input to the transmitting machine
        through this program.

        Remember, this program will NOT check if the connection is secure
        or the receiver/transmitter is still running this program. It is
        the responisibility of the user to be aware of this information.
        Results of the aforementioned situation are unknown, so be careful.
]]--

local connection = {} -- This will be the connection table for the connection that
                      -- is made to this particular computer.
local session = {}    -- This will be the shell session that has been started
                      -- over the starting shell that will capture and
                      -- resend events to the receiving computer.
local modemSide = nil -- This will be the side on which the lan cable exists.
local receiving = false -- Whether or not this particular computer is receiving
                        -- the terminal output or is transmitting it.



-- Parses the arguments passed from the top level shell. The proper useage
-- of the program is printed if the arguments are in some way incorrect.
-- Params : nil
-- Returns: true or false - Depending on whehter or not the arguments were correct.
function processArguments()
        if #tArgs == 2 then
                if tArgs[1] == "transmit" or tArgs[1] == "receive" then
                        if tArgs[1] == "receive" then
                                receiving = true
                        end

                        if tonumber(tArgs[2]) then
                                connection.id = tonumber(tArgs[2])
                                return true
                        end
                end
        end

        print("Useage: " .. fs.getName(shell.getRunningProgram()) .. " <transmit/receive> <connectionID>")
        return false
end

-- Attempts to locate a lan cable on any side of this particular computer.
-- This function will wrap the lan cable if one is found and return it, along
-- with the side the cable was found on.
-- Params : nil
-- Returns: rednet, rednetSide - The wrapped lan cable and the side its on.
function locateAndWrapLanCable()
        local sides = rs.getSides()
        for sideIndex, side in ipairs(sides) do
                if peripheral.getType(side) == "modem" then
                        modemSide = side
                        rednet.open(side)
                        return
                end
        end
end

-- Creates a new local shell session in a thread.
-- Params : nil
-- Returns: shellThread
function newShellThread()
        local shellThread = coroutine.create(function() shell.run("rom/programs/shell") end)
        return shellThread
end

-- Wraps all of the native terminal functions into new functionst that will
-- transmit themselves over lan to the receiving computer in a serialized table.
-- Params : nil
-- Returns: wrappedTerminal - The new native terminal functions that redirect output.
function wrapNativeTerminalFunctions()
        local oldTerminal = {}
        local wrappedTerminal = {}

        local function wrapFunction(functionName)
                return function( ... )
                        rednet.send(connection.id, textutils.serialize({functionName = functionName, params = { ... }}))
                        return oldTerminal[functionName]( ... )
                end
        end

        for functionName, functionObject in pairs(term.native) do
                oldTerminal[functionName] = functionObject
        end

        for functionName, functionObject in pairs(term.native) do
                wrappedTerminal[functionName] = wrapFunction(functionName)
        end

        return wrappedTerminal
end

-- Transmits information from this particular computer to the receiving computer
-- over lan in a serialized table. Also, this program handles the output on the
-- terminal after the computer has wrapped the native terminal output to be
-- sent over lan.
-- Params : nil
-- Returns: nil
function transmitTerminalOutput()
        -- Wrap all of the native terminal functions into a new table that
        -- will transmit their information to the receiving machine. Then
        -- redirect terminal output through aforementioned table.
        local redirectedTerminal = wrapNativeTerminalFunctions()
        term.redirect(redirectedTerminal)

        -- Create a new shell instance and resume it, all the while handling
        -- messages passed from the receiving computer.
        local shellThread = newShellThread()
        coroutine.resume(shellThread)

        while true do
                -- Wait for an event and handle it accordingly. Use a table to
                -- capture all parameters returned by any event.
                local event = {os.pullEvent()}

                -- If the event was a cable message, then go ahead and check
                -- to see if it was from the receiving machine. In the case
                -- that it is, then handle it like an event.
                if event[1] == "rednet_message" and event[2] == connection.id then
                        event = textutils.unserialize(event[3])
                        -- The message should have been an event table. Swap
                        -- the sent event for the current one and update the
                        -- screen.
                end

                -- Update the current shell thread with the event that was
                -- captured this iteration.
                coroutine.resume(shellThread, unpack(event))
        end
end

-- Receives terminal output from the transmitting machine and draws it to the
-- screen. This function also handles local events and redirects them to the
-- transmitting machine. However, rednet events are not redirected, nor are
-- lan cable events to prevent confusion on the transmitter's side.
-- Params : nil
-- Returns: nil
function receiveTerminalOutput()
        -- Runs any function from the transmitting machine from within the
        -- native terminal functions.
        local function runTerminalFunction(functionTable)
                if term.native[functionTable.functionName] then
                        return term.native[functionTable.functionName](unpack(functionTable.params))
                end
        end

        -- Create a new shell instance and resume it, all the while handling
        -- messages passed from the transmitting computer. Also, send
        -- user interaction events that are passsed.
        local shellThread = newShellThread()
        coroutine.resume(shellThread)

        while true do
                -- Wait for an event and handle it accordingly. Use a table to
                -- capture all parameters returned by any event.
                local event = {os.pullEvent()}

                        -- If the event was a cable message from the transmitting computer,
                        -- then go ahead and attempt to process it as a function call.
                        if event[1] == "rednet_message" and event[2] == connection.id then
                                -- Get the function table from the message by
                                -- unserializing the message, then execute it.
                                local functionTable = textutils.unserialize(event[3])
                                runTerminalFunction(functionTable)
                        -- Any other event can be serialized and sent to the
                        -- transmitting machine.
                        else
                                rednet.send(connection.id, textutils.serialize(event))
                        end

                -- Update the current shell thread with the event that was
                -- captured this iteration.
                -- coroutine.resume(shellThread, unpack(event))
        end
end

-- Proccess the given arguments to make sure the program has the necessary information
-- to execute properly.
if not processArguments() then
        return
end
-- Attempt to locate and wrap any land cable that can be found.
locateAndWrapLanCable()
if not modemSide then
        print("No modem found.")
        return
end

-- If we are not to be the receiving computer, then begin transmitting terminal
-- output to the receiving computer and listen for events from the receiving computer.
if not receiving then
        transmitTerminalOutput()
else
        receiveTerminalOutput()
end

http://pastebin.com/kDw9Zm7L

While I appreciate the input, and the obvious effort you've put into that program, it does not seem relevant to this standard. Your program seems to be a screen sharing program (and for that, your method is sufficient, I think), while this standard establishes a full terminal redirect over rednet to another computer. For this, two-way communication is necessary, so a simple wrapper function for each term function is insufficient. Your program accomplishes a similar thing, but not quite the same thing. For instance, a full TRoR could have either end on a turtle or a computer and, as long as the program in question was written to work on either, it would function correctly on the target computer; your system would not exhibit this behavior between a turtle and a computer.

It was certainly interesting reading through your code, though, so thanks for posting! :)/>

Lyqyd, on 28 November 2012 - 06:24 PM said:

While I appreciate the input, and the obvious effort you've put into that program, it does not seem relevant to this standard. Your program seems to be a screen sharing program (and for that, your method is sufficient, I think), while this standard establishes a full terminal redirect over rednet to another computer. For this, two-way communication is necessary, so a simple wrapper function for each term function is insufficient. Your program accomplishes a similar thing, but not quite the same thing. For instance, a full TRoR could have either end on a turtle or a computer and, as long as the program in question was written to work on either, it would function correctly on the target computer; your system would not exhibit this behavior between a turtle and a computer.

It was certainly interesting reading through your code, though, so thanks for posting! :)/>

I probably should have read your whole post before posting my solution, sorry.
However, I am going to go ahead and apply the concept of multiple connections on a single server computer to my design.

If I were to accomplish something that met the standards that you have described, I think I would have to redesign my entire script. Thanks for the response.

The standard described is meant more as a structure for the communication of the screen data, for greater interoperability. In that way, any program that needed to utilize TRoR for whatever purpose could be made to work with a generalized TRoR client, which may make a slightly more advanced concept accessible to more coders in the community. If you wished to utilize the communication method described in the TRoR standard, it should be a relatively simple conversion, as all of the example implementation code in the first post is in the public domain and free for use by any and all, for the purposes of implementing TRoR communications in accordance with this standard.

As a side note, I think the standard should not specify the failure-mode for communication timeouts (that is, for requesting the cursor position, screen size, and color status of the client), so that individual programs may ignore failures in these for purposes of screen broadcasting or similar usages. I welcome any commentary on that thought as well.

Lyqyd, on 09 December 2015 - 07:34 PM said:

Yes, though there is an additional "textTable" packet type (in which the buffer contents are sent whole) that vncd will only use, and that nsh will use when the framebuffer API is available. I should add that to the RFC post, thanks.

I believe you are still yet to do this

True, thanks for the reminder! I don't remember the table structure offhand, so I'll try to remember to post that tonight tomorrow night.