BBPack - Pastebin uploader / downloader

pastebin get cUYTGbpb bbpack

(Formerly called "Package")

This script is basically a modified version of the fabled "pastebin" script - it can send or receive files to or from your computer.

However, it has a couple of differences. First off, it supports multiple files per paste - tell it to upload a directory, and it'll do just that. The whole lot can then be downloaded elsewhere with one command. Whole system backups made easy!

On top of compressing with LZW, it also translates the upload into something ASCII-compatible, meaning that ComputerCraft won't strip out certain symbols as it would if you were using the regular pastebin script. Although this means the results aren't easily human-readable, it allows transmission of data that would otherwise be difficult to obtain when playing on an online MineCraft server (such as NBS files for Note, or schematic files for this schematic builder).

Note that bbpack's uploads will usually trigger pastebin's spam filtering system. You'll hence likely need to look up your paste in a web browser and fill in a captcha to complete the uploading process.

As of v1.6.0, BBPack implements a file system wrapper when loaded. By default this only adds a RAM drive (accessible via the root of your drive as "ram"), though it can also be configured to compress all of your files transparently, mount online files as if they were on your local drive, and connect to "clusters" - other ComputerCraft systems acting as combined storage drives.

It can either be executed as a regular script, or loaded as an API:

Script Usage

Uploads specified file or directory:
bbpack put [file / directory name]

Dumps paste into specified file or directory:
bbpack get <pasteid> [file / directory name]

Saves to a file, instead of uploading:
bbpack fileput [file / directory name] <archive>

Reads from a file, instead of downloading:
bbpack fileget <archive> [file / directory name]

If the [file / directory name] to pack / unpack to is excluded, it'll use the system root instead (eg, "bbpack put" uploads everything).

Mounts the specified web URL as a local file (or directory, if the URL is for a GitHub repo), or the specified cluster as a local directory:
bbpack mount (<URL> <localPath>) / (<cluster>)

Compresses your HDD (will also automatically compress files added in the future):
bbpack compress

Decompresses your HDD (or as much as it can before running out of space):
bbpack decompress

Turns the system into a server for the specified cluster:
bbpack cluster <cluster>

Updates bbpack on the current system and reboots, also instructing all mounted cluster server servers to do the same:
bbpack update

API Usage

bbpack.toBase64(table byte-values) => string text
Converts a given table of byte-values into a base64-based string (note: length will be increased by a third, eg three bytes becomes a four-character string).

bbpack.fromBase64(string text) => table byte-values
Converts a given base64-based string into a table of byte-values (eg a four character string becomes three bytes).

bbpack.compress(table byte-values / string text [, number value range]) => table byte-values
Applies LZW compression to a given table of byte-values or to a string. The value range must be a power of two that is higher than the largest byte value that may be in your input stream; the lower this is, the smaller the archive will be. Defaults to 256, though 128 is suitable for text.

bbpack.decompress(table byte-values [, boolean outputText [, number value range]]) => table byte-values / string text
Decompresses a given table of LZW-compressed byte-values. Type of the returned result depends on whether outputText is defined as true. The value range must match whatever the content was compressed with, defaulting to 256 if unspecified.

bbpack.uploadPaste(string name, string content) => string pasteID
Uploads a given string to pastebin.com (as a guest, no expiry), then returns the new paste ID.

bbpack.downloadPaste(string pasteID) => string content
Downloads and returns the content of a given pastebin.com paste.

bbpack.open(string file name, string file mode [, number value range]) => table file handle
Returns a file handle similar to that produced by fs.open() (usage is the same), but linked to bbpack's compression system. These handles include an additional function, file.extractHandle(), which returns a copy of the actual file handle being used.

bbpack.lines(string file name) => function iterator
It's io.lines(), but rigged to work with text-mode files created with bbpack.open().

Note: Each of the above functions (except downloadPaste) alternatively accepts file handles produced by fs.open(). If used, they'll automatically read out the contents and then close the handle. Be sure to specify mode "r" for text, or "rb" for binary (byte-values). Also note that bbpack.open() requires binary-mode handles, and bbpack.lines() requires text-mode bbpack.open() handles.

bbpack.fileSys()
Controls BBPack's filesystem wrapper. Accepts various parameters:

bool true/false: Compresses or decompresses the filesystem.

string cluster: Mounts the specified cluster as a local directory.

string url, string path: Mounts the specified web URL as a local file, or if the path is to a GitHub repo, as a directory.

bbpack.update()
Updates BBPack and reboots. Also instructs all mounted cluster servers to do the same.

File names are included in each paste for all files, so if you eg upload a single file, it's up to you whether you download it under a new name (by specifying one with the "get" command) or not (by using the "get" command on its own).

For example, if you wanted to download this pre-packed set of NBS files (as included with Note Block Studio), you'd enter:

bbpack get CYRmLz78 Songs

Version History

2015/01/29
1.0.0
Initial release.

2015/02/03
1.1.0
Can now be loaded as an API.

2015/04/17
1.2.0
Script:
Now records empty directories.
No longer includes external disk drives when performing uploads of root.
Added fileput / fileget parameters (for saving to / loading from the computer's own drive, as opposed to pastebin).

API:
Added bbpack.open().
Added bbpack.lines().

2015/04/19
1.2.1
bbpack.open() now accepts an optional third parameter, "value range", same as compress / decompress.

2015/12/11
1.3.0
Because I'm an idiot, base64 implementation was non-standard. This build corrects it and hence breaks compatibility with all base64 strings produced using older bbpack builds.

2015/12/21
1.3.1
Updated for new pastebin RAW access path.

2017/06/06
1.5.0
Compression ~20% faster, decompression ~100% faster.
Updated bbpack.open() to match new fs.open() behaviours (binary mode: includes readAll (returns string), read supports numeric argument (returns string if used), write accepts strings).
Renamed from "package" to "bbpack" due to alternate "package" API's inclusion into mainline ComputerCraft releases.

2017/07/28
1.6.0
BBPack now installs an FS wrapper when loaded, providing a RAM drive, web mounts, distributed filesystems, and a compressed filesystem. Controllable via bbpack.fileSys().
Also added bbpack.update().

2017/09/07
1.6.1
Can now mount GitHub repos.

Very nice! Very well written.

Thanks. :)/>

I've updated the paste with a version that can also be loaded as an API. This makes it easier for other scripts to work with compression or base64 conversions - handy for eking extra disk space out of computers, or for transmitting odd files over rednet and so on.

This is something that will be very handy, it'll be really useful for installers. And nice to see that you've updated it.
I'll certainly be using this sometime in the future :D/>

Made a few tweaks, such as adding a package.open() function to the API side of things. It returns a handle that's pretty much the same as what you'd get out of fs.open(), but rigged to pass any data you hand it through its compression system. Which you could already do yourself using the existing functions, albeit without the convenience.

Eg, just as you can do this:

local myFile = fs.open("somefile", "w")
myFile.writeLine("some text")
myFile.close()

… you can also do this:

local myFile = package.open("somefile", "w")
myFile.writeLine("some text")
myFile.close()

Text-mode handles get a slightly better compression ratio than binary, though unless you're planning on writing a decent amount of data to disk there's not all that much of a difference.

Bomb Bloke, on 29 January 2015 - 06:30 AM said:

- snip -

I tried using this to pack a load of files and folders together and it just errors with:

package:85: too many results

I used:

package fileput omi test

Huh.

What it's saying is that one of your files - not sure which, but it should've printed its name before it errored - is producing too many characters for either unpack() or for string.char() to handle. Probably the former.

Thing is, I can't reproduce that, even when handing in a table with ten million elements (and I'm willing to bet all your files combined wouldn't generate a table even a tenth of that size, bearing in mind that that area of code is only handling individual files…).

What version of ComputerCraft are we talking about, and would you consider uploading the problematic file somewhere?

Bomb Bloke, on 29 April 2015 - 01:16 AM said:

Huh.

What it's saying is that one of your files - not sure which, but it should've printed its name before it errored - is producing too many characters for either unpack() or for string.char() to handle. Probably the former.

Thing is, I can't reproduce that, even when handing in a table with ten million elements (and I'm willing to bet all your files combined wouldn't generate a table even a tenth of that size, bearing in mind that that area of code is only handling individual files…).

What version of ComputerCraft are we talking about, and would you consider uploading the problematic file somewhere?

I am using 1.6 currently… on an emulator (everything else works) and I can do?

Been a while since I've actually advertised an update for this, but there's some new functionality in 1.6.0 that might be of use about the place.

Personally I've gotten used to developing code in SSP, where I've configured ComputerCraft to up drive sizes considerably. Playing online recently I hence felt severely cramped by the default allowance of a singular manufacturer's megabyte, and turned to BBPack to alleviate the strain.

Loading BBPack (either by simply running it or by loading it as an API) now implements an fs wrapper. By default, the only visible change to your system is the addition of a RAM drive (dubbed "ram", accessible from the root of your HDD) - you can stick pretty much whatever you like in there, although as nothing within it is actually written to disk, all its content will be lost should your system shut down.

BBPack's wrapper can implement additional features via a few commands:

bbpack compress / bbpack.fileSys(true)
These commands iterate through all content on your computer's HDD and compresses it. Files compressed in this manner take on an additional .bbp extension, which is hidden from you so long as BBPack is loaded (the only visible effect is hence a reduction in filesize). Additional files written to your drive in future will also be automatically compressed, although files on external drives are unaffected. In practise I find this increases the capacity of a system by about 3x on average.

startup, .settings, bbpack and its config file (.bbpack.cfg) are unaffected by compression, as are any GIF files.

bbpack decompress / bbpack.fileSys(false)
Reverts filesystem compression, decompressing all files. May leave some files compressed if it runs out of space during the decompression process, in which case the command line version will print a report, or the function version will return false.

bbpack cluster <clusterName>
The system will become a NAS, opening attached modems for rednet usage and granting other systems access to its storage. Multiple systems configured to join the same cluster will share their space, allowing for simple storage of much more data than a single system could hold.

bbpack mount <clusterName> / bbpack.fileSys(<clusterName>)
Opens attached modems for rednet usage and mounts the specified cluster under a local directory of the same name. A cluster can be demounted by rm / fs.delete()ing it. Cluster-stored content is compressed regardless as to the setting for the local drive. Clusters may be mounted by multiple systems at a time. If a server system is added to / removed from a cluster, client systems must be instructed to remount that cluster.

bbpack mount <URL> <path> / bbpack.fileSys(<URL>, <path>)
Mounts the specified web path as a local file, without actually storing it on your local drive. Only handles single files at a time. The mount can be removed by rm / fs.delete()ing it, but is otherwise treated as read-only.

BBPack uses .bbpack.cfg to remember mounts, and to track whether local filesystem compression is active. It'll automatically recall this information when reloaded (eg after a reboot). It's recommended to add os.loadAPI("bbpack") to the very top of your startup file should you wish a given system to use the compression or mounting features. Cluster servers need only shell.run("bbpack cluster <clusterName>").

If BBPack is configured to mount any cluster servers, it'll automatically open any attached modems whenever it's loaded.

Reading from / writing to locally compressed files, or files in a cluster, will cause your system to yield. In particular, shell autocomplete becomes quite slow while browsing clusters - consider disabling autocomplete if this is a problem for you.

Compressed files are difficult to access via external editors. If you're looking for more storage in a single-player environment, you may be better off increasing filesystem sizes within ComputerCraft.cfg.

Ooooh, wow. I've seen people talk on the forums before about disk compression, but never seen a working implementation. This is very nice! I'm now intrigued what's taking up all the space - I've only ever hit the limit on my development "computer", never in-world.

Is there ever a time where compressing files would be increase file size and, if so, how does it handle it? I'm just thinking about files which may already be compressed in some form or another.

Bomb Bloke, on 28 July 2017 - 02:31 PM said:

In particular, shell autocomplete becomes quite slow while browsing clusters - consider disabling autocomplete if this is a problem for you.

I wonder if you could put auto-completion in a child coroutine, so it doesn't block input but still provides (albeit delayed) suggestions? Obviously you'd have to override read as well, which isn't ideal.

if file.writeLine or term.setPaletteColour then
	file.write(content)
else

Well, that's an interesting way of determining version. It's nice to see people using the extended binary functions though.

SquidDev, on 28 July 2017 - 05:43 PM said:

I'm now intrigued what's taking up all the space - I've only ever hit the limit on my development "computer", never in-world.

Basically the fact that I can wget directly from imgur now. Although I did mention that the compression system intentionally ignores GIFs (because they already use LZW compression - hence why my GIF-handling API requires this API!), I've rigged up BLittle to process whole animations, and it most certainly helps with the BLT files that outputs.

Plus Cranium's NBS collection adds up to over a meg just on its own. I've rigged up a single download pack for that, by the way.

SquidDev, on 28 July 2017 - 05:43 PM said:

Is there ever a time where compressing files would be increase file size and, if so, how does it handle it?

Files which're already sufficiently dense may indeed get larger when compressed, in which case BBPack will simply go ahead and save the compressed versions anyway - it doesn't perform any comparisons (though since you mention it, I can see a spot where it should). The single current allowance I've made for this is the filetype blacklist system, to which I can easily add more entries. Hmm. I suppose I should've added ZIPs.

However, most types a user would actually save to a ComputerCraft system (ie text) should see a net reduction. It is worth noting again that startup isn't compressed - a lengthy startup file is hence better off split into a "just load BBPack and then shell.run() whatever else you want" version.

SquidDev, on 28 July 2017 - 05:43 PM said:

I wonder if you could put auto-completion in a child coroutine, so it doesn't block input but still provides (albeit delayed) suggestions? Obviously you'd have to override read as well, which isn't ideal.

Overriding read is the last thing I want to do, especially with the constant stream of people trying to change the mainline version's behaviour.

… or, maybe I should try and push a coroutine-based version into mainline myself. Mwuaha.

More likely I'll leave it for people to figure out on their own. Using a GUI-based file manager would skirt the issue nicely, for example.

SquidDev, on 28 July 2017 - 05:43 PM said:

Well, that's an interesting way of determining version.

I'd prefer to be able to use something like the _HOST constant, but it's not terribly backwards compatible and I don't even trust that it'll be forwards compatible. _CC_VERSION certainly wasn't.

SquidDev, on 28 July 2017 - 05:43 PM said:

It's nice to see people using the extended binary functions though.

And it's very nice to have them! I'll likely never know why they were omitted in the first place.

SquidDev pointed out how easy it is to get the directory listing of a GitHub repo, so BBPack can mount those as virtual drives now (in addition to the single web files and other things it could mount before). Eg:

bbpack mount https://github.com/alekso56/ComputercraftLua/tree/master/treasure gamez
cd gamez/dan200/alongtimeago/
alongtimeago

You can mount different branches / commits / sub-folders / whatever - just head to the point you want using your browser, and copy/paste the URL. In code, you can also produce mounts using "bbpack.fileSys(url, driveName)".

All web-mounts are read-only, but you can unmount them by rm / fs.delete()'ing them. If you copy their contents you'll end up with a local version you can edit (eg "cp gamez localGamez"). Directory listings for repos are only refreshed when you first mount them (or the next time BBPack is loaded after a reboot - I recommend sticking os.loadAPI("bbpack") at the top of your startup file), but file contents are pulled fresh from the servers every time you access them.