Reading bytes

Hey guys,
I'm working on a project where I'm trying to read some data out of a png, and I'm having some issues. I'm adapting a lot of code from others and kind of mixing it together, and one of the issues I have is actually reading data from the png file. Since it's binary, it presents some difficulties. Here is the code I'm trying, but it will go through and read bytes up until it errors saying "attempt to perform arithmatic __mult on nil and number.

local function readByte(stream)
	return stream.read()
end
local function readInt(stream, bps)
	local bytes = {}
	bps = bps or 4
	for i=1,bps do
		bytes[i] = readByte(stream)
	end
	return bytesToNum(bytes)
end
local function readChar(stream, num)
	num = num or 1
	local bytes = 0
for i = 0, num - 1, 1 do
  bytes = bytes + (readByte(stream) * (256 ^ i))
end
end

So it seems that readByte is giving readChar a nil for one of the bytes. Obviously I'm doing something wrong. BTW this code is adapted from SquidDev's bitmap library. The other difficulty I am having is downloading binary files. This one is a creation of my own, and it works well for downloading binary, but when it downloads a png and I try to open it with Windows Photo Viewer, it says that it's missing or corrupt. Thoughts?

function downloadBinaryFile(url, saveFile)
local httpHandle = http.get(url)
if not httpHandle then
  return false
end
local data = httpHandle.readAll()
httpHandle.close()
print(#data .. " bytes")
local fhandle = fs.open(saveFile, "wb")
for i = 1, #data do
  fhandle.write(string.byte(data, i))
end
fhandle.close()
return true
end

downloadBinaryFile(<image url>,"skin.png")

HTTP binary support was a long standing bug that was patched in 1.80pr1. If you are using an older version it will not work.

Well putting aside the http portion of my question, what do you think about the byte reading part?

I have very little experience with reading binary in CC, but my first thought is that num is larger than the number of bytes available for some reason.

I can tell you that fileHandle.read() returns nil once you reach EOF (and never before), but otherwise, there's not enough information here to debug your first block of code. Provide the rest of it, along with the file you're using for your "stream".

Regarding the http API, even with later builds of ComputerCraft, http.get uses text mode for web handles by default, mangling anything that isn't strictly ASCII. For binary mode, you need to pass in true as a third argument (eg http.get("someAddress.com", nil, true). Under older builds, there's nothing for it but to use binary-to-text encoding methods such as base64 - BBPack can help you with this.

Another perk of the more recent versions is that binary mode file handles also accept strings, allowing you to dump out all of your data to disk in a single write operation. If you really want to deal with numeric byte representations, though, it can be a bit faster to cut down on your function calls by doing the conversion all in one go:

data = {data:byte(1, #data)}
for i = 1, #data do
  fhandle.write(data[i])
end

https://pastebin.com/mQ6CVG8V is the entirety of the code. It includes a bunch of stuff for implementing require because the libraries I'm using need it. Also uses deflateLua. I commented out line 359 for testing, but when all is working correctly it should pass that if condition. The image I'm using is here.

Line 342: you're using io.open(). File handles created in this manner require the use of colon notation for their function calls, like so (meaning you'd want stream:read() on line 151). If you wish to use dot notation instead, use fs.open().

Fixing that (and also uncommenting line 359) yields an error "Not a png". This leads me to believe that it's still not reading bytes correctly. I manually read through the each byte at the very beginning of the png, and it matches correctly with the if condition on line 358.

The readChar() function used in that comparison statement doesn't return anything. I'm guessing you meant to do:

local function readChar(stream, num)
    num = num or 1
    local bytes = {}
    for i = 1, num do
        bytes[i] = string.char(readByte(stream))
    end
    return table.concat(bytes)
end

I still seem to be getting the "Not a png" error. This is what has been troubling me for the past few days :wacko:/>

If you're not reading in the header you expect, then output what you are reading. If you can't figure things out from there, then provide that output here.

if readHeader ~= expectedHeader then
  print(readHeader:byte(1, #readHeader))
  error("Not a PNG")
end

I'm now getting an error that reads "window:94 Arguments must be the same length". I haven't made any changes to the code myself other than what you have said here.

This is why you're getting that error.

And this should help you locate the exact line within your own script that's causing the failure within the window API.

I've actually used trace before. Using it here actually prints nothing, however strange that may be. And I checked out BB's code for note posted in the other thread and when attempting to use it, it says attempt to call nil in reference to a function called unpack().

unpack has been renamed to table.unpack in later versions.

That's not exactly true - unpack() is only disabled if disable_lua51_features is set to true within ComputerCraft.cfg. By default it isn't, and since it's only good for performing partial forwards compatibility testing, there typically isn't any good reason to have it set.

Well with all this, it seems to work fine (to a point) with any PNG except the one I need. It seems that what I have may not be in the correct format for a PNG. I guess I'll have to sort that out later. The "to a point" I mentioned before refers to an error on line 382. It says attempt to add nil and number.

That line falls within StringStream.read(), which 1) needs to be called with colon notation and 2) needs a "num" parameter specifying the amount of bytes to read. It's called from line 151 (in readByte()) with neither of these, try changing that to:

return stream:read(1)

You could alternatively ditch the 1 parameter by adding a new line to the top of StringStream.read(), picking a default value for num:

num = num or 1

Note that either way this will force you to use io.open() on line 342, and that I only have access to an old version of your code.

Right, well here is a new version of my code so you can make an updated diagnosis. I do see what you are saying about the num variable and I tried to feed it to readByte but I was unsuccessful.

This is the change I would go with (around line 399):

    StringStream = {
        str = table.concat(output),
        read = function(self, num)
            num = num or 1
            local toreturn = self.str:sub(1, num)
            self.str = self.str:sub(num + 1)
            return toreturn
        end  
    }

I tried that before too, also to no avail. The error I get is "imgtest:354: Unsupported filter type: " which is because it doesn't seem to read anything on line 332. Not sure why though.

StringStream.read() is returning an incorrect data type - strings, instead of the numeric values that seem to be expected. Better to rig it to work entirely with numbers:

    printV("Deflating...")
    deflate.inflate_zlib {
        input = chunkData.IDAT.data,
        output = function(byte)
            output[#output+1] = byte  --# Ditch the string.char call here
        end,
        disable_crc = true
    }

    local callCount = 1

    StringStream = {
        read = function(self, num)
            num = num or 1
            local start, fin = callCount, callCount + (num or 1) - 1
            callCount = callCount + num
            return unpack(output, start, fin)
        end  
    }

This is why you're not seeing a "filter type" in that error message, as the filterType variable either currently holds a value that can't be rendered as a character (not so much of a problem, but less likely), or it's just an empty string (very much a problem, and unfortunately more likely).

If you find the latter to be true (and the error starts telling you the filter type is nil), that'd imply that the deflate.inflate_zlib() call on line 392 isn't actually putting any data into the "output" table. In that case, first check that the table parameters used in the inflate call are formatted correctly, and then check whether chunkData.IDAT.data actually holds some data.

I now get "imgtest:332: attempt to call nil"

Well, assuming that line is still only trying to call readByte(), I suppose you could go and check to see whether you've accidentally deleted that function's definition.

The function readByte() still exists and is declared before the getPixelRow() function.

If that's true, then 332 has changed since I've seen it last. I'm afraid my telepathy is failing me, so I can't comment further without access to the current version of the code.

Ah BB, that infallible sense of humor of yours…
Code: https://pastebin.com/NUwwJNcW

That code won't produce that error on that line. Check that you're running the correct copy of your script file.

By the way, digging a bit further I notice you're omitting DelusionalLogic's copyright notice. Be sure to include it in all copies of your code that include this work.

Yeah, I had cut it out temporarily to have less clutter, but in the end I plan on having all of the png library in it's own file with the appropriate copyright info.

Anyway, this is the code I am running as I uploaded it to pastebin via the in game pastebin script. So what else could this be?

The only other option is that you mixed up your error messages.

While I'm not really sure what you mean by mixing up my error messages, I can provide a screenshot of my terminal.
https://ibb.co/mosXVe

Any ideas?

emanuel12324, on 08 August 2018 - 03:35 PM said:

Any ideas?

line 332:

local filterType = readByte(stream)

Line 332 will not cause that error as bomb bloke said. Make sure line 332 is that line in the program you tried to run as well. If it is not then post your code to pastebin again give us the link here.

Well this is why I'm so troubled. The pastebin posted above is the exact code I'm using and it errors on that line. I have done everything from renaming the file to closing and reopening my world to relaunching my game. My suspicion is that the attempt to call nil isn't actually on line 332, but instead when readByte() tries to call StringStream:read()

emanuel12324, on 08 August 2018 - 09:05 PM said:

Well this is why I'm so troubled. The pastebin posted above is the exact code I'm using and it errors on that line. I have done everything from renaming the file to closing and reopening my world to relaunching my game. My suspicion is that the attempt to call nil isn't actually on line 332, but instead when readByte() tries to call StringStream:read()

If that was the case, trace would show it to you.

Edit: Just checked myself, in line 332 you are calling readByte(), which calls stream.read(stream).
However stream.read requires 2 Arguments, the stream and the number of bytes to read, the second one is not passed to it.

Edit 2: Also BB already told you about that error, which further makes me believe, that you are not providing us your current code.

I will once again post my code here, but you will see it has not changed since my most recent upload. I suspect you are not looking at the most recent paste that I posted. And about trace, well for whatever reason, it never shows me anything at all. Even the expected prints from my code don't show up.

Luca_S, on 08 August 2018 - 09:43 PM said:

emanuel12324, on 08 August 2018 - 09:05 PM said:

My suspicion is that the attempt to call nil isn't actually on line 332, but instead when readByte() tries to call StringStream:read()

If that was the case, trace would show it to you.

If that were the case, the error would directly point to line 171, where StringStream:read() is called: there wouldn't be any need to use trace to get that info. We're only interested in what's at the top of the stack, not what's lower down.

Luca_S, on 08 August 2018 - 09:43 PM said:

However stream.read requires 2 Arguments, the stream and the number of bytes to read, the second one is not passed to it.

It's been changed so that the second argument is optional. Even if it hadn't, though, that still wouldn't trigger the stated error on the stated line.

emanuel12324, on 09 August 2018 - 04:13 AM said:

Even the expected prints from my code don't show up.

This is another strong indication that you're not running the script file you think you're running.

How are you uploading the code to pastebin - with the "pastebin put" command, or by manually copying the file out of the computer's folder within your world save?

In the latter case, I'd say there's a good chance you're looking at the wrong computer folder!

Bomb Bloke, on 09 August 2018 - 05:39 AM said:

If that were the case, the error would directly point to line 171, where StringStream:read() is called: there wouldn't be any need to use trace to get that info. We're only interested in what's at the top of the stack, not what's lower down.

Not always. As it's used as a tail call, readByte will have exited before StringStream.read is actually called, meaning the error location will attributed to wherever readByte is called instead.

Ah, well spotted.

As for why it's erroring at all, it occurred to me that, in the unlikely event that emanuel is running the posted code, the error may be coming from another file. For example, if deflatelua is rigged to run a second copy of imgtest buried in a subfolder, then that could crash before the primary copy does.

Of course, that would be a very difficult mistake to make accidentally!

Having had a further look at the code, there's three functions which could be nil here (it's hard to tell which due to tail call):

• readByte - almost definitely not this
• StringStream.read - again, almost definitely not
• unpack - Ahhhr!

Can you try changing this to use table.unpack and see if it works? Alternatively, print unpack somewhere and see if it's nil. If you've got 5.1 features disabled in the configs, it's possible it won't be set.

Yaaahhhrrr

That's almost certainly it - and it in turn almost certainly means someone's using a tekkit pack.

Anyway, you'll be wanting to make sure disable_lua51_features is set to false, emanuel.

http://www.computercraft.info/wiki/ComputerCraft.cfg

Congratulations ladies and gentlemen, we have success. It was indeed the unpack not being called correctly, and adding table. to the front fixed it right up. So this project is shaping up. Now all I have to do is figure out how to successfully download a png. Thanks for all your help guys.