This is a read-only snapshot of the ComputerCraft forums, taken in April 2020.
KidBrine's profile picture

Number conversion

Started by KidBrine, 28 June 2019 - 06:32 AM
KidBrine #1
Posted 28 June 2019 - 08:32 AM
Basically I want to make an accurate way of converting numbers between bases
I like to test my method using base 10 to 16 and 10 to 16
but when the numbers get a little larger (34A BFC6 3278 ACBF) specificaly in this case there is a lack of accuracy…
if I convert 34ABFC63278ACBF to decimal and back i get 34ABFC63278ACC0 which is 1 greater than 34ABFC63278ACBF is.

before you tell me to use string.format() or tonumber(), like I have seen others do on the internet, they also have the same problem.

is there any way y'all could think if to make it more accurate?

PS. also not asking for, "No one needs to be that accurate"
Edited on 28 June 2019 - 06:34 AM
SquidDev #2
Posted 28 June 2019 - 09:11 AM
Lua, and so ComputerCraft represents numbers using doubles. As a result, when you start working with large numbers, you'll begin to lose some precision. 0x34ABFC63278ACBF is in the range of 257, and so is well beyond the region that doubles can represent accurately (doubles can handle integers up to 252 for those curious). There's not really a nice solution to this, so your best bet is to use one of the many big integer libraries out there instead. They effectively split the number up into smaller bits and store it in a table, allowing you to have arbitrarily large numbers.
KidBrine #3
Posted 28 June 2019 - 06:54 PM
Lua, and so ComputerCraft represents numbers using doubles. As a result, when you start working with large numbers, you'll begin to lose some precision. 0x34ABFC63278ACBF is in the range of 257, and so is well beyond the region that doubles can represent accurately (doubles can handle integers up to 252 for those curious). There's not really a nice solution to this, so your best bet is to use one of the many big integer libraries out there instead. They effectively split the number up into smaller bits and store it in a table, allowing you to have arbitrarily large numbers.

And I now need to make them CC friendly.