Arithmetic errors in large numbers

[dcantor]

TCC 10.00.68 Windows XP [Version 5.1.2600]

This FOR command is on one physical line.

for /l %%numzeroes in (0,1,15) do echo %@format[2,%numzeroes] %@format[19,0x1%@repeat[0,%numzeroes]] %@eval[0x1%@re
peat[0,%numzeroes] =x]
0 0x1 0x1
1 0x10 0x10
2 0x100 0x100
3 0x1000 0x1000
4 0x10000 0x10000
5 0x100000 0x100000
6 0x1000000 0x1000000
7 0x10000000 0x10000000
8 0x100000000 0x100000000
9 0x1000000000 0x1000000000
10 0x10000000000 0x10000000000
11 0x100000000000 0x100000000000
12 0x1000000000000 0x1000000000000
13 0x10000000000000 0x10000000000004
14 0x100000000000000 0xFFFFFFFFFFFFDC
15 0x1000000000000000 0x1000000000000BD0


Note lines 13, 14, and 15.

What's going on here?

 

[rconn]

dcantor wrote:

> TCC 10.00.68 Windows XP [Version 5.1.2600]
>
> This FOR command is on one physical line.
>
> for /l %%numzeroes in (0,1,15) do echo %@format[2,%numzeroes] %@format[19,0x1%@repeat[0,%numzeroes]] %@eval[0x1%@re
> peat[0,%numzeroes] =x]
> 0 0x1 0x1
> 1 0x10 0x10
> 2 0x100 0x100
> 3 0x1000 0x1000
> 4 0x10000 0x10000
> 5 0x100000 0x100000
> 6 0x1000000 0x1000000
> 7 0x10000000 0x10000000
> 8 0x100000000 0x100000000
> 9 0x1000000000 0x1000000000
> 10 0x10000000000 0x10000000000
> 11 0x100000000000 0x100000000000
> 12 0x1000000000000 0x1000000000000
> 13 0x10000000000000 0x10000000000004
> 14 0x100000000000000 0xFFFFFFFFFFFFDC
> 15 0x1000000000000000 0x1000000000000BD0
>
>
> Note lines 13, 14, and 15.
>
> What's going on here?

I have no idea what you're trying to do here -- can you reduce this to a
single line with an explanation of the intent?

(Note that if you're using hex, you're limited to 64-bit integers.)

Rex Conn
JP Software

 

[vefatica]

On Thu, 18 Jun 2009 20:15:28 -0500, rconn <> wrote:

|I have no idea what you're trying to do here -- can you reduce this to a
|single line with an explanation of the intent?
|
|(Note that if you're using hex, you're limited to 64-bit integers.)

Here's one of the errors:

v:\> echo %@eval[0x100000000000000=x]
0xFFFFFFFFFFFFDC

Another:

v:\> echo %@eval[0x1000000000000000=x]
0x1000000000000BD0

They're both less than 0x7FFFFFFFFFFFFFFF which is INT64_MAX.

Can't your "unlimited" arithmetic lib convert to/from hex?
--
- Vince

 

[rconn]

vefatica wrote:

> Can't your "unlimited" arithmetic lib convert to/from hex?

No. All hex numbers are limited to int 64's.

(Which isn't much of a limitation unless you're counting electrons.)

Rex Conn
JP Software

 

[dcantor]

dcantor wrote:


I have no idea what you're trying to do here -- can you reduce this to a
single line with an explanation of the intent?

(Note that if you're using hex, you're limited to 64-bit integers.)

Rex Conn
JP Software

I was building a 64-bit value one byte at a time in big endian order by extracting a byte from a file which had the values stored in ASCII representation of hexadecimal, multiplying the previous value by 256 and adding the value of the byte just acquired.

The file had values like this:

FE DC BA 98 76 54 32 10
representating the value 0xFEDCBA9876543210

The code looks something like this:

set nextbyte=%@trim[%@word[%bytenumber,%currentline]]
^ This caused nextbyte to have a 2-hex-digit value
set value=%@eval[0x%value%%nextbyte%]
:: and loop

There are many ways to do this loop, but the point is that I discovered that for a sufficiently large value, the computation fails.

I reduced that to the simple case that
%@eval[0x1000000000000000=x] doesn't equal 0x1000000000000000

Shouldn't it? It's fewer than 64 bits.

Dave C.

 

[rconn]

dcantor wrote:

> I reduced that to the simple case that
> %@eval[0x1000000000000000=x] doesn't equal 0x1000000000000000
>
> Shouldn't it? It's fewer than 64 bits.

When you input a hex number, it's converted to a double before being
passed to the large math library. So you're not going to get the full
64-bit input. (This does not affect hex output.)

Rex Conn
JP Software

 

[Steve Fabian]

rconn wrote:
| When you input a hex number, it's converted to a double before being
| passed to the large math library. So you're not going to get the
| full 64-bit input. (This does not affect hex output.)

Does this not negate the benefits of the 64-bit integers? For V11 I'd like
to see direct converisions between external (characterstring)
representations and large math library representations, e.g., using enhanced
forms of @convert.
--
Steve

 

[dcantor]

When you input a hex number, it's converted to a double before being
passed to the large math library. So you're not going to get the full
64-bit input. (This does not affect hex output.)

By "double", I assume you mean double-precision floating point, right?

Rex, I appreciate the acknowledgment that this is indeed a restriction that is not likely to be resolved. It would be helpful if there were a statement somewhere in the help file indicating just what range of 64-bit integers can be relied on to compute correctly. It's apparent that I can no longer build 64-bit integers from 64-bit hex values input from files. (I'm talking ASCII or Unicode, binary can be converted with @BALLOC, @BREAD, and @BPEEK.)

You say that it does not affect hex output, but it does:

echo %@eval[0x1000000000000000=x]

yields 0x1000000000000BD0 .

 

[dcantor]

My solution to my problem is

SET BH=%@BALLOC[8]
SET X=%@BPOKE[%BH,7,1,0xFE]
SET X=%@BPOKE[%BH,6,1,0xDC]
SET X=%@BPOKE[%BH,5,1,0xBA]
SET X=%@BPOKE[%BH,4,1,0x98]
SET X=%@BPOKE[%BH,3,1,0x76]
SET X=%@BPOKE[%BH,2,1,0x54]
SET X=%@BPOKE[%BH,1,1,0x32]
SET X=%@BPOKE[%BH,0,1,0x10]
ECHO %@CONVERT[10,16,%@BPEEK[%BH,0,8]]
fedcba9876543210

That's correct. That's just what I need.

 

[rconn]

dcantor wrote:

> ---Quote (Originally by rconn)---
>
> When you input a hex number, it's converted to a double before being
> passed to the large math library. So you're not going to get the full
> 64-bit input. (This does not affect hex output.)
> ---End Quote---
> By "double", I assume you mean double-precision floating point, right?
>
> Rex, I appreciate the acknowledgment that this is indeed a restriction that is not likely to be resolved. It would be helpful if there were a statement somewhere in the help file indicating just what range of 64-bit integers can be relied on to compute correctly. It's apparent that I can no longer build 64-bit integers from 64-bit hex values input from files. (I'm talking ASCII or Unicode, binary can be converted with @BALLOC, @BREAD, and @BPEEK.)
>
> You say that it does not affect hex output, but it does:
>
>
> Code:
> ---------
> echo %@eval[0x1000000000000000=x]
> ---------
> yields 0x1000000000000BD0 .

Right -- that's because of your hex input, not the output.

Rex Conn
JP Software

 

[Howard Goldstein]

Dave Cantor wrote:

> You say that it does not affect hex output, but it does:
>
>
> Code:
> ---------
> echo %@eval[0x1000000000000000=x]
> ---------
> yields 0x1000000000000BD0 .
>

No, it's your hex input that's causing the error. Try
echo %@eval[16**15=x]
and you'll see that the output is correct.

--
Howard

 

[vefatica]

On Sat, 20 Jun 2009 20:12:42 -0500, Howard Goldstein <>
wrote:

|Dave Cantor wrote:
|
|---Quote---
|> You say that it does not affect hex output, but it does:
|>
|>
|> Code:
|> ---------
|> echo %@eval[0x1000000000000000=x]
|> ---------
|> yields 0x1000000000000BD0 .
|>
|---End Quote---
|No, it's your hex input that's causing the error. Try
|echo %@eval[16**15=x]
|and you'll see that the output is correct.

Perhaps I missed it. Why is the hex input causing an error?

@EVAL accepts hex input. That number is only 61 bits. [And @CONVERT can
correctly convert it to decimal.]
--
- Vince

 

[dcantor]

I think I understand now.

I've done some more experimentation, and have discovered that when using precision specification =h or =x in @EVAL,

1. The fractional part of the resultant value is truncated. (This isn't exactly unexpected, but it's not documented.)

2. Numbers equal to or greater than 2**63 are reported as the number 0x7FFFFFFFFFFFFFFF (with or without the '0x') and numbers equal to or (algebraically) less than -(2**63) are reported as the number 0x8000000000000000 (with or without the '0x'). In other words, =h and =x always return a signed 64-bit integer.

Now that I understand it, I can deal with it.

I suggest and request that the help topic f_eval.htm document this in the section entitled "Hexadecimal display format specification".

Also, something should be said about the way that the double precision floating point calculations are done. I still say it seems weird to input only integers (even if they're specified in hexadecimal), do no real calculation on them, and output them, and get a number that's different from the input (e.g., %@eval[0x1000000000000000 =x]). That effect should be documented, preferably with the range of integers for which this does not happen.

The very good news is that, apparently, if integer numbers are input in decimal, and output in decimal, the range of values is well beyond +and- 2**64.

Perhaps there should be a warning that when computing with large magnitude integers, and hexadecimal output is needed, use @CONVERT[10,16,@EVAL[expression]] rather than @EVAL[expression =X].