1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Arithmetic errors in large numbers

Discussion in 'Support' started by dcantor, Jun 11, 2009.

  1. dcantor

    Joined:
    May 29, 2008
    Messages:
    507
    Likes Received:
    3
    TCC 10.00.68 Windows XP [Version 5.1.2600]

    This FOR command is on one physical line.

    for /l %%numzeroes in (0,1,15) do echo %@format[2,%numzeroes] %@format[19,0x1%@repeat[0,%numzeroes]] %@eval[0x1%@re
    peat[0,%numzeroes] =x]
    0 0x1 0x1
    1 0x10 0x10
    2 0x100 0x100
    3 0x1000 0x1000
    4 0x10000 0x10000
    5 0x100000 0x100000
    6 0x1000000 0x1000000
    7 0x10000000 0x10000000
    8 0x100000000 0x100000000
    9 0x1000000000 0x1000000000
    10 0x10000000000 0x10000000000
    11 0x100000000000 0x100000000000
    12 0x1000000000000 0x1000000000000
    13 0x10000000000000 0x10000000000004
    14 0x100000000000000 0xFFFFFFFFFFFFDC
    15 0x1000000000000000 0x1000000000000BD0


    Note lines 13, 14, and 15.

    What's going on here?
     
  2. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,854
    Likes Received:
    83
    dcantor wrote:

    I have no idea what you're trying to do here -- can you reduce this to a
    single line with an explanation of the intent?

    (Note that if you're using hex, you're limited to 64-bit integers.)

    Rex Conn
    JP Software
     
  3. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,938
    Likes Received:
    30
    On Thu, 18 Jun 2009 20:15:28 -0500, rconn <> wrote:

    |I have no idea what you're trying to do here -- can you reduce this to a
    |single line with an explanation of the intent?
    |
    |(Note that if you're using hex, you're limited to 64-bit integers.)

    Here's one of the errors:

    v:\> echo %@eval[0x100000000000000=x]
    0xFFFFFFFFFFFFDC

    Another:

    v:\> echo %@eval[0x1000000000000000=x]
    0x1000000000000BD0

    They're both less than 0x7FFFFFFFFFFFFFFF which is INT64_MAX.

    Can't your "unlimited" arithmetic lib convert to/from hex?
    --
    - Vince
     
  4. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,854
    Likes Received:
    83
    vefatica wrote:


    No. All hex numbers are limited to int 64's.

    (Which isn't much of a limitation unless you're counting electrons.)

    Rex Conn
    JP Software
     
  5. dcantor

    Joined:
    May 29, 2008
    Messages:
    507
    Likes Received:
    3

    I was building a 64-bit value one byte at a time in big endian order by extracting a byte from a file which had the values stored in ASCII representation of hexadecimal, multiplying the previous value by 256 and adding the value of the byte just acquired.

    The file had values like this:

    FE DC BA 98 76 54 32 10
    representating the value 0xFEDCBA9876543210

    The code looks something like this:

    set nextbyte=%@trim[%@word[%bytenumber,%currentline]]
    ^ This caused nextbyte to have a 2-hex-digit value
    set value=%@eval[0x%value%%nextbyte%]
    :: and loop

    There are many ways to do this loop, but the point is that I discovered that for a sufficiently large value, the computation fails.

    I reduced that to the simple case that
    %@eval[0x1000000000000000=x] doesn't equal 0x1000000000000000

    Shouldn't it? It's fewer than 64 bits.

    Dave C.
     
  6. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,854
    Likes Received:
    83
    dcantor wrote:


    When you input a hex number, it's converted to a double before being
    passed to the large math library. So you're not going to get the full
    64-bit input. (This does not affect hex output.)

    Rex Conn
    JP Software
     
  7. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    rconn wrote:
    | When you input a hex number, it's converted to a double before being
    | passed to the large math library. So you're not going to get the
    | full 64-bit input. (This does not affect hex output.)

    Does this not negate the benefits of the 64-bit integers? For V11 I'd like
    to see direct converisions between external (characterstring)
    representations and large math library representations, e.g., using enhanced
    forms of @convert.
    --
    Steve
     
  8. dcantor

    Joined:
    May 29, 2008
    Messages:
    507
    Likes Received:
    3
    By "double", I assume you mean double-precision floating point, right?

    Rex, I appreciate the acknowledgment that this is indeed a restriction that is not likely to be resolved. It would be helpful if there were a statement somewhere in the help file indicating just what range of 64-bit integers can be relied on to compute correctly. It's apparent that I can no longer build 64-bit integers from 64-bit hex values input from files. (I'm talking ASCII or Unicode, binary can be converted with @BALLOC, @BREAD, and @BPEEK.)

    You say that it does not affect hex output, but it does:

    Code:
    echo %@eval[0x1000000000000000=x]
    yields 0x1000000000000BD0 .
     
  9. dcantor

    Joined:
    May 29, 2008
    Messages:
    507
    Likes Received:
    3
    My solution to my problem is

    SET BH=%@BALLOC[8]
    SET X=%@BPOKE[%BH,7,1,0xFE]
    SET X=%@BPOKE[%BH,6,1,0xDC]
    SET X=%@BPOKE[%BH,5,1,0xBA]
    SET X=%@BPOKE[%BH,4,1,0x98]
    SET X=%@BPOKE[%BH,3,1,0x76]
    SET X=%@BPOKE[%BH,2,1,0x54]
    SET X=%@BPOKE[%BH,1,1,0x32]
    SET X=%@BPOKE[%BH,0,1,0x10]
    ECHO %@CONVERT[10,16,%@BPEEK[%BH,0,8]]
    fedcba9876543210

    That's correct. That's just what I need.
     
  10. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,854
    Likes Received:
    83
    dcantor wrote:

    Right -- that's because of your hex input, not the output.

    Rex Conn
    JP Software
     
  11. Howard Goldstein

    Joined:
    Jun 1, 2008
    Messages:
    111
    Likes Received:
    1
    Dave Cantor wrote:


    No, it's your hex input that's causing the error. Try
    echo %@eval[16**15=x]
    and you'll see that the output is correct.

    --
    Howard
     
  12. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,938
    Likes Received:
    30
    On Sat, 20 Jun 2009 20:12:42 -0500, Howard Goldstein <>
    wrote:

    |Dave Cantor wrote:
    |
    |---Quote---
    |> You say that it does not affect hex output, but it does:
    |>
    |>
    |> Code:
    |> ---------
    |> echo %@eval[0x1000000000000000=x]
    |> ---------
    |> yields 0x1000000000000BD0 .
    |>
    |---End Quote---
    |No, it's your hex input that's causing the error. Try
    |echo %@eval[16**15=x]
    |and you'll see that the output is correct.

    Perhaps I missed it. Why is the hex input causing an error?

    @EVAL accepts hex input. That number is only 61 bits. [And @CONVERT can
    correctly convert it to decimal.]
    --
    - Vince
     
  13. dcantor

    Joined:
    May 29, 2008
    Messages:
    507
    Likes Received:
    3
    I think I understand now.

    I've done some more experimentation, and have discovered that when using precision specification =h or =x in @EVAL,

    1. The fractional part of the resultant value is truncated. (This isn't exactly unexpected, but it's not documented.)

    2. Numbers equal to or greater than 2**63 are reported as the number 0x7FFFFFFFFFFFFFFF (with or without the '0x') and numbers equal to or (algebraically) less than -(2**63) are reported as the number 0x8000000000000000 (with or without the '0x'). In other words, =h and =x always return a signed 64-bit integer.

    Now that I understand it, I can deal with it.

    I suggest and request that the help topic f_eval.htm document this in the section entitled "Hexadecimal display format specification".

    Also, something should be said about the way that the double precision floating point calculations are done. I still say it seems weird to input only integers (even if they're specified in hexadecimal), do no real calculation on them, and output them, and get a number that's different from the input (e.g., %@eval[0x1000000000000000 =x]). That effect should be documented, preferably with the range of integers for which this does not happen.

    The very good news is that, apparently, if integer numbers are input in decimal, and output in decimal, the range of values is well beyond +and- 2**64.

    Perhaps there should be a warning that when computing with large magnitude integers, and hexadecimal output is needed, use @CONVERT[10,16,@EVAL[expression]] rather than @EVAL[expression =X].
     

Share This Page