Strange Rounding

Terry Pickering pickering@4j.lane.edu
Wed, 09 Oct 2002 11:08:05 -0700


Here is a small example of code that duplicates a problem I've encountered. 
Basically, I'm taking a text field that contains a number with a decimal 
point (from an Excel PRN file), and converting to a standard integer field.

DEFINE T-AMT CHAR*8 = PARM PROMPT "ENTER AMOUNT "
DEFINE N-AMT INT*8 = NCONVERT(T-AMT) * 100

REPORT &
    T-AMT &
    N-AMT PIC "^^^,^^^.^^"


If you enter the number "518.30" for T-AMT, it reports 518.29 for N-AMT. At 
first thought I assumed it must be rounding issue with the NCONVERT 
function, but then I tried "18.30" and "1518.30" and both of them worked ok.

Another number I've encountered with this strange occurrence is "2351.44"

I've tried splitting the define of N-AMT into two separate defines. I've tried

DEFINE X-AMT = NCONVERT(T-AMT)
DEFINE N-AMT INT*8 = X-AMT * 100

The workaround that seems to fix the problem is:

DEFINE N-AMT INT*8 = ROUND(NCONVERT(T-AMT * 100))

I think we have lots of code that uses the first method of conversion. Any 
ideas why it is doing what it does? Why does it work for "18.30" but not 
"518.30"?


____________________________
Terry Pickering