Strange Rounding

[Witkopp, Richard]

>From my experience, floating point (which is supposedly
used for phouse numeric operations) roundoff error is
not predictable.

> -----Original Message-----
> From: Terry Pickering [mailto:pickering@4j.lane.edu]
> Sent: Wednesday, October 09, 2002 11:08 AM
> To: powerh-l@lists.swau.edu
> Subject: Strange Rounding
> 
> 
> Here is a small example of code that duplicates a problem 
> I've encountered. 
> Basically, I'm taking a text field that contains a number 
> with a decimal 
> point (from an Excel PRN file), and converting to a standard 
> integer field.
> 
> DEFINE T-AMT CHAR*8 = PARM PROMPT "ENTER AMOUNT "
> DEFINE N-AMT INT*8 = NCONVERT(T-AMT) * 100
> 
> REPORT &
>     T-AMT &
>     N-AMT PIC "^^^,^^^.^^"
> 
> 
> If you enter the number "518.30" for T-AMT, it reports 518.29 
> for N-AMT. At 
> first thought I assumed it must be rounding issue with the NCONVERT 
> function, but then I tried "18.30" and "1518.30" and both of 
> them worked ok.
> 
> Another number I've encountered with this strange occurrence 
> is "2351.44"
> 
> I've tried splitting the define of N-AMT into two separate 
> defines. I've tried
> 
> DEFINE X-AMT = NCONVERT(T-AMT)
> DEFINE N-AMT INT*8 = X-AMT * 100
> 
> The workaround that seems to fix the problem is:
> 
> DEFINE N-AMT INT*8 = ROUND(NCONVERT(T-AMT * 100))
> 
> I think we have lots of code that uses the first method of 
> conversion. Any 
> ideas why it is doing what it does? Why does it work for 
> "18.30" but not 
> "518.30"?
> 
> 
> ____________________________
> Terry Pickering
> 
> 
> = = = = = = = = = = = = = = = = = = = = = = = = = = = =
> Mailing list: powerh-l@lists.swau.edu
> Subscribe: "subscribe" in message body to 
> powerh-l-request@lists.swau.edu
> Unsubscribe: "unsubscribe" in message body to 
> powerh-l-request@lists.swau.edu
> http://lists.swau.edu/mailman/listinfo/powerh-l
> This list is closed, thus to post to the list you must be a 
> subscriber.
>