Strange Rounding

[Deskin, Bob]

Welcome to the wonderful world of floating point numbers. Floating point
represents decimal numbers in binary and that's why you see what you see.
The NCONVERT takes the character number and converts it to a floating point
number. Why? Because we do all of our arithmetic in floating point. So you
get a floating point number which is multiplied by 100 and then moved into
an integer. The floating point number is not going to be exactly what you
entered because it's a binary representation of a decimal number. Moving it
to an integer truncates any fractional portion. If you're lucky, the float
representation was slightly larger than your original number. In some cases,
it's slightly lower and it looks like your number is truncated.

The ROUND function was an attempt to resolve this issue, but it can be
confusing. The best solution I've found is always to add .5 and FLOOR as in

define N-AMT INT*8 = FLOOR((NCONVERT(T-AMT) * 100) + .5)

You take the number you want and add .5 to it. The FLOOR drops the result to
the largest integer not greater than the number. For example, you entered
518.30. If this is represented in float as 518.2989457629 (just as an
example), multiplying by 100 gives 51829.89457629. Adding .5 gives
51830.39457629. FLOORing that gives 51830, the largest integer not greater
than 51830.39457629.

Hope this helps.

Bob Deskin              
PowerHouse Web Product Manager, Application Development Tools, Cognos Inc.
bob.deskin@cognos.com (613) 738-1338 ext 7268 FAX: (613) 727-1178
3755 Riverside Drive P.O. Box 9707 Stn. T, Ottawa ON K1G 4K9 CANADA

-----Original Message-----
From: Terry Pickering [mailto:pickering@4j.lane.edu]
Sent: Wednesday, October 09, 2002 2:08 PM
To: powerh-l@lists.swau.edu
Subject: Strange Rounding


Here is a small example of code that duplicates a problem I've encountered. 
Basically, I'm taking a text field that contains a number with a decimal 
point (from an Excel PRN file), and converting to a standard integer field.

DEFINE T-AMT CHAR*8 = PARM PROMPT "ENTER AMOUNT "
DEFINE N-AMT INT*8 = NCONVERT(T-AMT) * 100

REPORT &
    T-AMT &
    N-AMT PIC "^^^,^^^.^^"


If you enter the number "518.30" for T-AMT, it reports 518.29 for N-AMT. At 
first thought I assumed it must be rounding issue with the NCONVERT 
function, but then I tried "18.30" and "1518.30" and both of them worked ok.

Another number I've encountered with this strange occurrence is "2351.44"

I've tried splitting the define of N-AMT into two separate defines. I've
tried

DEFINE X-AMT = NCONVERT(T-AMT)
DEFINE N-AMT INT*8 = X-AMT * 100

The workaround that seems to fix the problem is:

DEFINE N-AMT INT*8 = ROUND(NCONVERT(T-AMT * 100))

I think we have lots of code that uses the first method of conversion. Any 
ideas why it is doing what it does? Why does it work for "18.30" but not 
"518.30"?


____________________________
Terry Pickering


= = = = = = = = = = = = = = = = = = = = = = = = = = = =
Mailing list: powerh-l@lists.swau.edu
Subscribe: "subscribe" in message body to powerh-l-request@lists.swau.edu
Unsubscribe: "unsubscribe" in message body to
powerh-l-request@lists.swau.edu
http://lists.swau.edu/mailman/listinfo/powerh-l
This list is closed, thus to post to the list you must be a subscriber.

This message may contain privileged and/or confidential information.  If you
have received this e-mail in error or are not the intended recipient, you
may not use, copy, disseminate or distribute it; do not open any
attachments, delete it immediately from your system and notify the sender
promptly by e-mail that you have done so.  Thank you.