Rounding question
Guy Werry
guy.werry@hbms.ca
Tue, 3 May 2005 15:37:06 -0500
Brian:
Can't agree with that statement. Floating-point inaccuracies are well known
in the FORTRAN area ... I've done enough FORTRAN to know that you can't
simply test for "I = 0" when floating point is involved.
Any claim about "server architecture" would deal with the PROCESSOR, as
opposed to disk format ... before the data ever touches the disk it has to
be manipulated in the processor (main memory), which will determine the
format of the data. It's more than just fieldvalue ... I had the same
problem in Quiz, which is where I was doing my testing.
Guy L. Werry
Senior Systems Analyst
Hudson Bay Mining & Smelting Co., Limited.
-----Original Message-----
From: brian_matthewsbrian matthews
[mailto:brian_matthews_bmw@hotmail.com]
Sent: Tuesday, May 03, 2005 3:25 PM
To: guy.werry@hbms.ca; powerh-l@lists.sowder.com
Subject: RE: Rounding question
I always read that Cognos claimed the server architecture was to blame for
the inaccuracy of float size 4 items, but this shows that even fieldvalue
will hit the problem, and clearly the fieldvalue has'nt yet touched the disk
- I wonder what happened to the so-called six dp accuracy for 4 byte floats
and how can this be claimed when even the window to the disk storage is
flawed before anything is written - no doubt this is an old ph7 specific bug
:)
regards Bri,
>From: Guy Werry <guy.werry@hbms.ca>
>To: "PH Mailing List (E-mail)" <powerh-l@lists.sowder.com>
>Subject: RE: Rounding question
>Date: Tue, 3 May 2005 14:13:20 -0500
>
>Thanks once again, Listers!
>
>Yes, this was the problem ... the floating point math winds up giving me
>0.149999999 instead of 0.015.
>
>So, I now do my round as:
> ROUND(FIELDVALUE + 0.000001,NUM-DECIMALS,NEAR)
>and this works fine.
>
>Of course, the reason I've never had issues like this with COBOL is that
>COBOL does its internal math in (I think) packed decimal, not float.
>
>Thanks again,
>Guy L. Werry
>Senior Systems Analyst
>Hudson Bay Mining & Smelting Co., Limited.
>
>
>-----Original Message-----
>From: David Morrison - Corporate [mailto:dmorrison@mcbrideelectric.com]
>Sent: Tuesday, May 03, 2005 10:56 AM
>To: Guy Werry; PH Mailing List (E-mail)
>Subject: RE: Rounding question
>
>
>Guy,
>
>How about adding something like .0000000001 to inp_val, prior to rounding?
>
>Thanks.
>
>David Morrison
>McBride Electric
>
>-----Original Message-----
>From: powerh-l-admin@lists.sowder.com
>[mailto:powerh-l-admin@lists.sowder.com]On Behalf Of Guy Werry
>Sent: Tuesday, May 03, 2005 8:47 AM
>To: PH Mailing List (E-mail)
>Subject: Rounding question
>
>
>Ph 7.33.D3, Ux 10.20
>
>The rounding function is giving me fits. Specifically, I have a value in a
>FLOAT data item and was rounding it like this:
> round(inp_val,3,near)
>
>So I'm trying to round to 3 decimals. If I enter 0.0015, it rounds DOWN to
>0.001!
>
>I have tried various combinations of the "multiply by 1000, round it,
>divide
>by 1000" rigamarole, with no success.
>
>Is there a clean, neat solution to this? Is there some way of telling the
>stupid thing to round UP on that 5? Note that the "UP" and "DOWN" options
>don't give the results that I need.
>
>[The frustration mirrored here is largely due to the fact that COBOL rounds
>wonderfully and predictably and I've NEVER found another language that
>could
>round its way out of a wet paper bag!]
>
>Thanks,
>Guy L. Werry
>Senior Systems Analyst
>Hudson Bay Mining & Smelting Co., Limited.
>
>
>
>--
>= = = = = = = = = = = = = = = = = = = = = = = = = = = =
>Mailing list: powerh-l@lists.sowder.com
>Subscribe: "subscribe" in message body to powerh-l-request@lists.sowder.com
>Unsubscribe: "unsubscribe <password>" in message body to
>powerh-l-request@lists.sowder.com
>http://lists.sowder.com/mailman/listinfo/powerh-l
>This list is closed, thus to post to the list you must be a subscriber.
>--
>= = = = = = = = = = = = = = = = = = = = = = = = = = = =
>Mailing list: powerh-l@lists.sowder.com
>Subscribe: "subscribe" in message body to powerh-l-request@lists.sowder.com
>Unsubscribe: "unsubscribe <password>" in message body to
>powerh-l-request@lists.sowder.com
>http://lists.sowder.com/mailman/listinfo/powerh-l
>This list is closed, thus to post to the list you must be a subscriber.
_________________________________________________________________
Be the first to hear what's new at MSN - sign up to our free newsletters!
http://www.msn.co.uk/newsletters