Rounding question
brian_matthewsbrian matthews
brian_matthews_bmw@hotmail.com
Tue, 03 May 2005 20:56:52 +0000
many thanks for the update - I always assumed that fieldtext and fieldvalue
were internal quick buffers and independent of the server - but this is
based on ignorant assumption only - so I am always happy to be corrected :)
regards Bri,
>From: Guy Werry <guy.werry@hbms.ca>
>To: powerh-l@lists.sowder.com
>Subject: RE: Rounding question
>Date: Tue, 3 May 2005 15:37:06 -0500
>
>Brian:
>
>Can't agree with that statement. Floating-point inaccuracies are well
>known
>in the FORTRAN area ... I've done enough FORTRAN to know that you can't
>simply test for "I = 0" when floating point is involved.
>
>Any claim about "server architecture" would deal with the PROCESSOR, as
>opposed to disk format ... before the data ever touches the disk it has to
>be manipulated in the processor (main memory), which will determine the
>format of the data. It's more than just fieldvalue ... I had the same
>problem in Quiz, which is where I was doing my testing.
>
>Guy L. Werry
>Senior Systems Analyst
>Hudson Bay Mining & Smelting Co., Limited.
>
>-----Original Message-----
>From: brian_matthewsbrian matthews
>[mailto:brian_matthews_bmw@hotmail.com]
>Sent: Tuesday, May 03, 2005 3:25 PM
>To: guy.werry@hbms.ca; powerh-l@lists.sowder.com
>Subject: RE: Rounding question
>
>
>I always read that Cognos claimed the server architecture was to blame for
>the inaccuracy of float size 4 items, but this shows that even fieldvalue
>will hit the problem, and clearly the fieldvalue has'nt yet touched the
>disk
>
>- I wonder what happened to the so-called six dp accuracy for 4 byte floats
>and how can this be claimed when even the window to the disk storage is
>flawed before anything is written - no doubt this is an old ph7 specific
>bug
>
>:)
>
>regards Bri,
>
>
>
>
> >From: Guy Werry <guy.werry@hbms.ca>
> >To: "PH Mailing List (E-mail)" <powerh-l@lists.sowder.com>
> >Subject: RE: Rounding question
> >Date: Tue, 3 May 2005 14:13:20 -0500
> >
> >Thanks once again, Listers!
> >
> >Yes, this was the problem ... the floating point math winds up giving me
> >0.149999999 instead of 0.015.
> >
> >So, I now do my round as:
> > ROUND(FIELDVALUE + 0.000001,NUM-DECIMALS,NEAR)
> >and this works fine.
> >
> >Of course, the reason I've never had issues like this with COBOL is that
> >COBOL does its internal math in (I think) packed decimal, not float.
> >
> >Thanks again,
> >Guy L. Werry
> >Senior Systems Analyst
> >Hudson Bay Mining & Smelting Co., Limited.
> >
> >
> >-----Original Message-----
> >From: David Morrison - Corporate [mailto:dmorrison@mcbrideelectric.com]
> >Sent: Tuesday, May 03, 2005 10:56 AM
> >To: Guy Werry; PH Mailing List (E-mail)
> >Subject: RE: Rounding question
> >
> >
> >Guy,
> >
> >How about adding something like .0000000001 to inp_val, prior to
>rounding?
> >
> >Thanks.
> >
> >David Morrison
> >McBride Electric
> >
> >-----Original Message-----
> >From: powerh-l-admin@lists.sowder.com
> >[mailto:powerh-l-admin@lists.sowder.com]On Behalf Of Guy Werry
> >Sent: Tuesday, May 03, 2005 8:47 AM
> >To: PH Mailing List (E-mail)
> >Subject: Rounding question
> >
> >
> >Ph 7.33.D3, Ux 10.20
> >
> >The rounding function is giving me fits. Specifically, I have a value in
>a
> >FLOAT data item and was rounding it like this:
> > round(inp_val,3,near)
> >
> >So I'm trying to round to 3 decimals. If I enter 0.0015, it rounds DOWN
>to
> >0.001!
> >
> >I have tried various combinations of the "multiply by 1000, round it,
> >divide
> >by 1000" rigamarole, with no success.
> >
> >Is there a clean, neat solution to this? Is there some way of telling
>the
> >stupid thing to round UP on that 5? Note that the "UP" and "DOWN"
>options
> >don't give the results that I need.
> >
> >[The frustration mirrored here is largely due to the fact that COBOL
>rounds
> >wonderfully and predictably and I've NEVER found another language that
> >could
> >round its way out of a wet paper bag!]
> >
> >Thanks,
> >Guy L. Werry
> >Senior Systems Analyst
> >Hudson Bay Mining & Smelting Co., Limited.
> >
> >
> >
> >--
> >= = = = = = = = = = = = = = = = = = = = = = = = = = = =
> >Mailing list: powerh-l@lists.sowder.com
> >Subscribe: "subscribe" in message body to
>powerh-l-request@lists.sowder.com
> >Unsubscribe: "unsubscribe <password>" in message body to
> >powerh-l-request@lists.sowder.com
> >http://lists.sowder.com/mailman/listinfo/powerh-l
> >This list is closed, thus to post to the list you must be a subscriber.
> >--
> >= = = = = = = = = = = = = = = = = = = = = = = = = = = =
> >Mailing list: powerh-l@lists.sowder.com
> >Subscribe: "subscribe" in message body to
>powerh-l-request@lists.sowder.com
> >Unsubscribe: "unsubscribe <password>" in message body to
> >powerh-l-request@lists.sowder.com
> >http://lists.sowder.com/mailman/listinfo/powerh-l
> >This list is closed, thus to post to the list you must be a subscriber.
>
>_________________________________________________________________
>Be the first to hear what's new at MSN - sign up to our free newsletters!
>http://www.msn.co.uk/newsletters
>--
>= = = = = = = = = = = = = = = = = = = = = = = = = = = =
>Mailing list: powerh-l@lists.sowder.com
>Subscribe: "subscribe" in message body to powerh-l-request@lists.sowder.com
>Unsubscribe: "unsubscribe <password>" in message body to
>powerh-l-request@lists.sowder.com
>http://lists.sowder.com/mailman/listinfo/powerh-l
>This list is closed, thus to post to the list you must be a subscriber.
_________________________________________________________________
Want to block unwanted pop-ups? Download the free MSN Toolbar now!
http://toolbar.msn.co.uk/