Rounding question
brian_matthewsbrian matthews
brian_matthews_bmw@hotmail.com
Wed, 04 May 2005 20:00:38 +0000
I would expect the round function to work as field displays work; namely, if
you increase the number of decimals for the field you will see more of the
true digits present, by the same token if you reduce the number of decimals
you will see fewer digits, and the last digit will be ‘rounded’ based on the
now invisible digits.
If a value of .0001499999… is held on disk, and the value is displayed in a
field of 5dps the value will appears as .00015 and not as .00014, the round
function should behave in the same way. In order for this to work, a fourth
parameter could be added to the round function in order to tell the round
function to pre-round the value at a certain point ( as must be the case
with field displays now ) this new value is then used as the input to the
round function.
I know it seems odd to round before rounding, but it’s better than having to
code as Guy is having to for every similar scenario.
regards Bri,
>From: Guy Werry <guy.werry@hbms.ca>
>To: PowerHouse List <powerh-l@lists.sowder.com>
>Subject: RE: Rounding question
>Date: Wed, 4 May 2005 11:26:30 -0500
>
>Thanks Robert ... you are pretty close. It's a data format issue, relating
>to the format of FLOAT data on the HP. Any language working with a float
>would have the same problem.
>
>In terms of using a different data type, I did some experimentation there
>but with no success, because Powerhouse uses float for its arithmetic. No
>matter what data type I start with, I wind up with FLOAT when it counts, in
>the actual calculation.
>
>So, I used the "add 0.000001" before rounding trick.
>
>Does anyone remember the days of HP's Transact language, where if you had
>multiple calculations on one line it would use the LEAST precise number of
>decimals for the internal calculations? We once had a percentage
>calculation that had a rounding error of around $500 because one of the
>data
>items involved was an INTEGER, which meant that the whold calculation
>rounded to ZERO decimals all the way through! That was fun tracking down
>....
>
>Anyways, we've got the problem licked. Thanks all!
>
>Guy L. Werry
>Senior Systems Analyst
>Hudson Bay Mining & Smelting Co., Limited.
>
>-----Original Message-----
>From: Robert Edis [mailto:robeconsult@sbcglobal.net]
>Sent: Wednesday, May 04, 2005 11:12 AM
>To: PowerHouse List
>Subject: RE: Rounding question
>
>
>G'day Guy
>
>This may not be a PH issue but rather a database
>issue. The database may be storing your floating
>point number of .0015 not as .001500000000000000
>(whatever the number of positions is the default for
>the data type) but as .00149999999999999991. SQL will
>return the correct value as .0015 but when the ROUND()
>function is applied PH (or SQL) may think the value is
>closer to .001 than .002.
>
>I have seen this as a problem in SQL Server many
>times. Have you tried making the data type a DECIMAL
>instead? Have you tried converting the FLOAT to a
>DECIMAL and then rounded?
>
>Blue
>
>
>--- David Morrison - Corporate
><dmorrison@mcbrideelectric.com> wrote:
> > Guy,
> >
> > How about adding something like .0000000001 to
> > inp_val, prior to rounding?
> >
> > Thanks.
> >
> > David Morrison
> > McBride Electric
> >
> > -----Original Message-----
> > From: powerh-l-admin@lists.sowder.com
> > [mailto:powerh-l-admin@lists.sowder.com]On Behalf Of
> > Guy Werry
> > Sent: Tuesday, May 03, 2005 8:47 AM
> > To: PH Mailing List (E-mail)
> > Subject: Rounding question
> >
> >
> > Ph 7.33.D3, Ux 10.20
> >
> > The rounding function is giving me fits.
> > Specifically, I have a value in a
> > FLOAT data item and was rounding it like this:
> > round(inp_val,3,near)
> >
> > So I'm trying to round to 3 decimals. If I enter
> > 0.0015, it rounds DOWN to
> > 0.001!
> >
> > I have tried various combinations of the "multiply
> > by 1000, round it, divide
> > by 1000" rigamarole, with no success.
> >
> > Is there a clean, neat solution to this? Is there
> > some way of telling the
> > stupid thing to round UP on that 5? Note that the
> > "UP" and "DOWN" options
> > don't give the results that I need.
> >
> > [The frustration mirrored here is largely due to the
> > fact that COBOL rounds
> > wonderfully and predictably and I've NEVER found
> > another language that could
> > round its way out of a wet paper bag!]
> >
> > Thanks,
> > Guy L. Werry
> > Senior Systems Analyst
> > Hudson Bay Mining & Smelting Co., Limited.
> >
> >
> >
> > --
> > = = = = = = = = = = = = = = = = = = = = = = = = = =
> > = =
> > Mailing list: powerh-l@lists.sowder.com
> > Subscribe: "subscribe" in message body to
> > powerh-l-request@lists.sowder.com
> > Unsubscribe: "unsubscribe <password>" in
> > message body to powerh-l-request@lists.sowder.com
> > http://lists.sowder.com/mailman/listinfo/powerh-l
> > This list is closed, thus to post to the list you
> > must be a subscriber.
> >
> > --
> > = = = = = = = = = = = = = = = = = = = = = = = = = =
> > = =
> > Mailing list: powerh-l@lists.sowder.com
> > Subscribe: "subscribe" in message body to
> > powerh-l-request@lists.sowder.com
> > Unsubscribe: "unsubscribe <password>" in
> > message body to powerh-l-request@lists.sowder.com
> > http://lists.sowder.com/mailman/listinfo/powerh-l
> > This list is closed, thus to post to the list you
> > must be a subscriber.
> >
>--
>= = = = = = = = = = = = = = = = = = = = = = = = = = = =
>Mailing list: powerh-l@lists.sowder.com
>Subscribe: "subscribe" in message body to powerh-l-request@lists.sowder.com
>Unsubscribe: "unsubscribe <password>" in message body to
>powerh-l-request@lists.sowder.com
>http://lists.sowder.com/mailman/listinfo/powerh-l
>This list is closed, thus to post to the list you must be a subscriber.
>--
>= = = = = = = = = = = = = = = = = = = = = = = = = = = =
>Mailing list: powerh-l@lists.sowder.com
>Subscribe: "subscribe" in message body to powerh-l-request@lists.sowder.com
>Unsubscribe: "unsubscribe <password>" in message body to
>powerh-l-request@lists.sowder.com
>http://lists.sowder.com/mailman/listinfo/powerh-l
>This list is closed, thus to post to the list you must be a subscriber.
_________________________________________________________________
It's fast, it's easy and it's free. Get MSN Messenger 7.0 today!
http://messenger.msn.co.uk