Rounding question

[Guy Werry]

Thanks Robert ... you are pretty close.  It's a data format issue, relating
to the format of FLOAT data on the HP.  Any language working with a float
would have the same problem.

In terms of using a different data type, I did some experimentation there
but with no success, because Powerhouse uses float for its arithmetic.  No
matter what data type I start with, I wind up with FLOAT when it counts, in
the actual calculation.

So, I used the "add 0.000001" before rounding trick.

Does anyone remember the days of HP's Transact language, where if you had
multiple calculations on one line it would use the LEAST precise number of
decimals for the internal calculations?  We once had a percentage
calculation that had a rounding error of around $500 because one of the data
items involved was an INTEGER, which meant that the whold calculation
rounded to ZERO decimals all the way through!  That was fun tracking down
....

Anyways, we've got the problem licked.  Thanks all!

Guy L. Werry
Senior Systems Analyst
Hudson Bay Mining & Smelting Co., Limited. 

-----Original Message-----
From: Robert Edis [mailto:robeconsult@sbcglobal.net]
Sent: Wednesday, May 04, 2005 11:12 AM
To: PowerHouse List
Subject: RE: Rounding question


G'day Guy

This may not be a PH issue but rather a database
issue.  The database may be storing your floating
point number of .0015 not as .001500000000000000
(whatever the number of positions is the default for
the data type) but as .00149999999999999991.  SQL will
return the correct value as .0015 but when the ROUND()
function is applied PH (or SQL) may think the value is
closer to .001 than .002.

I have seen this as a problem in SQL Server many
times.  Have you tried making the data type a DECIMAL
instead?  Have you tried converting the FLOAT to a
DECIMAL and then rounded?

Blue 


--- David Morrison - Corporate
<dmorrison@mcbrideelectric.com> wrote:
> Guy,
> 
> How about adding something like .0000000001 to
> inp_val, prior to rounding?
> 
> Thanks.
> 
> David Morrison
> McBride Electric
> 
> -----Original Message-----
> From: powerh-l-admin@lists.sowder.com
> [mailto:powerh-l-admin@lists.sowder.com]On Behalf Of
> Guy Werry
> Sent: Tuesday, May 03, 2005 8:47 AM
> To: PH Mailing List (E-mail)
> Subject: Rounding question
> 
> 
> Ph 7.33.D3, Ux 10.20
> 
> The rounding function is giving me fits. 
> Specifically, I have a value in a
> FLOAT data item and was rounding it like this:
> 	round(inp_val,3,near)
> 
> So I'm trying to round to 3 decimals.  If I enter
> 0.0015, it rounds DOWN to
> 0.001!
> 
> I have tried various combinations of the "multiply
> by 1000, round it, divide
> by 1000" rigamarole, with no success.
> 
> Is there a clean, neat solution to this?  Is there
> some way of telling the
> stupid thing to round UP on that 5?  Note that the
> "UP" and "DOWN" options
> don't give the results that I need.
> 
> [The frustration mirrored here is largely due to the
> fact that COBOL rounds
> wonderfully and predictably and I've NEVER found
> another language that could
> round its way out of a wet paper bag!]
> 
> Thanks,
> Guy L. Werry
> Senior Systems Analyst
> Hudson Bay Mining & Smelting Co., Limited. 
> 
> 
> 
> -- 
> = = = = = = = = = = = = = = = = = = = = = = = = = =
> = =
> Mailing list: powerh-l@lists.sowder.com
> Subscribe: "subscribe" in message body to
> powerh-l-request@lists.sowder.com
> Unsubscribe: "unsubscribe &lt;password&gt;" in
> message body to powerh-l-request@lists.sowder.com
> http://lists.sowder.com/mailman/listinfo/powerh-l
> This list is closed, thus to post to the list you
> must be a subscriber.
> 
> -- 
> = = = = = = = = = = = = = = = = = = = = = = = = = =
> = =
> Mailing list: powerh-l@lists.sowder.com
> Subscribe: "subscribe" in message body to
> powerh-l-request@lists.sowder.com
> Unsubscribe: "unsubscribe &lt;password&gt;" in
> message body to powerh-l-request@lists.sowder.com
> http://lists.sowder.com/mailman/listinfo/powerh-l
> This list is closed, thus to post to the list you
> must be a subscriber.
> 
-- 
= = = = = = = = = = = = = = = = = = = = = = = = = = = =
Mailing list: powerh-l@lists.sowder.com
Subscribe: "subscribe" in message body to powerh-l-request@lists.sowder.com
Unsubscribe: "unsubscribe &lt;password&gt;" in message body to
powerh-l-request@lists.sowder.com
http://lists.sowder.com/mailman/listinfo/powerh-l
This list is closed, thus to post to the list you must be a subscriber.