Rounding question
Robert Edis
robeconsult@sbcglobal.net
Wed, 4 May 2005 12:54:01 -0700 (PDT)
Guy
I had a problem like this back in about 1989/1990.
The uni I was working for took a large $ amount and
distributed it across disiplines based on a report
created in PH Quiz. The fractions got down to VERY
small numbers. The report was written in PH 5.? and
rounding was used.
When 6.00 of PH came out (VAX_VMS, Rdb) the report no
longer worked as expected and gave some really stupid
results. Cognos support could not provide an answer.
I finally figured out that the default data type used
by PH had changed between versions and to get around
this I converted all numbers to integers (IICRC) and
did some other tricks.
When version 6.10 was released and we upgraded the
problem went away and we could go back to the original
report logic.
Have a great day :)
Blue
--- Guy Werry <guy.werry@hbms.ca> wrote:
> Thanks Robert ... you are pretty close. It's a data
> format issue, relating
> to the format of FLOAT data on the HP. Any language
> working with a float
> would have the same problem.
>
> In terms of using a different data type, I did some
> experimentation there
> but with no success, because Powerhouse uses float
> for its arithmetic. No
> matter what data type I start with, I wind up with
> FLOAT when it counts, in
> the actual calculation.
>
> So, I used the "add 0.000001" before rounding trick.
>
> Does anyone remember the days of HP's Transact
> language, where if you had
> multiple calculations on one line it would use the
> LEAST precise number of
> decimals for the internal calculations? We once had
> a percentage
> calculation that had a rounding error of around $500
> because one of the data
> items involved was an INTEGER, which meant that the
> whold calculation
> rounded to ZERO decimals all the way through! That
> was fun tracking down
> ....
>
> Anyways, we've got the problem licked. Thanks all!
>
> Guy L. Werry
> Senior Systems Analyst
> Hudson Bay Mining & Smelting Co., Limited.
>
> -----Original Message-----
> From: Robert Edis [mailto:robeconsult@sbcglobal.net]
> Sent: Wednesday, May 04, 2005 11:12 AM
> To: PowerHouse List
> Subject: RE: Rounding question
>
>
> G'day Guy
>
> This may not be a PH issue but rather a database
> issue. The database may be storing your floating
> point number of .0015 not as .001500000000000000
> (whatever the number of positions is the default for
> the data type) but as .00149999999999999991. SQL
> will
> return the correct value as .0015 but when the
> ROUND()
> function is applied PH (or SQL) may think the value
> is
> closer to .001 than .002.
>
> I have seen this as a problem in SQL Server many
> times. Have you tried making the data type a
> DECIMAL
> instead? Have you tried converting the FLOAT to a
> DECIMAL and then rounded?
>
> Blue
>
>
> --- David Morrison - Corporate
> <dmorrison@mcbrideelectric.com> wrote:
> > Guy,
> >
> > How about adding something like .0000000001 to
> > inp_val, prior to rounding?
> >
> > Thanks.
> >
> > David Morrison
> > McBride Electric
> >
> > -----Original Message-----
> > From: powerh-l-admin@lists.sowder.com
> > [mailto:powerh-l-admin@lists.sowder.com]On Behalf
> Of
> > Guy Werry
> > Sent: Tuesday, May 03, 2005 8:47 AM
> > To: PH Mailing List (E-mail)
> > Subject: Rounding question
> >
> >
> > Ph 7.33.D3, Ux 10.20
> >
> > The rounding function is giving me fits.
> > Specifically, I have a value in a
> > FLOAT data item and was rounding it like this:
> > round(inp_val,3,near)
> >
> > So I'm trying to round to 3 decimals. If I enter
> > 0.0015, it rounds DOWN to
> > 0.001!
> >
> > I have tried various combinations of the "multiply
> > by 1000, round it, divide
> > by 1000" rigamarole, with no success.
> >
> > Is there a clean, neat solution to this? Is there
> > some way of telling the
> > stupid thing to round UP on that 5? Note that the
> > "UP" and "DOWN" options
> > don't give the results that I need.
> >
> > [The frustration mirrored here is largely due to
> the
> > fact that COBOL rounds
> > wonderfully and predictably and I've NEVER found
> > another language that could
> > round its way out of a wet paper bag!]
> >
> > Thanks,
> > Guy L. Werry
> > Senior Systems Analyst
> > Hudson Bay Mining & Smelting Co., Limited.
> >
> >
> >
> > --
> > = = = = = = = = = = = = = = = = = = = = = = = = =
> =
> > = =
> > Mailing list: powerh-l@lists.sowder.com
> > Subscribe: "subscribe" in message body to
> > powerh-l-request@lists.sowder.com
> > Unsubscribe: "unsubscribe <password>" in
> > message body to powerh-l-request@lists.sowder.com
> > http://lists.sowder.com/mailman/listinfo/powerh-l
> > This list is closed, thus to post to the list you
> > must be a subscriber.
> >
> > --
> > = = = = = = = = = = = = = = = = = = = = = = = = =
> =
> > = =
> > Mailing list: powerh-l@lists.sowder.com
> > Subscribe: "subscribe" in message body to
> > powerh-l-request@lists.sowder.com
> > Unsubscribe: "unsubscribe <password>" in
> > message body to powerh-l-request@lists.sowder.com
> > http://lists.sowder.com/mailman/listinfo/powerh-l
> > This list is closed, thus to post to the list you
> > must be a subscriber.
> >
> --
> = = = = = = = = = = = = = = = = = = = = = = = = = =
> = =
> Mailing list: powerh-l@lists.sowder.com
> Subscribe: "subscribe" in message body to
> powerh-l-request@lists.sowder.com
> Unsubscribe: "unsubscribe <password>" in
> message body to
> powerh-l-request@lists.sowder.com
> http://lists.sowder.com/mailman/listinfo/powerh-l
> This list is closed, thus to post to the list you
> must be a subscriber.
> --
> = = = = = = = = = = = = = = = = = = = = = = = = = =
> = =
> Mailing list: powerh-l@lists.sowder.com
> Subscribe: "subscribe" in message body to
> powerh-l-request@lists.sowder.com
> Unsubscribe: "unsubscribe <password>" in
> message body to powerh-l-request@lists.sowder.com
> http://lists.sowder.com/mailman/listinfo/powerh-l
> This list is closed, thus to post to the list you
> must be a subscriber.
>