QTP - Rounding

[Chris Sharman]

> I am having a rounding problem with qtp. When I convert using 
> NCONVERT from a flat text file and write it to a subfile my 
> number is OK (9233.61), However when I update/add the value into 
> the image database I get 9233.610000000001. 
> 
> Any ideas?

16 digits is at or beyond the precision limit for 64 bit floats
(D float gives 16, G float & T float give only 15).
If you display too many digits, you'll always see "noise" on the end
of a floating point number.
If you really need more precision, use 128 bit floats (33 digits), if
your architecture supports them.

Chris


-----------------------------------------------------------------------

Any views expressed in this message are those of the sender and not
necessarily those of CCA Group.  The unauthorized use, disclosure,
copying or alteration of this message is forbidden.  The contents of
this message may be confidential and/or privileged, copyright CCA Group
and are intended solely for the use of the individual or entity to whom
they are addressed.  Whilst this message has been scanned, CCA Group
cannot guarantee that it is virus free or compatible with your systems
and accepts no responsibility for any loss or damage arising from its
use. The recipient is advised to run their own anti-virus software. If
you receive this message in error please contact
postmaster@ccagroup.co.uk immediately, destroy any copies and delete it
from your computer systems.