Rounding question

[Chris Sharman]

brian_matthewsbrian matthews wrote:
> I always read that Cognos claimed the server architecture was to blame 
> for the inaccuracy of float size 4 items, but this shows that even 
> fieldvalue will hit the problem, and clearly the fieldvalue has'nt yet 
> touched the disk - I wonder what happened to the so-called six dp 
> accuracy for 4 byte floats and how can this be claimed when even the 
> window to the disk storage is flawed before anything is written - no 
> doubt this is an old ph7 specific bug :)

Floating point items of any size on any architecture are inaccurate by 
design - by contrast with integers (which are exact) and fixed point 
items (which can be exact).
Floating point is designed to give limited accuracy across a wide range, 
whereas integer is designed to be exact within a (relatively) small 
fixed range. Float size 4 gives approx 8 bits exponent and 23/24 bits 
mantissa (the fractional part) depending on architecture. 2**23 is 
approx 8*10**6, which is where the 6-7 digits accuracy comes from.
These accuracy issues are inherent in the design of all floating point 
architectures - there's no blame, just misunderstanding.

Cognos are only responsible for the (arguably poor) decision to force 
all arithmetic (even integer arithmetic) into floating point, making it 
susceptible to floating point rounding errors. With float*8 and 
appropriate rounding, no precision need be lost from an int size 4 - you 
only lose precision from int size 8 (and VMS timestamps). However, I 
suspect you need an explicit 'round' (or add 0.5) before assigning your 
float*8 arithmetic result back to an int, otherwise you'll get the 
default truncate behaviour.

0.0014999999 and 0.0015 are equivalent (to 7 digits accuracy) - but 
rounding to the nearest thousandth, when half of one-thousandth can't be 
exactly represented - is error-prone - anything that relies on exactness 
with floating point numbers is error-prone in this way - you can only 
have an approximation.

Chris


Any views expressed in this message are those of the sender and not necessarily those of CCA Group.  The unauthorized use, disclosure, copying or alteration of this message is forbidden.  The contents of
this message may be confidential and/or privileged, copyright CCA Group and are intended solely for the use of the individual or entity to whom they are addressed.  Whilst this message has been scanned, CCA Group cannot guarantee that it is virus free or compatible with your systems and accepts no responsibility for any loss or damage
arising from its use. The recipient is advised to run their own anti-virus software. If you receive this message in error please contact
postmaster@ccagroup.co.uk immediately, destroy any copies and delete it from your computer systems.