Floating Point Precision Problems

A question on Quora :

prompted me to look at how these numbers are handled in Excel, in VBA called from Excel, and in Python called from Excel via pyxll. The results are shown in the screenshot below:

Column A shows the operation performed on each line, with results from on-sheet calculations, VBA user defined functions (UDFs), Python UDFs, and Python UDFs using Numpy float64 data type, in the following columns.

For the operations using (0.1+0.2)+0.3 and 0.1+(0.2+0.3):

  • Both Excel and VBA show the two expressions as being equal to 0.6, with the difference exactly 0 and the two expressions as equal.
  • Both Python and Numpy, when called from Excel, show the two expressions as exactly equal to 0.6, but a small difference between them, and the expressions as unequal.
  • When the expressions are entered directly in Python, (0.1+0.2)+0.3 returns 0.600000000000001, with the other results as when called from Excel (see screen-shot below).

For the operations involving a division by zero:

  • The Excel on-sheet calculation returns #DIV/0! for all three cases.
  • The VBA code returns #VALUE! for all three. Note that it is possible to check for zero divisions, and return #DIV/0! with additional code.
  • Python using Python float values returns a zero division error message for all three cases.
  • Python using Numpy float values returns 1.7977E+308 for the first two cases and #NUM! for the third when called from Excel.
  • Entering the divisions by zero directly in Python returns the same error message as when called from Excel, but when using Numpy Float64 values, it returns inf, -inf, and nan, respectively, together with a warning (see screen-shot below).

Finally, comparing or subtracting two large numbers, with more than 15 significant figures, creates an inconsistency in Excel:

  • The numbers actually entered (formatted as text) are shown in column A, but when these numbers are entered as values the final 9 of the smaller one is truncated to 0, and the displayed number is 9999999999999990. Nonetheless, subtracting the two numbers returns zero, but checking for equality returns FALSE.
  • VBA, Python and Numpy all return the same results with these numbers. When the numbers are passed from Excel the truncated value of the smaller number is preserved, and subtraction returns 10.0 (rather than the exact result of 1.0), and checking for equality returns FALSE. When the numbers are entered directly in VBA or Python code the smaller number is rounded up to 1.0E16, so subtraction returns 0 and the equality check returns TRUE.
  • When entered in the VBA editor, the editor updates the display automatically changes the rounded up value to display as 1E16 (see screen-shots below).

In summary:

  • Excel does not fully comply with the IEEE 754 format, since it displays results close to zero as zero, and it has no equivalent of inf, -inf, or nan.
  • The Excel rounding or truncating of numbers is preserved when the values are passed to VBA or external code.
  • Checking for equality in Excel, the results are not always consistent with the displayed values.
  • When checking for equality VBA returned the same results as Excel (at least for the examples used here), even though it did not always return the same subtraction result.
  • The VBA equality and subtraction results were self-consistent (for the values checked).
  • When checking if two numbers are equal, either in on-sheet calculations or VBA or external code, always either used rounded values, or check that the relative absolute difference is below a specified tolerance.

For more examples, details, and discussion see: When does 35 not equal 35?

This entry was posted in Excel, Link to Python, Maths, Newton, NumPy and SciPy, PyXLL, UDFs, VBA and tagged , , , , , , , . Bookmark the permalink.

1 Response to Floating Point Precision Problems

  1. Bill Harvey says:

    A good friend, used to computing in structures, once reminded me thar 1+1=3 for any sufficiently large value of 1.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.