Spreadsheet cell value behaving oddly

Post here for help on using FreeCAD's graphical user interface (GUI).
Forum rules
and Helpful information
IMPORTANT: Please click here and read this first, before asking for help

Also, be nice to others! Read the FreeCAD code of conduct!
Post Reply
Jonathan8146
Posts: 7
Joined: Sat Apr 01, 2023 2:36 pm

Spreadsheet cell value behaving oddly

Post by Jonathan8146 »

Code: Select all

OS: Windows 10 Version 2009
Word size of FreeCAD: 64-bit
Version: 0.20.1.29410 (Git)
Build type: Release
Branch: releases/FreeCAD-0-20
Hash: f5d13554ecc7a456fb6e970568ae5c74ba727563
Python 3.8.10, Qt 5.15.2, Coin 4.0.1, Vtk 8.2.0, OCC 7.6.3
Locale: English/United Kingdom (en_GB)
Installed mods: 
  * fasteners 0.4.55
  * Help 1.0.3
Not sure if this is the correct place but couldn't find a better option or a similar issue.

I am parameterising my design and am starting in the spreadsheet.

One of my parameters has a value of 8.2 but the spreadsheet always seems to change this to 8.199999999999999. It doesn't seem to upset subsequent calculations based on the cell, but it is a bit disconcerting.

Am I being incredibly stupid, or is this some sort of "feature"?
User avatar
Roy_043
Veteran
Posts: 8450
Joined: Thu Dec 27, 2018 12:28 pm

Re: Spreadsheet cell value behaving oddly

Post by Roy_043 »

This is not unique to FreeCAD. What you see is a normal rounding error caused by floating point conversions. Not every decimal value can be represented accurately in the binary format that computers use internally. This may come as a total surprise, but as already mentioned, this is completely normal. If you don't want to see this then you should just reduce the number of decimals.

See: https://docs.python.org/3/tutorial/floatingpoint.html
Ulicad
Posts: 311
Joined: Tue Aug 16, 2022 10:35 am
Location: de_DE

Re: Spreadsheet cell value behaving oddly

Post by Ulicad »

Roy_043 wrote: Sat Apr 01, 2023 4:39 pm ... Not every decimal value can be represented accurately in the binary format that computers use internally. ...
This perrhaps is just curiosity - could you explain, why this value is not represented as 8.200000000000000 ?

Thanks a lot!
Regards, Ulicad / Uli
---
en: New FC-user, virgin to previous CAD-work, labouring on his new hobby Ulis Blog
de: Neuer FC-Nutzer, ohne vorhergehende CAD-Erfahrung, beschäftigt mit seinem neuen Hobby Ulis Blog
User avatar
Shalmeneser
Veteran
Posts: 9474
Joined: Wed Dec 23, 2020 12:04 am
Location: Fr

Re: Spreadsheet cell value behaving oddly

Post by Shalmeneser »

Code: Select all

d8.2 = b1000.001100110011001100110011001100110011001100110011001100110011001100110011001100110011...

Code: Select all

b1000.00110011001100110011 = d8.19999980926513671875
Jonathan8146
Posts: 7
Joined: Sat Apr 01, 2023 2:36 pm

Re: Spreadsheet cell value behaving oddly

Post by Jonathan8146 »

Thanks Roy.

I suppose I should have realised this, but I was playing around in Excel, notorious for its "quirky" behaviour, and it was fine so wasn't really expecting it.
User avatar
onekk
Veteran
Posts: 6144
Joined: Sat Jan 17, 2015 7:48 am
Contact:

Re: Spreadsheet cell value behaving oddly

Post by onekk »

Jonathan8146 wrote: Sat Apr 01, 2023 9:36 pm ...
It is simply IEEE floating point, some software do roundings "by design" but this way you introduce another error in the game.

It is a simple matter to know this behaviour. Internally most of FreeCAD is using for his calculations a precision on 1e-7 and internal units are mm so it is not too bad. And is using 64bit floats.

It is an engineering CAD kernel, build to make automotive and aerospace things not to design chips or high precision things. (but even when manufacturing chips are printed using photoreduction so masters are not printed at nm scale).

Kind Regards

Carlo D.
GitHub page: https://github.com/onekk/freecad-doc.
- In deep articles on FreeCAD.
- Learning how to model with scripting.
- Various other stuffs.

Blog: https://okkmkblog.wordpress.com/
Jonathan8146
Posts: 7
Joined: Sat Apr 01, 2023 2:36 pm

Re: Spreadsheet cell value behaving oddly

Post by Jonathan8146 »

A good job I am not creating anything requiring much precision then, just a storage box for a tap set.
GeneFC
Veteran
Posts: 5373
Joined: Sat Mar 19, 2016 3:36 pm
Location: Punta Gorda, FL

Re: Spreadsheet cell value behaving oddly

Post by GeneFC »

This topic has gone in a strange direction.

Two somewhat different items are being discussed.

Precision relates to the number of significant digits, regardless of numerical size (location of the decimal or the number of zeros).

Scale refers to the size in comparison to ordinary units such as meters or mm.

The precision of FreeCAD is limited in several ways to a usable 6 or 7 significant digits. These limits comes from various tolerance values incorporated in the determination of overlap, coincidence, etc. Some come from OCC, some come from the FreeCAD core, and others come from individual workbenches and external libraries.

Scale is usually not a limiter for any ordinary design until the sizes become sub-micron or multi-km. There have been reports of problems in both directions. I believe these are not the same sort of intrinsic limitations seen with precision, but rather choices made while creating various code elements. In that case it may be necessary to create a scaled design and later expand or contract externally.

Gene
openBrain
Veteran
Posts: 9034
Joined: Fri Nov 09, 2018 5:38 pm
Contact:

Re: Spreadsheet cell value behaving oddly

Post by openBrain »

Jonathan8146 wrote: Sat Apr 01, 2023 2:51 pm One of my parameters has a value of 8.2 but the spreadsheet always seems to change this to 8.199999999999999. It doesn't seem to upset subsequent calculations based on the cell, but it is a bit disconcerting.
While there are technical reasons for this, I can only confirm it's disconcerting and shouldn't be presented to user.
I found the issue and submitted a fixing PR : https://github.com/FreeCAD/FreeCAD/pull/9163
Many thanks for reporting.
Jonathan8146
Posts: 7
Joined: Sat Apr 01, 2023 2:36 pm

Re: Spreadsheet cell value behaving oddly

Post by Jonathan8146 »

openBrain wrote: Sun Apr 02, 2023 6:44 pm While there are technical reasons for this, I can only confirm it's disconcerting and shouldn't be presented to user.
I found the issue and submitted a fixing PR : https://github.com/FreeCAD/FreeCAD/pull/9163
Many thanks for reporting.
Thank you
Post Reply