Too many decimal places in floats
F143 - Failure Feedback Forum
% view a:F def 2.85 | get
2.84999990463
Is there a way to preserve just the two simple decimal places on 2.85? I'm not familiar with issues of representing floating point values, so I don't know how difficult this is. I know Tcl can handle this the way I want:
% expr 2.85 2.85
But you are probably not able to use Tcl at that layer in your code?
Summary
This is a limitation of 32-bit floats. Use doubles.
Imported
This is tricky. Vlerq uses real floats at this level - not dual objects like Tcl (and also 4 bytes vs 24). You'll get a better result with doubles (view a:D def 2.85), because they are more accurate and will probably do just the right thing. It's the float -> double conversion which introduces these errors.
Using doubles works great, thanks.
