Odd Behavior With Decimals

[Charles Hoens]

I was going along so well and just ran into a problem I just cannot solve. I have a list of a 1000 invoices or so. I performed a tablesum on the amount field and I got an amount with 11 decimal points. Figuring that there must be a bad number somewhere I added just 100 records at a time until I narrowed this down to the offending record. I put the amount back in and saved, but no luck, it still comes out strange. So I deleted the amount for that record and changed it to $100. and all worked fine. So, I went back in and put the correct amount back in $22.62 and the eleven decimals came back again!

Then I created a new field and using the update function copied each of the numbers in the amount field to another field amount2, but this time I used the round() function for 2 decimal points. Didn't seem to matter, the issue is still there.

Does anyone have ANY idea about this????

Charlie

[Stan Mathews]

Re: Odd Behavior With Decimals

It's just so much easier to work through these things with a sample table.

[G Gabriel]

Re: Odd Behavior With Decimals

Is the problem that:
1-You got the wrong total with 11 decimals? or
2-You got the correct total but wondering why 11 decimals?
And where did you perform that? On a form? Report? Browse?
Have you tried to enter 22.61 instead of 22.62 to see what happens?
My gut feeling, this has to do with a corrupted form/record..don't think it has to do with the calculation itself unless you hit the jackpot by accident.
Delete that record altogether, pack the table, update the indexes, compact your db then re-enter a new record with that value and see what happens.

[Charles Hoens]

Re: Odd Behavior With Decimals

#2 I did the tablesum() in a script file. I did compact the whole app, but I'll try the other things and see what happens. Very odd behavior though!

Charlie

[Ray in Capetown]

Re: Odd Behavior With Decimals

Its not atypical behaviour of the 8088 series processor. If you tablesum with a rounded result or format mask you wouldn't see it.

[Charles Hoens]

Re: Odd Behavior With Decimals

It's kind of disconcerting though!






Yes, I do bowl overhand!

[Ray in Capetown]

Re: Odd Behavior With Decimals

Shouldnt be - its standard floating point arithmetic, as long as you've been programming its been happening, you just view the numbers formatted at the level that Alpha functions handle it. It does a pretty good job doing so. You do need to know it exists so you dont get surprised.
It can handle up to 18 decimals. I just entered random numbers in the IW, try this
v=654.22*8765
?v
= 5734238.3
?str(v,30,18)
= "5734238.299999999800000" - thats what the value really contains

[Charles Hoens]

Re: Odd Behavior With Decimals

Hmmm. Very surprised. I remember I came across this years ago and I guess I forgot about it. I would have thought that the tablesum function would have eliminated this somehow?

Oh well!

[Ray in Capetown]

Re: Odd Behavior With Decimals

maybe someone at AS forgot to attend to that one

[Tom Cone Jr]

Re: Odd Behavior With Decimals

Charles said...
I would have thought that the tablesum function would have eliminated this somehow?

They can't read minds any better now than when you first came across this years ago :) The level of precision in a computed result varies from app to app and from person to person. Why would you think they'd automatically round the value at some smaller arbitrary number of places for all users?

[G Gabriel]

Re: Odd Behavior With Decimals

Before calling it an FPE and possibly mask a more sinister issue, these questions remain to be answered:
1-Decimals aside, are you getting the correct total?
2-Have you attempted 22.61 instead of 22.62?

[G Gabriel]

Re: Odd Behavior With Decimals

Shouldnt be - its standard floating point arithmetic, as long as you've been programming its been happening, you just view the numbers formatted at the level that Alpha functions handle it. It does a pretty good job doing so. You do need to know it exists so you dont get surprised.
It can handle up to 18 decimals. I just entered random numbers in the IW, try this
v=654.22*8765
?v
= 5734238.3
?str(v,30,18)
= "5734238.299999999800000" - thats what the value really contains

Ray:
The decimals were rounded to 3. That's not an FPE.

Here is an illustration of FPE:

dim a as n=456.123
?mod(a,1)
= 0.123

[COLOR="#0000FF"]Well, anyone would agree, that is the correct answer[/COLOR]

But then try this:

dim a as n=123.456
?mod(a,1)
= 0.456000000000003

[COLOR="#0000FF"]There is your FPE. Do you see it[/COLOR]?

[Ray in Capetown]

Re: Odd Behavior With Decimals

Yes G I kind of see, that is very specific. I'm just lumping rounding errors generally into one category not being too careful about the terminology.
Generally whatever the cause - there was an issue with the Pentium5 FPU too, I remember.

[Tom Cone Jr]

Re: Odd Behavior With Decimals

Charles,

perhaps you'll find this article helpful. For me, though, it made my head start hurting!

[Charles Hoens]

Re: Odd Behavior With Decimals

Really interesting article Tom. My head hurt too after reading it, but it seems to makes sense and to at least give us some idea of what is going on. My concern though is, if we can't get an exact answer to adding 100 or so values, then how are more complicated scientific calculations made possible, or do I have to get out my Cray?