[ferret_users] Wrong result in division

To: ferret_users@xxxxxxxx
Subject: [ferret_users] Wrong result in division
From: "'Fabio' via _OAR PMEL Ferret Users" <ferret_users@xxxxxxxx>
Date: Sat, 18 Jul 2020 23:19:59 +0200
In-reply-to: <0fde9cf2-7047-98ca-0780-1fc3a2fe9f75.ref@yahoo.it>
List-archive: <https://groups.google.com/a/noaa.gov/group/ferret_users/>
List-help: <https://support.google.com/a/noaa.gov/bin/topic.py?topic=25838>, <mailto:ferret_users+help@noaa.gov>
List-id: <ferret_users.noaa.gov>
List-post: <https://groups.google.com/a/noaa.gov/group/ferret_users/post>, <mailto:ferret_users@noaa.gov>
Mailing-list: list ferret_users@xxxxxxxx; contact ferret_users+owners@xxxxxxxx
References: <0fde9cf2-7047-98ca-0780-1fc3a2fe9f75.ref@yahoo.it>
Reply-to: Fabio <xxtraloud@xxxxxxxx>
Sender: owner-ferret_users@xxxxxxxx
User-agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0

Dear all,

I am sorry to send another question but I am puzzled as to why thissimple division is giving the wrong result. The same calculations in Rgive the correct result.

 
I have 2 variables calculated with the following
 
let myvar1 = var1[k=@sum] * 30 / .12
let nw = var2 * var3 * 30 / 1000
let myvar2 = nw[k=@sum]
 
and my result
let result = myvar1/myvar2
 
MYVAR1  MYVAR2  RESULT
---- J:30
25E   / 103:  0.0054  0.0049   1.09
27E   / 104:  0.0356  0.0344   1.03
---- J:31
25E   / 103:  0.0490  0.0390   5.54
27E   / 104:  0.0458  0.0338   1.36
---- J:35
25E   / 103:  0.1618  0.1680  47.26
27E   / 104:  0.0028  0.1906   0.20
 
But clearly RESULT is wrong, why is this happening?
 
Thanks in advance

Follow-Ups:
- Re: [ferret_users] Wrong result in division
  - From: Ryo Furue

Previous by thread: [ferret_users] Randomly distributed time series plots
Next by thread: Re: [ferret_users] Wrong result in division

[Thread Prev][Thread Next][Index]