[Thread Prev][Thread Next][Index]

[ferret_users] memory problems in ferret 7.20 (pyferret)

Hi,
 
I find a strange behaviour with pyferret 7.20
 
<phy-8:mschmidt>ferret
NOAA/PMEL TMAP
PyFerret v7.2 (optimized)
Linux 4.4.85-22-default - 09/19/17
19-Sep-17 23:27
the binary is build on openSuse 42.3. netcdf/hdf/szip are build separately, all other ingredients come from the system. The compiler is gcc-5, python is 2.7.13.
The following example illustrates the problem. Averaging and integrating a bigger dataset. Data are model results kept on a public THREDDS, so it should be reproducable. I leave things as they are without streamlining. Don't ask for the meaning ...
use http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/genus_run_88_5day_ocean.nc 
set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-23.5:-22.5
 
let n2_prod_anamm = P_DET_ANAM_N2 * 13.25
let n2_prod_denit = P_DET_DENIT_NH4 * 5.3/2
let n2_prod_h2s = P_H2S_OXNO3_SUL * 0.2
let n2_prod_sul = P_SUL_OXNO3_SO4*3/5
let n2_prod = (n2_prod_anamm + n2_prod_denit + n2_prod_h2s + n2_prod_sul)
 
list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
 
yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL) 
DATA SET : MOM4 Baltic Sea 3 n.m.
FILENAME : genus_run_88_5day_ocean.nc
FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/ 
LONGITUDE: 13.5E to 14.7E (XY integ.)
LATITUDE : 23.5S to 22.5S (XY integ.)
DEPTH (m): -0.0009867 to 5700 (ZT summed)
TIME : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN 
1.263E+09
 
Now increase the region, the result should be growing significantly
 
set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-28:-22.5 ! Note y!
 
list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
 
yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL) 
DATA SET : MOM4 Baltic Sea 3 n.m.
FILENAME : genus_run_88_5day_ocean.nc
FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/ 
LONGITUDE: 13.5E to 14.7E (XY integ.)
LATITUDE : 28S to 22.5S (XY integ.)
DEPTH (m): -0.0009867 to 5700 (ZT summed)
TIME : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN 
8.011E+08
This is strange, since the quantity is positive everywhere. Check with stats (not shown). 
 
Now play with the memory size:
 
set memory/size=30 ---> 3.244E+08
 
set memory/size=50 ---> 3.244E+08 ! Is this stable? No!
 
can memory
 
set memory/size=50 ---> 8.262E+08
 
can memory
 
set memory/size=300 ---> 4.973E+09
 
I think, this is enough to show, that something does not work correctly.
The classical ferret shows similar results. Aggregating withouth THREDDS gives 4.973E+09 for the larger region without increasing the memory size! 
 
Now a previous ferret version:
 
ferret
NOAA/PMEL TMAP
PyFERRET v7 (opt)
Linux 2.6.32-642.1.1.el6.x86_64 - 07/01/16
20-Sep-17 00:02
 
CAN MODE LOGO
SET MEM /SIZE=25.6
Cached data cleared from memory
SET MODE VERIFY
yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL) 
DATA SET : MOM4 Baltic Sea 3 n.m.
FILENAME : genus_run_88_5day_ocean.nc
FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/ 
LONGITUDE: 13.5E to 14.7E (XY integ.)
LATITUDE : 23.5S to 22.5S (XY integ.)
DEPTH (m): -0.0009867 to 5700 (ZT summed)
TIME : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN 
1.232E+09
 
Similar, but not the same as before! Make the region bigger now.
 
set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-28:-22.5
 
list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
 
yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL) 
DATA SET : MOM4 Baltic Sea 3 n.m.
FILENAME : genus_run_88_5day_ocean.nc
FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/ 
LONGITUDE: 13.5E to 14.7E (XY integ.)
LATITUDE : 28S to 22.5S (XY integ.)
DEPTH (m): -0.0009867 to 5700 (ZT summed)
TIME : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN 
4.760E+09
 
Not in spectrum of the previous results!
 
Now with another elder ferret:
 
ferret
NOAA/PMEL TMAP
FERRET v6.96
Linux 2.6.18-406.el5 64-bit - 12/02/15
20-Sep-17 00:09
 
CAN MODE LOGO
SET MEM /SIZE=25.6
Cached data cleared from memory
SET MODE VERIFY
use "http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/genus_run_88_5day_ocean.nc"; 
....
 
list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL) 
DATA SET : MOM4 Baltic Sea 3 n.m.
FILENAME : genus_run_88_5day_ocean.nc
FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/ 
LONGITUDE: 13.5E to 14.7E (XY integ.)
LATITUDE : 23.5S to 22.5S (XY integ.)
DEPTH (m): -0.0009867 to 5700 (ZT summed)
TIME : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN 
1.232E+09
 
!The same as with 7.0
 
set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-28:-22.5
 
yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL) 
DATA SET : MOM4 Baltic Sea 3 n.m.
FILENAME : genus_run_88_5day_ocean.nc
FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/ 
LONGITUDE: 13.5E to 14.7E (XY integ.)
LATITUDE : 28S to 22.5S (XY integ.)
DEPTH (m): -0.0009867 to 5700 (ZT summed)
TIME : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN 
4.759E+09
Again another result - with higher precision the difference does not become small.
Is it possible to reproduce this finding, or is this specific for my build? Any help is very welcome! 
 
Greetings,
 
Martin
[Thread Prev][Thread Next][Index]