Timelapse analysis - exception raised concerning error values

[phoegenauer]

Question

Hi,
I tried to do a simple timelapse inversion using 3 timesteps but I am struggeling with the following exception.

Here is my code:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pygimli as pg
import pygimli.meshtools as mt
from pygimli.physics import ert
from pygimli.viewer import pv

DATA = []
file_paths = [
    r"N:\Python\Timelapse\Data\Conrad_Grad Res2DInv 01-01-2024 08-09-42.dat",
    r"N:\Python\Timelapse\Data\Conrad_Grad Res2DInv 01-07-2024 09-17-30.dat",
    r"N:\Python\Timelapse\Data\Conrad_Grad Res2DInv 01-10-2024 09-16-37.dat"
]
fn = r"N:\Python\Timelapse\Data\electrodes_coordinates.txt"
df = pd.read_csv(fn, sep='\t')
coordinates = df[['X', 'Z']].values

DATA = [ert.load(filename) for filename in file_paths]
for data in DATA:
    data.setSensorPositions(coordinates)
    data.createGeometricFactors()
    data["rhoa"] = data["r"] * data["k"]
    data.remove(data['rhoa']<=0)
    data.remove(data['rhoa']>10000)
    data.set('err', ert.estimateError(data, relativeError=0.05, absoluteError=5e-5))
    pg.info('Filtered rhoa (min/max)', min(data['rhoa']), max(data['rhoa']))
    print(data)
    
tl = pg.physics.ert.TimelapseERT(DATA)
print(tl)


tl.invert(lam=100, verbose=True)
ax = tl.showAllModels()

but what I get is:

13/02/26 - 12:56:25 - pyGIMLi - �[0;32;49mINFO�[0m - Found 2 regions.
13/02/26 - 12:56:25 - pyGIMLi - �[0;32;49mINFO�[0m - Region with smallest marker set to background (marker=1)
13/02/26 - 12:56:25 - pyGIMLi - �[5;31;1;49mCRITICAL�[0m - <class 'pygimli.physics.ert.ertManager.ERTManager'>._ensureError(c:\Users\hoephi\.conda\envs\pg\Lib\site-packages\pygimli\frameworks\methodManager.py:379)
All error values need to be larger then 0. Either give and err argument or fill dataContainer  with a valid 'err'  0.0 0.05001093418692392

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[7], line 1
----> 1 tl.invert(lam=100, err=0.05, verbose=True)
      2 ax = tl.showAllModels()
      3 # for a in ax.flat:
      4 #     pg.viewer.mpl.drawPLC(a, combined_plc, fillRegion=False, fitView=False)
      5 # # tl.showAllModels(ratio=True, rMax=2, orientation="vertical")  #  test

File c:\Users\hoephi\.conda\envs\pg\Lib\site-packages\pygimli\physics\ert\timelapse.py:526, in TimelapseERT.invert(self, t, reg, regTL, **kwargs)
    524 for i, ti in enumerate(np.atleast_1d(t)):
    525     self.mgr.setData(self.chooseTime(ti))
--> 526     self.model = self.mgr.invert(startModel=startModel, **kwargs)
    527     if i == 0 or creep:
    528         startModel = self.model.copy()

File c:\Users\hoephi\.conda\envs\pg\Lib\site-packages\pygimli\frameworks\methodManager.py:832, in MeshMethodManager.invert(self, data, mesh, startModel, **kwargs)
    829 self.fop._refineP2 = kwargs.pop('refineP2', False)
    831 dataVals = self._ensureData(self.fop.data)
--> 832 errorVals = self._ensureError(self.fop.data, dataVals)
    834 if self.fop.mesh() is None:
    835     pg.critical('Please provide a mesh')

File c:\Users\hoephi\.conda\envs\pg\Lib\site-packages\pygimli\frameworks\methodManager.py:379, in MethodManager._ensureError(self, err, dataVals)
    377 try:
    378     if min(vals) <= 0:
--> 379         pg.critical("All error values need to be larger then 0. Either"
    380                     " give and err argument or fill dataContainer "
    381                     " with a valid 'err' ", min(vals), max(vals))
    382 except ValueError:
    383     pg.critical("Can't estimate data error")

File c:\Users\hoephi\.conda\envs\pg\Lib\site-packages\pygimli\core\logger.py:301, in critical(*args)
    299 def critical(*args):
    300     logger.critical(whereAmI(nr=2) + "\n" + _msg(*args))
--> 301     raise Exception(_msg(*args))

Exception: All error values need to be larger then 0. Either give and err argument or fill dataContainer  with a valid 'err'  0.0 0.05001093418692392

I don't really get the message, as I am using ert.estimateError(data, relativeError=0.05, absoluteError=5e-5) at the top...

I followed the sample code from here:
https://www.pygimli.org/_examples_auto/3_ert/plot_09_ert_timelapse.html

[florian-wagner]

Hi @phoegenauer,

this sounds similar to #825, which @andieie fixed in #889 (September last year). Which version of pyGIMLi are you using? (Please note that we have an issue template, where such information is asked for, so we do not have to ask for it. Makes our life a bit easier and issue solving faster.).

Cheers

[phoegenauer]

Hi @florian-wagner
Sorry, I forgot to mention that..

I am using pygimli version 1.5.5

Thanks!

[halbmy]

I cannot reproduce this error. After the changes of @andieie , chooseTime() should always come up with an error (even if it is not there is estimates one) and unless it is negative it should work. Please check that your errors are >0 throughout.

[phoegenauer]

I checked again and all my errors are > 0
Attached you can find the example files from my code snippet, that I use

tl_exampledata.zip

[andieie]

Dear @phoegenauer,

I tried to replicate your issue and found the problem. In your case you initialized the class with the list of Data containers called DATA. In the code, this groups the data for different times into a matrix; however, when combining multiple datasets, some measurements don't exist in all timesteps. These get masked in DATA but their errors are set to 0.
The inversion can't handle zero errors (division by zero), so we replace them with the median error from valid data. This is fine because masked data points are ignored during inversion anyway - they just need a valid placeholder to avoid numerical issues.

For now, the easiest fix is to add this to your code, after initializing the class:
tl.ERR[tl.ERR == 0] = np.median(tl.ERR[tl.ERR > 0])

and always check tl.ERR to make sure there are no 0's. I will add a fix to this to pygimli's next release.

[florian-wagner]

Thanks a lot @andieie!

[phoegenauer]

Thank you @andieie!