Exponential increase in CalcImage processing time (quant maps)

[Julien]

Hi there,

I have an interesting problem that appears so far only on my microprobe computer (the fastest computer I have, rated 7.8/8.0 according to Windows...). When I started my calculation, the time estimate suggested 1.5 hours. The processing was going super fast, and I was so happy that I decided to launch a second one in parallel (since it cannot use multiple thread / CPU anyway...). The second quant mapping process started, and to my good surprise, again the estimated time was great (like 1 hour or so).

Both maps I was treating are ca. 200000 and 300000 pixels, 10 elements (2 passes), no peak interference, MAN background. Using FFAST and Armstrong ZAF.

I left the computer and came back 2 hours after thinking all would be done... Well, the first map had the time estimate to 9 hours and the second one to 14 hours. The next morning (so 12 hours after), the time estimates had jumped to 20 and 24 hours!!! I decided to cancel one and leave only one (the one with "only" 20 hours of time estimate...). Now, after over 24 hours (!!!), I have only one CalcImage processing, and the time estimate is now 32 hours... So I guess I hit an exponential progress, and basically the treatment will never be done!?!?

On a side note, the only other program running was Thermo NSS. The CPU is still very low (only one CPU working at 100% out of the 8...), RAM is largely available.

Is there a problem of buffer? Like CalcImage is slowly creating gigantic files that are constantly read & write & save? I can imagine this would be a reason for a serious slow down the more the process advances...

Anybody had seen this problem?

Julien

[John Donovan]

Is there a problem of buffer? Like CalcImage is slowly creating gigantic files that are constantly read & write & save? I can imagine this would be a reason for a serious slow down the more the process advances...

Anybody had seen this problem?

Hi Julien,
All of CalcImage's quant calculations take place in memory, and are only written to disk at the end, so it's not likely a disk issue. That said, could there be a backup going on in background?  Can you upload to Dropbox the MDB and PrbImg files for me to try?

You should try on another computer just to see if that is different.  Most CalcImage quant maps are processed in 10 to 20 minutes...
john

[John Donovan]

Hi Julien,
Thanks for sending me your raw data files. These are large maps, but I'm not seeing any exponential increase in quantification time, so suspect it's a computer specific issue .  By the way, the program calculates the remaining time after each scan line as seen here:

' Calculate remaining time
currentsec! = CSng(Timer)
If currentsec! < dStart! Then dStart! = dStart! - SECPERDAY#     ' in case timer goes through midnight
dSecLeft! = ((currentsec! - dStart!) / ipixels&) * (npixelx& * npixely& - ipixels&)
astring$ = MiscConvertTimeToClockString(dSecLeft! / SECPERDAY#)
Next ipixely&

It's a simple calculation but it automatically adjusts for changing CPU loads, so something is going on with your computer I suspect.

On the 2016-07-21 Quantitative Astimex_Grt M08-16 (1)_00040_.CIP project I saw about 3 hours after the first scan line, and it seems to be staying around there, though slowly decreasing of course.

By the way, if you open the Log Window from the Window menu you can see the calculation details, though it will slow things down since there is a lot of output to the log window.  The only thing I notice that is amiss is that the run is generating a lot of excessive intensity drift warnings for the Mn standard, e.g., "Warning: standard drift is -2.51% per hour for Mn ka on spectrometer 5", but that shouldn't affect the calculation time, unless the log window is open...

Right now your calculation is around 10% complete and it is stating around 2 hours, 50 min remaining.  It's getting late here, but I will upload your results to the Dropbox tomorrow morning, though you might want to try another computer for the other quant mapping project, until you figure out what the issue is with the computer you were using.
john

[John Donovan]

Right now your calculation is around 10% complete and it is stating around 2 hours, 50 min remaining.  It's getting late here, but I will upload your results to the Dropbox tomorrow morning, though you might want to try another computer for the other quant mapping project, until you figure out what the issue is with the computer you were using.

Hi Julien,
So now that I look this morning, I *am* seeing a slow down in the quantification and it appears to be related to the excessive standard intensity drift warnings that your run is generating in the log window:

Warning: standard drift is -2.51% per hour for Mn ka on spectrometer 5

What I think is happening is that even though I don't write the quant details to the log window when it is not visible, these standard intensity drift warnings *are* being written (from another part of the application)!  And eventually, they start to get large enough that the long window buffer starts getting too full.

I have a solution that I am working on...
john

[John Donovan]

Hi Julien,
Alright. Because so many warnings were being generated from the excessive drift correction on the Mn standard intensity, the log window buffer was getting too full, causing the app quant calculations to slow down.  So I added a small code to clean that out automatically, even when the log window isn't visible.

Here are the oxide calculations complete



and the Surfer output is attached below (be sure to log in to see attachments).   Cool that the garnet is so zoned in Mn and Fe, but the stoichiometric oxygen is so consistent. I'm no geologist but I assume that because both the Fe and Mn are the same valence?

I am re-compiling and the updated CalcImage will be ready to download in a few hours.
john

[Julien]

Thanks a bunch John, you rock!

Any chance you can send me the full files already processed? The only files attached are the JPG outputs (not the SRF files - and yes, I am logged in), but I need to readjust the color scale, for instance for the Ca Ka as there are apatite inclusions that disallow to actually see the garnet zoning. About the Mn drift, yes, I should have checked all my standards. I did run a first time my standards, but (a) I had an abnormal temperature in the lab and (b) the beam was drifting quite significantly. Also, I've been using this run for quite a while (over 2 months, so there are very old calibration, and the earliest one where "bad" - I should delete them). Maybe I should simply create a new run and re-import only the latest standardization. That will certainly help... Quick question: When you import the standards from a run into a new file (using the load file setup), does it also import automatically the standard that are associated to the MAN assignments? Knowing you, it sure does, but I wanted to make sure

Julien

[John Donovan]

Any chance you can send me the full files already processed? The only files attached are the JPG outputs (not the SRF files - and yes, I am logged in), but I need to readjust the color scale, for instance for the Ca Ka as there are apatite inclusions that disallow to actually see the garnet zoning.

Sure, I've only reprocessed the 040 maps, but I put a zip in the PFE Dropbox.  Note that the PFE update will be ready to download in about 5 min, so you can process the remaining maps yourself no problem.

About the Mn drift, yes, I should have checked all my standards. I did run a first time my standards, but (a) I had an abnormal temperature in the lab and (b) the beam was drifting quite significantly. Also, I've been using this run for quite a while (over 2 months, so there are very old calibration, and the earliest one where "bad" - I should delete them).

Absolutely. You can disable those older standards so they don't generate the drift warning.

Maybe I should simply create a new run and re-import only the latest standardization. That will certainly help...

That would work also.

Quick question: When you import the standards from a run into a new file (using the load file setup), does it also import automatically the standard that are associated to the MAN assignments? Knowing you, it sure does, but I wanted to make sure

Yes. The Load File Setup feature loads all standards (when you click yes), including primary, interference and MAN standards.  It might not load any secondary standards though.

Looks like it just finished uploading the new update. Go ahead and run the Help | Update PFE menu and you should be good to go!
john

[Julien]

Awesome! Doing it soon - although I should be heading the mountains with Heather soon, but I'll get back on it tonight!

Thanks!

Julien

[John Donovan]

Awesome! Doing it soon - although I should be heading the mountains with Heather soon, but I'll get back on it tonight!

Seems to be all fixed now. See below and note the BSE image optionally included with the oxide quant results (login to see attachments). 
john