Author Topic: Batch Processing of Image Files?  (Read 3611 times)

Dave Wark

  • Student
  • *
  • Posts: 3
Batch Processing of Image Files?
« on: June 23, 2017, 02:16:04 PM »
We tend to set up multiple maps to run over several days – usually they are all essentially the same but it in different locations of samples. Each map set usually involves two passes of data to accumulate up to ten elements. We find that processing the data for quantification and viewing can be cumbersome. Following are some considerations to facilitate the final steps . . .

1.   When mapping is completed, we have to step through four processes (open database, specify quant parameters, output parameters, quantify maps) for each set of maps. Is there any way to set these up in “batch mode” so that we can process the maps with one instead of four steps?

2.    It would be even more useful if we had a way to set up batch processing of multiple map sets. Example: I may collect ten sets of maps (using the same parameters) in ten different locations over a long weekend run. When I return to the lab on Monday, I’d like to be able to command the program to sequentially process all of the maps.  At present, I process only one map set at a time (which can take 45 minutes or more), after which I have to run back to the lab to set up the next set every hour or so. I could avoid all of this (and avoid tying up the probe computer during the day), if I could instead set it up to process all of those map files during the evening while I’m home sipping a cold beer. Any chance this option exists?

3.   Once I output all of my map images to Surfer, I’d like to be able to compare all ten maps of Ni, for example.  Ideally, to compare them visually I’d have them all scaled the same. But Surfer appears to automatically scale each map to the min and max pixel values. It’s very tedious to manually rescale all those maps. Is there any way to set up a template that would specify the scale for each map so that they are all scaled the same when output to Surfer? I looked and couldn’t find anything, so I suspect the answer is no, but I thought it might be worth a try....

Thanks!

John Donovan

  • Administrator
  • Emeritus
  • *****
  • Posts: 3275
  • Other duties as assigned...
    • Probe Software
Re: Batch Processing of Image Files?
« Reply #1 on: June 24, 2017, 10:01:22 AM »
We tend to set up multiple maps to run over several days – usually they are all essentially the same but it in different locations of samples. Each map set usually involves two passes of data to accumulate up to ten elements. We find that processing the data for quantification and viewing can be cumbersome. Following are some considerations to facilitate the final steps . . .

Hi Dave,
I'll respond below as best as I can to your suggestions.  And yes, I am trying to make the quant mapping process as automatic (and still flexible) as possible, for example see this recent post:

http://probesoftware.com/smf/index.php?topic=41.msg6005#msg6005

But there are a number of things one can do already. But first let me make some general comments about quant map (or any) data processing.

1. No one should be processing or re-processing data on the probe computer. The probe computer should be running acquisitions all the time and instead we should be processing data on an off-line computer. The software license from Probe Software allows users to install the software on as many computers as they like. Every student or customer in my Oregon lab gets a complete copy of the software (and the Thermo EDS software) for complete flexibility in reprocessing data on any computer, whether their laptop, office or home computer.  The same goes for any lab that has purchased our software.

2. We need to be computing in parallel, not in serial. That means that if we have 5 map acquisitions we want to process, we should start 5 instances of CalcImage on an off-line computer and run the quant calculations in parallel. This is not a problem for modern multi-core computers. I do it all the time. Yes, this will cut into your "beer time", but that is your call. As Calvin's dad (in Calvin and Hobbes) once said "If they really wanted to improve the quality of life, they would invent *slower* computers"!   ;D

3. Alternatively you can install PFE/CalcImage on multiple off-line computers in your lab and run CalcImage map processing on all of them. In addition to giving the software to all students and commercial customers in our lab, we also have three off-line computers in the lab dedicated for re-processing data. We have Filezilla set up on all computers and allow a "read only" access to the probe computer's UserData and UserImages folders so users can grab their data and raw map data whenever they want. One could also use shared folders.

1.   When mapping is completed, we have to step through four processes (open database, specify quant parameters, output parameters, quantify maps) for each set of maps. Is there any way to set these up in “batch mode” so that we can process the maps with one instead of four steps?

I think the new project wizard is pretty fast and easy. Let's review the process. After the maps have been acquired, one simply opens the probe database, clicks on a single sample for the element setup, then clicks on a single map for each spectrometer pass. That means for a 10 element (two spectrometer passes) map acquisition with say a BSE map on each spectrometer pass, that is (10 + 2 =) 12 maps total for MAN acquisitions and (10*3 + 2 =) 32 maps total for off-peak acquisitions, you only need to click on *one* map from each acquisition and they are all loaded automatically!  So 4 or 5 mouse clicks for the entire new project wizard to load the mapping project!

For the quant parameters, you simply click the Specify Quantitative Parameters menu and it loads all the parameters automatically for you. You simply need to select the additional map types you want to output, so another click for analytical sensitivity and another click for detection limits and you are done. 

Finally one mouse click to start the quantification...  and again, you can start the quantification of all the maps you acquired overnight, in parallel as discussed above.  So how could this be improved? I am open to specific suggestions.

2.    It would be even more useful if we had a way to set up batch processing of multiple map sets. Example: I may collect ten sets of maps (using the same parameters) in ten different locations over a long weekend run. When I return to the lab on Monday, I’d like to be able to command the program to sequentially process all of the maps.  At present, I process only one map set at a time (which can take 45 minutes or more), after which I have to run back to the lab to set up the next set every hour or so. I could avoid all of this (and avoid tying up the probe computer during the day), if I could instead set it up to process all of those map files during the evening while I’m home sipping a cold beer. Any chance this option exists?

See my original comments above. Simply run multiple copies of CalcImage for quantifying your maps on a single *off-line* computer or even on multiple computers. If a single map takes 45 minutes and you have 10 instances of CalcImage processing 10 map acquisitions, it will still only take about 45 minutes total for all 10 acquisitions.  Barely enough time for a cold beer!   :)

3.   Once I output all of my map images to Surfer, I’d like to be able to compare all ten maps of Ni, for example.  Ideally, to compare them visually I’d have them all scaled the same. But Surfer appears to automatically scale each map to the min and max pixel values. It’s very tedious to manually rescale all those maps. Is there any way to set up a template that would specify the scale for each map so that they are all scaled the same when output to Surfer? I looked and couldn’t find anything, so I suspect the answer is no, but I thought it might be worth a try....

This is something that could be done, but the problem is how?  The easiest way I can think of is for you to write a short script in Surfer that takes all your maps and asks you what scale you would like them rescaled to.  I would suggest you take one of the duplicate scripts utilized in this menu:



and modify it as necessary.  This script will never get overwritten, so any changes you make will be preserved though subsequent updates of the software.  I've attached one of these "customizable" scripts below for you to edit using the Scripter app from Golden Software.  If you copy the script to the application data folder, usually C:\ProgramData\Probe Software\Probe for EPMA, the CalcImage app will use that script for output of your maps.

See how easy that was (for me)!   :D   But seriously, this is exactly what these customizable scripts are for.
john
« Last Edit: April 13, 2020, 09:58:38 PM by John Donovan »
John J. Donovan, Pres. 
(541) 343-3400

"Not Absolutely Certain, Yet Reliable"

John Donovan

  • Administrator
  • Emeritus
  • *****
  • Posts: 3275
  • Other duties as assigned...
    • Probe Software
Re: Batch Processing of Image Files?
« Reply #2 on: January 18, 2024, 08:37:58 AM »
Scott Boroughs suggested we improve our ability to quantify x-ray maps, so we've made a couple of small changes to CalcImage that may be helpful for those processing large numbers of quant maps.

First we added code when creating a new CalcImage quantitative x-ray map project so it now asks if you want to utilize the settings from an existing project, in order to create a new project:



Clicking "Yes" on this dialog will cause CalcImage to ask you for the name of an existing CalcImage project, then it will proceed to ask you to confirm the sample setup from the MDB file, and then ask you to select the x-ray images for quantification. Clicking "No" will give one the original wizard behavior (select MDB, select sample setup, etc.)..

After selecting a previous CalcImage project, all the calculation options and flags will be loaded from the previous project into the new project.  That should save a few mouse clicks!

There's also a small change in how CalcImage processes existing CalcImage projects from the CIP_Batch.Bat batch file:



Previously when the CalcImage CIP_Batch.bat file was run (by double-clicking on it from the Windows File Explorer), it would launch CalcImage, load each project, quantify all the pixels for all specified calculation options, then output all selected output types (elemental quant, oxides, formulas, detection limits, etc.) to the Surfer application for presentation quality output.

https://probesoftware.com/smf/index.php?topic=73.msg5130#msg5130

Now, it will perform the quantification as before, but it will also check that the Surfer application is installed on the current computer before attempting to output to Surfer.  That way, if Surfer is not installed, it will simply skip that part of the process and go on to quantify the next CalcImage project in the batch file.

Hope this helps those of you with lots and lots of quantitative x-ray maps to process!
« Last Edit: January 18, 2024, 08:58:47 AM by John Donovan »
John J. Donovan, Pres. 
(541) 343-3400

"Not Absolutely Certain, Yet Reliable"