Batch Processing

[Gareth D Hatton]

I have a question for the community:

Do you think we need batch processing on CalcImage?

What I mean by this is I have just done 28 samples looking for the same elements each time with the same conditions  Currently each one has to be processed as a single project.

This takes a while.

Does anyone else do this type of analysis and would this feature be useful?

[John Donovan]

Do you think we need batch processing on CalcImage?

Hi Gareth,
You make a valid point.

There are a number of ways this could be implemented. For example I could implement the program to process a command line argument which contains various information, for example the CIP project name.

But because the CIP project file contains the calculation flags, e.g., atomic, oxide, formula, etc, those need to be specified first before the quant mapping calculation starts.

So you'd have to create the projects first (but don't run the quant calculation yet) and then create a batch file like this:

call CalcImage "<projectname1>.CIP"
call CalcImage "<projectname2>.CIP"
call CalcImage "<projectname3>.CIP"

etc.

This way you could have all the quant calculations automated,  though you'd still have to create all the projects with their appropriate calculation flags specified first.

Then for the quant, you could process all the project without user intervention (I would have to set a flag to skip the user confirmation after each project is complete).

Would that help?
john

[Gareth D Hatton]

Hi John,
I think that it could be a useful if it is easy to implement, I can always just plug myself in and churn through.

Since the flags are all the same in all of the projects I could in theory copy and paste into a new file using Vim or some other and save the file under the next file name.  However I have noticed that there are other long numbers to contend with (such as 41690.6915856482).  What are these and is there any way to find them?

[Probeman]

Since the flags are all the same in all of the projects I could in theory copy and paste into a new file using Vim or some other and save the file under the next file name.  However I have noticed that there are other long numbers to contend with (such as 41690.6915856482).  What are these and is there any way to find them?

Good question and sorry the documentation isn't quite up to date on this topic!

Here is a pictorial description of the CIP file format. The lastest CalcImage help file also contains more information on page 12 of the CalcImage.pdf file but I will put that in a dropbox for you because it is too big to post as an attachment.



So, would you like me to add the ability to process each CIP file from a batch file without user intervention?

[John Donovan]

Some people have asked me what units the time values in the CalcImage CIP file are in.

These date/time values are the same as Microsoft Excel uses, which are in the number of days elapsed since 1899. Where day 1 (one) of the Excel dates is Jan 1, 1900.  The values to the right of the decimal point are for time as a fractional day.

So recent date/time values are on the order of 40000 or so.  Don't blame me, blame Bill Gates!   

[Ben Buse]

call CalcImage "<projectname1>.CIP"
call CalcImage "<projectname2>.CIP"
call CalcImage "<projectname3>.CIP"

etc.

Hi John,

I guess the above would help. But more useful would be when the mdb & configuration using for quantifying the maps has not changed. Apply the mdb & configuration to multiple sets of maps. But I guess this would be pretty difficult?

Ben

[John Donovan]

Hi Ben,
I'm not exactly sure what you have in mind here.

What is the goal you are trying to accomplish?   Can you provide an example of what you want to do?
john

[Ben Buse]

I guess this is similar to what Gareth is doing (is that right Gareth?), but if you were to batch - were it would be useful is in the following situation

Trace Si and Zn in many steel samples.

MDB - calibrate for trace Si and Zn, set up blank correction for Si and Zn as unknown sample, set up MAN backgrounds.

PI - run a series of Si,Zn maps on many samples.

Tell calcimage to process collectively the many samples using the MDB and unknown sample setup (rather than setting up individual projects).

Ben

[John Donovan]

I guess this is similar to what Gareth is doing (is that right Gareth?), but if you were to batch - were it would be useful is in the following situation

Trace Si and Zn in many steel samples.

MDB - calibrate for trace Si and Zn, set up blank correction for Si and Zn as unknown sample, set up MAN backgrounds.

PI - run a series of Si,Zn maps on many samples.

Tell calcimage to process collectively the many samples using the MDB and unknown sample setup (rather than setting up individual projects).

Hi Ben,
One difficulty is that each CalcImage project might consist of more than one spectrometer pass (if more than 5 elements are acquired).  So how would the "batch processor" know which sets of maps go together?

Maybe the best solution is to modify CalcImage to take a .CIP file as a command line argument and process it automatically as I originally described above with Gareth.

Also now that we can output all data types automatically from CalcImage, one could quant each project and then output them all by calling CalcImage with each project file as originally described:

call CalcImage "<projectname1>.CIP"
call CalcImage "<projectname2>.CIP"
call CalcImage "<projectname3>.CIP"
etc.

The only remaining task would be that the user would have to generate a script that creates a CalcImage project file based on the MDB, sample setup and maps specified by the user.  For a clever guy like you that should be easy!     The CIP project file format is documented above and in the CalcImage help file.

How does that sound?

[Gareth D Hatton]

I guess this is similar to what Gareth is doing (is that right Gareth?), but if you were to batch - were it would be useful is in the following situation

Trace Si and Zn in many steel samples.

MDB - calibrate for trace Si and Zn, set up blank correction for Si and Zn as unknown sample, set up MAN backgrounds.

PI - run a series of Si,Zn maps on many samples.

Tell calcimage to process collectively the many samples using the MDB and unknown sample setup (rather than setting up individual projects).

I am certainly doing multiple analyses using the same set up on many samples (for instance at the moment I am setting up to do one run of elements on 18 samples which all are pretty much the same).  I currently only process them by opening multiple instances of Calcimage.  I am not sure there is a viable workaround...  I had a quick play with call but could only get it to launch CalcImage if I used the entire path and also then it would not automatically open the CIP file so I left it there.

[John Donovan]

Hi Gareth,
Having CalcImage perform processing of a CIP file from the command line is a proposal, it's not implemented yet!  Actually it is implemented as of last night but I haven't released it!

First I need to update the CalcImage Help file and I want to re-arrange some of the menus, but I will have something interesting later today or tomorrow for you to try.  But yes, you will need to run your batch file from the PFE/CI app folder or call CalcImage with the full application path as you describe. Probably better to specify the full app path in your batch file as then you can keep the batch files in the same folder as your CIP projects...

One interesting application for this batch processing method in CalcImage is that if one wanted to tweak one calculation option, since they are generally stored in the MDB file, you will be able to make the change once, and then reprocess all the maps with a single batch call.

[John Donovan]

Hi Gareth and Ben,
We think we have something that will do what you want for batch processing of quant map projects in CalcImage.

The first thing we did was to modify CalcImage to accept a command line argument. This command line argument must be a CalcImage .CIP project file with the full path. Here is an example of a batch file that will launch CalcImage and automatically load the specified CIP project:



There are two parts to each line, first a string containing the full path to the CalcImage app in double quotes. This first part will always stay the same. The second part is the full path to the CalcImage project file to be automatically reprocessed, also in double quotes.

We also modified the code for creating a new project. That is when you create a new CalcImage project using the CalcImage project wizard, the software will now automatically append that project to a file (CIP_Batch.bat) in the user's current project folder with the new project name, similar to that shown in the example above.  Of course one is also welcome to create their own batch files to load their CalcImage projects automatically.

Once each CalcImage project is loaded from the command line, CalcImage will automatically load the raw intensity files, re-calculate the quant maps, and automatically output all quant maps. All based on the existing element to map assignments and data output types selected by the user.

The batch file will process each project file one at  time.  Of course one could run multiple batch files simultaneously.  Please let me know if this is useful for your batch processing needs.

[John Donovan]

I should mention again that this command line argument batch processing in CalcImage will not work unless the user has already assigned the maps to the quant elements using the Specify Quantitative Parameters menu.

Of course there is nothing to prevent anyone from creating their own .CIP project files with the x-ray maps already assigned to the elements and the data types specified from a script of their own devising.  Towards that goal we have documented the CIP file format in detail in the latest CalcImage help file as seen here:



To further assist your own script development for batch processing I have posted here the actual code used to write the CalcImage project file:

Code: [Select]

' Project file does not contain full path (uses current CIP folder)
Open CalcImageProjectFile$ For Output As #CalcImageProjectFileNumber%

Print #CalcImageProjectFileNumber%, CalcImageProjectVersion%, VbDquote$ & CalcImageProjectTitle$ & VbDquote$
Print #CalcImageProjectFileNumber%, VbDquote$ & CalcImageSampleNameTitle$ & VbDquote$
Print #CalcImageProjectFileNumber%, VbDquote$ & MiscGetFileNameOnly$(ProbeDataFile$) & VbDquote$
Print #CalcImageProjectFileNumber%, CalcImageProbeDataSample%, CalcImageNumberofImageFiles%, CalcImageCalculateTotalsImageFlag%, CalcImageCalculateStoichiometricOxygenImageFlag%, CalcImageCalculateExcessOxygenImageFlag%, CalcImageCalculateNetIntensitiesFlag%, CalcImageCalculateBgdIntensitiesFlag%, CalcImageCalculateKratioIntensitiesFlag%, CalcImageCalculateQuantPercentsFlag%, CalcImageCalculateAtomicPercentsFlag%, CalcImageCalculateOxidePercentsFlag%, CalcImageCalculateFormulaBasisFlag%, CalcImageCalculateElementByDiffImageFlag%, CalcImageCalculateDetectionLimitsFlag%, CalcImageCalculateAnalyticalSensitivityFlag%, CalcImageCalculateLogWeightPercentsFlag%, CalcImageCalculateDetectionLimitsBlankingFlag%, CalcImageCalculateAnalyticalSensitivityBlankingFlag%, Int(0)

' Write on and off peak image files
For i% = 1 To CalcImageNumberofImageFiles%
Print #CalcImageProjectFileNumber%, CalcImageImageTypes%(i%), CalcImageElementRows%(i%), Format$(CalcImageCountTimes!(i%), f83$), CalcImageBeamCurrents!(i%), CalcImageBeamCurrents2!(i%), CalcImageTimeOfAcqs#(i%), CalcImageTimeOfAcqs2#(i%), VbDquote$ & MiscGetFileNameOnly$(CalcImageImageFiles$(i%)) & VbDquote$
Next i%

' Now write CPQ files and parameters
Call CalcImageCPQReadWriteProjectFile(mode%, CalcImageProjectFileNumber%)
If ierror Then Exit Sub

' Now write misc parameters
Print #CalcImageProjectFileNumber%, CalcImageTitleAppendDataFlag%, CalcImageSurferOutputTemplateFlag%, CalcImageSurferSliceTemplateFlag%, CalcImageSurferPolygonTemplateFlag%, CalcImageSurferStripTemplateFlag%, itemp5%, itemp6%

' Now write EDS spectrum image flag and filename
Print #CalcImageProjectFileNumber%, IntegrateEDSSpectrumImagingFlag, VbDquote$ & IntegrateEDSSpectrumImagingFilename$ & VbDquote$

' Now write scan type and stage type
Print #CalcImageProjectFileNumber%, CalcImageScanTypeFlag%, CalcImageStageTypeFlag%

' Now write analog signal image output flags and names
Print #CalcImageProjectFileNumber%, CalcImageAnalogSignalFlags(1), VbDquote$ & CalcImageAnalogSignalLabels(1) & VbDquote, CalcImageAnalogSignalFlags(2), VbDquote$ & CalcImageAnalogSignalLabels(2) & VbDquote, CalcImageAnalogSignalFlags(3), VbDquote$ & CalcImageAnalogSignalLabels(3) & VbDquote

' Now write analog signal file names
Print #CalcImageProjectFileNumber%, VbDquote$ & MiscGetFileNameOnly$(AnalogFilenames$(1)) & VbDquote$, VbDquote$ & MiscGetFileNameOnly$(AnalogFilenames$(2)) & VbDquote$, VbDquote$ & MiscGetFileNameOnly$(AnalogFilenames$(3)) & VbDquote$

' Now write number of TDI files and number of TDI intervals
Print #CalcImageProjectFileNumber%, CalcImageNumberOfTDIImageFiles%, CalcImageNumberOfTDIImageIntervals%

' Write TDI image files
For i% = 1 To CalcImageNumberOfTDIImageFiles%
Print #CalcImageProjectFileNumber%, CalcImageTDIImageIntervals%(i%), CalcImageTDIElementRows%(i%), Format$(CalcImageTDICountTimes!(i%), f83$), CalcImageTDIBeamCurrents!(i%), CalcImageTDIBeamCurrents2!(i%), CalcImageTDITimeOfAcqs#(i%), CalcImageTDITimeOfAcqs2#(i%), VbDquote & MiscGetFileNameOnly$(CalcImageTDIImageFiles$(i%)) & VbDquote
Next i%

Close #CalcImageProjectFileNumber%

This plus the explanation should make it easy to develop your own CalcImage batch scripts.

[John Donovan]

I should also point out that one very cool advantage to this CIP_Batch.bat file is that once you manually (or through your own scripting code) create a .CIP project file (and the project is appended to the CIP_Batch.bat file), one can easily re-calculate and output all of one's projects by simply re-running the CIP_Batch.bat file from their user data folder.

For example, lets say that one decides to utilize a different matrix correction, or change the background model, or re-assign a standard or add an interference correction or change the element or formula by difference... since all these parameters are stored in the MDB file (which contains the element setup and standard intensities and is therefore usually the same for a related set of quant x-ray maps), one just makes the desired change to the element setup in PFE or CalcImage and then one simply double-clicks the CIP_Batch.bat file and all the CalcImage projects in the batch file are automatically loaded, re-calculated and output again.

I suspect that this is something that everyone will find useful even if you don't want to get into writing your own .CIP files from a script.   

It's difficult to show with still images how fun it is to watch CalcImage load, reprocess and output many projects, all automatically without user intervention, but here is an image of CalcImage running from a batch file (see command prompt window in upper left), with the quant calculation being re-run:



Next is an image of the quant map reprocessing completed and loaded:



and finally an image of all the map data types (elemental, atomic percent, oxide, k-ratios, detection limits, etc., etc.) being automatically output from Surfer:



This new CalcImage batch processing capability is now available with PFE v. 12.1.9 which one can obtain from the PFE Help menu.

[JonF]

I was just about to add a batch processing feature to the CalcImage wish list thread... guess I'm behind the curve by 8 months!

Nice, thanks John!