Reading PRBIMG files directly

[Dan R]

We are doing rapid screening and need to process  lots of maps collected in probe image. Is there a way that we can read the raw intensities directly from the PRBIMG file. I can open one in a text editor and see the header and footer information, but the row data look like nonsense. Is there anyway we can convert this to something meaningful?
Thanks,
-Dan
 
Example:
Row605=AECXRACgB0UAIBlFAAAgRQDgJkUAQDBFAIA7RQDAKEUAACVFAGA4RQDAQUUAwC1FAKA+RQCgSEUAQCZFAKAWRQAAOUUA4CFFAGAaRQBANUUAIBlFACAZRQBAU0UAoCpFAMAoRQAAJUUAAC9FACAeRQBgM0UAIEtFAKAqRQAgS0UAwEFFAEBTRQBgM0UA4DpFAGA9RQDAPEUA4FhFAMAyRQAgGUUAoDRFAEArRQCgQ0UAQDpFAIATRQBAP0UAQDVFAMBLRQBgM0UAwF9FACBuRQBAXUUA4F1FAGBbRQBAWEUAIHhFAOB7RQDAUEUA4GJFAKBDRQAgMkUA4DVFAMA8RQBgGkUAwGRFAGBHRQDgJkUA4BxFAKBDRQDAGUUAQBxFAMAtRQDgMEUAIB5FAGAkRQCgIEUAoBtFAMAURQAAG0UA4CZFAOAmRQDALUUAIDJFAGAuRQBAJkUAACBFAGAuRQDgHEUAYEJFAEAXRQCgJUUAwChFAEA/RQCgOUUAABtFACAtRQCgQ0UAIDJFAKA0RQDgSUUAAGFFAKBSRQBAXUUA4F1FAEBnRQCgUkUAAEhFAGAzRQAAPkUAoEhFAGApRQBAF0UA4ElFAIBARQDARkUA4CZFAGAuRQBgLkUAYDhFAOA1RQDAKEUAgDFFAKAlRQCAMUUAAD5FAAAqRQDgP0UAoEhFAABrRQCAaEUAYEJFAIBURQCgV0UAQERFAMAURQBAF0UA4BxFAKA0RQAgLUUAoBZFAGAVRQBAF0UAQDBFAAAbRQBgBkUAYClFAKAqRQCgG0UAQDVFAGA4RQCAQEUAIBRFAGAQRQAAEUUAABZFAEAXRQAgFEUAwEFFAGBMRQBgQkUAYFtFAGBvRQDgYkUA4ERFAEBYRQDAVUUAIEZFAOBTRQBAU0UAoGFFAGAuRQBAREUAAFdFAGBMRQBgOEUAYEdFAEA/RQCgSEUAgCxFAGApRQBAMEUAIC1FACAyRQCgUkUAgEpFAGAkRQAANEUAQDpFAEA/RQDAN0UAoEhFACBkRQCgcEUAQF1FAABIRQAgeEUAAJFFAGCLRQAgfUUA4IhFAICERQBgakUAwFBFAIBPRQCgSEUAgFlFAOBERQCgTUUAIEFFAABNRQDgK0UA4CtFAKA+RQDAQUUAQCtFACBBRQDAPEUA4FhFAEBTRQBgTEUAAHBFAPCHRQBgb0UAAFJFAABNRQCgPkUAwF9FAAA5RQAANEUAIFVFACBLRQBgPUUAwDJFAEA1RQCALEUAIDxFAEBERQCgUkUAAEhFAIBKRQAgbkUAQDBFAOBTRQDgK0UA4ERFAIBoRQAAUkUA8IdFAEBTRQAAPkUAoGZFAEBsRQCQiEUAAGFFAGB5RQCAfEUA8IJFAKB6RQDwh0UA0J1FAJCXRQBQpUUAoK9FAKClRQBAukUAEKlFAAC5RQCQoUUAoKpFAICnRQCws0UAAL5FAECmRQBgmkUAkJJFADCJRQCwhkUA0IlFAICdRQCAp0UAwKhFANCxRQAAuUUAoK9FAJCcRQDgpkUAgKJFAACMRQDAikUAkJJFABCGRQDgkkUAAH9FAGCBRQAgj0UAMIlFAGCfRQDQjkUAEJpFAHCtRQAAr0UAoMhFAJCwRQBwrUUAcJ5FABCQRQBQjEUAQHZFACB9RQDwgkUAgHxFAECSRQBAg0UAwI9FABCVRQBQkUUAIJ5FAODERQCA3kUAEOVFAFDrRQCw20UAINBFABCzRQAAm0UAIJRFAMCFRQCgf0UAgHJFAOCDRQBgmkUAEKRFAECrRQBApkUAgKJFAFCWRQDQk0UAwH1FACB9RQBgeUUAAHpFAICJRQDgkkUA8JtFADCxRQAAr0UA0MBFAMCyRQBQw0UAEMxFAODiRQDw+kUA8OZFAMDkRQCA1EUAIO5FABDlRQAA10UAwMZFANDZRQCgzUUAkNhFAJDERQAwykUAUMNFAJDORQDAvEUAIMZFACDBRQDwvkUAUNdFAGDMRQBwy0UA4LVFAKDIRQBgzEUAAMhFALDHRQAgy0UAgMVFAAC0RQCAsUUAUKVFALCfRQAwokUAoJFFAMCPRQAQpEUAYLhFACDLRQCw4EUAwORFAPD6RQCg/0UA0OhFAMDGRQCQv0UAMMBFAFDDRQBAzkUAcLJFAIDARQAA0kUAcMtFAJC6RQCQ00UAEMxFAIDARQDAxkUAYLhFACDBRQDQsUUA4L9FACDQRQDgsEUAsMJFACDQRQBQvkUAENZFAEDERQAwz0UAILxFAIC7RQDg2EUAoMNFAFDSRQAA0kUAMMBFAHDLRQDgxEUAwMtFAGDWRQBQ3EUAQM5FAJDORQCgw0UAENtFAFDIRQAwxUUA4L9FANCxRQDAxkUAcLxFAFDDRQDQ1EUA0MBFABDHRQCAz0UAELNFAMCoRQBwnkUAMJ1FAJCcRQAwykUA0PJFADD3RQBQ60UA4PZFADD3RQCg5kUA0LtFAHCyRQBQyEUAwO5FADDoRQDwuUUAgMVFAOC6RQAwykUAQMRFABDMRQCwx0UA0M9FABDMRQBgx0UAAM1FAPDIRQAQzEUA0MBFAFDSRQDgyUUAoNJFAMDkRQDQ40UA0ONFABDMRQCAykUAAMNFANCxRQBwskUAgMBFAGDWRQCAz0UA4OdFANDZRQDQykUA8MNFACCyRQAQuEUA0NRFAJDERQBwwUUAIMFFANDURQAAw0UAILxFAEC6RQAA10UAAM1FAJDJRQAA0kUA0MBFAIDARQDAy0UAENZFALDMRQAAw0UAIMFFAAC0RQBAukUAsMxFABC9RQDw10UAcMtFAFDDRQDAt0UAILxFANCxRQBQpUUAkKFFAIDARQDgzkUAoNJFAIDKRQAgy0UAEMxFADDjRQDQ2UUAIMFFADCORQAQn0UAAHpFAECIRQAApUUAQKZFANCTRQAQs0UAwMtFABDlRQCQ50UAcNVFAGDMRQCQxEUAEMJFADDKRQBwy0UAEMdFACDLRQBgzEUAUMNFAGC4RQAgskUAcMFFACDBRQCgw0UAgMpFADDPRQAQzEUA8M1FAEDORQCg0kUA8NJFAMC8RQBAyUUA4M5FALDRRQCwwkUAULlFAOCmRQBApkUA8JFFAPCRRQCAsUUAQL9FAECwRQDAskUA4LpFAEC6RQDg00UAsMJFAGDHRQBAyUUAUK9FAPC+RQDgxEUAYMdFACC8RQAAvkUAgLtFAJC/RQBAv0UAULRFAPDIRQBQuUUAcMFF
 

[John Donovan]

Hi Dan,
The pixel values in the raw x-ray intensity PrbImg files are encoded in Base64:

https://en.wikipedia.org/wiki/Base64

The values are in cps (not normalized to beam current).

There are several possibilities for you:

1. Write your own Base64 decoder using a library such as MatLab.

2. Use my VB code to decode the data. See files Base64Coder.bas and Base64Reader.bas on the Openmicroanalysis site:

https://github.com/openmicroanalysis/calczaf

3. Use the menu in CalcImage to load a PrbImg file, and the app will automatically create a .GRD file which is much easier to read.

Does any of this help?
john

[Dan R]

That definitely helps! Thanks!

Here is what I found for MATLAB.

The base64decode.m file can be found here:
https://www.mathworks.com/matlabcentral/fileexchange/39526-byte-encoding-utilities?focused=3773211&tab=function

Once this code is run, you will need to run one additional function to convert the data, so the script looks like this:
packed_data=base64decode(raw_data);
unpacked_data = typecast(packed_data,'single');

Dan

[remidelville]

Hi,
We also like to process directly in MATLAB data raw cps data for mapping. The ASCII format given by PeakSight was very handy: one could quickly check in the text editor some of the main acquisition parameters and the matrix of raw counts was directly visible and easily uploadable in MATLAB. Now we moved to probesoftware and I have difficulties with the loading of PRBIMG in MATLAB. I quickly tried that the base64decode according to the instructions of Dan Ruscitto but at first sight, it seems that a bit more scripting will be necessary to handle the header and extract an actual matrix of cps. If anybody is willing to share a working Matlab script (using the base64decode) to extract the matrix of cps, I am very much interested.
Rémi

[John Donovan]

You might find it easier to use CalcImage to convert the PrbImg files into GRD files.  Once they are in GRD files, you can get the intensities (in cps) exported in Surfer to ASCII. 

I could also make an ASCII export option menu in CalcImage. Would you like a "Convert all PrbImg Files To ASCII" menu?  What ASCII format would be best for your work?  Would you like other information in the ASCII file, or just the intensities?

I'm sure you have your own (nuclear specific!) reasons for performing the quant in MatLab as opposed to CalcImage, but I'm curious as to exactly why.  What exactly are you doing in MatLab that CalcImage isn't doing?

[John Donovan]

Hi Remi,
I don't know if this will do what you need, but I just added a new menu to CalcImage to convert all PrbImg files in a selected folder to ASCII (tab delimited) files.  These ASCII files can be easily loaded into, say Excel or MatLab, to obtain the x-ray map intensities.

To find this feature, open CalcImage (no need to load a project), and click on the Window | Log Window menu. Then select the new menu as shown here:



It will ask you to select one PrbImg file and then when you click Open it will read and convert all the map intensities for all .PrbImg files in that folder into ASCII files with the same name, but with a .DAT extension.

Right now I'm just putting the file name of the original PrbImg file in the top and then the x by y pixel intensities as seen here:

"C:\UserImages\Roberts\Pos_0001\PrbImg\171117 cassiterites_00001_WDS1_Sn_PETJ_.PrbImg"
46.666668      43.333332      43.333332      30.000000      26.666666      23.333334      36.666668 
56.666668      33.333332      43.333332      16.666666      40.000000      30.000000      40.000000 
43.333332      20.000000      40.000000      33.333332      46.666668      33.333332      36.666668 
40.000000      40.000000      36.666668      23.333334      63.333332      46.666668      23.333334 
46.666668      30.000000      56.666668      33.333332      40.000000      56.666668      36.666668 
33.333332      23.333334      36.666668      46.666668      50.000000      60.000000      33.333332 
30.000000      36.666668      40.000000      50.000000      26.666666      16.666666      60.000000 
16.666666      30.000000      33.333332      46.666668      23.333334      26.666666      63.333332 
46.666668      33.333332      30.000000      26.666666      50.000000      30.000000      40.000000 
etc.

I can re-format this output to whatever works best for you, but it's a start at least.

[Ben Buse]

Hi John,

"Convert All PrbImg Files in Folder to ASCII" is excellent.

I can now open them in ImageJ using open TextImage.

Two things

(1) Any chance you can remove the first line [the file name] for I have to delete it before opening it in ImageJ?

(2) It be really good if you could do this for grd files as well.

Thanks

Ben

[John Donovan]

"Convert All PrbImg Files in Folder to ASCII" is excellent.

I can now open them in ImageJ using open TextImage.

Two things

(1) Any chance you can remove the first line [the file name] for I have to delete it before opening it in ImageJ?

(2) It be really good if you could do this for grd files as well.

Hi Ben,
I can certainly remove the filename at the beginning of the file.   

On (2), you want all PrbImg files converted to GRD as well?    Should that be a separate menu or do you always want ASCII text and GRD files output together?
john

[Ben Buse]

On (2), you want all PrbImg files converted to GRD as well?    Should that be a separate menu or do you always want ASCII text and GRD files output together?
john

Hi John,

No what I meant was grd -> ASCII. So I can open the quant maps in imagej - would this be possible?

Thanks

Ben

[John Donovan]

On (2), you want all PrbImg files converted to GRD as well?    Should that be a separate menu or do you always want ASCII text and GRD files output together?
john

Hi John,

No what I meant was grd -> ASCII. So I can open the quant maps in imagej - would this be possible?

Thanks

Ben

Ah, OK.  I'll make a separate menu for GRD to ASCII conversion.

One more question:  I'll remove the filename at the beginning of the ASCII file, but would it be helpful to include the X/Y pixel dimensions in the ASCII files?  That is the first line could be "800 <tab> 600" indicating that the map is 800 pixels wide by 600 pixels high...

Since Remi originally requested this I hope this is helpful for him as well.
john

[Ben Buse]

Not for imagej import (text image) - you don't want any first line header

That be great

Thanks

Ben

[John Donovan]

Not for imagej import (text image) - you don't want any first line header

That be great

Thanks

Ben

In that case maybe I'll make the file with a .txt extension with just the intensities?  Then I can also make the .dat file with some header information.

[remidelville]

Dear John,

Thanks for adding this extension in extension in CalcImage. It certainly makes export easier.
The reason we are using directly count maps in Matlab  is that we have a developed an alternative procedure to quantify mapping not relying on the usual ZAF quantification method (which would be punishingly long for decent size mapping). MATLAB also allows us integrating other image processing steps very specific to our needs in a single automated script. We also do batch processing of  multiple ASCII files. We still need to explore the possibilities of CalcImage since we only recently implemented the full ProbeSoftware on our machine.

Having a header in the ASCII/txt file is for us not an issue since most advanced soft can easily discriminate it from the data. Experience tells us that is actually quite handy to have a .txt file containing the data and a header that one can quickly open to check main conditions of acquisitions. That allows us to spot rather quickly possible mistakes in the acquisitions settings. A scripting program like MATLAB can also automatically extract some data from the header that may be useful for the processing. In our case, we need for example to normalize for the beam current so the script get the current used for the map in the header itself. Below an example of the header that PeakSight generates when data is exported to a .txt file. As you can see most basic acquisition parameters are included. Something similar would of course be very nice to have as an export from the ProbeSoftware suite.


FileName :   no name (Current dataset : 1)
Signal(s) Used :  Vs1 SE , Vs2 BSE Z, Cr Ka, O  Ka, Pu Mb, U  Ma
Spectromers Conditions :   Sp1 PET,  Sp2 PC1,  Sp3 PET,  Sp4 PET
Full Spectromers Conditions :   Sp1 PET(2d= 8.75,K= 0.000144),  Sp2 PC1(2d= 60,K= 0.01483),  Sp3 PET(2d= 8.75,K= 0.000144),  Sp4 PET(2d= 8.75,K= 0.000144)
Column Conditions :  Cond 1 : 20.2keV 202.6123nA 
Date :  16-oct-2017
User Name :  sx
Setup Name :  Bv 20170831.impSet
DataSet Comment :  Image Mosaic
Comment :   
Analysis Date :  16/10/2017 09:28:49
Project Name :   
Sample Name :   
Analysis Parameters :     
 Sp   Elements   Xtal   Position   Bias   Gain   Dtime   Blin   Wind   Mode   
 Sp1   Cr Ka      PET   26190      1318   1042   3   2124   2876   Auto   
 Sp2   O  Ka      PC1   39940      1510   770   3   486   4514   Auto   
 Sp3   Pu Mb      PET   40077      1847   982   3   1132   2726   Auto   
 Sp4   U  Ma      PET   44667      1849   988   3   946   2584   Auto   
Peak Position :   Sp1 26190,  Sp2 39940,  Sp3 40077,  Sp4 44667
Start :   X = -3300 Y = 130 Z = -218
Stop :   
Dwell Time :  0.02 Sec 
Acquisition type :   Stage Grid 
Step Number :  1024
Line Number :  1024
Beam Size :  0 µm
Step Size :  1.
Y Unit : cts
Image  : Pu Mb

[John Donovan]

Hi Remi,
There are many ways to get this sort of information out of Probe for EPMA and Probe Image.

For example one can simply open the PrbImg file in any text editor and all the conditions for that map are there in ASCII. The following is a sample of some of that information (it's just a Windows INI file format):

Code: [Select]

[ProbeImage]
ImageFile=FeCu_Sulfide-1_00050_SP1_Si_PET_.tif
Type=prbimg
Version=1.2.0
Notes=
Probe=Demo Microprobe JEOL
InstrumentModel=Demo
ScanOrientation=Cartesian
ScanType=Beam
[BasicInfo]
Author=
Computer=
Description=
Make=
Model=
Organization=
[UUID]
Application=d9f25482-54b8-4f10-a4ca-cffcee884cfe
ImageSet=d3021183-b051-4958-ad6a-f5187090f4a7
Image=ecfe46be-29a0-4434-a342-7ca13e4502fa
[Registration]
X1Pixel=63
Y1Pixel=63
X2Pixel=0
Y2Pixel=0
X3Pixel=63
Y3Pixel=0
X1Real=48.9564
Y1Real=49.0423
Z1Real=8
X2Real=49.0423
Y2Real=48.9564
Z2Real=8
Z3Real=8
[ColumnConditions]
BeamCurrent=1e-008
BeamSize=1
ChannelNumber=1
ChannelLabel=SP1
Crystal=1
CrystalName=PET
Element=14
HighVoltage=15
Line=1
LineName=Ka
NegOffset=-4.41
Magnification=4353
OffsetType=0
PixelTime=20
PosOffset=4.411
Position=227.844
SignalName=SP1
ScanRotation=2.21175
TakeoffAngle=40
Bias=1674
Baseline=1
Deadtime=0
Gain=32
Mode=0
Window=9

For a summary of the map setup for all the mapped elements, since the map setup is based on the sample setup in Probe for EPMA, simply use the Report button in the Analyze! window. Here is a post that discusses this report output:

http://probesoftware.com/smf/index.php?topic=42.msg5982#msg5982

The Report information can also be exported to Excel.  An example of this sample setup output in tabular format is attached below in both ASCII (.DAT) and Excel (.XLXS) formats.

Later today I will post about the new export menus I've added to CalcImage along the lines that you and Ben requested.

[John Donovan]

Hi Remi and Ben,
I added several new file conversion menus to the CalcImage "log window" as seen here:



Basically for the conversion to ASCII, the .TXT file contains only the intensities as tab delimited values.  The .DAT file contains a number of useful values in addition to the tab delimited intensities as seen here in this example:

"C:\UserImages\Carpenter\Wi-108-2b_00423_SP1_Na_TAP_.prbimg"
800   800
15   0.05
100.05   100.05
14.331   11.934   53.6757   51.2787

Where the first line is the full path to the file, the second line is the X and Y pixel dimensions, the third line is the keV and pixel dwell time in seconds, the fourth line is the before and after beam current in nA, and the fifth line is the stage extents of the map in millimeters (XMIN, XMAX, YMIN, YMAX).

The latest version of PFE (update from the Help menu), contains these new CalcImage menus.  Please let me know if you need any additional changes to this output.