Author Topic: .impDAT format  (Read 2436 times)

Probeman

  • Emeritus
  • *****
  • Posts: 1822
  • Never sleeps...
    • John Donovan
.impDAT format
« on: June 05, 2017, 02:16:48 pm »
Does anyone have any information on what parameters are in the header of the Cameca .impDAT x-ray map file?  Ideally a list of the byte offsets, data types and what info they contain.
john

The only stupid question is the one not asked!

sem-geologist

  • Post Doc
  • ***
  • Posts: 10
Re: .impDAT format
« Reply #1 on: March 20, 2018, 02:00:57 am »
yes, I have RE (reverse engineered) it (as did with brukers hypermap bcf format which is fully supported in hyperspy), but I am not sure about licensing (it is under GPL).

sem-geologist

  • Post Doc
  • ***
  • Posts: 10
Re: .impDAT format
« Reply #2 on: March 20, 2018, 02:04:29 am »
I mean it is not in hyperspy, but it is already published in internet as GPL'ed code in my github repository. My experience shows that Cameca breaks compatibility between files; the structure changes with some peaksight versions (e.g. peaksight 6.2 saved impDat cant be opened with older versions). My RE implementation reads files produced with peaksight 5 to 6.2.

Probeman

  • Emeritus
  • *****
  • Posts: 1822
  • Never sleeps...
    • John Donovan
Re: .impDAT format
« Reply #3 on: March 20, 2018, 08:42:31 am »
Hi SEM-Geo,
Thanks for the info.  Do you have a link to your GitHub repository?

Good to know that Cameca breaks the ImpDat format with version changes. I assume they store the version number in the .ImpDat so one can handle this?
john
The only stupid question is the one not asked!

sem-geologist

  • Post Doc
  • ***
  • Posts: 10
Re: .impDAT format
« Reply #4 on: March 22, 2018, 08:19:35 am »
It is in this repository https://github.com/sem-geologist/QSEM-viewer; It is quite messy, as I had to pause on RE and had to finalise my PhD. My idea is developing python/cython based software which would work natively on all OS'es and be capable to open all data formats produced by machines in our lab facilities. At the beginning I used standard tools for RE, I used mainly Hexinator and wxHexEditor (I live and develop mainly on linux/ thus this limited bunch of tools), These tools were more than enought for sorting out bruker format, but camecas binary formats (inheritant .net "feature") is messy as hell. Under lib folder you can find my python implementation (also at root there is jupyter notebook which uses the lib). I had roughtly sorted out WDS scans, and images, but qtiDat is still work in progress as I can't get what is so much changed in between last and previous versions. All parts have versions: main file header have version, and every struct and struct of structs and structs of structs of structs and ... (oh God, should I write that).. have versions, and my code have checks for those.

Finally I found some incredible RE tool: Kaitai Struct http://kaitai.io/ which I started to use (it is quite memory hungry), and so I started rewriting the library using Kaitai Struct as I find it order of magnitude more productive in RE than my previous workflow. Kaitai struct of cameca format at this moment is saved just in the root folder as cameca.ksy. One incredible thing about Kaitai is that it can produce libraries for most of the popular languages, so I think You could find some form which You could test without python.

However I warn: You can't use/include the code straight in the PfE as I am publishing/published it under GPLv3, unless PfE would comply with GPLv3. I had choose this license to protect myself from any possible sues of IP infringement. (think about libre/open office being capable of opening doc/docx files under interoperability umbrella). You are however free to download, examine, reimplement your closed source implementation in your language.
« Last Edit: March 22, 2018, 02:40:22 pm by John Donovan »

Probeman

  • Emeritus
  • *****
  • Posts: 1822
  • Never sleeps...
    • John Donovan
Re: .impDAT format
« Reply #5 on: March 22, 2018, 12:36:45 pm »
Very interesting and thank-you for the info.

We've already started writing a PeakSight x-ray image convertor based on their text output so no worries.  I think this will be fine because one can convert all the files in a PeakSight map acquisition with a mouse click or two.  This convertor is only something that some people need for special applications, as normally they would just simply acquire the x-ray maps directly in our Probe Image software.

Thanks again, and very interesting work you are doing.
The only stupid question is the one not asked!

John Donovan

  • Administrator
  • Emeritus
  • *****
  • Posts: 2376
  • Other duties as assigned...
    • Probe Software
Re: .impDAT format
« Reply #6 on: March 22, 2018, 02:17:58 pm »
...All parts have versions: main file header have version, and every struct and struct of structs and structs of structs of structs and ... (oh God, should I write that).. have versions, and my code have checks for those.

Don't get me started on version checking...   sometimes I think most of our code is dealing with version info and error trapping!

Have you dealt with the JEOL EPMA stage?  As you may know it is opposite of Cameca (and all other SEM stages including JEOL SEMs!). Here is a discussion of the stage scan acquisition differences:

http://probesoftware.com/smf/index.php?topic=101.0

In addition to that we have the unit differences of um for Cameca vs. mm for everyone else (because it would be too easy if all vendors used the same stage units), not to mention the spectrometer units!

But to me the biggest pain is that JEOL EPMA stages are the opposite orientation (anti-Cartesian where the max X/Y is in the *lower left*) compared to every other instrument stage out there!   :(
John J. Donovan, Pres. 
(541) 343-3400

"Not Absolutely Certain, Yet Reliable"

John Donovan

  • Administrator
  • Emeritus
  • *****
  • Posts: 2376
  • Other duties as assigned...
    • Probe Software
Re: .impDAT format
« Reply #7 on: March 25, 2018, 10:23:53 am »
I'm looking at the PeakSight text export file I received frrom Philipp Poeml (created from exporting ones .impDAT files to text) and have a question.

Here is the section that contains the spectrometer parameters and what they call the "Peak Position".

Analysis Parameters :     
 Sp   Elements   Xtal   Position   Bias   Gain   Dtime   Blin   Wind   Mode   
 Sp1   Zr La      PET   69723      1400   1120   3   539   4000   Diff   
 Sp2   Pu Ma      QTZ   55079      1900   1200   3   552   4000   Diff   
 Sp3   Am Ma      QTZ   53747      1900   1000   3   540   4000   Diff   
 Sp4   Si Ka      TAP   27139      1400   1350   3   580   4000   Diff   
Peak Position :   Sp1 69723,  Sp2 55079,  Sp3 53747,  Sp4 27139

In this map example, the "Position" in the "Analysis Parameters" and the position listed in the "Peak Position" line are the same values. I assume because this is an on-peak map.  But what about off-peak maps?  I assume that PeakSight can also acquire off-peak maps?

So my question is: if one acquires an off-peak map in PeakSight will the "Position" in the "Analysis Parameters" section be different from the position in the "Peak Position" line?

I ask because in Probe Image one can acquire no off-peak maps (and utilize the MAN correction), or one can acquire only high side maps (high only off-peak correction), or one can acquire only low side maps (low only off-peak correction), or one can acquire high *and* low side maps and utilize any of the off-peak interpolation methods (linear, exponential, etc.).  That is, how does PeakSight denote that the txt export x-ray map is on-peak or off-peak?
« Last Edit: March 25, 2018, 12:09:20 pm by John Donovan »
John J. Donovan, Pres. 
(541) 343-3400

"Not Absolutely Certain, Yet Reliable"

sem-geologist

  • Post Doc
  • ***
  • Posts: 10
Re: .impDAT format
« Reply #8 on: March 26, 2018, 04:47:56 am »
Unfortunately I have no idea about Jeol's EPMA, our facilities have only Jeol SEM, and stage is not automated/connected with software. But believe me your called "Anti-Cartesian" stage is not so bad. I had seen instruments with normal polarity for X but reverse for Y (some LA-ICP instruments); some reverse polarity is used on some of optical microscopes.

Version checking and error handling are the fundamentals in programming which uses some IO (like 99.99% of software/firmware). Without versions of formats we would have no progress, we would have to stick with single format which normally after some heavy use shows its weak sides, and needs some improvement. Version of the files makes possible for older/newer (less complete/more complete in features) files to be opened with older/newer software.

About text files... no bad feelings,  but Cameca's export to ascii simply sucks. It is super slow, outputs very limited information... thus it is not the coincidence that my fingers got itchy for reverse engineering Cameca's binary formats (impDat sucks too, but a bit less). What is so bad about ascii impDat output? everything. It spews separate files per signal and dataset, it does not fill in information about off-peak position if bkgd mapping is allso present. In peaksight 6.1 there were even no export option into ascii. Oh and mind that peak position given in metadata is after applying offset implied by Xtal verification.

So answering to the last question: yes the PeakSigh (mind no 't') is able to acquire single background map (only single off-peak per line, + and - bkgd is imposible). How it is converted into "low peak" or bkgd directly under peak is not known to me, there is also no options for linear (no slope) or exponential correction. When exported into ascii, both peak and bkgd txt files contain the exactly same metadata, The only way to know that we are looking into bkgd map and not into peak map is due to file name of txt file.

How efficient is mapping functionality in PfE? can you do multi area dataset?

John Donovan

  • Administrator
  • Emeritus
  • *****
  • Posts: 2376
  • Other duties as assigned...
    • Probe Software
Re: .impDAT format
« Reply #9 on: March 26, 2018, 08:38:03 am »
Unfortunately I have no idea about Jeol's EPMA, our facilities have only Jeol SEM, and stage is not automated/connected with software. But believe me your called "Anti-Cartesian" stage is not so bad. I had seen instruments with normal polarity for X but reverse for Y (some LA-ICP instruments); some reverse polarity is used on some of optical microscopes.

Yes, JEOL's EPMA is "anti-cartesian".  I had also heard from Nick Botto at UC Davis that some LA-ICPMS stages are also "half-cartesian" with as you say, an inverted Y axis. But we now can handle these situations nicely in PictureSnapApp, for moving images and stage coordinates between instruments with different stages:

http://probesoftware.com/smf/index.php?topic=1020.msg6937#msg6937

By the way, PictureSnapApp is a free app in the "demo" and "text input" modes. In "text input" mode one has to enter the stage calibrations using the keyboard, but it works for sample navigation on instruments that we don't yet have a stage interface for (or one doesn't want to "cough up" the US$499 for the hardware interface driver).  PictureSnapApp can be downloaded here:

http://probesoftware.com/download/PictureSnapApp.msi

Version checking and error handling are the fundamentals in programming which uses some IO (like 99.99% of software/firmware). Without versions of formats we would have no progress, we would have to stick with single format which normally after some heavy use shows its weak sides, and needs some improvement. Version of the files makes possible for older/newer (less complete/more complete in features) files to be opened with older/newer software.

Absolutely. Version checking is essential.  I bump my software version probably once a month! Usually only when I have to add a new parameter to the database, or when I make a big change to the code. As for error trapping, yes also. I would much rather trap an error than have the software lock up. The main challenge is when to bother the user and when to deal with the issue silently...

About text files... no bad feelings,  but Cameca's export to ascii simply sucks. It is super slow, outputs very limited information... thus it is not the coincidence that my fingers got itchy for reverse engineering Cameca's binary formats (impDat sucks too, but a bit less). What is so bad about ascii impDat output? everything. It spews separate files per signal and dataset, it does not fill in information about off-peak position if bkgd mapping is allso present. In peaksight 6.1 there were even no export option into ascii. Oh and mind that peak position given in metadata is after applying offset implied by Xtal verification.

So answering to the last question: yes the PeakSigh (mind no 't') is able to acquire single background map (only single off-peak per line, + and - bkgd is imposible). How it is converted into "low peak" or bkgd directly under peak is not known to me, there is also no options for linear (no slope) or exponential correction. When exported into ascii, both peak and bkgd txt files contain the exactly same metadata, The only way to know that we are looking into bkgd map and not into peak map is due to file name of txt file.

PeakSigh... ha, ha.

On my question on off-peak maps, sigh.  You confirm my suspicions about the PeakSight ASCII export and the lack of noting the background positions in the meta data.  OK, so I guess the user could denote that the exported maps are off-peak maps in the filename by some agreed on naming convention. But then the user would have to remember to include that identifying string in the sample/filename.  Or perhaps the app should just bother them at the time they run the conversion to our mapping format (.PrbImg), and ask, "are these maps on, high or low off-peak maps?"

Regarding the Veri spectro command, we always avoid that command in the lab. That way we know the absolute spectro positions and also can observe when things are changing mechanically.

How efficient is mapping functionality in PfE? can you do multi area dataset?

The mapping functions are in our Probe Image software. It utilizes the same firmware calls as PeakSight, but it does automatically handle acquisition of off-peak maps:



This is an example from a JEOL instrument, but the cool thing is that the software automatically acquires the off-peak maps specified (based on the off-peak positions from Probe for EPMA) after the on-peak acquisition. Of course most people nowadays just acquire on-peak maps and utilize our MAN correction which gives better sensitivity and in roughly half the acquisition time:

http://probesoftware.com/smf/index.php?topic=307.0

Question:  do you know how backward compatible is Cameca's ASCII export format for the .impDAT files?  I mean, has Cameca changed the ASCII export format with different versions of PeakSight?   I don't know which version of PeakSight Philipp Poeml used to create these ASCII files that I am working with, but I wonder if we will be able to read older and newer PeakSight ACSII export files?
« Last Edit: March 26, 2018, 08:47:54 am by John Donovan »
John J. Donovan, Pres. 
(541) 343-3400

"Not Absolutely Certain, Yet Reliable"

sem-geologist

  • Post Doc
  • ***
  • Posts: 10
Re: .impDAT format
« Reply #10 on: March 26, 2018, 09:54:32 am »
I don't know how much they could change the ascii output during windows era (I am 100% sure it is different than from old good unix PeakSight). I had just checked out this on new Peaksigh 6.2 (and reported here); on 6.1 (and I guess 6.0, but that we simple skipped) there was even no option to export to ascii, on 5.x I tried something to export to ascii few years ago, but when I saw the speed of that... I just decided that rather better I will erase presence of that option from my head (I am speed junky, if something is not working after clicking/executing in 100ms, I start to swear).

I have a few more questions about PfE (if this is right post to ask about it). While I looked over all those screenshots and this last one, I get impression that all options and setups are put into same windows as acquisition. I guess PfE is not using the same convention as cameca: setup->acquisition->interpretation/results. Putting all the setup options at the same window as the acquisition list looks a bit overcomplicated (Particularly that lots of our customers concentrates not on analytical difficulties, but on what to analyse, leaving complexity to operator, which we solve in the setup phase). Is my impression right?
What about mapping? If you use same cameca firmware for map acquisitions does not PfE suffer from the same limitation: in particular the video card memory size (I guess 64MB) which makes it not possible to run High definition mappings (e.g. 2048*2048 pixels) when selecting 5 WDS spectrometers, two analog signals (e.g. BSE, CL). Does PfE suffer from this?

This is only little connected with above, Cameca's PeakSigh also hold limit for WDS scans which is 2048 points (I guess it is impossed by binary format where wdsDat covers partly with imgDat), does PfE have such hard limitation for WDS scan?

Edit by John: I split this topic into another topic here:

http://probesoftware.com/smf/index.php?topic=1054.0
« Last Edit: March 26, 2018, 12:02:18 pm by John Donovan »

neko

  • Professor
  • ****
  • Posts: 57
Re: .impDAT format
« Reply #11 on: March 30, 2018, 11:35:20 am »
THANK YOUUUUUU

I was literally, literally just working on reverse engineering .impDat and if you've already got that, damn that's great news.

I did just this morning "reverse engineer" the CAMIMG "format" that you can save images from the command line with - turns out it's actually just impDat with a different header and more footer. Once I copy/pasted in the header/footer with a hex editor, and added the .impDat (it has to be a capital D too, otherwise the program wont open it), et voilĂ , I have a perfectly working, extractable file.

Now I just need to write a script to automatically fudge the header/footer from camimg to impdat (unless you've already done that, in which case bless your heart), and assuming your extractor works with peaksight 4.0 files, I can automate image acquisition from the command line by writing a python script to write an interpretor script to do so (and then implement better mosaic making than Cameca can manage themselves). I know it's ridiculous and PfE will just solve that problem for me, but after 7 years of PS4.0 it's a grudge match and I will not be defeated *shakes tiny fist*

neko

  • Professor
  • ****
  • Posts: 57
Re: .impDAT format
« Reply #12 on: March 30, 2018, 01:46:59 pm »
Or perhaps the app should just bother them at the time they run the conversion to our mapping format (.PrbImg), and ask, "are these maps on, high or low off-peak maps?"

Is this implying that if I have some old, very large xray maps + low side background of something kinda interesting like an internet famous meteorite from Russia, that, upon upgrading to PfE, we can import these files and quantitatively analyze them?

Probeman

  • Emeritus
  • *****
  • Posts: 1822
  • Never sleeps...
    • John Donovan
Re: .impDAT format
« Reply #13 on: March 30, 2018, 02:31:43 pm »
Or perhaps the app should just bother them at the time they run the conversion to our mapping format (.PrbImg), and ask, "are these maps on, high or low off-peak maps?"

Is this implying that if I have some old, very large xray maps + low side background of something kinda interesting like an internet famous meteorite from Russia, that, upon upgrading to PfE, we can import these files and quantitatively analyze them?

Yes, but only if you also have the standard intensities acquired in Probe for EPMA. That said, I have gone back and re-acquired standard intensities in PfE "after-the-fact" and been able to quantify previously acquired x-ray maps. This of course assumes that your spectrometer intensities are fairly reproducible, and most Cameca instruments seem to have very stable intensities over time.

Also note that even if you only have the on-peak x-ray maps from your "old, very large xray maps", you can still perform a quantitative background correction on the x-ray maps using the MAN method. 

Let's just say that it would be worth trying.
« Last Edit: March 31, 2018, 08:24:17 am by Probeman »
The only stupid question is the one not asked!

neko

  • Professor
  • ****
  • Posts: 57
Re: .impDAT format
« Reply #14 on: April 02, 2018, 01:10:54 pm »
Yes, but only if you also have the standard intensities acquired in Probe for EPMA.

I captured some maps of standards + backgrounds at the same time, but I'm guessing that can't be converted into PfE standard intensities?

It'd be interesting to try in any case, as people currently plot their meteorites on the Urey-Craig diagram via guesses. And this is probably the first internet famous meteor landing. :D