Author Topic: Cameca .Tiff file outputs  (Read 195 times)

jon_wade

  • Professor
  • ****
  • Posts: 64
Cameca .Tiff file outputs
« on: June 14, 2019, 10:22:40 am »
Dear collected wisdom

I know this isn't the place, and I know John's (very valid) answer, but that may have to wait until we win the lottery...

Anyone have experience of exporting images from peak sight 6?  I have a *lot* collected in a mosaic, but each one is being exported with a different intensity, suggesting that, somewhere, they are being auto ranged.
All I really want is raw counts so I can stuff them into python and, if necessary, fix the ranging, but I cannot work out if they are ranged, and if so how, or even how to get the unadulterated raw data out of PS6. 
CSV data looks like 32 bit tiff which appear to be raw counts, but its really not clear how or what is going on in the software.

Anyone got any magic insights?  (granted, its a lot of data... but each .imDAT file is 3+Gb.  Nope, no idea what is in that 3 Gb.....pixies?  unicorns?  analysts tears? ;) )


John Donovan

  • Administrator
  • Emeritus
  • *****
  • Posts: 2426
  • Other duties as assigned...
    • Probe Software
Re: Cameca .Tiff file outputs
« Reply #1 on: June 14, 2019, 01:03:42 pm »
I know this isn't the place, and I know John's (very valid) answer, but that may have to wait until we win the lottery...

Listen up mate, you talking about our software?  "Win the lottery"?  You're at Oxford University, right?  I think you meant to say, "hold a bake sale".  Either that or lotteries in the UK are pretty disappointing!   ;)

Anyone have experience of exporting images from peak sight 6?  I have a *lot* collected in a mosaic, but each one is being exported with a different intensity, suggesting that, somewhere, they are being auto ranged.
All I really want is raw counts so I can stuff them into python and, if necessary, fix the ranging, but I cannot work out if they are ranged, and if so how, or even how to get the unadulterated raw data out of PS6. 
CSV data looks like 32 bit tiff which appear to be raw counts, but its really not clear how or what is going on in the software.

Anyone got any magic insights?  (granted, its a lot of data... but each .imDAT file is 3+Gb.  Nope, no idea what is in that 3 Gb.....pixies?  unicorns?  analysts tears? ;) )

Don't know anything about the ImDAT file format, but from my failing memory the .ImpDat image file format has a 1 (or 2) kilobyte header and then just long integers (X by Y).  It's a binary file but it would be easy to write some code to process these files if all you need are the intensity values.

And yes, we have asked Cameca repeatedly for the format of the .ImpDat header and they simply refuse to provide it, saying: just use the ASCII export file.  Unfortunately the ASCII export format leaves some things ambiguous, not to mention that they modify it on occasion. So the binary format would be much better to parse, especially if one wants the raw intensities. But maybe someone out there will specify information on the .ImpDat header format as part of their next Cameca microprobe purchase...?
« Last Edit: June 14, 2019, 03:57:49 pm by John Donovan »
John J. Donovan, Pres. 
(541) 343-3400

"Not Absolutely Certain, Yet Reliable"

jon_wade

  • Professor
  • ****
  • Posts: 64
Re: Cameca .Tiff file outputs
« Reply #2 on: June 15, 2019, 08:27:09 am »
cheers John - I can see myself getting Hexedit out at some point  :P

Looking at consecutive BSE Tiff's of the same material, the grey levels are varying.  Is this a scaling thing in the software?

Honestly, I can promise you that should we have a windfall we'll definitely be investing PfEPMA, but cash, for a variety of reasons, appears short right now.

I'd be interested to know from others, but I get the distinct smell in the UK that geoscience demand for EPMA and analysis in general is declining.... :(

« Last Edit: June 15, 2019, 10:15:36 am by John Donovan »

Probeman

  • Emeritus
  • *****
  • Posts: 1876
  • Never sleeps...
    • John Donovan
Re: Cameca .Tiff file outputs
« Reply #3 on: June 15, 2019, 10:21:25 am »
Looking at consecutive BSE Tiff's of the same material, the grey levels are varying.  Is this a scaling thing in the software?

Hi Jon,
You're asking about the TIFF files from PeakSight?  I have no idea, but I do know that we have to set the BSE mode from "Differential" to "Ground", and then adjust the brightness until the image looks good, otherwise the BSE images show streaks when the BSE signal changes from pits and cracks.

Also on my system we have in the past seen gradients from left to right in the BSE brightness at lower magnifications:

https://probesoftware.com/smf/index.php?topic=583.msg3318#msg3318

This can be fixed by adjusting some electronics.
The only stupid question is the one not asked!

neko

  • Professor
  • ****
  • Posts: 60
Re: Cameca .Tiff file outputs
« Reply #4 on: July 09, 2019, 11:54:20 am »
Looking at consecutive BSE Tiff's of the same material, the grey levels are varying.  Is this a scaling thing in the software?

Based on the 3gb file size you've generated, I'm guessing you're not assembling these maps in PeakSigh (one might hope they've removed the 4 megapixel limitation since version 4 but I doubt it's large enough now) but are outputting individual cells for re-assembly in ImageJ or similar (have they fixed the rows/column designations being backwards to literally every other application on the planet yet?). Assembling the map in PeakSight will actually result in everything being set to 0-255 (or whatever their highest normalization value is), but when you output them individually, they're all normalized based on the brightest pixel value. WHICH IS TERRIBLE AND SHOULD NEVER EVER BE DONE IN THE FIRST PLACE EVEN WITH ONLY A SINGLE IMAGE BECAUSE THIS IS SCIENCE DAMNIT.

Someone did write an open-source Python program for extracting impdat, but unfortunately they wrote it in a weird Python GUI instead of making it command line so I'm not sure how useful it might be (link is here: https://probesoftware.com/smf/index.php?topic=938 ).

The Ascii format does just contain an array of pixel values (AFAIK 0-255 for BSE, 0-? for Xray channels), but the headers suck and ImageJ needs them stripped before it can import them - Writing something to strip the BSE headers, then running them through a batch importer in ImageJ as 8 bit greyscale should work (not sure if the unix tool ImageMagic works with ascii images but if so, command lines are great for batch processing, and who knows, might even have a header-stripping tool built in). They are n-length integers but ImageJ and Matlab understand them when importing.

This is definitely a problem I'm interested in solving so feel free to contact me about it, because I've had to go through and de-normalize (eg set high value to 255) tons of images by hand when people want them to be directly comparable to each other BECAUSE THIS IS SCIENCE AND THAT IS IMPORTANT AND WHY DOESN'T CAMECAAAAARGIGIHTNEHUONSTHUSANOTBKT *deep breath* I just can't for the life of me understand why Cameca doesn't understand that imaging is actually important. I had been hoping they'd have fixed that by 6.2 but I guess I was wrong.

Probeman

  • Emeritus
  • *****
  • Posts: 1876
  • Never sleeps...
    • John Donovan
Re: Cameca .Tiff file outputs
« Reply #5 on: July 09, 2019, 12:52:52 pm »
Hi Nick,
Yes. It is weird that they would do this.

Thermo doesn't do this in their mosaics and they are also using a .tif format. The Stage app in Probe for EPMA saves the raw intensity data as floating point in GRD format as seen here for one of my standard mounts:



https://probesoftware.com/smf/index.php?topic=324.0

I guess you can wait for them to fix this (it shouldn't be that hard), or maybe someday you and Jon Wade will get it together and get some better software!   ;)

Attached below are some high resolution BSE and CL mosaics acquired on our probe.  I downsized them so they aren't so huge for uploading.  The Surfer GRD mosaics themselves are each over 1 GB.
« Last Edit: July 09, 2019, 12:58:20 pm by Probeman »
The only stupid question is the one not asked!