Correction of negative result in quant

[VincentW]

I had a technical problem with PFE then I sent an email to Paul Carpenter for help.  Here is the email conversation of problem and solution. 

Hi, Paul:
I have a problem when I perform analysis on “pure” ZnO polycrystals (powder) for impurities on trace level (hundreds ppm).  I selected 20 elements to run and quant.  Result showed negative numbers on most of elements, meaning there were no such elements detected above detection limit.  However I do not want to show customer the negative numbers in result sheet.  In prowbein.ini file there is a line ForceNegativeKratiosToZero=0  to force the negative number to be zero, but I do not see the change to zero on quant result by either signing the line to 0 or 1.  Are there any other way in quant module in which we can sign the result to be positive or zero, not negative?

Here is the answer from Paul:

Hi Vincent,

The line in the .ini file is the default value for the checkbox that is in:
Analytical (Menu) -- Analysis options, see right hand side of window for Force K-ratios to zero.
This checkbox if set in a run overrides the ini file value.

I would not expect the Zn K line to interfere with many elements, but if you are using a high kV (like 20-25) then there is efficient exication of Zn k-alpha and k-beta and of course the higher order reflections can be seen on the LiF, PET, and TAP crystals. A typical interference is for Zn L-alpha on Na K-alpha.  You can use the Jeol L-value application or tables to see what possible interferences there are (put the L value Jeol utility in append mode to select multiple elements for display and be sure to set the order to a high number to inspect the interferences).  The PFE software and Standard also can be used to search for interferences.

Beware of having the Force K-ratios to zero flag turned on all the time as that can mask bad backgrounds. For the systems that you analyze, which have a lot of elements, there can be many interferences.

Cheers,

Paul

[Probeman]

Hi, Paul:
I have a problem when I perform analysis on “pure” ZnO polycrystals (powder) for impurities on trace level (hundreds ppm).  I selected 20 elements to run and quant.  Result showed negative numbers on most of elements, meaning there were no such elements detected above detection limit.  However I do not want to show customer the negative numbers in result sheet.  In prowbein.ini file there is a line ForceNegativeKratiosToZero=0  to force the negative number to be zero, but I do not see the change to zero on quant result by either signing the line to 0 or 1.  Are there any other way in quant module in which we can sign the result to be positive or zero, not negative?

Hi Vincent,
The line in the .ini file is the default value for the checkbox that is in:
Analytical (Menu) -- Analysis options, see right hand side of window for Force K-ratios to zero. This checkbox if set in a run overrides the ini file value.

Beware of having the Force K-ratios to zero flag turned on all the time as that can mask bad backgrounds. For the systems that you analyze, which have a lot of elements, there can be many interferences.

Paul is absolutely correct. It is important to pay attention to negative results as they may indicate an over correction of the background. Also, remember that if one is measuring a zero concentration, one would expect to see some concentrations reported as negative- in fact about half of them!

Why is this?  Because there will always be a variance about the measurement and if the average yields zero, some individual results will be positive and some will be negative as seen in this example:

ELEM:       Ti      Fe      Al       K      Na
BGDS:      EXP     LIN     AVG     LIN     LIN
TIME:   180.00  180.00  180.00  180.00  180.00
BEAM:   100.07  100.07  100.07  100.07  100.07
AGGR:                                         

ELEM:       Ti      Fe      Al       K      Na   SUM 
    37 -.00201  .00020  .00382  .00090 -.00141 100.000
    38 -.00070  .00140  .00350 -.00027  .00150 100.000
    39 -.00042  .00315  .00340 -.00062  .00077 100.000
    40 -.00209  .00276  .00258 -.00054  .00027 100.000
    41  .00046  .00021  .00141  .00140 -.00010 100.000

AVER:  -.00095  .00154  .00294  .00018  .00021 100.000
SDEV:   .00109  .00139  .00097  .00092  .00108  .00000
SERR:   .00049  .00062  .00043  .00041  .00048
%RSD:  -114.73 89.7468 32.9528 523.272 525.567
STDS:       22     395     374     374     336


In the above example we measured Ti, Fe, Al, K and Na in a synthetic quartz. The matrix was specified as SiO2 to assist in the matrix correction. Now although some of the individual data points and even the averages of some elements are negative, these values are just as "real" as the positive values for the reasons mentioned above.

How can we tell this? Because the average values in the AVER: line are all within 1 or 2 standard deviations (the line labeled SDEV:) from zero, except for Al. Now assuming Gaussian statistics we expect that 99% of our data should fall within 3 standard deviations of the accepted value, correct?  But that means that 1% of the time our variance will *exceed* 3 standard deviations so the 0.0029 wt % (29 PPM) reported could be a statistical fluctuation, but the odds are against it.

Again, why? Because we haven't made 100 measurements, only 5 measurements. And in fact, this particular SiO2 material has been checked using bulk techniques and is known to have about 1.4 PPM Ti and 15 PPM Al.

So the 0.0029 wt% measured Al is easily within 2 standard deviations of 0.0015 wt% (15 PPM) accepted value for this standard, as one might expect. So we were correct to be suspicious that this was merely a statistical outlier. It is actually an accurate though not very precise measurement for trying to determine 15 PPM concentrations (Conditions were 15 keV, 100 nA and 180 sec on-peak and 180 off-peak).

In fact, if you use the "Force negative k-ratios to Zero" option, you will actually be biasing your results high because you are eliminating half of the valid measurements (the statistically negative ones).

Why do we have this option?  Personally I would rather not, but some users insist on it, and there may be some valid uses for it, such as when an interference correction is over applied because the interference standard contains more than one interfering element and the amount of over correction is less than the under correction of not using the interference correction at all...

Edit by John: There is also an entire topic on the subject of "negative values" here:

http://probesoftware.com/smf/index.php?topic=30.0

[Probeman]

It should be clear that the same sort of zero statistics occur for MAN corrected acquisitions as well as off-peak acquisitions. Here are the "raw" intensity data from a glass where the major elements were acquired using MAN backgrounds and the minor/trace elements were acquired using traditional off-peak corrections:

Un   34 MB2 IW2 C4-ext-3
TakeOff = 40.0  KiloVolt = 15.0  Beam Current = 20.0  Beam Size =   10
(Magnification (analytical) =   8000),        Beam Mode = Analog  Spot
Number of Data Lines:   6             Number of 'Good' Data Lines:   6
Sample is a Time Dependent Intensity (TDI) Self-Calibration Acquisition Type Sample

On and Off Peak Positions:
ELEM:    na ka   si ka    k ka   al ka   mg ka   ca ka   ti ka   mn ka   fe ka    p ka   cr ka
ONPEAK 46343.0 81470.0 42799.0 32480.0 38495.0 83405.0 68239.0 52197.0 48080.0 23982.0 56852.0
OFFSET 19.8477 -15.523 -22.215 -14.074 4.15234 29.9375 52.3984 35.9023 35.3867 -10.539 46.5156
HIPEAK    ----    ---- 43607.5    ----    ----    ---- 68692.4 52715.5    ---- 25393.6 57410.9
LOPEAK    ----    ---- 42120.1    ----    ----    ---- 67785.6 51533.0    ---- 23225.2 56355.3
HI-OFF    ----    ---- 808.500    ----    ----    ---- 453.398 518.500    ---- 1411.60 558.898
LO-OFF    ----    ---- -678.90    ----    ----    ---- -453.40 -664.05    ---- -756.76 -496.70


On-Peak (off-peak corrected) or MAN On-Peak X-ray Counts (cps/30nA) (and Faraday Current):
ELEM:    na ka   si ka    k ka   al ka   mg ka   ca ka   ti ka   mn ka   fe ka    p ka   cr ka   BEAM
BGD:       MAN     MAN     OFF     MAN     MAN     MAN     OFF     OFF     MAN     OFF     OFF
SPEC:        1       2       2       4       1       5       3       3       5       4       3
CRYST:     TAP    LPET    LPET     TAP     TAP     LIF    LLIF    LLIF     LIF     TAP    LLIF
ORDER:       1       1       2       1       2       1       1       2       2       2       3
  300G   238.4 10429.7   205.1  3613.6  1289.2   209.2    75.9    58.4   269.5    79.7     5.9  19.831
  301G   236.1 10408.9   276.2  3628.0  1304.2   210.6    76.6    55.3   310.4    79.4     6.5  19.831
  302G   234.4 10454.6   272.6  3600.6  1294.4   207.6    78.4    57.2   313.9    77.6     5.7  19.829
  303G   237.3 10456.4   274.5  3593.1  1277.1   209.9    76.4    61.3   309.7    82.9     7.4  19.829
  304G   234.9 10403.7   271.3  3627.9  1297.0   212.5    74.5    59.6   311.9    80.1     6.0  19.828
  305G   235.0 10423.7   274.7  3575.3  1280.8   208.3    78.2    57.0   314.3    78.7     4.8  19.828

AVER:    236.0 10429.5   262.4  3606.4  1290.4   209.7    76.7    58.1   304.9    79.8     6.0  19.829
SDEV:      1.6    22.2    28.1    20.8    10.2     1.7     1.5     2.1    17.5     1.8      .9    .001
1SIG:      2.1    22.7     3.3    11.6     5.7     2.0     2.5     2.4     2.8     1.7     1.0
SIGR:      .74     .98    8.46    1.78    1.79     .87     .58     .88    6.30    1.05     .87
SERR:       .6     9.1    11.5     8.5     4.2      .7      .6      .9     7.1      .7      .3
%RSD:      .66     .21   10.72     .58     .79     .83    1.90    3.65    5.73    2.24   14.07

Off-Peak (calculated) X-ray Counts (cps/30nA):
ELEM:    na ka   si ka    k ka   al ka   mg ka   ca ka   ti ka   mn ka   fe ka    p ka   cr ka
TYPE:     NONE    NONE  LINEAR    NONE    NONE    NONE  LINEAR  LINEAR    NONE EXPONEN  LINEAR
COEF1:    ----    ----    ----    ----    ----    ----    ----    ----    ----  8.0000    ----
COEF2:    ----    ----    ----    ----    ----    ----    ----    ----    ----    ----    ----
COEF3:    ----    ----    ----    ----    ----    ----    ----    ----    ----    ----    ----
  300G    ----    ----    29.4    ----    ----    ----     6.0    18.7    ----    37.9    13.2
  301G    ----    ----    29.8    ----    ----    ----     6.7    18.4    ----    36.1    12.4
  302G    ----    ----    29.1    ----    ----    ----     7.8    17.6    ----    34.9    13.8
  303G    ----    ----    31.1    ----    ----    ----     6.2    17.1    ----    35.8    11.8
  304G    ----    ----    30.6    ----    ----    ----     7.7    18.7    ----    36.4    13.3
  305G    ----    ----    30.0    ----    ----    ----     7.5    20.7    ----    34.6    12.9

AVER:     ----    ----    30.0    ----    ----    ----     7.0    18.5    ----    36.0    12.9
SDEV:     ----    ----      .8    ----    ----    ----      .8     1.3    ----     1.2      .7


Note that the calculated (interpolated) background intensities are shown only for the off-peak elements.  Why is this?  Because the MAN background is calculated during the matrix correction iteration based on the average atomic number (Z-bar), which in turn is based on the calculated composition of the unknown.

Obviously for standards, the MAN background intensities can be calculated at any time since their composition is already known!  But even when "analyzing" standards from the Analyze! window, standard samples are treated as unknown samples for comparison as secondary standards!

This is even more apparent when we examine the raw data for a standard that contains none of the elements are measuring such as NiO:

On-Peak (off-peak corrected) or MAN On-Peak X-ray Counts (cps/30nA) (and Faraday Current):
ELEM:    na ka   si ka    k ka   al ka   mg ka   ca ka   ti ka   mn ka   fe ka    p ka   cr ka   BEAM
BGD:       MAN     MAN     OFF     MAN     MAN     MAN     OFF     OFF     MAN     OFF     OFF
SPEC:        1       2       2       4       1       5       3       3       5       4       3
CRYST:     TAP    LPET    LPET     TAP     TAP     LIF    LLIF    LLIF     LIF     TAP    LLIF
ORDER:       1       1       2       1       2       1       1       2       2       2       3
  360G     7.2    18.3     7.1    27.3    16.2     2.6    -1.4   -10.2    15.5   -15.0      .1  19.807
  361G     6.1    17.0    -3.4    33.1    16.2     2.8      .8    -3.0    16.0    -3.7      .6  19.807
  362G     7.4    14.5    -3.1    28.7    15.8     2.9      .8     2.9    16.8   -12.1     1.8  19.805
  363G     8.0    11.9    -5.3    29.2    16.8     2.9     -.3     -.8    15.9    -3.2     4.0  19.807

AVER:      7.2    15.4    -1.2    29.6    16.3     2.8      .0    -2.7    16.0    -8.5     1.6  19.806
SDEV:       .8     2.8     5.6     2.5      .4      .2     1.0     5.5      .6     5.9     1.7    .001
1SIG:       .7     1.2     2.0     1.5      .9      .3     1.3     2.3      .9     2.1     1.7
SIGR:     1.12    2.25    2.81    1.65     .46     .47     .77    2.37     .62    2.85    1.01
SERR:       .4     1.4     2.8     1.2      .2      .1      .5     2.8      .3     3.0      .9
%RSD:    11.48   18.18 -471.82    8.34    2.56    5.52-2681.91 -201.87    3.48  -69.98  107.17


In the case of NiO, we can see that elements measured off-peak are already background corrected and the statistically significantly positive intensities for the MAN acquired elements are simply the on-peak intensities (no off-peak intensity subtraction). But this is exactly how the MAN background correction works!  If we were attempting to measure these elements in a material with a similar average Z, then this NiO measurement would provide us with an accurate estimation of the background intensities since the x-ray continuum intensity is essentially dependent on average atomic number.

A more extreme example, here is the "raw" intensity output from an unknown sample with *all* elements acquired using MAN calibrations for the background correction:

On-Peak (off-peak corrected) or MAN On-Peak X-ray Counts (cps/1nA) (and Faraday Current):
ELEM:    Na ka   Mg ka   Al ka   Si ka    K ka   Ca ka   Fe ka   Ti ka   Cr ka    P ka   Mn ka   Ni ka   Co ka    S ka   BEAM
BGD:       MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN
SPEC:        1       1       2       2       4       4       3       5       5       4       3       3       5       4
CRYST:     TAP     TAP     TAP     TAP    PETJ    PETJ     LiF    LiFH    LiFH    PETJ     LiF     LiF    LiFH    PETJ
ORDER:       1       2       2       1       3       4       2       1       2       1       1       3       3       2
  392G    1.22   66.22  104.69  199.56     .50   29.39    2.42    1.04     .49     .10     .11     .12     .63     .21  25.065
  393G    1.11   65.50  103.92  201.30     .51   29.16    2.42     .99     .53     .11     .14     .15     .63     .20  25.040
  394G    1.25   65.52  103.62  200.58     .55   29.79    2.44    1.18     .49     .11     .10     .13     .58     .20  25.060
  395G    1.20   64.07  102.43  197.63     .53   29.11    2.36    1.17     .47     .10     .09     .13     .58     .23  25.090
  396G    1.09   65.02  104.49  199.55     .57   29.38    2.45    1.13     .50     .10     .11     .13     .63     .21  25.065
  397G    1.15   62.81  103.28  197.34     .57   29.11    2.52    1.14     .48     .10     .08     .14     .60     .22  25.090

AVER:     1.17   64.86  103.74  199.33     .54   29.32    2.44    1.11     .49     .10     .10     .14     .61     .21  25.068
SDEV:      .06    1.23     .83    1.57     .03     .26     .05     .08     .02     .01     .02     .01     .02     .01    .019
1SIG:      .03     .25     .32     .44     .03     .24     .06     .04     .03     .01     .01     .01     .03     .02
SIGR:     1.86    4.83    2.58    3.54     .92    1.09     .90    1.99     .72     .44    1.34     .87     .85     .55
SERR:      .03     .50     .34     .64     .01     .11     .02     .03     .01     .00     .01     .00     .01     .00
%RSD:     5.42    1.89     .80     .79    5.62     .90    2.11    6.90    4.59    6.16   18.52    8.59    3.96    5.39


As you can see all the intensities are statistically above zero because they are the "raw" MAN (on-peak) intensities. Here are the same data but now with the matrix and MAN background corrections applied:

ELEM:       Na      Mg      Al      Si       K      Ca      Fe      Ti      Cr       P      Mn      Ni      Co       S       O
TYPE:     ANAL    ANAL    ANAL    ANAL    ANAL    ANAL    ANAL    ANAL    ANAL    ANAL    ANAL    ANAL    ANAL    ANAL    CALC
BGDS:      MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN     MAN
TIME:    40.00   40.00   40.00   40.00   20.00   20.00   30.00   30.00   20.00   20.00   20.00   30.00   30.00   20.00
BEAM:    25.07   25.07   25.07   25.07   25.07   25.07   25.07   25.07   25.07   25.07   25.07   25.07   25.07   25.07

ELEM:       Na      Mg      Al      Si       K      Ca      Fe      Ti      Cr       P      Mn      Ni      Co       S       O   SUM 
   392    .179   8.279  11.290  19.746    .038   8.607   6.370    .684    .120    .010    .070   -.116    .048    .033  43.898  99.256
   393    .153   8.196  11.205  19.911    .040   8.535   6.361    .648    .147    .016    .139    .060    .047    .027  43.970  99.454
   394    .184   8.196  11.170  19.832    .052   8.722   6.398    .795    .122    .017    .044   -.047   -.004    .027  43.966  99.476
   395    .175   8.008  11.035  19.539    .048   8.522   6.202    .784    .104    .011    .012   -.053   -.004    .039  43.232  97.655
   396    .148   8.132  11.262  19.736    .058   8.599   6.429    .753    .123    .011    .070   -.068    .046    .029  43.828  99.158
   397    .163   7.863  11.128  19.508    .058   8.513   6.633    .764    .111    .010   -.005    .006    .019    .033  43.298  98.102

AVER:     .167   8.113  11.182  19.712    .049   8.583   6.399    .738    .121    .013    .055   -.036    .025    .031  43.699  98.850
SDEV:     .015    .152    .093    .159    .009    .079    .139    .059    .015    .003    .051    .061    .025    .004    .341    .775
SERR:     .006    .062    .038    .065    .004    .032    .057    .024    .006    .001    .021    .025    .010    .002    .139
%RSD:     8.74    1.87     .83     .81   17.71     .92    2.17    7.98   12.02   26.51   93.09 -168.98   99.08   14.14     .78
STDS:     1500     450    1545    1039    1524    1039    1715    1710    1713    1940     473     472     124     113       0


Now we observe that all elements at or close to zero (e.g., Ni and Co) contain both negative and positive concentrations, just as is the case when measuring off-peak elements as shown in the previous posts in this topic.  This is due to the natural statistics of the measurement regardless of whether the background is off-peak or MAN corrected.

Now some people may object that negative concentrations aren't physically real and they should be eliminated, but although one can "Force All Negative K-Ratios To Zero" from the Analysis Options dialog, that will bias one's data positively since both the negative and positive concentration results for a zero concentration measurement are equally statistically valid.

Bottom line: negative concentrations are expected when measuring a zero concentration. In fact, if they weren't observed we would need to worry!

[John Donovan]

Now we observe that all elements at or close to zero (e.g., Ni and Co) contain both negative and positive concentrations, just as is the case when measuring off-peak elements as shown in the previous posts in this topic.  This is due to the natural statistics of the measurement regardless of whether the background is off-peak or MAN corrected.

Now some people may object that negative concentrations aren't physically real and they should be eliminated, but although one can "Force All Negative K-Ratios To Zero" from the Analysis Options dialog, that will bias one's data positively since both the negative and positive concentration results for a zero concentration measurement are equally statistically valid.

Bottom line: negative concentrations are expected when measuring a zero concentration. In fact, if they weren't observed we would need to worry!

And for those who can take a joke, here are the latest results on negative mass:  http://arxiv.org/abs/1304.1566

[Probeman]

I'm returning to the topic of "negative concentrations" with a graphical explanation.

When we measure a zero concentration of an element using EPMA (e.g., below 1 PPM) we are measuring both the peak intensity and the continuum intensity (let's assume off-peak measurements). When the concentration is essentially zero, the peak intensity and the continuum intensity are essentially equal within statistics. And in fact, the distribution of the net intensities (and resulting concentrations) should ideally be equally distributed around zero. Of course the distribution around zero will be a Poisson distribution, but it will be very close to a Gaussian distribution because both the peak and the continuum raw intensities are well above zero with any reasonable sensitivity.



Above is a schematic plot of a Gaussian distribution around zero, which represents the net intensity (or concentrations) resulting from a peak and background net intensity measurement. Of course most of our measurements will result in net intensities or concentrations close to zero, but there will be some outliers on both sides of zero. In fact half of our net intensity calculations will result in negative intensities, when measuring a zero concentration of an element (e.g., < 1 PPM in EPMA). So there is nothing "non-physical" about working with negative net intensities. They are a result of the measurement process. And in fact they are sometimes extremely useful when they might indicate a problem with the background correction!

But when these negative net intensities are converted to (negative) concentrations, some might object that one cannot have negative concentrations because negative concentrations are "non-physical". I will agree that by themselves these negative concentrations are "non-physical", but what happens when we are calculating the average of these net intensities or concentrations?

Because values that are below zero have equal statistical significance compared to values that are above zero, if we do not include these negative intensities or concentrations in our average, we will bias the average towards more positive values.  So in order to have accurate results for trace elements, I contend we need to include all our measurements (positive and negative) in the calculation of the average.  Otherwise we will bias our results.

Just a few thoughts for the day.

[Probeman]

Here's a slide from my recent webinar on improving accuracy and sensitivity of trace element analyses:

https://www.youtube.com/watch?v=9KM5lU403VY&t=1478s



This slide is based on a slide sent to me by one of our commercial EPMA colleagues who asked to remain anonymous, but I think it makes the point well, that we should not be zeroing our negative intensities when measuring trace elements close to zero. If you are seeing consistently negative intensities, instead figure out what is wrong with your analytical approach. Maybe you have:

1. a spectral interference on one (or both) of your off-peak positions (check a wavescan and adjust your off-peak positions)
2. a curved background and you are using a linear interpolation (switch to a curved background model, e.g., exponential or polynomial)
3. perhaps you have a "hole" in your background under your on-peak position (apply a blank correction)

Regarding the last point, note that there are two types of "holes" in the continuum. One is due to an instrumental artifact, e.g., the hole under the Ti Ka peak position on some PET crystals, the other is due to an absorption edge in the sample, e.g., measuring Au Ma in iron sulfides.  The latter requires a zero blank sample that (roughly) matches your unknown matrix, the former could be any high purity material since the artifact will show up in all materials (though in the case of Ti Ka, high purity SiO2 is readily available).

The discussion of this issue is found at time stamp: 47:05 in the webinar video.

[Nicholas Ritchie]

What John says is correct.  I'd like to say one more thing.  It is totally appropriate as a final step to average together a series of acquisitions for a single element and then set the average value to zero if it is less than zero for reporting purposes.  Be sure to continue propagating uncertainties so you have a sense how negative the value is to check if your backgrounds etc. are reasonable.

Also: It is also necessary to set negative values to zero to compute quantities like the analytical total or matrix correction factors.  It can be subtle when to or not to set values which can not be less than zero to zero.   Usually the answer has to do whether you will subsequently average the values together - in which case, average first then truncate.

[Probeman]

What John says is correct.  I'd like to say one more thing.  It is totally appropriate as a final step to average together a series of acquisitions for a single element and then set the average value to zero if it is less than zero for reporting purposes.  Be sure to continue propagating
uncertainties so you have a sense how negative the value is to check if your backgrounds etc. are reasonable.

So you are saying that if one zeros a negative average, one should still report the variance which includes the negative individual points?  Hadn't thought about it, but that makes sense.  Then everyone could see that the range of the variance drops below zero. Though I think I'd still leave the negative value as it is and let the user decide to do that (or not) in their reporting.

Also: It is also necessary to set negative values to zero to compute quantities like the analytical total or matrix correction factors.  It can be subtle when to or not to set values which can not be less than zero to zero.   Usually the answer has to do whether you will subsequently average the values together - in which case, average first then truncate.

Yes, I agree.  For matrix corrections we set negative k-ratios to a very small number (10^-9 I think) because not only can we not take the SQRT of a negative number, sometimes having an actual zero causes problems in other calculations.

But I'm still thinking about the necessity of not including negative concentrations in the analytical total.  I have two problems with this: first the user may just look at the analytical total and think that it is OK and overlook a specific element or two that is quite negative. And second, it bothers me that a column or row of concentrations would no longer (literally) add up to the printed total.

I think seeing negative k-ratios or concentrations (or low analytical totals) should (hopefully!) prompt the user to look more closely into problems in their analytical setup.

[Nicholas Ritchie]

The key phrase is "for reporting purposes".  If you intend on using a value in subsequent calculations, use the non-truncated value.  I'd suggest that it is probably best to include both columns in data tables (non-truncated and truncated).  Each has its use.

I believe that the analytical total should be calculated with values truncated at zero because 1) we know that negative masses are not physical; and 2) including a negative mass will bias the analytical total.

Consider the following artificial example:

          C          T[C]
Fe      80.0       80
Cr      12.0       12
Ni       9.0        9
Mo     -1.0       0
Sum 100.0      101.0

The total looks too good in the C column but this is an artifact because the negative Mo value drags the value down and we know Mo simply can't be negative (non-physical).  The T[C] column sum reflects this reality better.

[John Donovan]

Consider the following artificial example:

          C          T[C]
Fe      80.0       80
Cr      12.0       12
Ni       9.0        9
Mo     -1.0       0
Sum 100.0      101.0

The total looks too good in the C column but this is an artifact because the negative Mo value drags the value down and we know Mo simply can't be negative (non-physical).  The T[C] column sum reflects this reality better.

Yes, I get that. And of course one can have a spuriously good total even without a negative concentration present. That is, one element could be too low (for whatever reason), and another element could be too high (for whatever reason), so they "cancel out" in the total.

Maybe it's just a pedantic thing for me, but I worry someone might think that all is "good enough" when they see a 101% total. But if they saw the -1.0% they would go, oh, something definitely isn't correct!

Maybe the issue is just a "reporting" as you mentioned.  I think we should keep negative values in the program output and then the user can decide whether or not to "report" them (along with the adjusted total or not).

[Stefan Baunack]

This is an interesting discussion.
Some years ago, I stumbled over the “Error” values in the EDAX TEAM EBSD/EDS Software. The only information I found (2019-10-02 [1]) was:
“I asked EDAX and here's the answer (roughly): The estimated error is based on all random (usually 1 sigma) and systematically sources (Processing of spectra, models etc.) available.
The TEAM software includes full calculation of analytical uncertainty plus an estimation of error propagation.” [1]

Later I saw results from Oxford software containing also columns “Wt%” and “Wt% Sigma”.
I don’t know if the formalism is described somewhere.

For a physicist, a concentration given as c_X = (-0.02 ± 0.05) would mean: “The concentration of element X is zero within the confidence level.”
The same would hold for c_X = (0.00 ± 0.05).

We have to discuss two cases:
(1) What is the effect of negative intensities?
If intensity_value < 0 transfers not only into negative number of atoms, but also into negative values for contributions of the yet to be determined average sample Z, backscattering, adsorption, fluorescence, a.s.o. this would be definitely non-physical, but the effect may be small, because only a small number of atoms will be treated this way.

(2) Where are the negative counts from? IMHO negative counts are the result of peak fitting and/or background subtraction.
Linear least squares fit (of spectra) one could do (i) with the constraint that the intensity of the fitted spectra must be non-negative [2], or (ii) using the fit results of our preferred software AND take their confidence intervals into account.
The confidence intervals for the fitting parameters are easily calculated by modern software (e.g. polyfit, polyval in Matlab/Octave; regression tool in Excel).
Method (i) will yield always non-negative intensities.
Using method (ii) any intensity_value < confidence_value should be set to 0. Here the statistics of the input data (counting noise) is taken into consideration.

I don’t know what option is used in the commercial programs.

[1] https://www.researchgate.net/post/Whats_the_meaning_of_the_Error_value_in_EDAXs_TEAM_EBSD_EDS_Software

[2] J. A. Taillon, An Open Evaluation of Hyperspectral Unmixing Strategies for EDS Analysis, Microscopy and Microanalysis, 24(S1), 752-753, doi:10.1017/S1431927618004257

[Nicholas Ritchie]

In the language of the ISO Guide to Uncertainties in Measurement, there are Type A and Type B errors. 

Type A are those that can be improved by making replicate measurements - statistical fluctions.
Type B are model or parameter dependent - making more measurements won't improve these.

Count statistics in X-ray spectra are Type A.  Most vendors report uncertainties due to count statistics.

The models we use to convert k-ratios (intensity ratios) to measures of composition are model and parameter dependent sources of uncertainty.  They are Type B sources of uncertainty.  Most vendors don't attempt to report these. (Maybe EDAX does.)

(1) You are correct that including negative mass fractions in calculations of "average sample Z, backscattering, adsorption, fluorescence" is definitely non-physical and should be avoided.
(2) Negative k-ratios are a result of overestimating the background (in peak fitting or background subtraction.)  For EDS, you can (and should) constrain the fit to only produce non-negative values otherwise the negative value may be compensated in the fit by another element leading to an overestimate of this element.  (DTSA-II does this.)

Vendor's estimates of uncertainties are always best case and likely an underestimate of the true uncertainty.  (The vendors are motivated to have you think their software is more accurate than their competitors.)

[John Donovan]

(1) What is the effect of negative intensities?
If intensity_value < 0 transfers not only into negative number of atoms, but also into negative values for contributions of the yet to be determined average sample Z, backscattering, adsorption, fluorescence, a.s.o. this would be definitely non-physical, but the effect may be small, because only a small number of atoms will be treated this way.

Yes, the situation of negative net intensities or k-ratios occurs when performing a background correction, but can also occur in the case of WDS net intensities or k-ratios when performing a spectral interference correction and one other item mentioned below.

By the way, Probeman was not correct above when he said that we make negative intensities equal to a very small number for the matrix corrections.  We used to do that, but for many years we actually propagate the negative k-ratios through all the matrix calculations, but we handle any situations where a SQRT or whatever could cause the calculation to blow up.  The code is on GitHub in the Open Microanalysis CalcZAF repository.

This means that a negative k-ratio has essentially has the same effect as a zero concentration, but the code preserves negative k-ratios so the user can obtain a proper distribution around a zero concentration.  Remember, as described in my webinar here (see time stamp 04:05):

https://www.youtube.com/watch?v=9KM5lU403VY&ab_channel=ProbeSoftwareInc

as you mentioned above, the matrix effect is essentially unaffected by trace elements, and visa versa, trace elements are essentially unaffected by matrix corrections.  What we should be worrying about with trace elements are these three items:

1. background corrections (including the blank correction)
2. interference corrections
3. secondary fluorescence from boundaries

These three items are critical for trace element accuracy and all three can produce negative net intensities or k-ratios.  And for proper sensitivity calculations we must, as Nicholas and Probeman mentioned above, preserve negative net intensities and/or k-ratios for a proper error distribution.  But the user may choose to zero these negative concentrations when reporting results.