Cameca Gun Ion Pump Glitches

[sem-geologist]

I think this is good place to share my 2 cents.

This is going to be about SXFive FE, but don't discard this fast, particularly if you are planning to run LaB6 or CeB6 on your SX100 as that I think applies to that too (unless your ion pump is noble diode or StarCell type). So I see you guys had described here some serious leaks, and after fixing Ar leak from spectrometers the issues were fixed for your machines. If you are running with tungsten tip that is good approach and probably you won't need to look any further.

But for tips operating continuously a year and further you should be aware that there are also not-obvious Ar contamination sources. First of all very tiny part of Ar diffuses through separation window (despite tight seal). Secondly, there is always residue air on sample surface, pores, and air contains 1% of argon. This can look insignificant in short term, but is of key importance in long-term. These argon bits accumulate in the diode ion getter pump(s), but differently than other getterable gases, argon is temporary buried on cathode by sputtering process. The problem is that then pump gets critical mass of argon in the pump it is no more capable to manage to hold it all, and even worse - it looses ability to hold already collected argon... and so pump releases most of the argon in an "avalanche" fashion (thus You would see in the logged data huge spike).

I am describing here argon cycles, there can be other issues on SXFive FE like leaky metal gaskets (these can be checked or ruled out by changing room temperature as that is sensitive to that), and can produce similar patterns, which is hard to distinguish (devil in the details). It is good to understand how argon capture works, and how spiking can be overcome.

SXFive FE is equipped with two ion getter pumps (both simple Diode type). The big primary pump is situated close to the Schottky's emitter, and small (20 l) is situated below the primary. This small one buffers column vacuum in between chamber and gun containment section, and as discussed latter is first in the line to catch and suffer from argon accumulation. The small one starts to be Argon saturated after more-less 1 year (that is with excellent separation windows and also careful procedures limiting leakage of samples in the chamber; if spectrometer would leak argon, then pump would start to suffer much more sudden).

So whats going with that released argon from the pump? In closed system (sealed vacuum containment and single ion pump) that would go into "perpetual" well defined periodic spiking, as after argon release the same pump tries to catch that argon back.  But after crossing some stability threshold it releases all captured argon again and repeats the process many times. On the microprobe it is a partially open system and part of released argon travels further in vacuum system where it is taken care. And so the affected pump with every cycle needs to deal with lesser amount of argon and duration in between spikes is increasing until finally it stops spiking. A small part of released argon escapes through column down to the chamber (if column-chamber valve is not closed) where it is taken out with turbo pumps, another small part is captured by other, in our case primary, pump. The affected pump keeps spiking despite lessened amount of argon as, differently to initially gradually accumulated argon, it needs to pump huge amount of argon at once. The newly formed argon burial structures are very unstable and so pump releases argon again very shortly after previous spike (minutes to hours to days). If part of argon can escape during spiking, then cycles will increase periodicity (longer time between spikes) as after some days/weeks/months there will be manageable amount of argon left and ion pump will cease spiking, but not forever, just for some longer time (months, half year...).

Looks good? Problem is self-solved? Not so fast. During smaller (secondary) pump release of argon, the primary ion pump is significantly being contaminated with argon with every spike of secondary pump, which is not good as primary pump being of simple Diode type is also susceptible for argon release. It is a larger pump and thus it can capture (temporary) much more argon... and so release much more argon. The safety vacuum threshold for Schottky emitter is set lax so that it would not get triggered with spikes originated in the secondary ion pump, but when primary pump gets argon saturated and releases all its argon - vacuum goes few orders over safety limit and trips the emitter.

If nothing is done with argon accumulation, then we experience that Primary pump releases argon in about 1.5 years from last baking. It actually did for us last week, that is why I am writing it here. I still have fresh mind after doing research on ion pump workings and workarounds. This is not first time it happen to us. We had similar "1.5 Year old" ion pump incident while running previous emitter. It then went worse than now as amount of argon released by primary pump was so large that secondary pump had gave up (that was 1.5 year ago, during Christmas-New Years eve brake, I was quite shocked to find such a mess getting back from holiday). This time secondary ion pump had survived, but I guess it is as I run it with manually set 5000V voltage, so it has much better response to pressure increase. But it was still enough to trip the field emission gun.

So I thought up some very simple procedure to get rid of Argon without baking the pumps, and I found it worked for our SXFive FE. It is based on 2 actually 3 assumptions:

1) If I want that no argon would be caught by any of the ion pumps, then both ion pumps should be off, as there is no possibility to isolate one from other.
2) Excess argon needs some open way to get out.
3) ion getter (Diode) pump needs energy to keep the Ar from escaping.

So with these in mind the procedure is clear:
(1) shut both ion pumps off
(2) while keeping column-chamber valve (EP 6) open.
(and obviously - the FEG needs to be shutdown before)

I had monitored the vacuum by separate vacuum gauge (IMG2(?)), which is installed near primary ion pump. The vacuum instantly worsened up.... but then as expected by my assumption, started to improve. As it had plateaued in a half hour, I had restarted ion pumps - vacuum got improved - no more spikes. Actually funnily, the primary pump did not wanted to restart, and controller showed "too low pressure" (too good vacuum) - seriously. But if waiting for a bit it gets correctly. It got back to ultimate vacuum in an hour.

BTW, emitter is today back on and running for SXFiveFE record, I have plans for it to be able to work stable (200 nA/1h within specification) up to 18000 hours... 5000h more to go. When FEG is run correctly it runs stable and also the vacuum baseline is really good. After argon riddance, IP1 (primary pump) shows 1.7E-7 Pa  - that is with Shottky tip operational (!), before spikes (two weeks ago) it increased from 1.8E-7 to 2.0E-7 Pa. From previous experience, when FEG is run in nonsense default mode - the primary vacuum would raise to 2.8E-7 Pa in a few months from 1.7E-7 Pa achieved during tip installation. This demonstrates that actually any baking of ion pumps is not needed on FEG instrument (after installment) at all when everything is run using brains. (just for comparison - our ZEISS FEG SEM manuals highly recommend baking ion pumps once 1-3 months)

[Probeman]

The long saga of our ion pump glitches (Turn Gun Valve to Position 3) may finally be over.  And it appears that the problem was not due to a bad ion pump though we may have had some Ar contamination problems in the past.

Here is how our instrument engineer explained it to my lab manager:

Hi Julie,
       I may have stumbled upon the cause of the GV3 errors! I was going to pull a common mode choke off of the breaker assembly when I saw scorch marks around the wire contact holes on the bottom of the breaker. The wires crimped there were heavily oxidized, which would make for a very poor connection at this point. Since these wires feed the Physics Subsystem, that subsystem was likely running in an undervoltage condition, leading to instability there.

Also, current cutting in and out because of the bad connections would lead to noise transients that triggered the Ion Pump and Penning safety circuits.  I've noticed that such transients bleed right into the power supply to those safety circuits.

    Tomorrow AM I'll clean up the breakers contacts and trim back the wires to fresh unoxidized copper. We'll see if this is the cause.

     Cameca is working on getting us the proper part this time. I reckon we should allow them to send it to us since we're not 100% sure the breaker problem is the cause.
Cheers,

Steve

He then followed that up with this further explanation to me:

Hi John,
      Read the email below to see what was found as the cause of the GV3 error.

Normally our fix for the error was to bang ion the ion pump to knock out arcing ash, or replace a column separation window to get Chamber and Gun pressures low enough that system noise would not trigger the Vac CPUs TTL fault monitor inputs. When this new version of the GV3 error crept in, nothing could prevent it except for lying (via a Lie Generator Box I had to build) to the pressure measurement  system and the Vac CPU's TTL fault monitor inputs.

     Regarding the note to Julie below, I wanted that common mode choke to check if it could reduce the noise bleed through that the +/-15V power supply exhibits to a ridiculous degree on the test bench. That's when I discovered the scorched connections on the circuit breaker. Bark up the right tree and you'll eventually spot  the prey! Note that it turns out that the +/-15V supply would not have benefited from that common mode choke because that supply is on the exit side of the choke and would thus be exposed to full amplitude noise from the bad connections on the very same circuit node. Also note that screws of the scorched terminals of the breaker were under nearly zero torque (i.e. they took almost no torque to loosen with a screw driver). All owners of SX100's would do well to re-tighten these screws. If they find scorching and overheated wires, they should wire brush the breakers wire crimps and replace the overheated wires (same 10AWG stranded hookup wire or larger).

Reusing overheated wire is not a good idea as thermal runaway will likely recur.
Cheers,

Steve

[Probeman]

Although the frequency of the vacuum glitches described in this topic have decreased since they began, we are still seeing them occasionally.

I've asked our instrument engineer to summarize the situation in case any one has seen similar issues on their Cameca instrument and/or any suggestions on how to further mitigate these glitches:

DESCRIPTION OF THE PROBLEM (GV3 Error):
UO's Cameca SX100 microprobe suffers from vacuum shut downs that occur whether or not the beam is on. There is some evidence that the problem occurs more frequently when the beam is on, which is understandable since the extra electrical noise  background is higher with high voltage on. See further details below. These shutdowns can interrupt a long analysis run, or a mapping run.  We call these shutdowns GV3 errors because the user interface error simply says Please Set Gun Valve to Position 3. When we comply with this dialog request, the vacuum system begins a sequence which restores full diffusion pump and Ion pump operation.

CAMECA SX100 Vacuum Pressure Measurement System:
There are six vacuum pressure gauges feeding vacuum pressure readings to the vacuum control board. Four of these are thermocouple gauges for measuring rough vacuum. One is a penning gauge, and the remaining is the ion pump for the electron gun chamber. Each of these six gauges provide an analog voltage that corresponds to the measured pressure and these values are displayed in the user interface.  These analog voltages are also monitored by safety comparator circuits that will detect when the readings of any given gauge have risen above a safe operating level. The logic level output of these comparators feed the vacuum control processor and will cause a vacuum system shutdown when they go active high (+5V). The vacuum pressure readings from the analog signals can also cause a shutdown when they read too high, even if the comparator outputs have been defeated. So there are redundant means determining if a given gauge is experiencing out of bounds pressure: Analog pressure readings that Cameca software will respond to, and Safety comparators (who's inputs are those Analog voltages). If you were to defeat a comparator output (force it Low), a high Analog voltage caused by a truly high pressure level will still cause a shutdown, although with an unknown latency.

Determining which gauge triggered the vacuum shutdown:
To our knowledge, there is no logfile that captures these errors and discloses which of the six gauges had seen the pressure excursion that led to the shutdown. We have installed a custom state logger that can capture these comparator outputs and we now have knowledge of which gauge experienced a pressure burst or noise. Only a logger such as this with a time resolution of 10mS or faster can reveal what's happening because once a comparator has been tripped, several more will trip due to the vacuum system shut down. This solution to the Who Dunnit problem works, but it does require importing the logger contents into a spreadsheet and producing signal vs time plots.

We also have defeated these comparator logic level outputs during tests to see if we could determine which thermocouple gauge was source of GV3 errors, and this did work. It turns out that our Varian TC controller has some noise issues since we have all new TC gauges, but still have GV3 errors related to them.

MEASURES TO REDUCE GV3 ERROR RATE:

•     Replace the gun ion pump elements (and clean out sputter ash) if the gun pressure is consistently 5x10e-5 Pa or higher
•     Replace column separation windows with pinhole leaks that lead to Col. pressure (penning) readings consistently 5x10e-4 or higher
•     Clean any debris on gate valve O-ring that leads to Col. pressure (penning) readings consistently 5x10e-4 or higher
•     Replace thermocouple gauges that have gone noisy (readings on the Varian controller can reveal this noise)

The 4 measures above will keep the gauge analog voltages (which represents pressure) further away from the safety comparator trip thresholds, thus preventing nuisance GV3 errors when system noise riding on top of the analog readings crosses the thresholds.

ADDITIONAL MEASURES (To no avail):
We have also doubled or tripled the value of capacitor C1 in both penning and ion pump power supplies in order to suppress transients due to arcs. These changes had no effect on GV3 errors.

GV3 ERRORS ARE LESS FREQUENT, BUT STILL OCCUR:
We still get the GV3 errors, though much less frequently due to the measures taken so far.  The design of the vacuum system seems sound when it comes to timely response to real vacuum pressure excursions. The problem is that this same safety circuit can respond to system noise and  cause nuisance shutdowns. Data logger runs (not the State Logger mentioned above) have not  revealed any real pressure burst when one of these GV3 shutdowns has occurred. This leads us to  believe system noise is responsible for the tripped comparator thresholds.

This state of affairs has us considering designing a comparator signal vetting circuit which will reject short duration events from the comparators. If the event duration exceeds perhaps 10mS, or manifests as a burst of short pulses, the signal will be immediately passed on to the vacuum processor.

For further safety, we'll use combinatorial logic to pass the fault immediately if two or more gauges is seeing an event.