Diffusion pump on SX100

[sem-geologist]

Sorry for very silly question.

Has anyone a user or service manual of Alcatel Crystal 102 Diffusion pump (the diffusion pump installed on SX100)?

While I maintain all our primary pumps (changing oil regularly, and getting service inspections), the Diffusion pump is quite an enigma to me. It was just working up to now, until it stopped working (I guess, or it could be wrong gauge readings). Alternative source of failure (blown window, leak) were checked and not found. The bottom part is hot while operational, thus probably it is not burnt out heating element to blame (albeit it could be failing and thus produce less heat (?)). I started to look for manual, could not find any on internet for this particular vendor, but found disturbing information in manuals of other vendor diff pumps. I.e. oil level and color should be checked every 6-9 months... from 2014 (from when I am around the lab; the pump is from 1998 - when that machine was assembled) I don't remember anyone do that. It was not inspected also during last service two years ago... Albeit I have some traces that it probably was serviced at least once (There is a small glass bottle with handwritten "diff. oil" in the wardrobe). The manuals of other vendors mentions that oil should not be stored more than 7-9 years... so I guess if it is working inside the pump it also looses its properties during time.

I have no idea how to check the oil (there are this small pipe with two screws - I guess it would be used to drain and fill oil without disassembling it from the place...?)
Neither I have any idea what kind of oil is currently filled (In case I would want to fill new oil, I should clean the diff pump from old oil, as different kind of oils should not be mixed). (ha, I actually could drain a sample and check under our new RAMAN spectrometer)
In some french vac shop I could find on sale a spare heater and dedicated silicon oil. However, having no manual I have no idea how much oil I should buy and fill (after cleaning).

Thus I am looking to buy new diff pump, at least I would get manuals and clear instructions how to proceed with it. I looked to few possible replaceable models (ISO100):
* Edwards Diffstak 100/300F as it looks compatible (probably). But the price just punch me in the face, How diffusion pump - one of most simplistic pumps in design - can cost more than dryscroll?
* Agilent HS-2 (albeit still don't know the price, but I am starting to see why Edwards design costs so much - maybe price is worth it)

For ISO100 there is also Agilent VHS-4 which I think is a complete overkill as requires 1450W for heating.

Heater power comparison:
* Alcatel Crystal 102 is 500W (at least so is rated the replacement heater part; is the original the same rated?)
* Edwards Diffstak 100/300F is 650W (larger by 150W, but the power supply wires can handle that increase in demand)
* Agilent HS-2 is 450W (the pumping speed is twice smaller 160L/s than of Edwards (280L/s); but is it smaller than Alcatel Crystal?)

P.S. is Agilent HS-2 really ISO-F 100? from pictures I see that it is only 4 places for  M8 screws, but ISO-F 100 should have 8.

P.S.S.
There is other problem, 25 years is far too much for 12V battery which is used to power isolation of diff pump from the chamber and unify spect and chamber vac in case of power failure. It is 12V 125306 (vendor SAFT) 100mAh. If I measured correctly, when instrument is on-line it is charged with constant voltage of 12.8V. I am scratching my head what replacement I could drop-in. Eventually maybe I could put there something replaceable, non-chargeable with 12V and push that through diode, so it would not be charged during being on-line. I am not sure if drain from battery is cut off after isolation valve is triggered.

Other idea is to put the high density capacitor, the battery (that is why it is small) needs to be able to provide enough power for single event directly after power failure. Maybe Capacitor would have enough punch to do that. Unfortunately I have no idea about original battery (what its internal resistance, output current...) so coming up with replacement is a bit tricky.

Any advice?

[Probeman]

Other idea is to put the high density capacitor, the battery (that is why it is small) needs to be able to provide enough power for single event directly after power failure. Maybe Capacitor would have enough punch to do that. Unfortunately I have no idea about original battery (what its internal resistance, output current...) so coming up with replacement is a bit tricky.

Any advice?

I was going to respond with a suggestion to put your entire instrument on a 240v 12 KW UPS because that's what we have on our SX100 (it only runs for 15 minutes, but within a few minutes of a power outage the diesel generators will kick on and they will run for days):

https://probesoftware.com/smf/index.php?topic=1288.msg10726#msg10726

but then I saw your post here:

https://probesoftware.com/smf/index.php?topic=1288.msg10727#msg10727

So, you are doing the same thing apparently?   Why then do you need a separate power backup for your diffusion pump?

[sem-geologist]

I was going to respond with a suggestion to put your entire instrument on a 240v 12 KW UPS because that's what we have on our SX100 (it only runs for 15 minutes, but within a few minutes of a power outage the diesel generators will kick on and they will run for days):

https://probesoftware.com/smf/index.php?topic=1288.msg10726#msg10726

but then I saw your post here:

https://probesoftware.com/smf/index.php?topic=1288.msg10727#msg10727

So, you are doing the same thing apparently?   Why then do you need a separate power backup for your diffusion pump?

Without diesel generator the UPS for SX100 is useless (in diff pump prevention). We have beefy UPS supplying only to our SXFiveFE where it makes sense (and that I mentioned in those other topics). However our SX100 is not under UPS protection, and neither it would make a difference.

There are few crucial differences between SXFive(FE) and SX100 which makes UPS relevant or irrelevant.
SXFiveFE:
* has Schottky tip - can be destroyed with abrupt cutting of supply
* has column-gun motorized separation valve
* has turbomolecular pump
* uses valve(s) to separate the vacuum between chamber and spectrometers which requires pressure and energy to stay closed (loosing power will automatically open valves unify spectrometer and chamber vacuum).

SX100:
* W/LaB6
* manual column-gun valve
* diffusion pump
* motorized valve to separate or unify chamber and spectrometer vacuum directly mechanically connected with diffusion pump inlet valve - power is needed to switch it to opened or closed states.

Case with power-failure with no UPS for SXFiveFE:
0. Power down.
1. Tip gets off abruptly (for fractions of second being hot without any field containment), ion pumps releases Ar and some other lousily buried stuff - high risk of tip contamination.
2. motorized column-gun separation valve can't be closed as there is no power thus gun and chamber spaces keeps being connected.
3. turbomolecular pump starts naturally to slow down - the chamber vacuum gradually approaches primary vacuum during that... but because of point 2 UHV space of gun is getting contaminated by primary vacuum eventually.
4. valves open between spectrometer and chamber unifying the pressure
In case of P10 gas leak to spectrometer - P10 contaminates Gun chamber at that point.

Thus UPS (and corresponding software solution) for SXFiveFE is not just optional - it is a must.
See the difference with presented case with power-failure (lets say for 4 hours) and UPS present and equipped with communication and preventive measures (it is not enough to have an UPS. The UPS communication with PC and the state of UPS needs to be accessible to custom made daemon which will correctly prevent damages in correct steps or pull out from prevention sequence):
0. Power down, UPS keeps the system fully operational for 2 hours, (but Air conditioning is not under UPS, thus it is not wise to keep tip fully operational for 2 hours).
1. 30 min after power down, decrease tip heat by 0.2 A (custom script)
2. 1h - initiate full safe tip shutdown (takes about 10 min), (at older versions close the column-gun valve with corresponding script, at newest firmware it does automatically after gun is shutdown). (there is exe for gun shutdown in newest peaksight, thus can be executed with independent UPS software)
3. if it is possible trip the chiller - that will trigger preventive shutting of electronics - two biggest power consumers out - UPS got additional few hours which it can keep whole vacuum system running. As there is loss of communication with SX at that moment PC should be also triggered to shut down which will save even more battery.
After 4 hours power is back: the vacuum was kept fully functioning. After turning FEG back on in one day system is ready back to be used.

In case power would not come back - eventually all pumps would go off, but as all valves which needs to be shut and all valves which needs to be open are correct - It would take just additional few hours to return back to functional state, and again in one day system would be ready back.

In case of no UPS - it would require baking of UHV part, and would take at least 2 days to get back to functional state. Also it has some additional risks when engaging baking procedure.


Now lets look to sequence of events during power failure at SX100 (without UPS):
0. Power down
1. PC goes instantly off.
1. in about 0.5 second significant voltage drop triggers electronics to shutdown and cut off supply to other systems (SX100 is capable to go through very short sub second outages without a hitch due to most of integrated PSU's containing very huge capacitors).
2. Gun is forcibly off (still steaming hot, if it is LaB6 - that can be kind of catastrophic), diffusion pump heater is off (but oil does not cool instantly, it is still actively pumping for few seconds after), roughing pumps are off, ion pumps are off.
3. Pressure starts to raise in the spectrometers (as those have biggest leaks), however bottom of diffusion pump is still hot enough to boil the oil (and there is no cooling of diffusion pump as water chiller is off too).
4. The backing buffer is separated from roughing pump with electromagnetic valve which gets closed with power down. Thus diffusion pump has good vacuum reserve from backing side for cooling down. However inlet is in danger as pressure raises there.
5. And there comes this mentioned 100mA 12V battery, which powers single event of diff pump inlet valve closing, which prevents hot diffusion pump from interaction with gases from deteriorating vacuum in the chamber. In case of no leaks between spectrometer and chamber there is not such a big deal, but otherwise there is risk of deterioration of diffusion oil and its partial breakdown in reaction with contaminants from spectrometer (i.e. P10 gas).

To precise up what kind of battery I am blabbering here about, look to the pictures below.
The placement of 12V battery in SX100, here it is already taken out:


and this is how battery looks from closer:

marking on two lines are as follows:
10 VB 10
PC9602 REF 125306
vendor marking in background: SAFT

Now lets look how UPS with 15 min capability would help here (without diesel generator):
0. Power goes Down, UPS takes over.
1. PC can switch the Gun off. And that's all what can be achieved! (PC can also shutdown itself to prevent data-loss, albeit 15 min of LaB6 being off could make a difference for it to survive.)
2. Can't Halt vacuum (so that diff pump could cool down safely), can't do anything as all other vacuum states requires manual switching of gun-chamber valve to position 1.
3. after 15min the sequence of event is the same as in the list of events above for case without UPS... and finally it comes again to the moment where 100mA 12V battery is needed to prevent the diff pump deterioration by closing the diff pump inlet valve.
Of course if after all this, soon (~5-10min) the power will come back the valve will close automatically due to hard restart of vacuum system, thus there is a bit less of danger. The worst case (highest risks) without functional 12V battery is with long outage. So Probeman, lucky You, that Your lab/institution has well thought power backup system. At least someone at your institution thinks in advance.

Anyway we had for many years the UPS for SX100 as it helps in situations when power failure happens during work hours, but we had 3 consecutive break downs of UPS within single year, and just gave up after with UPS for SX100 (especially that it does not brings any crucial prevention).

And we are heading off-topic. 12V battery is important to prevent the diff pump oil from weak vacuum or contamination from spectrometer. The valve is complicated it is kind of double - it closes inlet of diffpump and at same time opens the inlet to spectrometers (equalizing vac between spectrometer and chamber). It is important to not deteriorate diffusion oil and probably during many last years (probe was assembled in factory in 1998) it could during some power outages.

However my main problem is Old diffusion pump itself. And as I got only single (and a bit off-topic) answer, and looking to the Cameca design I fear it is mounted for a life-time with no oil control in mind. I.e. to check the oil level the pump needs to be vented to air pressure first, and in the way how it is mounted and connected in SX100 - it looks that venting of it was never seen as any of options. Also lack of feedback means that no one probably had seen any maintenance work being done on those pumps during service visits.

 

[Probeman]

Thus UPS (and corresponding software solution) for SXFiveFE is not just optional - it is a must.
See the difference with presented case with power-failure (lets say for 4 hours) and UPS present and equipped with communication and preventive measures (it is not enough to have an UPS. The UPS communication with PC and the state of UPS needs to be accessible to custom made daemon which will correctly prevent damages in correct steps or pull out from prevention sequence):
...

I would agree that every EPMA instrument must have a UPS.   Every instrument in our lab is on a UPS because as you point out, some instruments are very sensitive!

I should also point out that it is not just the instrument that must be on the UPS. The computers, pumps, water cooler, and cryogenic compressor are all backed up by the UPS. This requires a roughly 12 KW UPS, which as I said only runs for 10 or 15 min, but it is backed up by diesel generators within a few minutes.  That way, we can monitor the instrument and UPS.

Anyway we had for many years the UPS for SX100 as it helps in situations when power failure happens during work hours, but we had 3 consecutive break downs of UPS within single year, and just gave up after with UPS for SX100 (especially that it does not brings any crucial prevention).

Sounds like a bad UPS!     

In our case the UPS units require battery replacements every 3 to 5 years, but other than that they run perfectly fine.  The other critically important consideration is that the UPS be of the "on-line" design. That is, the entire instrument and support equipment is *always* running on the batteries of the UPS, which are constantly being re-charged.

That way, when there is a power outage (and we have about one power outage a year), there is not even a millisec of power interruption.

Sorry if this a bit off-topic, but I just thought I should clarify that not all UPS units are the same.

[sem-geologist]

Sounds like a bad UPS!     

In our case the UPS units require battery replacements every 3 to 5 years, but other than that they run perfectly fine.  The other critically important consideration is that the UPS be of the "on-line" design. That is, the entire instrument and support equipment is *always* running on the batteries of the UPS, which are constantly being re-charged.

That way, when there is a power outage (and we have about one power outage a year), there is not even a millisec of power interruption.

Sorry if this a bit off-topic, but I just thought I should clarify that not all UPS units are the same.

Yes it was on-line type of UPS with generation of clear single phase sinusoid from 3-phase input. We have exact same model (EATON) powering our two high resolution Zeiss FESEM and works without a hitch so far. I could not find what had caused these marvelous burn down of UPS (I was at site during last failure, it happened during power shortage) then, neither could UPS service do that (scratched the heads). However, as I had to educate myself a lot about lightning and ground protection, as new microscope (ThermoFisher) is being installed, and its requirements were bizzare....
  So I think at last I know (have the most plausible explanation) what could bake them. Burned EATON (for SX100) and not burned Liebert (for SXFiveFE) are powered by the same 3 phase power line. The difference is that Liebert UPS has over-voltage protection installed before (at UPS input) and after UPS (at output), where EATON UPS had not. While output P and N is isolated from input P and N at UPS, the PE line is directly connected through both sides, so equalization of potentials in case of voltage surge will transfer to voltage change on PE at input, which also will transfer at UPS output, and UPS output side gets kicked in the guts because is not protected. The EATON UPS by our ZEISS SEM has such (both side) protection installed, and thus had not suffered through all these events. Another story is the protective earth, where intentionally designed for probes TT is unintentionally connected with TN-S... That is the price of having lab in old building where regulations, designing and execution of it take places in three different disconnected realities  .

But What I am point out is that old SX100 due to manual Gun valve, has no way to protect itself while being on UPS from aftermatch, when UPS battery is drained down if power does not return in those 15 minutes back. Without this Cameca initially installed functional 12V battery, even if there is UPS, the diffusion pump is left unprotected. If gun valve would be motorized - then valve could be closed and vacuum "halt" executed (which would with grace power down diffusion pump, protect its inlet and let it cool down with help of circulating water). But because it is manual, even with UPS present, when its battery gets low and powering down system is happening anyway - UPS just delays problematic event and does not eliminate it. Yes it reduce risks for power blackout events taking shorter than UPS battery capacity, but if event is longer - the outcome is the same. We probably will get UPS installed again for our SX100 as the cause of burning looks plausible and there is a promising way to eliminate the cause.

Anyway, getting back to the topic of Diffusion pump.

Anyone? Had you seen Diffusion oil pump being inspected, oil level checked of Your SX100 (of course we are excluding this question from these who had Turbo pump installed instead of diffusion). Today I had dig through pile of piles of old paper scraps collected till 2015 from 2000, and found out no record that there had been any inspection of diffusion pump. Albeit I found out that heating element was replaced at least twice while SX100 was still on warranty. I think the inspection of oil level is very tricky as physically design of SX100 stands very much in the way to conduct such inspection. AFAI could get from other vendor manuals the Diffusion pump oil inlet (where stick for oil level measurement should be attached) should not be opened while pump is under vacuum, and venting of pump to atmospheric pressure is required. It is however hard to do on SX100, there is no procedure, there is no nitrogen inlet where primary pump valve to vacuum buffer is placed (contrary to primary vacuum valve for spects and chamber). Opening one of pipe junctions could do the trick of venting the diffusion pump, but that will suck the dust which is nearly impossible to get rid completely in that space of probe. Any such inspection would provide more risks to the diffusion oil than good.

Another thing which I found in papers is that this secondary pening gauge worked erratically many times. Maybe this time is again the same problem reapearing. Maybe before dismantling Diffusion pump I should borrow some three-way ISO25 junction and mount another gauge to see if Cameca gauge is not giving wrong reading.

[sem-geologist]

At installation list from 2000 I see position with dibutyl phtalate with number 500 (ml?). Has anyone idea what it was used for? I just hope not as diffusion pump oil... As I see no positions with diffusion pump oil, I wonder if pump was drained during transport, or it was filled at factory (I tend to believe it was filled at factory as access for filling and draining diffusion pump in SX100 is so hindered).

1. Checked the heater resistance which is 96 ohm; that at AC 230V translates to ~520W - thus heater is rather OK (heater going bad should increase the resistance). It is also tightly attached to the pump.

[Probeman]

Yes it was on-line type of UPS with generation of clear single phase sinusoid from 3-phase input. We have exact same model (EATON) powering our two high resolution Zeiss FESEM and works without a hitch so far. I could not find what had caused these marvelous burn down of UPS (I was at site during last failure, it happened during power shortage) then, neither could UPS service do that (scratched the heads). However, as I had to educate myself a lot about lightning and ground protection, as new microscope (ThermoFisher) is being installed, and its requirements were bizzare....

Could have been a grounding problem...

But What I am point out is that old SX100 due to manual Gun valve, has no way to protect itself while being on UPS from aftermatch, when UPS battery is drained down if power does not return in those 15 minutes back. Without this Cameca initially installed functional 12V battery, even if there is UPS, the diffusion pump is left unprotected. If gun valve would be motorized - then valve could be closed and vacuum "halt" executed (which would with grace power down diffusion pump, protect its inlet and let it cool down with help of circulating water). But because it is manual, even with UPS present, when its battery gets low and powering down system is happening anyway - UPS just delays problematic event and does not eliminate it. Yes it reduce risks for power blackout events taking shorter than UPS battery capacity, but if event is longer - the outcome is the same. We probably will get UPS installed again for our SX100 as the cause of burning looks plausible and there is a promising way to eliminate the cause.

Yes, that is exactly why we have our instruments on a "on-line" UPS *plus* diesel backup power after several minutes of a power outage!  A few winters ago the entire campus went down for days, and our lab instruments continued to run just fine the entire time.

[Probeman]

At installation list from 2000 I see position with dibutyl phtalate with number 500 (ml?). Has anyone idea what it was used for? I just hope not as diffusion pump oil... As I see no positions with diffusion pump oil, I wonder if pump was drained during transport, or it was filled at factory (I tend to believe it was filled at factory as access for filling and draining diffusion pump in SX100 is so hindered).

I think dibutyl phtalate is the fluid used for the P-10 gas "bubblers...

[Probeman]

Our instrument engineer provides this advice:

500mL sounds right for that medium sized diffusion pump. Cameca loads them with Santovac 5.

You should warm the pump up on a hot plate to 45C before draining old oil so that oil viscosity lowers enough to drain properly. You should drain into a beaker so as to measure what volume of oil charge they were starting with. If the oil is slightly yellow (not clear as when it was new), that's still OK. If draining via the downward pointing fitting, you'll  have to replace that O-ring since it will be hardened and no longer seal properly. Have to scrape out (with wood or plastic, never metal tools)  old O-ring remnants that  peeled off of old O-ring.

I wouldn't worry about checking condition of Santovac 5 for the life of your instrument since its tuff stuff. If, however, the pump was running with no cooling, it may have been ruined. Note that should you try to measure the water flowing through the DP's cooling lines, don't expect more than a trickle! That long tube really presents a lot of flow resistance. If the upper ½ of the pump body is slightly below room temperature, you're OK.

Note that if Santovac 5 is too expensive for your taste, you'll have to very thoroughly clean out all traces of Santovac 5 from the pump before switching to a silicone-based fluid.

Steve

[JonF]

Another thing which I found in papers is that this secondary pening gauge worked erratically many times. Maybe this time is again the same problem reapearing. Maybe before dismantling Diffusion pump I should borrow some three-way ISO25 junction and mount another gauge to see if Cameca gauge is not giving wrong reading.

We have the turbo on our SX100, but to quote the person I learnt from: "Never trust the gauge!".

Adding a T piece and another penning is relatively trivial, so I'd go that route first. The penning can get contaminated and cause all sorts of chaos, whereas the diff pump is pretty simple and just... works. Especially if its hot at the bottom, as you say in your first post.
Assuming you can drop the penning as easily on a diff pump system as you can on a turbo (albeit needing to be a contortionist to get your arm in), and that the gauge is one of the Alcatel types, you can open them up easily enough to check it isn't massively contaminated. You can buy rebuild kits for the Alcatel gauges, or you can just clean up what you have and then reseal with fresh indium wire. Cameca usually just swap the whole gauge out.

What error are you actually getting from the instrument? Bad vacuum in the chamber?

[sem-geologist]

Probeman and JonF,
I really appreciate Your answers and effort with consulting with your engineers. Please, give my thanks to him.

Our instrument engineer provides this advice:

500mL sounds right for that medium sized diffusion pump. Cameca loads them with Santovac 5.

You should warm the pump up on a hot plate to 45C before draining old oil so that oil viscosity lowers enough to drain properly. You should drain into a beaker so as to measure what volume of oil charge they were starting with. If the oil is slightly yellow (not clear as when it was new), that's still OK. If draining via the downward pointing fitting, you'll  have to replace that O-ring since it will be hardened and no longer seal properly. Have to scrape out (with wood or plastic, never metal tools)  old O-ring remnants that  peeled off of old O-ring.

I wouldn't worry about checking condition of Santovac 5 for the life of your instrument since its tuff stuff. If, however, the pump was running with no cooling, it may have been ruined. Note that should you try to measure the water flowing through the DP's cooling lines, don't expect more than a trickle! That long tube really presents a lot of flow resistance. If the upper ½ of the pump body is slightly below room temperature, you're OK.

Note that if Santovac 5 is too expensive for your taste, you'll have to very thoroughly clean out all traces of Santovac 5 from the pump before switching to a silicone-based fluid.

Steve

so listening through all this, indeed, I believe he had done some maintenance of DP - That means that Diff pump on SX100 is not so fool-proof?
I am quite surprised with amount of 500mL oil, for example similarly sized pump of Edwards takes only 125mL, thus it is quite surprising for me.
As for my concern of its (oil) condition, it is due to instrument running on our site for 23 years (and there is no documented event of DP  inspection), and as most of QA (quality assessment) stickers on our SX100 are from 98, I have reason to believe that it was running in the factory before it landed to our lab already two years, thus 25 years in total. Reading description of Santovac 5 there is mentioned that shelf life of oil is 5 to 7 years. How can it be better protected while working in the diff pump, where it actively goes through endless thermal cycles and entraps and releases other gas molecules through pumping process, compared to cool staying in the sealed bottle on the shelf. It is quite counter intuitive to me. Yes, I am aware about old O-rings - that is why I am reluctant opening those without thinking and consulting it over. As for Water flow, there is ratemeter mounted on the SX100, if water would flow too slow, there is logical means built-in at SX100 which would switch off electronics and vacuum system. Also water supply is connected in series between consuming parts (HV-control board, Scanning and mag power boards, condenser lenses, diffusion pump), not parallel, thus blocking the flow in the pump would stop the flow, or kill the pump in the water chiller (albeit we need to replace the pump every 4 years). It is quite cool at the top of DP when running. Also we have water filter, and replace distilled water every half a year.

Another thing which I found in papers is that this secondary pening gauge worked erratically many times. Maybe this time is again the same problem reapearing. Maybe before dismantling Diffusion pump I should borrow some three-way ISO25 junction and mount another gauge to see if Cameca gauge is not giving wrong reading.

We have the turbo on our SX100, but to quote the person I learnt from: "Never trust the gauge!".

Adding a T piece and another penning is relatively trivial, so I'd go that route first. The penning can get contaminated and cause all sorts of chaos, whereas the diff pump is pretty simple and just... works. Especially if its hot at the bottom, as you say in your first post.
Assuming you can drop the penning as easily on a diff pump system as you can on a turbo (albeit needing to be a contortionist to get your arm in), and that the gauge is one of the Alcatel types, you can open them up easily enough to check it isn't massively contaminated. You can buy rebuild kits for the Alcatel gauges, or you can just clean up what you have and then reseal with fresh indium wire. Cameca usually just swap the whole gauge out.

What error are you actually getting from the instrument? Bad vacuum in the chamber?

Yes, I don't trust the Gauge - in particular that one. In the beginning I thought of borrowing one, but after digging through documented history of the machine yesterday, I find that it is probably most failing thing in SX100. Thus I think better idea would be to mount T junction and another independent vacuum detector permanently. This will save headache in the future and allow rapidly troubleshoot the vacuum. What could happen, the isolation in concentric cable between gauge and Cameca self-built controller could get fatigue due to age. The leak in the cable could tricks the controller to think that there is higher current flowing inside the gauge thus vacuum is worse. Contamination can be rather discarded as it is quite freshly refurbished gauge (few months). I found out from historical records that cable was already changed few times.

The error is Vacuum error, and green light at the back of vacuum box for chamber vac is not lighting up. It is not reaching the safe threshold of x*E-3 Pa, but sticks about 1.5E-3 Pa (x, as I dont remember exact value). As I am writing now, this hypothesis starts to make sense as if triggering and returning after sample exchange the vac for a brief moment (a second or two) gets down to 3E-3Pa, but then falls to 1.5E-3. As I think it is probably easily to check with some electrician tools. The resistance between axial and outer parts of the cable.

[Probeman]

so listening through all this, indeed, I believe he had done some maintenance of DP - That means that Diff pump on SX100 is not so fool-proof?

Yeah, well, there was an "incident" about 10 years ago during a routine maintenance when a student helper pulled the hose off the backing pump before the sample chamber was vented...  and then we had DP fluid spurting up into the sample chamber which coated the light optics and column, so the entire instrument had to be disassembled and cleaned.   

So, yes, not exactly "fool proof"!

[sem-geologist]

so listening through all this, indeed, I believe he had done some maintenance of DP - That means that Diff pump on SX100 is not so fool-proof?

Yeah, well, there was an "incident" about 10 years ago during a routine maintenance when a student helper pulled the hose off the backing pump before the sample chamber was vented...  and then we had DP fluid spurting up into the sample chamber which coated the light optics and column, so the entire instrument had to be disassembled and cleaned.   

So, yes, not exactly "fool proof"!

This is very important information concluding and letting to draw whole picture. So I was wondering why it would use 500mL of oil while other pumps of similar size use 125-150mL. The pump in this SX100 configuration is kind of boobytrap, and opening it is out of equation, as clearly outlet (backing) pipe is partly under oil. Other vendor pumps are recommended to be vented from the side of outlet backing, as also are recommended to do that while pump is still warm. Looks that following such borrowed recommendations from other vendors would lead straight to catastrophe.  For the same reason as described in the incident, opening filling or draining is forbidden while pump is under vacuum. (the drain/fill pipe is always completely drowned under oil). It is much more plausible that in my case the High voltage cable of gauge got fatigued and needs to be replaced. And unless I can prove the vacuum is bad (i.e. with help of other independent gauge mounted on T-junction) I should not do anything with diffusion pump. It explains why there is no records for our machine for any service of diff pump, or why no one else ever had seen (with exception to that single catastrophic case) Cameca service doing any inspection of diffusion pump.

[Jacob]

I'd be very wary of the 500mL number. That's very far outside of my experience of diffusion pumps of this size. A more typical value would definitely be 100-150mL. The 500mL number might be the amount of something that was ordered, however.

One of the reasons that the pump oil level has been ignored is that santovac is a much more robust diffusion pump fluid than the earlier formulations. Silicone fluids tended to break down and cake inside the pumps where as the santovac (polyphenyl ether) tended to crack into lighter components which were then pumped away. I've opened pumps that have been running for decades continuously and the fluid is perfect condition.

The largest problem I've found with diffusion pumps using santovac has been backstreaming of foreline pump oil which then mixes with the diffusion pump oil.

[sem-geologist]

We have the turbo on our SX100, but to quote the person I learnt from: "Never trust the gauge!".

Yes, I started to understand such narrative. But is the situation still the same as decades ago and can't we have anything better?

At first, Alcatel Penning gauge (gauge installed on early models of SX100) is indeed hard to trust - there is whole pipeline of things where stuff can go wrong... I dig up through piles of old emails (my late predecessor had printed all communications and emails, which I could look through and learn the whole history of repairs of our SX100), and learned that indeed the problems with vacuum measurements were re-emerging again and again and again... Not only gauge but also cable was changed many times. Investigation of complete construction had revealed major weaknesses of this type of gauge.

1) High Voltage is sent through coaxial cable. That is about 4kV which is constant. If dielectric material weakens somewhere in the cable it start to leak  and generates some current at HV source side. The whole vacuum measuring is based on precise current measurement through shunting resistor at returning path of HV generator, so such additional cable leakage current will sum up with current generated at gauge and worsen the displayed value of vacuum. But, in case of a huge leak in cable there will be no possibility to start penning gauge at all, as there will be significant voltage drop at gauge (leakage in cable will form something like voltage divider). Even if it will be clean gauge - bad cable will hinder its functioning. As for alcatel gauge cables - these are no more produced - all available at second hand shops are of questionable quality! Making new cable (DIY) from scratch is also hindered by requirement of HV reliable cable terminations (tools, expertise), and proper cable availability on the market (There are cables more think, and expensive).

2) Penning gauge, while being in simple construction, can give easily wrong idea about vacuum when not striking. Its lack of current (no strike event) could be compensated with small leak and final reading would look as good properly working gauge value. Experienced user knows that reading is wrong - but only if user tracks vacuum readings closely after the moment the vacuum valve is opened, and tracks the vacuum value response to that event. If vacuum suddenly just goes to 5E-5Pa from very rough vacuum - that is clearly unrealistic. However, if user leaves pumping unattended, and gets back to instrument after 10 min (and cable is leaky a bit), he can see the i.e. 4E-4Pa, even if gauge had not strike. Also Penning gauge needs constant high voltage, that have tendency to deteriorate the weakened isolation material in the cable and progress the problem(s).

Thus I came up with initiative, to make a replacement card which would interface modern gauges to SX100 (giving-up alcatel penning). That would be not jumping one tier in technology (like SXFive's used Agilent IMG) but two tiers. The next technological tier, IMG (inversed magnetron gauge), is more advantageous compared to Penning, but still requires the high voltage cabling and its correct termination and custom made current sensing circuits are difficult to do properly. Interfacing stock controlers of gauges is also hard as it would require changing firmware and finding some serial port on Cameca SX100 vacuum logical board. (That is rather completely out of equation).
Inside SX100 vacuum supply box, there is small board interfacing Alcatel gauge. It produces HV supply, measures the return current and sends the calculated vacuum as log10(P) as DC signal from 0 to 10V (10V roughting, 0V -ultimate vac) through coaxial cable to Vacuum logical board.

Thus naturally my attention got stolen by active gauges, which is higher tier than normal IMG. Active gauges work with low voltage cables, and produce all required HV for integrated IMG inside the gauges integrated electronics and send out only low voltage 0-10V signals (see the pattern?). Such gauges are also flexible as they can be powered from 14.5 to 40V, and such supply is already present in the Vacuum supply box. Also it looks that few different vendors have very similar connection based on 8C8P (or FCC68, or well known RJ45 ethernet cable/sockets), with same basic pin configuration: at least I found out few such gauges from Edwards, Curt J. Lesker and Inficon with such connections.

But what are advantages?
1) No HV cable. Basically simple Ethernet cable will do the job, as only 2W +15V is required to power such gauge, and signal is 0 to +10V. Replacement cable can be easily made or bought. (but it should not age at all, differently to high voltage cables).
2) These active gauges have builtin microcontrollers and LED(s) which show(s) status of gauge (i.e. if strike was successful) - the meaning of vacuum reading is clear. Situation then strike failed, and we have low current and very low vacuum reading is easily to catch. Also Active Gauges reduce high voltage to minimum after strike - this means less contamination of gauge - longer exploitation time between need of cleaning.
3) Edwards gauges (i.e. AIM200) has special multi-strike geometries which guaranties striking even in dirty environments.

How it will work?
My idea is to translate the voltage from such modern gauge to corresponding expected DC voltage by logical vacuum board of SX100. Basically my design is few OPAMPS for DC offset, scale and clip (values out of expected range). Two potentiometers will allow to easily recalibrate board to different kind of gauges from different vendor. It would be possible even to use WRG from Edwards (albeit pirani range would not be seen). The idea is to have seamless replacement (no firmware or software changes).

So how many people would be interested in such design?

I have schematics alreadynearly finished (see the attachement), and now I am at stage of PCB design.
I won't produce and sell the boards, the EU laws and handicaps are too enormous for me (Basically RoHS3 forbids me to make and sell reliable electronics). But I could share the design, gerber files, bill of materials and notes for proper assemblage at site overcoming RoHS hindrances. PCB when having these is possible to order in most of the world. Buying parts and assembling them on PCB should be not too difficult, board is designed to be hand solder-able (no surface mounted parts, everything "though hole" parts). Albeit design could be updated with surface mounted parts if someone would decide to produce and sell larger batch of such boards.