PeakSight 6.2 and DEP (Data Execution Prevention)

[UofO EPMA Lab]

We just installed the Cameca PeakSight 6.2 on a new Windows 7 64 bit computer.  The installation was quite automatic and went well.

The only issue was that we have a "custom" Bragg crystal (PC25 which is optimized for boron) on spectrometer 4, from when we bought the instrument in 2006, that was not in Cameca's crystal database, so Edgar is helping to get that sorted out.  Oh and we stupidly forget to note which sub systems (stage, wds and scanning boards) were "old" vs. "new", so we had to boot up to old Cameca XP computer for a moment to get that sorted out!   

In the PeakSight installation instructions most of the manual operating system configuration issues dealt with changing the Windows automatic updates to "download but let me decide when to install them", and modifying the Power Saving feature to never go to sleep (for overnight runs of course).  This is entirely reasonable and one does the same mods for the Probe for EPMA computer.

Now on the hardware side of things we simply bought a Dell Precision Workstation (T5810)   We also had to buy a Falcon Plus video card and some additional network cards as we also wanted a WWW connection and a network connection for our UPS monitoring software (Liebert MultiLink). 

Now that brings me to our questions. The PeakSight installation instructions are saying we should do a couple more things that we are not sure we are comfortable with on an Internet connected computer. First it wants us to turn off DEP (Data Execution Prevention).

A good explanation of DEP is found here, but DEP basically tries to protect the processor against various malware and viruses:

https://blogs.technet.microsoft.com/askperf/2008/06/17/to-dep-or-not-to-dep/

For the moment we have not done this (though we do have anti-virus software installed), but we think we are seeing a problem in PeakSight where it appears to lock up the computer completely (mouse, keyword and even <ctrl> <alt> <del>) after running about 10 or 20 minutes or so.  After this we can't even get the VNC connection to work from another computer.  And there is no error message on the screen, just nothing on the screen responds any more.

However, Probe for EPMA on the other computer still continues to run just fine and keeps on acquiring data normally, so the instrument itself is not affected by this at all. But the PeakSight computer is totally frozen and we have to hold down the power button and reboot it.  We did a test last night where we left Windows running but closed PeakSight and this morning everything is still fine.  So we don't think it's the computer hardware or OS. And we will try again, but we suspect PeakSight will lock up again, and so we guess we will have to turn off DEP in some manner.

But here's the thing: the Cameca instructions want us to issue the following instructions:



But this appears to us to turn off DEP for all apps and services.  However, in the Microsoft tech link above it offers the following suggestion to turn off DEP only for a specific application:



This seems much more reasonable to us. Has anybody dealt with these DEP issues with regard to PeakSight?  We know that PeakSight consists of a number of apps that run and communicate with each other as seen here:



We could perform some experiments, but does any one know exactly what PeakSight app(s) we need to turn off DEP for? Should we turn DEP off for all PeakSight apps?   We would really only like to turn off DEP for the PeakSight app(s) that are required and leave DEP running for the OS and other apps.

On a related note the PeakSight instructions also ask that we change the User Account Settings to "Never be notified about changes to your computer", which Microsoft says "Is not recommended".

Again, does anyone have experience with this account setting and is it dangerous for Internet connected computers?

Any suggestions?

[UofO EPMA Lab]

Ok, we may have been successful using the DEP GUI to turn off DEP for the Cameca PeakSight apps. PeakSight now seems to be running fine...

But let me provide some additional info, because the Microsoft link I posted previously didn't provide much details, so here is what we did on our Win 7 system:
 
1. Right click on My Computer or the Computer icon in Explorer and select Properties. This opens the Control Panel | All Control Panel Items | System window.

2. Next click the Advanced system settings link in the upper left. That opens up the System Properties window.

3. Next click on the Advanced tab and click the Settings button. That opens the Performance Options dialog. This is the window shown in the Microsoft link above.

4. Now click the Data Execution Prevention tab. You will see the DEP GUI. We had originally thought we would use the "Turn on DEP for all programs and services except those I select" option, but Windows informed us that most of the PeakSight apps could not have DEP turned off. So instead we used the option "Turn on DEP for essential Windows programs and services only" and that seems to have done the trick.



So far PeakSight has been running several hours and everything looks happy. We'll let it run over night and see what we see tomorrow.

With what's been going on with the Internet lately one can't be too careful.  Good thing we're not responsible for voting machines in Ohio! 

[BenjaminWade]

Hi John
This is very interesting to me for a couple of reasons. Last year I got exactly the same Dell computer upgrade from Cameca, and an engineer from Paris installed the new computer while he was fixing a multitude of other problems I was having at the time. I am assuming then he just disabled DEP entirely. I guess this could potentially be a problem and I need to follow your fix. But its not an outstanding issue for me at the moment due to the second reason this thread is interesting...

A day after installing the new computer with Peaksite 6.2 he had to put it back in its box and resintall the old computer! No matter what he tried he couldn't get EDS mapping in Esprit 2.1 to work with PeakSite 6.2 on the SXFive. All EDS maps just come back with zero counts. The factory had apparently heard of this problem in one other installation and at the time there was no resolution. I needed the probe working so I left it at that for the moment.

So you are running PeakSite 6.2 and Esprit 2.1.1? (my build is 2.1.1.16854). If so I am then assuming that your EDS mapping acquisition in PeakSite is OK?

Cheers

[UofO EPMA Lab]

Hi Ben,
I'm really sorry but I'm afraid I can't help much as I have a Thermo Pathfinder system on our Sx100 which is integrated with our Probe for EPMA software.

But you probably should start a new topic on this issue on this Cameca board as someone else will probably have more information for you on this.

[UofO EPMA Lab]

Well it looks like we are back to square one.

PeakSight ran for about 6 hours or so, but then locked up Windows completely again. So I guess we have no choice but to turn off DEP completely on the Cameca computer using the command suggested by the Cameca documentation:



Dang, too bad.  We were hoping to leave DEP on for better protection from malware but apparently that won't be possible with PeakSight.   I hope our anti-virus software is up to the task.

But it makes one wonder what PeakSight is doing that makes Windows think it's being a naughty program...   

[UofO EPMA Lab]

Ok, I guess we're back to square zero...  maybe we should first ask if anyone is successfully running PeakSight 6.2 on Windows 7 64?

We thought when we turned off DEP using the command prompt above that Cameca documents in the installation instructions, that it would solve the problem with PeakSight locking up the computer.  But within a few hours of PeakSight running, the Cameca computer was totally locked up again, keyboard, mouse, everything. Power reset again.

So we went ahead and implemented all the UAC and RegisterBase mods that the installation document specifies, but yet again, after a few hours of running PeakSight, the Cameca computer is totally locked up, requiring a power reset to re-boot the computer.

Just as a sanity check we left Windows running overnight *without* PeakSight running, and this morning the Cameca computer is totally happy and humming along.  So it appears the problem only occurs when PeakSight is running for more than a few hours.  By the way, we are running PeakSight in an administrative login until this issue is solved.

Has anyone else seen anything like this?  Does anyone have any suggestions on how to prevent PeakSight 6.2 from locking up the computer?  We don't know what else to try...  for example is there a subset of PeakSight apps that we can run to try and narrow down the issue?

[jrminter]

Have you tried clearing the system logs, letting PeakSight run to crash and looking at the errors? Might point you in an unanticpated direction.

[UofO EPMA Lab]

Have you tried clearing the system logs, letting PeakSight run to crash and looking at the errors? Might point you in an unanticpated direction.

That's a great idea.  I assume the log info will be retained even after a power switch reset?   Edgar Chavez is taking a look too via VNC, but I will certainly try that Monday if it's still happening then (I've already been in the lab today- Saturday(!), but had to go in to give an Intel Science Fair winners tour, so it was in a good cause).   

[neko]

Have you tried turning off the A/V software? Or at the very least, whitelisting cameca programs from being run through their run-time virus scanning? I've had problems with that in the past, especially with uncommon software - and that was only accounting industry uncommon, not unheard-of uncommon like PeakSight is outside of our labs.

[UofO EPMA Lab]

Have you tried turning off the A/V software? Or at the very least, whitelisting cameca programs from being run through their run-time virus scanning? I've had problems with that in the past, especially with uncommon software - and that was only accounting industry uncommon, not unheard-of uncommon like PeakSight is outside of our labs.

Hi Nick,
That's another good idea.  We've never had a problem with anti-virus with the PeakSight 4.2, but of course this is completely re-written product from Cameca, so maybe. Monday we will disconnect from the Internet and try another test with the anti-virus disabled.

We're just surprised that this latest PeakSight requires so many modifications to the Window system in order to run...  why would all these DEP, UAC mods and registry editing be required?  The Internet is dangerous enough without disabling all these defensive measures on a computer.

[neko]

Have you monitored the computer "vitals" while PeakSight is running? Processor/Memory usage and temperature/voltage monitors?

We recently had a problem where our computer just bluescreened after booting into windows. The fan on one of the two video cards had died again. After I got the new fan assembly and installed it, I got some monitoring software and sure enough, one video card is ~5˚C hotter than the other, the only place it can go is right next to the raid card. I bought a spare since the lifespan is about 2 years. 

[UofO EPMA Lab]

Have you monitored the computer "vitals" while PeakSight is running? Processor/Memory usage and temperature/voltage monitors?

We recently had a problem where our computer just bluescreened after booting into windows. The fan on one of the two video cards had died again. After I got the new fan assembly and installed it, I got some monitoring software and sure enough, one video card is ~5˚C hotter than the other, the only place it can go is right next to the raid card. I bought a spare since the lifespan is about 2 years. 

We wish it were just the computer, but the PC runs fine for days as long as PeakSight isn't running, so we're pretty sure it's PeakSight making Windows angry.  But we'll will watch the Window performance monitor next time we try running PeakSight in case it's a memory leak issue somehow.  We wish we knew if anyone else is running PeakSight 6.2 and if they had seen any of these issues.  It's a very recent release.

We had to replace the motherboard twice on the Dell Optiplex PC that Cameca originally provided.  It's those cheap capacitors that "pop" after a few years of running continuously. Elsewhere in the lab we always buy Dell Precision Workstations as they use tantalum caps that never fail.  So PeakSight is running on a new Dell Precision Workstation that we bought ourselves along with the Falcon Plus video frame grabber.

Blows our mind that Dell would try to save a few pennies on cheap capacitors and have it fail after only 4 or 5 years.  But the Precision Workstations seem to run forever.

[neko]

PeakSight could possibly be locking up threads over time until it builds into a full system lock. I can't remember if it's only in the server version of windows or not, but there's a program that will log performance to disk so you can track it up to the point of crash without having to sit there watching it (but I guess if it locks up with the static image, you'll see the last thing the monitor had to show you anyway.

 It is *very* strange that changing the DEP settings would cause it to crash slower... but it *is* windows and they've done some amazing work on making things...unique. The results program in PS4 definitely has some fun memory leaks, but it was built so long ago that it can't use enough memory on a modern computer to lock up anything else before it goes haywire

Have you tried pulling out the framegrabber (or starting up the framegrabber testing software so PeakSight starts up but doesn't launch the framegrabber software)? In case it's Matrox or whatever 3rd party software they use nowdays causing the issue. Also maybe pull the secondary network card, try turning down graphics or running on the built-in video as a test.

Do they recommend Windows 10 for 6? I think on our quote they mentioned a PC running Windows 10... which... no thank you, but that might cause backwards compatibility problems if it was designed with 10 in mind. Not that you want to, but you could try a second install with 10 and see if that changes any behavior. You could also have Cameca give you a manifest of what patches are and aren't installed on their test machines ("It works for me!") so you could tune your system to be exactly the same (assuming they're also running it on 7). If there are any Spectre/Meltdown mitigations you have installed that they haven't tested out yet, that could possibly cause some weird execution problems (I'm not sure how much high level programmers see of the predictive stuff, but I'm going to assume that there's some assembly somewhere in this and it's probably highly unique...).

Bad Dell caps have been a problem for like 15 years now, and I think they've been sued over it at least once as a class-action. I thought it was an exaggeration that they bought a billion of them but I guess not. I also love how Dell charges 200% more than top of the line samsungs for unbranded SSDs. If an optiplex is what was $2000 on our upgrade quote, cameca is dreaming. Our $60 spare precision T7400 is way better than any brand new optiplex, and it was built when Vista was out.

[UofO EPMA Lab]

Hi Nick,
We don't think changing the DEP settings affected anything really.  It probably would have been fine if we had just used DEP option from the Advanced Settings GUI to turn off DEP for all apps except Windows and essential services.

Edgar wrote and said that he thinks the problem is that there's a patched DLL for Aphelion on the PeakSight DVD that needs to be copied over manually to the Cameca program folder.  Unfortunately the Cameca docs didn't mention this!

The freeze of PC come from an update missing for Aphelion. The file Adcis_.Driver.Falcon.dll on sxpc-V6.2\Patches folder should be copied into C:\Program Files (x86)\Cameca SX\bin\Aphelion431.NET folder.

So we just copied it over this morning and now we're running PeakSight.  Hopefully this is good to go!

Once this is all working we might do some more research on how to restore the DEP for "Windows and essential services".  Since we issued the command prompt command line suggested in the Cameca docs, that DEP GUI window says we now have to use the bcdedit.exe  app to change these settings and suggests that we search the Microsoft KnowledgeBase for how to do that.

[BenjaminWade]

Hi John
Ah right yes sorry forgot you have a Thermo. I got a response back from Cameca and apparently its to do with Windows 10 , the BIOS on the Dell PC and something about the HDD...

Anyway, I am currently running Windows 7 Professional 64-bit and Peaksite 6.2 successfully with no problems. The service engineer did the upgrade to Peaksite 6.2 on my old Dell PC to use while they were figuring out what is going on with the new upgrade.  I assume he just turned DEP off for everything...

It looks like you are perhaps all running OK now though, but let me know if there is anything you want me to check on our current Win7/PeakSite6.2 PC.

Cheers