So yea, it is the SIEM. It is a really slow leak but my get-process dump over time pointed it out.
From: [email protected] [mailto:[email protected]] On Behalf Of Kennedy, Jim Sent: Monday, October 16, 2017 3:08 PM To: ntsysadm Subject: RE: [NTSysADM] RE: 2008 R2 Hyper V guests OoM I have a SIEM on each of them. The vendor is trustworthy, no reports of anyone else having this issue and the agent upgrades don’t coincide with this happening. Although an upgrade to Windows could certainly impact it. There was an upgrade to the SCOM agent that does line up pretty good with when this started. But you would think the world would be screaming if that were the case. I disabled the SCOM agent on all the 2008 R2 boxes for now. So far it has been fine, but still a tad too soon to blame that. From: [email protected] [mailto:[email protected]] On Behalf Of Andrew S. Baker Sent: Monday, October 16, 2017 2:54 PM To: ntsysadm Subject: Re: [NTSysADM] RE: 2008 R2 Hyper V guests OoM I was thinking antimalware myself. In fact, antimalware, some other agent software, and malware, are the three things that come to mind for this scenario -- especially if the devices experiencing the problem are not logged on to the console. Regards, ASB https://about.me/Andrew.S.Baker Providing CyberSecurity and IT Operations Consulting for the SMB market… GPG: 860D 40A1 4DA5 3AE1 B052 8F9F 07A1 F9D6 A549 8842 On Thu, Oct 12, 2017 at 6:50 PM, Richard Stovall <[email protected]<mailto:[email protected]>> wrote: I seem to remember Vipre causing that occasionally, in its early incarnations. On Tue, Oct 10, 2017 at 10:12 AM, Kennedy, Jim <[email protected]<mailto:[email protected]>> wrote: Still having this issue, and it has spread to many of my 2008 R2 servers including non hyper V guests. They all start with this: The server was unable to allocate from the system nonpaged pool because the pool was empty. Full on hangs, so I can’t get in to see what ate the memory. Not seeing anything in real time looking like too many handles. Any ideas here gang? From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Kennedy, Jim Sent: Monday, September 11, 2017 1:25 PM To: [email protected]<mailto:[email protected]> Subject: [NTSysADM] RE: 2008 R2 Hyper V guests OoM So yea, that is exactly what I did. TYVM sir. From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Michael B. Smith Sent: Monday, September 11, 2017 12:59 PM To: [email protected]<mailto:[email protected]> Subject: [NTSysADM] RE: 2008 R2 Hyper V guests OoM Don’t run overcommitted in production. From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Kennedy, Jim Sent: Monday, September 11, 2017 12:20 PM To: '[email protected]<mailto:[email protected]>' Subject: [NTSysADM] 2008 R2 Hyper V guests OoM Just started a couple of weeks ago. I suspect an August update so I may cross post this later over on Patch Management. 2012 R2 Hyper V host (2 of them) with a mixture of 2008 R2 and 2012 R2 guests. Only the 2008 R2’s are exhibiting this behavior, they are all low usage machines. They are all set to dynamic memory and have been running for years without issue. One is only an FTP server that accepts 4 connections a night for an automated data transfer. And the incoming connections are IP restricted on our ASA, so it isn’t like it is getting flooded with hacking attempts. These boxes are varied in their use FTP, internal only web server, RDP Gateway, generic file server…… They crash shortly after a 2019 from srv. “The server was unable to allocate from the system nonpaged pool because the pool was empty.” Setting them to a fixed memory on the slightly larger than what I would expect them to need seems to have fixed it. Any other ideas?
