N54L ESX rebooting when using additional NIC

[Raggeh]

Hello all.  I have an odd issue concerning an N54L Microserver, ESX 6 and an HP NC360T NIC.

 

I have 2 identical N54L's which were both running ESX 5.5, had 3 NICs (onboard and the NC360), both hooked up via iSCSI to a Synology DS412+.  All was good.  I've recently been looking at ESX 6 and after upgrading it appears that one of my hosts was rebooting whenever it saw any activity.  I narrowed it down to using the NC360 being the cause of the reboots - if I assigned iSCSI, vMotion, management etc all to the onboard NIC the host behaved fine.  As soon as I got the NC360 involved for anything and fired up a VM, the host would reboot.

 

I wiped the host and did a clean install of ESX 6 instead of an upgrade.

I've also been running a memtest for the past 12 hours which so far has been ok.

Simple then - replace the NIC.  So I did.  Nope, still has the same issue.

 

So on the basis that my other identically configured N54L is fine, I was wondering if anyone had come across a similar issue, or whether it's possible that the PCI slot has developed a fault somewhere, which doesn't affect the rest of the system until you actively use the dodgy slot?

 

Cheers,

R.

Edited June 19, 2015 by Raggeh

[schoondoggy]

Is the NC360 in the x16 PCIe slot or is it a modded card in the x1 PCIe slot?

Have you tried switching the NC360's between the N54l's just to rule out the card?

Have you tried going back to ESXi 5.5 to see if the issue goes away?

[Raggeh]

They're all un-modded cards in the x16 slot.

I bought another NC360 and that didn't solve the problem so the card is ruled out.

 

I'll try 5.5 actually, it'd be odd if that sorted it though.

Also I just thought I could go really basic and try different Ethernet cables and ports on the switch.

[jmwills]

I would also check to insure the card is properly seated.

[schoondoggy]

Use some compressed air to clean the slot, just to be sure nothing fell in it.

[chappys4life]

Did you see if the NC360 needs a driver for 6? Sounds like a driver issue more than hardware. 

[Raggeh]

No, I have 2 identical boxes, I'm only having issues with 1.

[Raggeh]

How weird.  I put a clean install of ESX 5.5 on the problem server and it was fine.  I upgraded to ESX 6 again and it is rebooting again.  It is identical to my other Microserver which is fine running ESX 6.

 

I don't get it.

 

I'll get VUM working and see if any updates can fix the issue.

[Raggeh]

Nope that didn't work either!  How can I have 2 identical setups where 1 reboots when using a particular software version and the other doesn't? :|

[schoondoggy]

I would still swap the NC360's to see if the problem moves. The NC360's have gone through many chip revisions and firmware levels.