Jump to content
RESET Forums (homeservershow.com)
Poppapete

10G Network

Recommended Posts

Poppapete

I too have come to the conclusion that I have either a DHCP problem or a faulty 10G nic in the server. I checked that the role was uninstalled on the server a few days back.  I can't swap cards as the 10G nic in the DT is part of the MB (Asus ROG Rampage VI extreme). The reason I started this exercise was that the DT already had a 10G nic onboard. I will remove the 10G card from the server and put it in another computer on the network and see what happens.  It may take a few days as I am going away for a bit.  I will not give up just yet. The fact that some DHCP server gave out the 169.254. IP address is I am sure significant as I have never seen it before!!

Share this post


Link to post
Share on other sites
itGeeks

Wow talk about community support, This is one of the best parts of HSS forums.

 

I have been sitting on the side lines just watching this post because it looked like you where in good hands. I do have some questions and advice for things to try. Please bare with me if you have already answered or tried anything I am about to suggest, With 4 pages of stuff here it may have already been covered. For starters that Asus motherboard is one amazing looking product and costly too.

 

1) What is the make and model of your router?

2) I see you where running the DHCP server on the Windows 2012 R2 Essentials, Unless your needing something special that your router can't handle like ( multiple DHCP roles) you should always assign the DHCP role to your router. I agree it does seem like something is wonky with DHCP.

 

3) Double check to make sure the DHCP roll is disabled on the server, Open a command prompt on both server and DT and type ipconfig/all

and make sure its showing the DHCP server is your routers IP address.

 

3) Remove the 1G NIC's on on both DT and server, If the are on-board aka built-in then see if you can disable them in the bios not threw the OS, This will make sure they are invisible to the OS then test the 10G.

 

4) One great suggestion was to hook a CAT6 cable from your DT 10G NIC to the servers 10G NIC and set an IP address manually and test, Did you do that and what was the results?

 

5) Make sure you have the Asus 10G card on the server installed in a PCIe 4x or above slot and make sure its not being shared with anything else you may have installed.

 

On closing what ever you have to do to get the 1G NIC's out of both systems to eliminate them either threw the bios or fiscal will be the best way to trouble shoot going forward. Making sure both systems are truly looking at the router for DHCP threw the ipconfig/all at the command prompt will at least clear all that as the problem.

 

If you have already done any of this then please forgive me.... Stick with it, If all this stuff was easy everybody would be doing it and besides what fun would that be..

 

 

 

Edited by itGeeks

Share this post


Link to post
Share on other sites
ShadowPeo
23 hours ago, Poppapete said:

EDIT:

I tried a CAT6 cable from server to 10G switch. As before all leds light up green (10G speed) but ip config shows nic as disconnected.

 

Really appreciate the time you have spent with me on this but I give up.

2

 

It sounds like a dead card. I would offer to lend you one, but the spares I have are all fibre/SFP+ based

Share this post


Link to post
Share on other sites
Poppapete

itGeeks,

1.Router is a TP-Link TL-ER5120 not a dinky router. 1G switch is TP-Link TL-SG3424 again not a dinky switch and both very manageable.

 

2&3. I do not use AD and have been using "skipdomainserver" on my servers for many years (ever since Tinkertry explained it.)  DHCP role is not activated on my 2012R2E box (I have checked twice). ipconfig shows DHCP on nics as 19992.168.0.1 which is the Router.

 

 

 

It seems everyone agrees that the villain is most probably the ASUS 10G Nic in the server.  When I connect the 10G to the 1G switch still no go and again all LED's show connection.  When I got the drivers for this nic from ASUS I chose 64 bit for W10.  There was no choice for any windows server.  Maybe the drivers just don't work for 2012R2E.  When I get time I will take the NIC out and put it in a W10 box and try that.

Share this post


Link to post
Share on other sites
JackoUK

Looking ahead to that happy day when the kit is basically working ...

... there will remain some configuration options for optimal performance.

 

1. The 10GbE NIC settings such as jumbo frames, send and receive buffer sizes, RSS factors, ... as per the Cinevate link on SMB I posted earlier. No doubt the 10GbE switch will have a few settings to tinker with too.

 

2. Windows handles multiple NIC's using a metric for each adapter (or does the job automatically using standard conventions). So the metric for the GbE adapter on my laptop is lower than that for wireless, hence the wired port will be used first by Windows programs, if connected (or maybe it's used first because of a 'higher bandwidth' rule). I suspect  that a working 10GbE connection will consume all traffic unless action is taken to override Windows' conventions. We can do better: what we really want is ... all access traffic e.g. Internet browsing, remote desktop, ... to go through the old 1GbE link ... and storage operations to go through the fast 10GbE link.

 

3. SMB 3 MC is far cleverer than your average Windows program and will attempt to find and use all network paths regardless of their metrics. Further, it can be configured to limit those paths to subclasses (say 10GbE) using powershell cmdlets. I envisage setting 1GbE links as high priority and limiting SMB MC to 10GbE bandwidth. That way access traffic will head for the 1GbE NIC (failing over to the 10GbE NIC) ... and storage traffic will head for the 10 GbE NIC (failing if the 10GbE NIC fails).

 

For those reasons I would keep ITGEEKS idea of disabling 1GbE NIC's in BIOS in reserve and hope fixing the supposedly faulty 10GbE NIC does the trick. Indeed you have already seen SMB failing over: when you had both the 1GbE and faulty 10GbE in place data transfer still occurred over the 1GbE links. And when you had a faulty 10GbE path you couldn't get internet access.

 

[I am planning to avoid 10GbE switching altogether (motivated by the high cost of the switch for one reason). By direct-connecting a 10GbE link in a workstation to a 10GbE link in a data server on a separate subnet, the server essentially becomes a DAS disk. This idea borrows from Microsoft's hyper converged 2-node cluster configuration see page 34 of

https://gallery.technet.microsoft.com/Understand-Hyper-Converged-bae286dd]

Share this post


Link to post
Share on other sites
itGeeks
On 2/26/2018 at 5:48 PM, Poppapete said:

1.If I connect the 1G ports to the 10G switch with 10G switch connected to the router:
Then I have internet on both machines, access to the files on the server from the DT.  So 10G switch is OK

2.If I connect only the 10G ports to the 10G switch:
The server immediately has no internet, but the DT is connected to internet.
All connections on both 10G switch and the ASUS Nic card in server show green flashing LED’s.  I have latest drivers on both 10G nics and both show working with best drivers in device manager.

Results of an ipconfig /all command on both machines:
Server:
Media State                 Media Disconnected

Description                  ASUS XG-C100C 10G PCI-E
Address                       88-D7 etc
DHCP enabled            Yes
Autoconfiguration      Yes

Desktop:
Connection-specific DNS Suffix  . :
Description . . . . . . . . . . . :     Aquantia 10G Ethernet connection
Physical Address. . . . . . . . . : B0-6E-BF-82-9A-13
DHCP Enabled. . . .. . . :         Yes
Autoconfiguration Enabled . . Yes
Link-local IPv6 Address . . . . fe80::a938:2622:773b:6464%7(Preferred)
IPv4 Address. . . . . . . . . . . :   192.168.0.5(Preferred)
Subnet Mask . . . . . . . . . .       : 255.255.255.0  
Lease Obtained. . . . . . . . .      . : Tuesday, 27 February 2018 7:48:17 AM
Lease Expires . . . . . . . . . . :    Tuesday, 27 February 2018 9:48:16 AM
Default Gateway . . . . . . . .     . : 192.168.0.1
DHCP Server . . . . . . . . . .      . : 192.168.0.1
DHCPv6 IAID . . . . . . . . . . . : 129003199
DHCPv6 Client DUID. . . . .    : 00-01-00-01-21-88-3F-07-B0-6E-BF-82-9A-14
DNS Servers . . . . . . . . . .       . : 203.0.178.191
                                                 203.215.29.191
NetBIOS over Tcpip. .. .         : Enabled

 

I have decided to snip this as my starting point in order to try and help you. Now I have questions-

 

1) At this point the only switch connect on your network should be the 10G switch connected directly to the router, Correct? If not please remove all other switches including the PoE switch for your cameras.

 

2) What is the make and model of the 10G switch? I comb the thread this morning but did not see it.

3) Are you using certified CAT6/6a patch cables to connect the devices to the 10G switch?

4) What driver are your using for the Asus 10G NIC on the server? If your using the Windows 10 driver this could very well be your trouble, Microsoft made major change to the network stack on Windows 10. Try a Windows 7/8 64-bit driver from Asus found here

5) Open up device manager on the server and open the properties for the NIC and see what the status is on the general tab, Does it say "this device is working properly"?

 

6) Remove ALL teaming you have setup on both the DT and server.

7) Remove ALL static IP's you have setup on both DT and server, If your using "Static DHCP on the router, Remove them and allow the systems to grab a new IP.

 

8) Remove ALL 1G NICs on both DT and server, If you can't remove them disable them in the Bios.

9) Are you running any VM's on the server? If so I would say disable them for now, The vNICs will be tied to the old 1G NICs.

10) why is your desktop computer using the public IP of Asia Pacific for DNS? That's a big no no, Other then for testing you "never" wan't to have clients on your internal network pointing to a public DNS server. This will break your internal network, A public DNS server knows nothing about your network (hence) you could/will have local DNS resolution problems such as trying to find a device on your local network. The correct way to setup your DNS is to point all your devices to in most cases your router (local) IP address, In this case 192.168.0.1

 

I would also encourage you to use either Google DNS or Open DNS servers instead of your ISP's DNS servers, You will in most cases see a performance improvement. The way you would do this is on the WAN interface of the router change the DNS settings to the followinging-

 

Google DNS1: 8.8.8.8

Google DNS2: 8.8.4.4

 

OpenDNS1: 208.67.222.222

OpenDNS2: 208.67.220.220

 

I think this checks off my laundry list of questions and advice for you for now. Once you get back to me with all the answers and you have completed all the steps I have outlined here we will all have a good foundation to continue trying to help you sort this out if your troube is still not fixed. The ultimate outcome will probably be a bad 10G NIC if at this point it still does not work, Only time will tell. 

Share this post


Link to post
Share on other sites
Poppapete

itGeeks,
10G switch is Netgear XS505M
I bought new short CAT6A cables for the exercise
I have gone to ASUS site and DL Win7 64 driver package. It has same name and size as the Win10 64 package but I unpacked and setup the drivers again.  Device Manage says 10G Nic is working properly on server.
I removed teaming and double 1G nics from server a week ago.
I removed static IP’s from NICs on server and DT  a week ago but still grabs old IPs in some cases.

DNS servers are ones I get from using Steve Gibsons DNS benchmark tool.  I am in Australia don’t forget.

 

SO.  I have just connected to DT and the Server to the 10G switch again after installing the Win7 drivers.  It has been an hour and DT seem to be connected to the internet but server is not.  I ran the internet trouble shooter on the server but it could not find the problem  I give you ipconfig printout of both 10Gnics.  Can’t wait to have time to pull the 10G card out of the server and put it in a W10Box but that will be next week sometime.

Ethernet adapter Ethernet 2 (Integrated 10G on DT):

   Connection-specific DNS Suffix  . :

   Description . . . . . . . . . . . : Aquantia 10G Ethernet connection

   Physical Address. . . . . . . . . : B0-6E-BF-82-9A-13

   DHCP Enabled. . . . . . . . . . . : Yes

   Autoconfiguration Enabled . . . . : Yes

   Link-local IPv6 Address . . . . . : fe80::a938:2622:773b:6464%7(Preferred)

   IPv4 Address. . . . . . . . . . . : 192.168.0.13(Preferred)

   Subnet Mask . . . . . . . . . . . : 255.255.255.0

   Lease Obtained. . . . . . . . . . : Thursday, 1 March 2018 2:54:17 PM

   Lease Expires . . . . . . . . . . : Thursday, 1 March 2018 5:10:08 PM

   Default Gateway . . . . . . . . . : 192.168.0.1

   DHCP Server . . . . . . . . . . . : 192.168.0.1

   DHCPv6 IAID . . . . . . . . . . . : 129003199

   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-88-3F-07-B0-6E-BF-82-9A-14

   DNS Servers . . . . . . . . . . . : 203.0.178.191

                                       203.215.29.191

   NetBIOS over Tcpip. . . . . . . . : Enabled

 

Ethernet adapter Ethernet 3 (PCIe on Server)

 

   Connection-specific DNS Suffix  . :

   Description . . . . . . . . . . . : ASUS XG-C100C 10G PCI-E Network Adapter

   Physical Address. . . . . . . . . : 88-D7-F6-A8-BA-8F

   DHCP Enabled. . . . . . . . . . . : Yes

   Autoconfiguration Enabled . . . . : Yes

   Link-local IPv6 Address . . . . . : fe80::992d:2f87:b47d:3a15%17(Preferred)

   IPv4 Address. . . . . . . . . . . : 192.168.0.14(Preferred)

   Subnet Mask . . . . . . . . . . . : 255.255.255.0

   Lease Obtained. . . . . . . . . . : Thursday, 1 March 2018 2:58:57 PM

   Lease Expires . . . . . . . . . . : Thursday, 1 March 2018 4:58:57 PM

   Default Gateway . . . . . . . . . : 192.168.0.1

   DHCP Server . . . . . . . . . . . : 192.168.0.1

   DHCPv6 IAID . . . . . . . . . . . : 512284662

   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-1A-D7-E1-D5-0C-C4-7A-00-C6-AE

   DNS Servers . . . . . . . . . . . : 203.0.178.191

                                203.215.29.191

   NetBIOS over Tcpip. . . . . . . . : Enabled

 

Share this post


Link to post
Share on other sites
ShadowPeo

Do me a favour and go into the advanced properties on the NIC on the server and disable VMQ (Virtual Machine Queuing) I discoverd this is an issue in some environments with Broadcom based cards, Intel does not have the issue FWIW

 

As I cannot be bothered writing it out again, I will point you to one I prepared earlier: https://justin.simmonds.id.au/blog/2013/02/06/hyper-v-high-ping-latency/

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×