Jump to content
RESET Forums (homeservershow.com)
schoondoggy

Ready to expirement 10Gb Ethernet?

Recommended Posts

marty_oc

Hi guys, I have ordered some 10GbE NICs on eBay (Mellanox ConnectX 2 with SFP+) and twinax SFP+ cable. I plan to direct connect my desktop PC with my Gen8 Microserver - adding switch will be really expensive way - and I really do not need it, because high bandwidth is used only between my desktop and the server. 

 

Commenting on above statement - that teaming two gigabits is enough - my drives provide about 260-280 MB/s read rate, so it would not be enough :)

 

I will post updates soon, I hope I will be picking the cards tomorrow.

Share this post


Link to post
Share on other sites
ShadowPeo

It will be interesting to see how this goes, I have used high-end Dell gear at work because we got an incredible deal on it due to the client being in state funded education, but most of that is way to expensive for home use, and as it has been pointed out, has sound issues, not an issue in a server room but not something you want in your house, let alone the cooling bill to keep them within warranty retaining temps.

 

I have been thinking about going down this path myself, but even at the prices I get the equipment for its way to expensive. However I have the core switches coming out of lease (and warranty) in a couple of years and they are essentially disposable items once the warranty is out so I might grab one of those. I am also having a hard time justifying the cost of this at this point, as, as has been pointed out in other posts, there is not much that requires 10GBE especially in the home environment, even on the long-haul fibre links to the out buildings (75-125 meters) I simply use bonded pairs and even this is overkill, there is nothing even touching this bandwidth for the most part I max out the disks throughput on the mass storage devices (I did cheap-out on the disks so they max out at about 113MB/s sustained write) before I max out the links (I just happened to get the transceivers free when we did the upgrade at work a couple of years back).

 

For Reference the work setup is as follows;

With these setups we use 80GB (2x Bonded 40GB QSFP) in the core stack (two independent switches for redundancy) and twin 10GBE over fibre links (one from each switch out to each endpoint switch) so the destruction of any one thing in the environment will be minimal as everything is redundant on the core infrastructure out to the endpoints. The endpoint switches themselves are mostly single units so there is no redundancy in the endpoint locations, only two locations have full redundant stacks. The only noticeable part of a downtime scenario is how long the fail-over of the infrastructure to its backup nodes (each sever as a 10GBE link to each switch for data, and another to teach switch to a dedicated storage VLAN)

Share this post


Link to post
Share on other sites
marty_oc

Okay, so my experience with 10 GbE:

 

 

  •  with Debian Linux on the Microserver and Windows 10 and Ubuntu Linux on desktop Mellanox cards work plug and play
  •  I used the cards to direct-connect my workstation to the server. This is perfectly okay for me because my workstation is the only machine in the network which would ever require faster connection to the server than gigabit.
  •  performance tests:
    • 10 GbE works well. Plain bandwidth test to /dev/null maked something about 1.2 GBps.
    • FTP transfer, source Samsung SSD 850 EVO on my workstation:
      • destination: single WD Green 1.5 TB: ~115 MB/s
      • destination: RAID 5 with 3 WD REDs 3 TB: ~125 MB/s
      • destination: single Intel 330 SSD: ~240 MB/s
    • FTP transfer, source RAID5 with 3 WD REDs 3 TB on the server:
      • destination: Samsung SSD 850 EVO on my wkst: ~ 250 MB/s
    •  FTP transfer, source single Intel 330 SSD on the server:
      •  destination: Samsung SSD 850 EVO on my wkst: ~ 280 MB/s

 

Generally now the network is no more bandwidth limit for all those storages. 

Edited by marty_oc
  • Like 1

Share this post


Link to post
Share on other sites
schoondoggy

Okay, so my experience with 10 GbE:

 

 

  •  with Debian Linux on the Microserver and Windows 10 and Ubuntu Linux on desktop Mellanox cards work plug and play
  •  I used the cards to direct-connect my workstation to the server. This is perfectly okay for me because my workstation is the only machine in the network which would ever require faster connection to the server than gigabit.
  •  performance tests:
    • 10 GbE works well. Plain bandwidth test to /dev/null maked something about 1.2 GBps.
    • FTP transfer, source Samsung SSD 850 EVO on my workstation:
      • destination: single WD Green 1.5 TB: ~115 MB/s
      • destination: RAID 5 with 3 WD REDs 3 TB: ~125 MB/s
      • destination: single Intel 330 SSD: ~240 MB/s
    • FTP transfer, source RAID5 with 3 WD REDs 3 TB on the server:
      • destination: Samsung SSD 850 EVO on my wkst: ~ 250 MB/s
    •  FTP transfer, source single Intel 330 SSD on the server:
      •  destination: Samsung SSD 850 EVO on my wkst: ~ 280 MB/s

 

Generally now the network is no more bandwidth limit for all those storages. 

Very interesting!

Thanks for posting. After the holidays I hope to play with 10GbE a bit more.

Share this post


Link to post
Share on other sites
schoondoggy

As with any purchase, check before you buy. With Ethernet NIC's it is easy to trust names like HP and Intel, but in 10GbE I am finding very poor support for older cards. HP NC510F appear to be good cards, but I don't see drivers for Server 2008 or 2012? No ESXi support. Maybe they have Linux support? Intel EXPX9501AFXSR seem to have some Window Server support, not sure on 2012, no ESXi.

Glad I did some research before I bought. 

Share this post


Link to post
Share on other sites
schoondoggy

Thanks to Ernesto on HGG for advising, HP 10GbE NIC's are going for $20 on ebay with a SFP+ cable. 

http://www.ebay.com/itm/671798-001-666172-001-MNPA19-XTR-HP-10GB-ETHERNET-NETWORK-INTERFACE-CARD-W-CABLE-/282042392246?hash=item41ab095eb6

 

These are Mellanox based cards and seem to have very good driver support.

 

Switches have dropped as well:

DLINK DGS-1510-20 $325.99 two 10GbE ports SFP+

https://www.amazon.com/D-Link-Systems-SmartPro-Stackable-DGS-1510-20/dp/B00H554JL4/ref=sr_1_2?ie=UTF8&qid=1471472556&sr=8-2&keywords=10Gbe+switch

DLINK DGS-1510-28x $535.93 four 10GbE ports SFP+

https://www.amazon.com/D-Link-Systems-SmartPro-Stackable-DGS-1510-20/dp/B00H554JL4/ref=sr_1_2?ie=UTF8&qid=1471472556&sr=8-2&keywords=10Gbe+switch

Share this post


Link to post
Share on other sites
Big Worm

Thanks to Ernesto on HGG for advising, HP 10GbE NIC's are going for $20 on ebay with a SFP+ cable. 

http://www.ebay.com/itm/671798-001-666172-001-MNPA19-XTR-HP-10GB-ETHERNET-NETWORK-INTERFACE-CARD-W-CABLE-/282042392246?hash=item41ab095eb6

 

These are Mellanox based cards and seem to have very good driver support.

 

Switches have dropped as well:

DLINK DGS-1510-20 $325.99 two 10GbE ports SFP+

https://www.amazon.com/D-Link-Systems-SmartPro-Stackable-DGS-1510-20/dp/B00H554JL4/ref=sr_1_2?ie=UTF8&qid=1471472556&sr=8-2&keywords=10Gbe+switch

DLINK DGS-1510-28x $535.93 four 10GbE ports SFP+

https://www.amazon.com/D-Link-Systems-SmartPro-Stackable-DGS-1510-20/dp/B00H554JL4/ref=sr_1_2?ie=UTF8&qid=1471472556&sr=8-2&keywords=10Gbe+switch

You are welcome.  Those are great cards at a great price.  Only I can't get myself to pull the trigger on is the switch.  But for now direct connect is working awesome!

  • Like 1

Share this post


Link to post
Share on other sites
flamaster
On 17.08.2016 at 3:25 PM, schoondoggy said:

Thanks to Ernesto on HGG for advising, HP 10GbE NIC's are going for $20 on ebay with a SFP+ cable. 

http://www.ebay.com/itm/671798-001-666172-001-MNPA19-XTR-HP-10GB-ETHERNET-NETWORK-INTERFACE-CARD-W-CABLE-/282042392246?hash=item41ab095eb6

 

These are Mellanox based cards and seem to have very good driver support.

 

Switches have dropped as well:

DLINK DGS-1510-20 $325.99 two 10GbE ports SFP+

https://www.amazon.com/D-Link-Systems-SmartPro-Stackable-DGS-1510-20/dp/B00H554JL4/ref=sr_1_2?ie=UTF8&qid=1471472556&sr=8-2&keywords=10Gbe+switch

DLINK DGS-1510-28x $535.93 four 10GbE ports SFP+

https://www.amazon.com/D-Link-Systems-SmartPro-Stackable-DGS-1510-20/dp/B00H554JL4/ref=sr_1_2?ie=UTF8&qid=1471472556&sr=8-2&keywords=10Gbe+switch

 

Anybody looked at the MikroTik Cloud Router Switch CRS210-8G-2S+IN ?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...