Jump to content
RESET Forums (homeservershow.com)

HP Proliant ML350p G8 Watercooling!


RyanG8
 Share

Recommended Posts

Hey guys, been reading thru this thread, and decided to join up!

 

My setup:

    HP ProLiant ML350p Gen8 (duh) Rackmount Server
    2x Intel® Xeon® Processor E5-2630 6 Core 2.30 GHz
    96 GBs ECC Ram
    8x 300GB SAS Drives
    (all bought as is for $850 (AUS) from a local IT store selling it on ebay!)

 

The twin core, and the 96GBs of RAM is what really sold this to me… plus, the addition of the 8 SAS Drives wasn’t too bad either.

 

RAM Issue:

 

I noticed that the RAM ever second boot cycle came up with an error.  I eventually figured out that it was one particular RAM module that was effecting a whole bank (3 DIMMS lit up with errors, and when I moved the problem module, to another bank, the problematic module caused the same errors for the following 2 DIMM modules…).  So, no biggy, going to take the RAM module back to the IT store to replace/swap!

 

CPU upgrading:

 

One thing I didn’t see mentioned was upgrading CPU’s to V2’s.  I’ve come to realise with the new firmware updates from HP, you are able to upgrade from 2600 Xeon’s, to the 2600 v2’s – meaning cores higher than 8, and higher clocks… (and temps!) :D

 

https://support.hpe.com/hpesc/public/docDisplay?docId=c03911173

 

Has anyone followed thru with this CPU upgrade?

 

I have updated my server to the newest firmware (I believe) from the bootable ISO: (P03093_001_spp-Gen8.1-SPPGen81.4.iso) bringing me up to firmware version 2.7…3… I think? (I’ll have to check that when i get home!)

 

From what I understand, the latest SPP update, includes the ability to use v2 Xeon, correct?


I couldn’t find anywhere that definitively answers this.

I also wanted to know if anyone has upgraded their system using the G10 service pack… I’m trying to find where I read that it was possible, tho, I doubt it (I mean, it’s completely different hardware… unless there were some Bios/iLO upgrades?)

Can anyone clarify if it’s possible to upgrade to the Gen10 SPP and if so, why and what are the benefits?

 

RUNNING SILENT:

 

I’ve been following the guides you’ve been sharing with the 'quietening' upgrades! I was considering water-cooling mine… but, seeing as easier it is to go down with the old-skool tried and tested method of 'heatsink and fan', I think I’m gonna be going down that route!

Mind if I pick your brain JamesT when the time comes mate? :D

I’m not a hunderpercent confident on the soldering side of things – I’ve done it before, it doesn’t look pretty, but it does the job :D I think it’s more the fans, placement mounting that im a little hesitant about.

I’m gonna order two of those sockets holder assemblies you purchased (unless you know of a better way! And order some 2011 heatsink and Noc’ fans to match.  Im still undecided about how to go about the 4 main fans tho…


If i set the fans on the CPU heatsink to 100%(or just power them directly), will the BIOS think that there’s something wrong with the system fan’s, and spit out an error?


I’m still a little confused as to what will (and wont) happen with regards to that... If not, i can start with just having the heatsink and fans, keeping the original fans connected (and only a little loud!) whilst i work out how i'm going to replace them...!

 

8 x SAS HDD's:

 

So, the system came with 8 300GB SAS HDD's (the 2.5" variety) and was wondering what to do with them... they're the 10K proper performance SAS type drives... should i keep them in RAID 0 using the HP p420 RAID controller, and use it as a swap/spare drive? (if it looses data, i won't be concerned) OR.. IT mode it, and integrate it with TrueNAS? I figured, keeping it in RAID, means less strain or work for TrueNAS, considering TrueNAS just sees it as another disk?

 

SIDE NOTE:

 

I have also bought 2 Dell external SAS controllers, and installed them – both will be connecting to an external JBOD.  The external JBOD is actually my old Plex server case with the HDD’s still in there, I figured no reason to spend more money on equipment ;)

 

I have also bought a PCIE card from amazon that takes the apple only standard of their 2013 M.2 SSD’s, so as to use it as normal M.2 Sata – great bang for buck, otherwise I probably would have turfed 2 of these old 256GB SSD’s that are great SSD’s and put them to use as L2arc drive cache in my server!

 

If anyone wants links to anything, please lemme know! 
 

My g8: (air duct shroud removed)
469876507_G8Server.thumb.jpg.4310f40af173ba0366cedfae627068ce.jpg

 

Apologies for the loooong post... but i'm "working" during lockdown, so i was REALLY bored... :P


P.S. I'm in Sydney Australia - if anyone's close by, give'us a shout! :D

 

-K

Link to comment
Share on other sites

  • 2 weeks later...
On 9/13/2021 at 12:46 AM, KinYam said:

Mind if I pick your brain JamesT when the time comes mate? :D

100%, anytime mate. Earlier in this thread (irrc it was XeonUnicorn or HJB) the fan daughterboard mod is explained; basically the "fan present" and RPM reporting is done though wiring; that little pcb is just a pluggable termination point, no smarts in it. So long as you connect up the wiring on it correctly, the ML will think all 4 fans are there, and that they are reporting the correct RPM. 

 

That's actually the circuit I am working on. It will be an updated fan board which aftermarket fans can plug into.

 

On 9/13/2021 at 12:46 AM, KinYam said:

Has anyone followed thru with this CPU upgrade?

 

Not on this forum, I don't think, but I read this: 

 

https://community.hpe.com/t5/ProLiant-Servers-ML-DL-SL/ML350p-Processor-Upgrade-Steps-and-parts/td-p/7080628

 

I believe that in the ML30p service/product manual lists units that were shipped with V2 processors. I am looking at either 2667V2 or 2690V2 for my project. Let me know how you get on.

 

On 9/13/2021 at 12:46 AM, KinYam said:

If i set the fans on the CPU heatsink to 100%(or just power them directly), will the BIOS think that there’s something wrong with the system fan’s, and spit out an error?

 

If you do not make the modifications to the fan board, the ML will get so hot that the matter in it will densify much like a dying star; blackholes follow shortly after. Also, I believe if you remove the 'CPU lane' fans, the machine will not boot.

 

On 9/13/2021 at 12:46 AM, KinYam said:

From what I understand, the latest SPP update, includes the ability to use v2 Xeon, correct?

I am also very keen to know the answer to this.

 

You are lucky with the drives being included too; mine was ex business use so they paid for "secure" (see: wood chipper) data destruction. Annoying part is that the caddies were destroyed too, and oem ones are like $80 bucks with shipping. Back to Aliexpress where I bought 6 generics for the same price.

 

On 9/13/2021 at 12:46 AM, KinYam said:

I’m not a hunderpercent confident on the soldering side of things – I’ve done it before, it doesn’t look pretty, but it does the job :D

 

You don't have to solder it, you can twist the relevant wires together. But honestly, if you're not confident with soldering, this is a perfect project to gain some. Everything is marked and HP makes good PCB's, so it will be harder to mess it up than it would be to do it right. You can do it!

 

On 9/13/2021 at 12:46 AM, KinYam said:

I’m gonna order two of those sockets holder assemblies you purchased (unless you know of a better way!

 

I chatted with the Aliexpress seller about this as he was interested in what I was doing. About 2 weeks after I bought those Foxconn ones, he sent me this ad for narrow ILM assemblies from another seller, but they were like $8 and unbranded. You could take a chance and see if they fit, no grinding etc, would be straight up plug and play if it fits. I wouldn't be surprised if its the ILM assemblies being used on Huananzhi Xeon mobos; I would bet that they would be fine.

 

https://www.aliexpress.com/item/4001013749863.html?srcSns=sns_SMS&spreadType=socialShare&bizType=ProductDetail&social_params=20279837182&aff_fcid=17513c384b834adb9fc2946defe6012c-1632283031948-04339-_mNpjQcf&tt=MG&spm=a2g0s.imconversation.0.0.553e3e5fS0lVhC&aff_fsk=_mNpjQcf&aff_platform=default&sk=_mNpjQcf&aff_trace_key=17513c384b834adb9fc2946defe6012c-1632283031948-04339-_mNpjQcf&shareId=20279837182&businessType=ProductDetail&platform=AE&terminal_id=fa00db65c97241d39d6395183d31ee51

 

 

I have a tendency to waffle, if you need anything else just let me know. Glad to be chatting directly with someone else about this thing. I love this machine now that its quiet lol.

 

Hope you are all safe and well over there. We got out of LVL4 today... YAAAAAAAAAY

 

-Jim

 

Link to comment
Share on other sites

My ML350p Gen8 already was reasonably silent. Best tip I found buried somewhere in another forum is to have at least one genuine HP drive connected to the 420i. That silences it as soon as (in my case) ESXi boots, mostly to 6% fan speed. However, since it still revs the fans for short periods on load, and I found that the revving up and down was especially annoying, I decided to see where I could get.

 

Mine had full fan redundancy, so fan louvres to block dead fans, plus the additional PCI lane baffles.

 

For now, I have removed the louvres and all air baffles, and installed two NH-D9DX i4 3U. As mentioned elsewhere, these fit without any changes to the mainboard. They can be mounted in any position, but as they are not perfectly square, they only fit this MB sideways. I have the one on CPU2 blowing away from the 420i, as not to add heat to that.

Controlled by an NA-FC1, but just set to 100% for now.

 

Both CPUs are consistently at 40 degrees, which I think is the lowest it will report (my Microservers are also at 40). I had CPU1 on 51-53 earlier, probably because it receives the hot air from CPU2. Delta fans stay at 6%. So for now, I'll leave it like this and see. If it has stopped revving up and down, I might just keep the Delta's in. They are silent at 6%. Otherwise, I'll replace those as well.

 

This is 2 x E5-2960 with 128GB, 6 x SSD (4 genuine HP).

 

IMG_1357.thumb.jpeg.540fd9c17b7111556c9cd54a6d36c5f5.jpeg

 

 

Link to comment
Share on other sites

On 9/24/2021 at 8:15 PM, brodgartech said:

For now, I have removed the louvres and all air baffles, and installed two NH-D9DX i4 3U. As mentioned elsewhere, these fit without any changes to the mainboard. They can be mounted in any position, but as they are not perfectly square, they only fit this MB sideways. I have the one on CPU2 blowing away from the 420i, as not to add heat to that.

Controlled by an NA-FC1, but just set to 100% for now.

 

 

Wait... the NH-D9DX i4 3u just... installs without any changes to the motherboard? 

 

Now I'm a little confused... JamesT said he had to Dremel out the rivets to use the allied expressed bought retention bracket... (i've also bought two of these, waiting for delivery).

I'm guessing you're just using the cooler itself to keep the CPU in place just like glob did? Am i missing something?

 

Either way...


I've ordered 2x NH-U9DX i4's, 2x 2011 retention brackets from Allied express, a Noctua NA-FC1 controller, and some thermal compound... ETA about 2 weeks... 😎

 

BTW, is anyone using their 'TreeFiddy as a TrueNas server at all? Or using Proxmox?

Mine is originally being used as a Truenas server... BUT... considering running a VM to run as my own desktop (when i've upgraded the CPU's...!) 😅

Link to comment
Share on other sites

On 9/22/2021 at 2:18 PM, JamesT said:

I am looking at either 2667V2 or 2690V2 for my project. Let me know how you get on.

 

Ditto here. I also notice sometimes they are sold in bulk on eBay... if you wanted to pool resources/found a great bargain, lemme know/count me in!

 

-K

Link to comment
Share on other sites

(might be leaning towards the 2667 v2 for the higher clock & boost performance fyi! - Better single thread passmark scores...)


Passmark e5 2667 v2 (8core 16thread 3.3Ghz Turbo 4.0Ghz)
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2667+v2+%40+3.30GHz&id=2154
 

Passmark e5 2697 v2 (12core 24thread 2.7Ghz Turbo 3.5Ghz):
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2697+v2+%40+2.70GHz&id=2009

Link to comment
Share on other sites

I found 2 normal E5-2690 in china (don't know if i can tell the sales site) at 10usd and buy. They arrived at my home in Brazil today after 22 business days

 

About SSD's using the 3.5" cage...... the western digital green, it goes through all post screens without any error, they are my production ssd's now.They are my favorites right now for my homelab.

  • Like 1
Link to comment
Share on other sites

  • 2 weeks later...
On 9/28/2021 at 1:33 AM, KinYam said:

Wait... the NH-D9DX i4 3u just... installs without any changes to the motherboard? 

 

Now I'm a little confused... JamesT said he had to Dremel out the rivets to use the allied expressed bought retention bracket... (i've also bought two of these, waiting for delivery).

 

Hi KinYam,

 

My focus was on trying to keep the functional description of the machine as close to original as possible. Considering my experience level, I was not prepared to take the chance of flowing air sideways across the CPU when I had already decided to reduce the cooling systems potential static pressure to less than 1/5th of what the delta fans are capable of.  I set the OEM air flow direction as a design parameter and did not look back. I suppose the way to view it is that I wanted to do what everyone else here had already proved worked, but keep the CPU lid. I am happy that Brodgartech has proved another method; many people won't be as prepared to get the dremel out; this would be a good option for those projects. 

 

I had to give my wife her dining room table back and we are renovating our house so the machine is packed away for a couple weeks, I can't do screenshots without hassle, but I am pretty sure I got 37degC when using NZXT software and stress testing the CPU. Not sure how accurate that would be or if 3degC below 40degC is significant enough to note; would be interested in Brodgartechs comments on this.

 

There are many Xeon CPU's on Aliexpress... prices seem competitive and some of the sellers have moved thousands of units with 4.9+ reviews. I've heard that China is upgrading its state owned data centres at a blistering pace, so perhaps this is where the stock comes from on such a scale that it is so cheap? I'm not sure how true that is. If anyone has experience or comments on this, please do share them. I have read some posts on reddit about needing to be careful with what you buy as even if the model numbers are the same, there are other things to watch out for like Engineering Sample CPU's and where they were produced? I don't know much about this, I'm still learning.

 

Anyway, good luck on your build!

 

 

 

 

Link to comment
Share on other sites

 

On 9/27/2021 at 2:33 PM, KinYam said:

 

Wait... the NH-D9DX i4 3u just... installs without any changes to the motherboard? 

 

Yes, but as JamesT correctly says, it changes the airflow inside the server, so it might have side effects I can't predict, especially if you use PCIe cards that produce heat as well.

 

However, as it originally doesn't have any fans on the CPUs, I though adding some in whatever direction would be a bonus as it gets more heat away from the CPUs. Blowing it to the sides does still require the front fans though, to get it out of the case, where JamesT could get away with only the CPU-fans I think? It was remarkably cooler with the added fans and original Deltas. These Noctua heatsinks are a lot bigger as well, of course.

 

Keeping the Deltas with just the added CPU cooling will probably work for a lot some people, as long as there's nothing in there that forces the Deltas up regardless. I've had an HP branded FC card in there that just set the Deltas to 40% from boot. So it's pretty useless in that scenario.

 

By the way, these specific heatsinks were mentioned here: https://forums.servethehome.com/index.php?threads/hp-ml350p-gen8-fan-replacement.21359/post-250038 Credit where it's due.

 

I haven't run into issues so far. iLO still thinks the fans only need 6%.

 

To follow up: as I wanted to control the speed of the CPU fans, I ended up changing the Deltas to 92mm Noctuas as well, and using an AquaComputer OCTO to control all the fans, with four temperature probes inside the server (2 x CPU, RAID and a random memory module). CPUs are cool enough. I'm happy so far.  However, I don't know for how much stress the server will be able to take before these Noctuas just aren't enough. I'm using it mainly for development and staging so will probably never hit that point the way I use it (I just need the cores and memory for the VMs).

 

IMG_1418.jpeg

IMG_1416.jpeg

 

 

Following up on JamesT's question, this is what AquaSuite tells me during normal usage at the moment (7 VMs, 20% ESXi load). I hadn't mentioned before, but my ML350p is dual E5-2690 with 128GB.

 

 

 

aquasuite.jpg

Edited by brodgartech
  • Like 1
Link to comment
Share on other sites

  • 2 weeks later...

I want to thank you all that pioneered in this forum, it was very helpful and I was able to make a much more silent server. I've replaced most with noctuas:

  • cooler D9DX-i4 3u (fit correctly afaik, but no stress yet to check) 
  • NF-A9 (bought as a 2nd for the cooler not using yet so put into the main fans),
  • NF-F12 Chromax PWM
  • industrial nf12 3000 
  • NF-A4x20 PWM for PSU
  • (corsair commander controller, liquidctl manual control for now)

I initially attach the fans to the fan cage and no baffle. With the Chromax that was not enough, the HD temp was sky high.
The industrial fair much better but was still needing to be on top speed making too loud.

I've decided to cut the baffle and attach the fans directly to it and it seems to keep the temperature better without needing to be full speed on the industrial. I'll experiment some other fans now that I might not need that full power.

It's not the nicest assembly, I'll do some better fixing of the fans.  You can see the picture of it there.

Next steps:

 - replace original sas HDDs with some SSDs to see what will be the impact in the hd temperature and hdd noise. 
 - second processor cooler, not sure I'll cut the baffle and use the same model or try a water version that might fit the baffle with less cutting.

 - commander pro temperature probes

 

To the people that used the commander pro, where did you locate your probes?

 

Kind regards and good modding.

UNADJUSTEDNONRAW_thumb_2aa6.jpg

  • Thanks 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Similar Content

    • gru
      By gru
      Hi! I am planning to buy a Microserver Gen10 Plus E-2224 for a small business, and want to add as much disk space as possible. I already have the older Gen10 with 4x4TB HDDs in RAID10 (effective capacity 8TB) and a 2TB SSD in the 5th slot (which is missing on the new G10 plus). The max stated capacity on the HPE site seems to be limited with their own enterprise HDDs, so I'd like to see if I can go for bigger disks. The server would be running a sort of an application which collects lots of data from devices, and the data tends to grow over time (1-2TB per year), and I'd prefer not to worry about the capacity for the next several years.
       
      My primary concern is the max capacity (max single drive capacity + max total RAID capacity with the Smart Array S100i) I can reliably install. If possible, some actual setup which has been shown to work with 32GB RAM and its limited 180W power supply? I was also planning on upgrading it to 32GB. If I am buying a 16GB RAM server, does this means it has both slots occupied with 8GB modules and I need a 2x32GB kit? For the os SSD, I won't have the PCIe slot available because we need to insert a certain GPS PCIe card that the server uses for time synchronization. So, if I want an SSD, this means I should use something like 1xSSD + 3xHDD in RAID5, instead of RAID10?  
      I've been planning to install 16TB Seagate Exos X16 drives, with rated max operating power at 10W (6.3W for random reads/writes). Does this seam feasible?
       
      Thanks a lot for your tips!
    • E3000
      By E3000
      Does anyone know if it is possible to get Agentless Management Service (AMS) running on Debian/Ubuntu Linux? It is always showing as Not Available in the iLO page and I have only ever seen this active in Windows.
    • E3000
      By E3000
      Hey guys,
       
      Has anyone around here been successful in setting up Proxmox on a Gen8 using a HBA for storage?
    • E3000
      By E3000
      Hey guys,
       
      As of June 2020, what are the latest versions of firmware for HP Microserver Gen8 to be installed on the system itself? Also do any other components have updatable firmware/os-independent software?
       
      So far my list is as follows:
      BIOS - J06 (04/04/2019) iLO 4 - v2.73 (11/02/2020) Intelligent Provisioning - v1.71 (20/02/2019)  
      and I guess the following could also be added (although both out of date):
      Smart Update Manager (HP SUM) - v8.1.0 (21/12/2017) Service Pack for ProLiant (06/11/2017)  
      Date formated as DD/MM/YYYY.
       
      I believe all the B120i Smart Storage stuff is part of Intelligent Provisioning.
    • E3000
      By E3000
      Hey guys, 
       
      Bit of a random question here, and I know it depends on many factors such as what they are doing etc, but how many VMs have you guys comfortably ran on Gen8 with an E3-1265v2 and 16GB RAM? When did you start to see a performance hit compared to what the Gen8 is generally capable of?
×
×
  • Create New...