Looking for advice on building compute pool

[RyanConway91]

I am loosing my current pool of old (~2010) servers that I use for model runs and have the oppurtunity to spec some new machines. Has anyone built a high performance machine or cluster for groundwater modeling?

Aspirational specs:
~300 threads so I can run an IES iteration in one shot
~2gb of RAM per thread
~10gb of SSD per thread
~should be spread over a couple machines to allow for multiple PEST runs/users.

Does not need to be cutting edge, but new enough to get another decade or so of use.

[briochh]

@p-ortega and I have been discussing this recently. We are about to set up some Mac mini, although at a smaller scale than your req. @cnicol-gwlogic and others are running AMD threadrippers. For me there is a logical balance between number of processors and other resources including comms channel bandwidth hence we are looking at more smaller units rather than a few big beasts. But maybe these new beasts handle memory and IO needs of heaps of simultaneous runs. Interested to see what others' thoughts are on this.

[cnicol-gwlogic]

Hi there, HNY and all that - yes, I am running a bunch of threadrippers and ryzens, plus a few older Intel i9s.

The newer threadrippers are 64 or 32 cores in my case, and I have some ryzens at 16 cores each (which are fast in parallel executions per core); the i9s are up to 18 cores each.

I find I can get away with 100 to 150 cores all up, and I have had to deal with more than a few chunky models. I mostly run ies, with 300-600 realz, but tend to aim for smaller / faster / a little more abstract models. But sure 300 cores would be great for those bigger models.

Re: the threadrippers, don't get older generations, as they seem to be severely hampered by memory bandwidth than new gen models (even with a brute parallel execution-specific bios setup). And the pro / workstation (WRX) models (as opposed to the TRX models) are worth the extra $$ for pest-style purposes, because of the additional (monster) memory channels/bandwidth. For example @neilmanewell has a bunch of 2nd gen threadripper servers but can only use half the cores or so...I can use all of my 32 cores or 60 of my 64 cores no problem - and that is with that same machine hosting the pest manager...

For details on how to eek the most performance from those threadrippers, see here: https://github.com/cnicol-gwlogic/pest-threadripper_settings. Without careful bios setup and memory hardware, the new threadrippers can feel like a waste of $$....so it is worth investing the time into exploring that side of things if you go down that threadripper road.

The nuc/mini PC option does seem more and more cost effective over time however, as core counts rise per dollar. I've not done the sums to compare that option to a few big workstations, but just eyeballing it quickly now, the mini PC option does seem to add up nicely. Eg I paid about aud13k for my 64 core threadrippers, whilst you can get 10 Mac mini cores for aud1k (or x64 nucs for less); I have not looked at what does /does not come with that though (eg ssds/memory). But with those minis, you don't have the hassle of playing around with bios settings etc, and cooling might be less of an issue (?).

Cheers.

[neilmanewell]

Yep - I'm still encountering memory channel traffic jams with my 2nd gen 32 and 24 core threadripper machines (4 memory channels). They were a bargain though (can get new 32 core 2990wx cpus for $800 AUD on Amazon sales). MFUSG and MF6 do not share memory very well, so without the correct bios/NUMA settings models can run very slowly. I find that if I run smaller MF models, I encounter less memory bandwidth issues. I can happily run about 22 small models on a 32 core threadripper. Compounding to this, I suspect simultaneous speed comparisons are not fair, because when you run a single model, that model runs at a boosted, hyperthreaded speed. When you run simultaneous models they revert to base cpu speed (but don't quote me).

Years ago I tested this memory issue with intel VTune Profiler and the analysis revealed that with large models, as soon as simultaneous n exceeds memory channel count, memory bound IO hit 100%, and model runtime scales linearly.

I have experimented with the NUC farm route. My Ryzen 7 7735HS 8 core minisforum NUC can only run around 4 large MFUSG models before runtimes bog down. Perhaps these smaller, mobile cpus cant handle lots of computations pushed through two memory channels. But they have potential for incredible $ per core (like $80/core compared to $300/core for a threadripper pro build).

So I think the jury is still out. Need to find the time to do some proper comparisons to work out exactly what is the best bang for buck configuration.

[cnicol-gwlogic]

@briochh - are you getting the m4 pro or non pro models? Also the 10core or higher model? I'd be interested in hearing how whatever you get performs. Was just reading up on the "performance" cores vs "efficiency" cores, so the higher core count models sound better for pest as they have more of the former. They are definitely interesting though.

[p-ortega]

Hi @cnicol-gwlogic

We got the M4 base model with more RAM (32GB), storage (1TB), and 10Gb Ethernet. I somewhat regret opting for the 1TB version now because there’s a new Thunderbolt 5 enclosure for NVMe drives, and I think setting up a shared file system with HTCondor isn’t too difficult.

The jury is still out on the P and E cores, but we’ve seen promising results from Rodrigo Herrera, who has many M1/M2 units and just got some M4s too.

The price (AUS) for 8 of those Mac Minis was about the same as 2xThreadrippers 7970X 32 core with 128GB RAM and 4TB SSD. I had a few chats with the people selling those and they were clueless about memory bandwidth and we heard the stories...

Also, scalability ftw I reckon.. It's easier to justify one more 2k node instead of 8-10k.

[neilmanewell]

https://tinyurl.com/3aah7p7e

I think these come out as $46/core compared to $250+/core for the M4. Might try one. Doubt you'd be able to run 14 models across 2 tiny memory channels though.

[p-ortega]

I was tempted by one of those too, but Reddit seems not to love NUCs from Minisforum, BeeLink, etc.

[RyanConway91]

Thanks for all the discussion. Sounds like we are transitioning to housing the servers in data center, so the mini-PCs are not a good option. We will pay for storage on a per-rack basis, so it seems like getting most performance per-rack will be a goal. Probably means threadrippers?

[cnicol-gwlogic]

Or hardcore server CPUs, depending on how flexible your provider is. But those are pricier than threadrippers.

[RyanConway91]

Here are the key hardware on the machine proposed by our supplier. I guess its their "AI" setup minus the GPU.

2x AMD EPYC 9754 2.25GHz, 128C/256T, 256M Cache (360W) DDR5-4800 338-CMDM [256C/512T total]
16x 64GB RDIMM, 5600MT/s, Dual Rank 370-BBRN – 16 [1024GB total]
4x 960GB Data Center NVMe Read Intensive AG Drive U2 Gen4 with carrier 400-BMTJ – [3840 GB total]

Here is the full setup if anyone wants a headache...

PowerEdge R6625 210-BFXO – 1
2.5 Chassis 379-BDTF - 1
NVMe Backplane 379-BDSX – 1
No Rear Storage 379-BDTE – 1
Trusted Platform Module 2.0 V3 461-AAIG - 1
2.5" Chassis with up to 8 NVMe HWRAID Drives, Smart Flow, Front PERC 11 321-BIIK - 1
AMD EPYC 9754 2.25GHz, 128C/256T, 256M Cache (360W) DDR5-4800 338-CMDM - 1
AMD EPYC 9754 2.25GHz, 128C/256T, 256M Cache (360W) DDR5-4800 338-CMDM - 1
Additional Processor Selected 379-BDCO – 1
Heatsink for 2 CPU configuration 412-ABEE – 1
Performance Optimized 370-AHLL - 1 5600MT/s RDIMMs 370-BBRX - 1
C31, No RAID with NVMe and front PERC 379-BEGI – 1
PERC H755N Front 405-AAZE - 1
Bracket for floating load on fPerc 750-ADSN – 1
No Hard Drive 400-ABHL – 1
Performance BIOS Settings 384-BBBL - 1
UEFI BIOS Boot Mode with GPT Partition 800-BBDM – 1
No Energy Star 387-BBEY - 1
High Performance Fan x4 384-BDHQ – 1
Dual, Redundant(1+1),Hot-Plug PSU,1800W MM HLAC(ONLY FOR 200-240Vac)Titanium,C16 Connector 450-AMIN – 1
Riser Config 4, 2 x16 LP (Gen5) 330-BCDB – 1
PowerEdge R6625 Motherboard 329-BHOU - 1
Broadcom 57414 Dual Port 10/25GbE SFP28, OCP NIC 3.0 540-BCOC – 1
Broadcom 5720 Dual Port 1GbE LOM 540-BDKD – 1
TRAY,W/LBLS,X8/X10,R6625 321-BIGJ – 1
Standard Bezel 325-BEVE – 1
BOSS-N1 controller card + with 2 M.2 480GB (RAID 1) 403-BCRU – 1
BOSS Cables and Bracket for R660 470-AFMG – 1
No Operating System 611-BBBF – 1
Dell Connectivity Client - Enabled 379-BFXS – 1
Dell Connectivity Module 634-CYDF – 1
iDRAC9, Enterprise 16G 528-CTIC – 1
OpenManage Enterprise Advanced 528-BIYY – 1
No Quick Sync 350-BBXM – 1
iDRAC,Factory Generated Password 379-BCSF – 1
iDRAC Service Module (ISM), NOT Installed 379-BCQX – 1
iDRAC Group Manager, Disabled 379-BCQY – 1
Cable Management Arm 770-BDMT – 1
ReadyRails Sliding Rails (A15) 770-BECD – 1
No Systems Documentation, No OpenManage DVD Kit 631-AACK – 1
PowerEdge R6625 Shipping 340-DDEC – 1
PowerEdge R6625 Shipping Material 4 340-DDCC – 1
PowerEdge R6625 No CCC or CE Marking 470-AFOY – 1
Basic Next Business Day 36 Months 709-BBFM – 1
ProSupport and Next Business Day Onsite Service Initial, 36 Month(s) 865-BBMY – 1
No Field Deployment 883-BBFN – 1
Infrastructure Deployment Selected 701-6538 – 1
64GB RDIMM, 5600MT/s, Dual Rank 370-BBRN – 16 [RAM; 1024GB total]
960GB Data Center NVMe Read Intensive AG Drive U2 Gen4 with carrier 400-BMTJ – 4
CORD,PWR,250V,13A,C15/14,NA,2M 450-AMKV – 2
Broadcom 57414 Dual Port 10/25GbE SFP28 Adapter, PCIe Low Profile, V2 540-BDGV - 1
No Media Required 605-BBFN - 1

[cnicol-gwlogic]

Hey, I've not got experience with the epycs so can't really comment much. But it'd be worth you testing one near-fully loaded with pest runs before you bite the bullet, if you know someone with one or your supplier has a test machine.

Note that some of those principles I learnt from my threadripper experience noted in this thread may apply to epycs too...so some frigging around may still be needed; spec to the vendor what type of loads you will be hitting it with (isolated instances that need as clear a CPU/memory path as is possible - little focus on multi threading necessary in our pest/mf work).

But even if they don't crank, as long as they run at reasonable pace, 250+ instances the would be pretty good.

One thing I did notice is that the nvme size is way too small for that many runs. Or did I misread that at <4tb? I'd say you'll be needing more like 12 or 13 tb. I am gonna look into those read intensive drives though, as my regular nvmes definitely get hammered, showing appreciable decline in condition in hdsentinel pretty quickly.

[RyanConway91]

Thanks for the feedback . You're correct - currently specing 3.8 TB for NVMe. I rarely build models that take up more than ~5 GB on the hard drive, but maybe doubling the NVMe is a good idea. We have a try before buy agreement with the vendor so I will get the chance to test it out before pulling the trigger.

Are there any "benchmark" problems out there that I should use to test the performance?

[cnicol-gwlogic]

No worries. Not sure re benchmarks. I'd just use one of your own bigger models, as that is closest to your specific worst case needs. Small models are neither here nor there, it's the bigger slower individual runs that are painful.

Re the nvme size, sure go on your estimated needs. I guess I am going by my use case, where I'll often have 2 or 3 projects on the drives, and am often archiving some biggish data sets like rasters etc. But that can be worked around if in an office environment with alt storage. Cheers.