[illumos-Discuss] CPU/Chipset architecture for high performance ZFS servers

[Richard Elling]

On Apr 1, 2011, at 6:03 AM, Alasdair Lumsden wrote:

> Hi All,
> 
> I was wondering if anyone could comment on the best CPU/Chipset architecture (Intel E56xx/X56xx vs AMD 61xx) for high performance ZFS/Nexenta based storage servers.
> 
> There will be two clustered storage heads, with dual 10GigE adapters attached to multiple SAS 6Gbps disk shelves with 20-80 15k RPM SAS disks, plus multiple ZeusRAM drives as ZIL devices, with the potential for whole disk shelves of L2ARC cache.
> 
> The storage would be a backend to Oracle 10g boxes running intensive batch jobs from high end Intel kit (32+ cores), so performance is critical.
> 
> For something like this I'm guessing 96GB+ of RAM per head is going to be needed, and I'm trying to figure out whether 2 CPU sockets is enough, or whether 4 sockets would be beneficial.

You won't run out of CPU, but you can run out of I/O.  For most 2-socket Intel boxes, there is
only one IOH. The 5520 IOH only provides 36 PCI-e lanes, so you will see different combinations
of PCI-e bridges on systems from various vendors. Obviously, fewer bridges is better and that tends
to become the biggest architectural difference in the machines.  Similar analysis applies to the AMD
chipsets.  For an interesting case of simplicity built for speed, look at the Sun X4500 design.

> 
> I'm trying to price this up as an alternative to a typical NetApp/EMC/Hitachi style solution, and would be interested to hear if anyone has had similar successes displacing proprietary vendor solutions with this type of thing.

At Nexenta, we live to do this every day :-)
 -- richard

-- 

Richard Elling
richard at nexenta.com   +1-760-896-4422
Nexenta European User Conference, Amsterdam, May 20
www.nexenta.com/corp/european-user-conference-2011