Testing 1, 2, 3 » LSI SAS3081E-R

Sometimes giving in is easier

Daedalus — Sat, 12 Dec 2009 13:18:26 +0000

OpenSolaris’ ZFS implementation recently picked up one of the tastiest things it possibly could: block level dedupe.

Except I no longer care.

Too impatient to wait for the RMA on the dead Asus P5Q-E (of which the replacement is now a spare swap-in board), thanks to an incredibly generous friend I picked up a Gigabyte GA-EP45-Extreme… which OpenSolaris b127 hated, and refused to boot with. After a few days of hair pulling and switching off almost everything I could in the BIOS to try and rectify the issue, I finally admitted OpenSolaris was not to be.

The Gigabyte GA-EP45 Extreme, great board, hated by OpenSolaris

Not willing to risk Nexenta, I dropped to FreeBSD 8, the last bastion of ZFS hope (no folks, FUSE does not count).

FreeBSD worked wonderfully from a compatibility front, but I soon discovered that when it came to virtualisation, it had the same options as a prisoner faced with the Spanish inquisition: basically none. There is, ironically, a version of Sun’s VirtualBox floating around, but it’s a hack job that hates 64-bit, and like most things FreeBSD if you’re not running from the command line you’re asking for pain.

And so, hoping that one day Larry Ellison would open up ZFS licensing a little more so the GPL crowd would stop whining and just integrate it already, I sighed, flicked the 3ware 9650SE into hardware RAID 6 and reached for the Ubuntu 9.10 64-bit disc.

It worked.

Post mortem: List of controller cards that will work with OpenSolaris

While I note with grim satisfaction that Areca has still failed to produce a Solaris driver for it’s ARC-1300ix series, here’s a list of PCI-Express cards known to work with OpenSolaris without requiring any RAID 0/JBOD workarounds, and being able to control at least eight drives.

LSI SAS3081E-R
Intel SASUC8I flashed with the SAS8031E-R’s IT (initiator target) firmware
3ware 9650SE series

Tiny, yes? The last, which I ended up with due to non-availability of the first two in Australia, is significantly more expensive as it has hardware RAID capability as well.

Post mortem: Final system

Rack: HP 10622
OS: Ubuntu 9.10
PSU: Corsair TX-850
CPU: Intel Q9550
Memory: 8GB Corsair Dominator PC-2 8500
Motherboard: Gigabyte GA-EP45 Extreme
GPU: Geforce 7600GS silent (to be swapped out with a PCI card when a second 3ware controller card is bought)
Controller card: 3ware 9650SE-8LPML
Network card: HP NC364T
Case: Chenbro RM41416B
UPS: APC Smart-UPS 750
Switch: Netgear GS724T
System drives: Samsung HD501LJ SATA
Array drives (RAID 6 w/XFS): WD RE3 1TB x3, Samsung HD103UJ 1TB x2, Seagate 7200.11 x2, Seagate 7200.12

The only problem left is the Seagate 7200.12, which seems to keep dropping from the array. I’ll have to see if a firmware update to the 3ware card fixes it, otherwise I may need to swap in a new drive (Update: turns out the ridiculously expensive Mini-SAS to SATA cables I bought were dodgy. Upon replacing, I’ve had no dropouts).

Curse of the server

Daedalus — Tue, 29 Sep 2009 12:10:00 +0000

Gasp! Intrigue! Another fucking motherboard!

I suppose at this point I really shouldn’t be surprised. The number of things that have gone wrong to date rival most government run projects.

Not covering old ground:

The Areca ARC-1300ix 16 was returned on the basis of no Solaris driver and it being a glorified port multiplier. I waited for about a month for the Adaptec 31605 on back order. After showing no signs of turning up any time soon, I cancelled the order and decided to pony up for the highly featured and crazily expensive 3ware 9650SE-8LPML instead. The day after, I find out Digicor has started distributing the SAS3081E-R again, a significantly cheaper option. Take in mind either choice locks me into a motherboard with at least three PCI-E x4 slots (due to needing 16 channels, and to equip the Intel quad gigabit Ethernet card).
The moment the 3ware arrives, I plug it in, and lo and behold, the machine no longer posts. Having seen a similar behaviour on the previous server (would or would not boot based on random hardware plugged in and how many times half the male population has scratched its crotch in the last hour while the wind is blowing west), I proceeded to disconnect everything until only RAM, CPU and GPU remained — and it still didn’t boot. Being that the only remaining part from the old server was the 850W CoolerMaster PSU, I ordered a Corsair TX-850 at AU$240 to remove all doubt, plugged it in AND;
The Corsair TX-850, a PSU with a beefy 12V rail and five year warranty - AU$240.
Discovered that the Asus P5Q-E motherboard, which replaced the exploded MSI was the dead part, despite no sparking, despite working a week ago, despite nothing being physically wrong with the board — it just stopped posting, meaning I’ve once again spent more cash than I have to.
The excellent Asus P5Q-E. Alas, it was not to be.

Well, fuck. That’s two motherboards gone in one build, which is making me wonder if the case is shorting something somehow. At this point I’ve had the chassis for over ten months, with no working system. It’s enough to make you want to buy a prebuilt NAS.

Meanwhile the brand new Netgear GS724T, APC SmartUPS 750 and HP 22RU rack just sit there, waiting for some action…

The mighty 3ware 9650SE-8LPML, coming to a server near you - AU$915.

Netgear GS724T - picked up for AU$280.

APC SmartUPS 750 - AU$231 on eBay.

HP 10622 rack - the one I bought on eBay for AU$180 likely has a bit more wear and tear than the one in this image.

More controller fuss, networking

Daedalus — Mon, 22 Jun 2009 13:40:48 +0000

I contacted PC-Pitstop to see if they would sell me the Areca ARC-1300ix 16, as it had started dribbling into stock in Canada and North America. They mentioned that one customer had an issue where the board could only see eight drives instead of the full 16, and wanted to wait until a new shipment to see if the problem persists before selling me anything.

Very nice of a store to say so. I’ve been waiting a long time though, and yet another card has risen to viability…

The LSI SAS3081E-R may provide salvation. Image courtesy of New Egg.

I’ve been told that LSI SAS3081E-R is capable of setting up drives individually, rather than requiring a RAID mode be set. It all seems to be hearsay and rumours though, no one can 100 percent confirm it. Fortunately it’s also sold with Sun gear, which makes compatibility for OpenSolaris a no-brainer. New Egg sells them for US$209, and apparently it’s on PCI-E x8, offering more bandwidth than the ARC-1300ix 16 and having to share it with less ports. Of course LSI can’t decide whether it offers 2400MB/s or 2500MB/s according to its site — given that PCI-E is 250MB/s unidirectional per lane, it can only offer a lower 2000MB/s.

In the meant time, I picked an HP NC364T on eBay Australia for AU$348, and am eagerly awaiting its arrival. The quad-gigabit Ethernet card uses dual Intel 82571EB chips over PCI-E x4, which are functionally identical to the 82751GB chips found on the Intel PRO/1000 PT Quad Port Low Profile adapter — for all intents and purposes making this a rebadge. Considering both cards go retail for around AU$1000, it’s an absolute steal, and should be fun to team or load balance over. By and large a dedicated network card should not only reduce the latency over an integrated, but may help maximise network throughput.

While I'm not quite ready to go 4GbE, the HP NC364T should help me grow a little.

More controller fuss, networking

Daedalus — Mon, 22 Jun 2009 13:40:48 +0000

Very nice of a store to say so. I’ve been waiting a long time though, and yet another card has risen to viability…

The LSI SAS3081E-R may provide salvation. Image courtesy of New Egg.

While I'm not quite ready to go 4GbE, the HP NC364T should help me grow a little.