(The information on this page is specific to the Linux 3.2 and 3.14 kernels, but its application is not limited to those kernels.)
OK, I'm going to start this page with a message to all the people on the internet who insist that you can't see the physical drives attached to an HP P400 SAS RAID controller from within the OS, or that you can't use an HP P400 with 3TB or larger physical drives.
BOLLOCKS.
Right, well now that's over we can get on with the LP. This is Wayne, my rubber plant...
The other week I was poking about on Amazon to see if I could find any bargains in their hard drives. What I actually found was a festering dog's arse. Unfortunately it wasn't listed as "festering dog's arse", it was listed as "Supermicro 2000GB SAS HDD Industrial 24/7, HDD-2000GB-SAS" with a price of twelve quid each. And they were down to the last 5 they had in stock. So I bought the lot. Well, what the fuck would you do?
Of course, I now had to get a SAS interface card as well. So I went looking for one. At first the only ones I found were stupidly fucking expensive, but I kept looking, and eventually found a Hewlett-Packard P400 SAS RAID controller card for 40 quid. So I ordered it, and in due course it turned up.
The hard drives didn't, though. That delivery took a couple more weeks to arrive and when it did it wasn't hard drives, it was mice. It was kind of like a technological version of Cinderella. So I emailed the seller to ask how long they'd been employing fairy godmothers in Dispatch.
Their reply informed me that Supermicro 2000GB SAS HDD Industrial 24/7, HDD-2000GB-SAS and Festering Dog's Arse happened to share a stock number, and this caused hard drives to turn into mice in the post. Since I had no use for any more mice, I sent them back for a refund, and when that came through I went back to the hard drive bargain hunt.
The first thing I found was that the Supermicro 2000GB SAS HDD Industrial 24/7, HDD-2000GB-SAS vs. Festering Dog's Arse confusion affected several more sellers. The bunch I'd originally ordered from weren't listing them any more but a bunch of others were, at similarly cheap prices. So to anyone else who happens across that listing - be warned. It isn't a hard drive and it isn't bleeding Christmas and it isn't worth spunking in your pants and buying as many as possible, because all you will get is a box of mice.
Further searching revealed that the best example of hard cheap driviness now available was a Seagate ST33000651SS for 80 quid odd. In comparison with other drives they had this is cheapish for an ordinary SATA 3TB drive, and fucking cheap for a 3TB SAS drive. Indeed, this thing apparently normally went for £480 odd. And it was being sold by Amazon themselves, not some other buggers, so I felt confident that this listing was actually for real and it would not turn into a mouse. So I bought it, and was eventually most pleased to discover that it really was a real hard drive.
Finally at bleeding last I was able to plug the card into the motherboard, connect the drive to it and boot the fucker up. Yep, here we go. The P400 is giving me a choice of... er... configuring a RAID0 array with one physical drive of capacity zero and then bombing out when it tried to save the configuration, or not doing this. Yeah, well, not to worry, at least it does see the drive; the OS'll sort it out and I'm not daft enough to actually want a RAID with one physical drive anyway.
Boot Linux. Poke through dmesg and /dev and /proc and /sys and this and that. Is there any sign of the drive? Is there arse. I can google around and download some utility with a name like a tribe of Eskimos that tells me the drive is "unsupported". Yes, you cunt, but I don't care if the RAID bit supports it or not, I just want to use the card as a straight HBA so it doesn't fucking matter does it. Poke around some more. Still no result. Google around some more and find a load of people asking about the same sort of question, and one and all receiving the answer that the HP P400 cannot be used as a straight HBA device so if you're trying to use one with a 3TB drive you're fucked. No, not even if you update the firmware. Not on this card. Is fockings. Is piss of sheet.
Oh, you great fucking cuntarsed shitewank of a dried up spunk dribble. Cunt, fuck, cunt, fuck, cunt, fuck, cunt.
In that case... what else have they got on Amazon? (No, not Ebay, fuck Ebay, every fucking thing on Ebay insists on Paypal and I can't fucking use Paypal because Paypal are fucking cunts who don't like me because I am called Pigeon. Yes, you did read that right. Paypal are such a bunch of fucking wankers that they have decided to make it impossible for me to use their service because they don't like my name. Fucking pieces of shit.) Er... there are some more cheap ones which (google google) ...also can't handle 3TB drives, some shiteily-described cheap ones which (google google google) ...there are plenty of other people selling with equally shitey descriptions but no actual fucking information whatsoever, and some stupidly fucking expensive ones, as in order of magnitude more expensive, which is taking the fucking piss so they can all get fucked. Anyway, this thing about it being impossible to get at the physical drives is surely bollocks; all the hardware is there, and if there's some bloody processor sitting in the way and complaining about how big everything is these days there has got to be some way of telling the thing to fuck off and not bloody look if it doesn't like it.
In the Linux kernel source there are comments in the code for this card to the effect that such-and-such might let the OS see the physical drives "and that would be bad". No it bloody wouldn't, you dolt, it's exactly what I want to happen. The FreeBSD driver author has taken a more enlightened attitude. That driver has a boot parameter which you can set to make sure it does happen (and by the looks of it it's had it for at least 10 years and servers round the world have not gone into meltdown as a result). Cool! (For ChrisOfAllTrades: "Masked devices can be exposed by setting the hw.ciss.expose_hidden_physical tunable to non-zero at boot time.") So how does it work then... er... oh fuck me, it hardly does anything. It just makes one bloody conditional always come out false. There is a bit in the LUN data returned by the controller from the drive which tells the driver to hide the drive, and all you have to do is ignore it.
From which it follows that if it is possible to find the place where the Linux driver tests that bit and tell it not to, there is a good chance that... FUCKING YES!!! FUCKING /dev/sdf FUCKING ROCK ON THERE IT FUCKING IS! Partition it and format it and mount it and THREE TB FUCKING THREE TB ERE WE GO ERE WE GO ERE WE GO ect ect ect.
So it is BOLLOCKS that you can't see the physical drives on an HP P400 from the OS. You can see them and you can use them. FreeBSD can do it out of the box and now Linux can do it too.
(For notbelgianbutdutch: your assertion "p400 doesn't do >2TB, line-stops-here. It just doesn't, don't bother trying" misses the point. The P400 in this setup isn't "doing" >2TB. All it's doing is acting as an interface. It is the OS that is "doing" >2TB. By enabling the OS to see the raw physical drives, we are bypassing the intelligence of the P400's chipset and in so doing bypassing its size limitations, so they don't count.)
Here is the patch for kernel 3.2.0.
Download: hpsa_expose_masked-3.2.0.patch.gz
--- hpsa.c.orig 2014-11-30 02:19:19.149567805 +0000 +++ hpsa.c 2014-11-30 02:26:54.081554358 +0000 @@ -80,6 +80,10 @@ module_param(hpsa_simple_mode, int, S_IRUGO|S_IWUSR); MODULE_PARM_DESC(hpsa_simple_mode, "Use 'simple mode' rather than 'performant mode'"); +static int hpsa_expose_masked; +module_param(hpsa_expose_masked, int, S_IRUGO|S_IWUSR); +MODULE_PARM_DESC(hpsa_expose_masked, + "Expose masked physical devices (cf. BSD hw.ciss.expose_hidden_physical)"); /* define the PCI info for the cards we can control */ static const struct pci_device_id hpsa_pci_device_id[] = { @@ -1964,10 +1968,12 @@ lunaddrbytes = figure_lunaddrbytes(h, raid_ctlr_position, i, nphysicals, nlogicals, physdev_list, logdev_list); /* skip masked physical devices. */ - if (lunaddrbytes[3] & 0xC0 && - i < nphysicals + (raid_ctlr_position == 0)) - continue; - + if (!hpsa_expose_masked) { + if (lunaddrbytes[3] & 0xC0 && + i < nphysicals + (raid_ctlr_position == 0)) + continue; + } + /* Get device type, vendor, model, device id */ if (hpsa_update_device_info(h, lunaddrbytes, tmpdevice, &is_OBDR)) @@ -2006,7 +2012,7 @@ ncurrent++; break; case TYPE_DISK: - if (i < nphysicals) + if ((!hpsa_expose_masked) && (i < nphysicals)) break; ncurrent++; break;
What this does is it adds a new module parameter hpsa_expose_masked which instructs the hpsa driver to act like the BSD one when similarly configured and ignore the hide-this-device flag. So you insert the module with
# modprobe hpsa hpsa_allow_any=1 hpsa_expose_masked=1
and now you can see the physical drives as normal SCSI devices and we are all as merry as lambkins on the lea.
(The only "wrinkle" is that the cciss module will probably be loaded by default and you have to unload it before the hpsa module will do anything, then blacklist it to make sure it isn't loaded in future.)
For kernel 3.14 the situation is slightly different. Although it is possible to apply the patch for 3.2 to the driver in 3.14, with a suitable allowance of fuzz, there have nevertheless been significant changes in the driver code, and the result is that the driver throws a page fault and locks up. (It is possible, of course, that using the unmodified 3.14 hpsa driver with a P400 by means of hpsa_allow_any also results in a crash; I haven't tried it.)
Fortunately, though, the 3.2 driver code still works when inserted into the 3.14 source tree in place of the 3.14 driver. The only extra bit needed is a minor edit to cope with a couple of #defines that are no longer present in 3.14. Yes, this is horrible. No, I don't care.
Download: hpsa_expose_masked-3.2.0_for_3.14.patch.gz
hpsa driver code from 3.2: linux-3.2-hpsa.tar.gz
--- hpsa.c.orig 2012-12-06 11:20:40.000000000 +0000 +++ hpsa.c 2015-03-26 18:51:33.591509696 +0000 @@ -80,6 +80,10 @@ module_param(hpsa_simple_mode, int, S_IRUGO|S_IWUSR); MODULE_PARM_DESC(hpsa_simple_mode, "Use 'simple mode' rather than 'performant mode'"); +static int hpsa_expose_masked; +module_param(hpsa_expose_masked, int, S_IRUGO|S_IWUSR); +MODULE_PARM_DESC(hpsa_expose_masked, + "Expose masked physical devices (cf. BSD hw.ciss.expose_hidden_physical)"); /* define the PCI info for the cards we can control */ static const struct pci_device_id hpsa_pci_device_id[] = { @@ -1946,10 +1950,12 @@ lunaddrbytes = figure_lunaddrbytes(h, raid_ctlr_position, i, nphysicals, nlogicals, physdev_list, logdev_list); /* skip masked physical devices. */ - if (lunaddrbytes[3] & 0xC0 && - i < nphysicals + (raid_ctlr_position == 0)) - continue; - + if (!hpsa_expose_masked) { + if (lunaddrbytes[3] & 0xC0 && + i < nphysicals + (raid_ctlr_position == 0)) + continue; + } + /* Get device type, vendor, model, device id */ if (hpsa_update_device_info(h, lunaddrbytes, tmpdevice, &is_OBDR)) @@ -1988,7 +1994,7 @@ ncurrent++; break; case TYPE_DISK: - if (i < nphysicals) + if ((!hpsa_expose_masked) && (i < nphysicals)) break; ncurrent++; break; @@ -4554,7 +4560,7 @@ static struct pci_driver hpsa_pci_driver = { .name = "hpsa", .probe = hpsa_init_one, - .remove = __devexit_p(hpsa_remove_one), + .remove = hpsa_remove_one, .id_table = hpsa_pci_device_id, /* id_table */ .shutdown = hpsa_shutdown, .suspend = hpsa_suspend, --- hpsa.h.orig 2015-03-26 18:48:52.930175930 +0000 +++ hpsa.h 2015-03-26 18:50:59.395225804 +0000 @@ -26,6 +26,9 @@ #define IO_OK 0 #define IO_ERROR 1 +#define __devinit +#define __devexit + struct ctlr_info; struct access_method {
It works, too. It doesn't alias sectors by losing the top bits or any shit like that. I wrote a 3TB file to it containing all different numbers and then read it back again and all the numbers were where they were supposed to be, it didn't wrap around and start overwriting the start of the file while leaving the rest of the disk untouched or anything. I have full and proper access to a 3TB physical drive using a Hewlett Packard P400 as a straight HBA and all the people who insist that that isn't possible can get fucked. They are WRONG ON THE INTERNET and I have proved it.
(I have encountered the WRITE SAME failed. Manually zeroing. bug while using this patch in conjunction with LVM. This is pretty much a "cosmetic" bug, though; no data corruption results, and the standard workaround of putting a suitably-edited command along the lines of echo '0' > /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/host7/target7:2:2/7:2:2:0/scsi_disk/7:2:2:0/max_write_same_blocks in the init scripts does the trick as usual.)
For 3.15 and up the situation is different again, and interestingly so, although I have not experimented with it. The hpsa driver in 3.15 was officially modified to expose the physical devices - as long as the card is in HBA mode. Unfortunately, the P400 doesn't actually have an HBA mode as such; it is possible to instruct the tribe of Eskimos to attempt to set it to HBA mode, but nothing actually happens. Still, when/if I get round to messing with it it might open the possibility of a more elegant hack.
That this hack works is especially gratifying because this whole thing of SAS cards being orders of magnitude more expensive than SATA is a load of fucking shit anyway. SAS and SATA are electrically the same, the difference is purely one of protocols. This means that building a SAS or SATA interface requires the same parts whichever it is, and the only difference is in software. Therefore anyone who charges a hundred times as much for a SAS interface as for a SATA one is perpetrating a ginormous fucking rip-off and being a ginormous fucking cunt. So it is very pleasing to be able to stick two fingers up at the bastards with nothing more than a couple of lines of code.
Of course this hack doesn't affect the BIOS not being able to see the physical drive, so you still can't boot off it, but that's no big deal. It just means you have to install some other drive to boot off that doesn't go through the card. Since all this has to have on it is /boot any old bleeding drive will do and if you spend a tenner on it you're spending too much. A freebie USB stick hard-wired to one of the spare USB headers on the motherboard would do. Me, I'm doing that thing where you put all the static OS files on a solid state drive anyway, so it doesn't matter at all.
Also of course it means that I won't be able to use the hardware RAID even if I do get another drive (which I more than likely will at some point). Again, this doesn't matter. I'd rather use Linux software RAID in any case because the giant massive arse with hardware RAID is that if the controller fails the whole bloody thing is fucked and you need an identical card to make it work again and even then it's not guaranteed. Performance? Well, I used Linux software RAID on a bunch of SCSI320 drives and it was fast as fuck on a 2GHz single-core 32-bit CPU. This box has a 3.5GHz 8-core 64-bit CPU so I'd not expect to notice any dissatisfaction with the performance at all.
Back to Pigeon's Nest
Be kind to pigeons