In our environment ZFS has become a critical component of our Storage Infrastructure.  We have been able to provision Fibre Channel storage from our ZFS file systems which give us the benefit of data integrity, deduplication, performance through use of the ARC, L2ARC, and ZIL as needed.  Additionally the real benefit to ZFS is our storage can run on commodity hardware which reduces the TCO significantly over the major Storage vendors.   Now administering ZFS is really quite simple, however getting Fibre Channel can be a bit more work.  However after reading this you ought to be able to handle this with your eyes closed, well maybe not with your eyes closed.  But you will be much better prepared.

You will need

  • Working ZFS Implementation on Solaris 11 Express.
  • QLogic FC HBA which supports the QLT driver.  We are using the QLE2562.
  • Of course a working fabric and clients in order to utilize the storage.

Install Prerequisite Packages

bash 
pkg install stmf fcoet srpt target
Packages to install:     4
Create boot environment:    No
Services to restart:     1
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  4/4       38/38      0.8/0.8

PHASE                                        ACTIONS
Install Phase                                138/138

PHASE                                          ITEMS
Package State Update Phase                       4/4
Image State Update Phase                         2/2

PHASE                                          ITEMS
Reading Existing Index                           8/8
Indexing Packages                                4/4

Ensure that the Fibre Channel HBA is Detected

bash 
prtconf -D | grep qlc
pci1077,15d, instance #0 (driver name: qlc)
pci1077,15d, instance #1 (driver name: qlc)

Determine the PCI Address of the Fibre Channel HBA

bash 
mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs sd ip hook neti arp usba uhci stmf stmf_sbd qlc fctl sockfs md lofs random idm crypto smbsrv nfs sppp ipc ptm fcp cpc fcip nsmb ufs logindmux ]
> ::devbindings -q qlc
ffffff01cb184040 pciex1077,2532, instance #0 (driver name: qlc)
ffffff01cb183d90 pciex1077,2532, instance #1 (driver name: qlc)
> $q

Now we will need this from the above output pciex1077,2532.

Deactivate the QLC Driver for the Fibre Channel HBA

Remove the default driver (qlc) from the adapter.

bash 
update_drv -d -i 'pciex1077,2532' qlc

Activate the QLT Driver for the Fibre Channel HBA

Add the target driver (qlt) to the adapter.

bash 
update_drv -a -i 'pciex1077,2532' qlt

Reboot the Machine

bash 
reboot

Confirm the HBA is using the QLT Driver

bash 
prtconf -D | grep qlt
pci1077,15d, instance #0 (driver name: qlt)
pci1077,15d, instance #1 (driver name: qlt)
bash 
mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci zfs sd ip hook neti arp usba uhci stmf stmf_sbd fctl sockfs lofs random idm crypto cpc smbsrv nfs fcip sppp ufs ptm ipc ]
> ::devbindings -q qlc
> ::devbindings -q qlt
ffffff01cb184040 pciex1077,2532, instance #0 (driver name: qlt)
ffffff01cb183d90 pciex1077,2532, instance #1 (driver name: qlt)
> $q

Configure the SCSI Target Service

bash 
svcs stmf
STATE          STIME    FMRI
disabled       13:27:08 svc:/system/stmf:default
bash 
svcadm enable stmf
bash 
svcs stmf
STATE          STIME    FMRI
online         13:29:54 svc:/system/stmf:default

Display the Status of the SCSI Target Service

The ALUA Status is what allows a LUN to be served via FC and iSCSI at the same time.  We do not require or want it.

bash 
stmfadm list-state
Operational Status: online
Config Status     : initialized
ALUA Status       : disabled
ALUA Node         : 0

Create a File System (Volume) to be Provisioned

In order to present a LUN we will need to create a ZFS file system as a volume.

bash 
zfs create -V 10G tank/test_lun0

Notice the lack of a mount point, this is due to it being provisioned as a volume.

bash 
zfs list tank/test_lun0
NAME             USED  AVAIL  REFER  MOUNTPOINT
tank/test_lun0  10.3G  1.15T  23.9K  -

Create a LUN for the Volume

We must create a LUN with the Volume, this basically gives it a WWN.

bash 
sbdadm create-lu /dev/zvol/rdsk/tank/test_lun0
Created the following LU:

GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f0e8efcb0000004e52bcc60001  10737418240          /dev/zvol/rdsk/tank/test_lun0

Retrieve the WWN of the Adapters

This will be needed to add members to the Target Group.

bash 
fcinfo hba-port
HBA Port WWN: 21000024ff309840
Port Mode: Target
Port ID: 0
OS Device Name: Not Applicable
Manufacturer: QLogic Corp.
Model: QLE2562
Firmware Version: 5.2.1
FCode/BIOS Version: N/A
Serial Number: not available
Driver Name: COMSTAR QLT
Driver Version: 20100505-1.05
Type: unknown
State: offline
Supported Speeds: 2Gb 4Gb 8Gb
Current Speed: not established
Node WWN: 20000024ff309840
HBA Port WWN: 21000024ff309841
Port Mode: Target
Port ID: 0
OS Device Name: Not Applicable
Manufacturer: QLogic Corp.
Model: QLE2562
Firmware Version: 5.2.1
FCode/BIOS Version: N/A
Serial Number: not available
Driver Name: COMSTAR QLT
Driver Version: 20100505-1.05
Type: unknown
State: offline
Supported Speeds: 2Gb 4Gb 8Gb
Current Speed: not established
Node WWN: 20000024ff309841

Lets trim the output for readability.

bash 
fcinfo hba-port | grep "HBA Port WWN"
HBA Port WWN: 21000024ff309840
HBA Port WWN: 21000024ff309841

Create Target Group

The Target Group is used to control which adapters are used to present LUNs.

bash 
stmfadm create-tg ust4zfs201

Add Members to our Target Group

Any WWN that we add to this group will be able to present the LUNs we associate with this Target Group via a view.

bash 
stmfadm add-tg-member -g ust4zfs201 wwn.21000024ff309840
stmfadm: STMF target must be offline

Since the SCSI Target service is already running we must stop it to add members to the Target Group.

bash 
svcadm disable svc:/system/stmf:default

We want to associate all of our WWN with this Target Group since we won’t be restricting LUNs to a specific adapter.

bash 
stmfadm add-tg-member -g ust4zfs201 wwn.21000024ff309840
bash 
stmfadm add-tg-member -g ust4zfs201 wwn.21000024ff309841

Now we can restart the SCSI Target service

bash 
svcadm enable svc:/system/stmf:default

Create Host Group

The Host Group is used to allow access to LUN(s) by Client(s).  I name my Host Groups functionally, so for example if the storage would be associated with a single client named testhostg then I would name the group that.  If it were to be used as shared storage for a VM then I would name it as the VM instead, since it would attach to multiple clients.

bash 
stmfadm create-hg testhostg

Add Members to the Host Group

Here we simply add the WWN of the client(s) to the Host Group.

bash 
stmfadm add-hg-member -g testhostg wwn.2100001b32878b01

Create a View

The view will tie everything together.  It associates the Host Group with the LUN and the Target Group.  Without this the LUN will not be usable.

bash 
sbdadm list-lu

Found 1 LU(s)

GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f0e8efcb0000004e52bcc60001  10737418240          /dev/zvol/rdsk/tank/test_lun0

600144f0e8efcb0000004e52bcc60001 is what we needed.  Now lets wrap everything up neatly in a View.

bash 
stmfadm add-view -h testhostg -t ust4zfs201 600144f0e8efcb0000004e52bcc60001

Now from your client you should see your storage as a block device.  You will need to perform a scan or a refresh from the client.