[illumos-Developer] Important - time sensitive: Drive failures and infinite waits

Alasdair Lumsden alasdairrr at gmail.com
Thu May 26 07:35:08 PDT 2011


Hi Garrett,

On 26 May 2011, at 15:18, Garrett D'Amore wrote:

> Please supply your zpool status -v  so that we can see your pools.

zpool status hangs, however the drive layout consists of:

zpool create -f data \
mirror c1t0d0 c7t0d0 \
mirror c1t1d0 c7t1d0 \
mirror c1t2d0 c7t2d0 \
mirror c1t3d0 c7t3d0 \
mirror c1t4d0 c7t4d0 \
mirror c1t5d0 c7t5d0 \
mirror c1t6d0 c7t6d0 \
mirror c1t7d0 c7t7d0 \
mirror c2t0d0 c8t0d0 \
mirror c2t1d0 c8t1d0 \
mirror c2t2d0 c8t2d0 \
mirror c2t3d0 c8t3d0 \
mirror c2t4d0 c8t4d0 \
mirror c2t5d0 c8t5d0 \
mirror c2t6d0 c8t6d0 \
mirror c2t7d0 c8t7d0 \
mirror c3t8d0 c9t0d0 \
mirror c3t9d0 c9t1d0 \
mirror c3t10d0 c9t2d0 \
mirror c3t11d0 c9t3d0 \
mirror c3t12d0 c9t4d0 \
mirror c3t13d0 c9t5d0 \
mirror c3t14d0 c9t6d0 \
spare c3t15d0 c9t7d0 \
log c3t0d0 \
 c3t1d0 \
 c3t2d0 \
 c3t3d0


Here's the output from iostat:

c6d0             Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Model: INTEL SSDSA2M04 Revision:  Serial No: CVGB006101SZ040 Size: 40.02GB <40020148224 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 
c5d1             Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Model: OCZ-VERTEX      Revision:  Serial No: ODV323H0G038DTD Size: 32.02GB <32017047552 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 
c2t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5381117 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c2t1d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5443548 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c2t2d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5678931 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c2t3d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5356168 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c2t4d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5549628 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c2t5d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5533204 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c2t6d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5470750 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c2t7d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5372521 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c1t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5404909 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c1t1d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5549732 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c1t2d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5444576 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c1t3d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5422458 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c1t4d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5678480 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c1t5d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5678100 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c1t6d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5655057 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c1t7d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5675652 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c8t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5478645 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c8t1d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATW0305838 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c8t2d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5549060 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c8t3d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5678513 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c8t4d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5510655 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c8t5d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5397990 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c8t6d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 1D05 Serial No: WD-WCATR0142053 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c8t7d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5424683 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c7t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5268905 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c7t1d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5655767 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c7t2d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5431042 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c7t3d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5549248 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c7t4d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5534719 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c7t5d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5495506 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c7t6d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5443019 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c7t7d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5663476 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c9t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5414895 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c9t1d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5547432 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c9t2d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5656489 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c9t3d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5656236 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c9t4d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5655697 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c9t5d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5285236 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c9t6d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5506168 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c9t7d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5655149 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 1 Predictive Failure Analysis: 0 
c3t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: MARVELL SD88SA02 Revision: D20R Serial No: 1002M020XF      
Size: 24.58GB <24575868928 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 5 Predictive Failure Analysis: 0 
c3t1d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: MARVELL SD88SA02 Revision: D20R Serial No: 1002M0218A      
Size: 24.58GB <24575868928 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 3 Predictive Failure Analysis: 0 
c3t2d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: MARVELL SD88SA02 Revision: D20R Serial No: 1002M0217Q      
Size: 24.58GB <24575868928 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 5 Predictive Failure Analysis: 0 
c3t3d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: MARVELL SD88SA02 Revision: D20R Serial No: 1002M021A8      
Size: 24.58GB <24575868928 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 3 Predictive Failure Analysis: 0 
c3t8d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5378260 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c3t9d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 1D05 Serial No: WD-WCATR0179048 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c3t10d0          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 1D05 Serial No: WD-WCATR0176705 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c3t11d0          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5425093 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 6 Predictive Failure Analysis: 0 
c3t12d0          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV4834948 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c3t13d0          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5546687 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c3t14d0          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5534965 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 4 Predictive Failure Analysis: 0 
c3t15d0          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD7501AALS-0 Revision: 0K05 Serial No: WD-WMATV5532272 
Size: 750.16GB <750156374016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 1 Predictive Failure Analysis: 0 


Running the mdb command "::walk zio_root | ::zio -r" shows:

> ::walk zio_root | ::zio -r
ADDRESS                                  TYPE  STAGE            WAITER

ffffff0918ca4cd0                         NULL  CHECKSUM_VERIFY  ffffff003da48c40
 ffffff0915a27c80                        WRITE VDEV_IO_START    -
 ffffff0918eafcc0                        WRITE VDEV_IO_START    -
 ffffff0918d09968                        WRITE VDEV_IO_START    -
 ffffff0918aadc98                        WRITE VDEV_IO_START    -
ffffff09195eb358                         NULL  OPEN             -
ffffff090dfecc88                         NULL  OPEN             -

> ffffff0918ca4cd0::zio
ADDRESS                                  TYPE  STAGE            WAITER          
ffffff0918ca4cd0                         NULL  CHECKSUM_VERIFY  ffffff003da48c40
> ffffff0918ca4cd0::zio -r
ADDRESS                                  TYPE  STAGE            WAITER          
ffffff0918ca4cd0                         NULL  CHECKSUM_VERIFY  ffffff003da48c40
 ffffff0915a27c80                        WRITE VDEV_IO_START    -
 ffffff0918eafcc0                        WRITE VDEV_IO_START    -
 ffffff0918d09968                        WRITE VDEV_IO_START    -
 ffffff0918aadc98                        WRITE VDEV_IO_START    -
> ffffff0918ca4cd0::print -t struct zio
struct zio {
    zbookmark_t io_bookmark = {
        uint64_t zb_objset = 0
        uint64_t zb_object = 0
        int64_t zb_level = 0
        uint64_t zb_blkid = 0
    }
    zio_prop_t io_prop = {
        enum zio_checksum zp_checksum = 0 (ZIO_CHECKSUM_INHERIT)
        enum zio_compress zp_compress = 0 (ZIO_COMPRESS_INHERIT)
        dmu_object_type_t zp_type = 0 (DMU_OT_NONE)
        uint8_t zp_level = 0
        uint8_t zp_copies = 0
        uint8_t zp_dedup = 0
        uint8_t zp_dedup_verify = 0
    }
    zio_type_t io_type = 0 (ZIO_TYPE_NULL)
    enum zio_child io_child_type = 3 (ZIO_CHILD_LOGICAL)
    int io_cmd = 0
    uint8_t io_priority = 0
    uint8_t io_reexecute = 0
    uint8_t [2] io_state = [ 0x1, 0 ]
    uint64_t io_txg = 0
    spa_t *io_spa = 0xffffff090e71a580
    blkptr_t *io_bp = 0
    blkptr_t *io_bp_override = 0
    blkptr_t io_bp_copy = {
        dva_t [3] blk_dva = [
            dva_t {
                uint64_t [2] dva_word = [ 0, 0 ]
            },
            dva_t {
                uint64_t [2] dva_word = [ 0, 0 ]
            },
            dva_t {
                uint64_t [2] dva_word = [ 0, 0 ]
            },
        ]
        uint64_t blk_prop = 0         
        uint64_t [2] blk_pad = [ 0, 0 ]
        uint64_t blk_phys_birth = 0
        uint64_t blk_birth = 0
        uint64_t blk_fill = 0
        zio_cksum_t blk_cksum = {
            uint64_t [4] zc_word = [ 0, 0, 0, 0 ]
        }
    }
    list_t io_parent_list = {
        size_t list_size = 0x30
        size_t list_offset = 0x10
        struct list_node list_head = {
            struct list_node *list_next = 0xffffff0918ca4dc0
            struct list_node *list_prev = 0xffffff0918ca4dc0
        }
    }
    list_t io_child_list = {
        size_t list_size = 0x30
        size_t list_offset = 0x20
        struct list_node list_head = {
            struct list_node *list_next = 0xffffff0918da5788
            struct list_node *list_prev = 0xffffff090e24c380
        }
    }
    zio_link_t *io_walk_link = 0
    zio_t *io_logical = 0
    zio_transform_t *io_transform_stack = 0
    zio_done_func_t *io_ready = 0
    zio_done_func_t *io_done = 0
    void *io_private = 0xffffff003da489e8
    int64_t io_prev_space_delta = 0
    blkptr_t io_bp_orig = {
        dva_t [3] blk_dva = [
            dva_t {
                uint64_t [2] dva_word = [ 0, 0 ]
            },
            dva_t {
                uint64_t [2] dva_word = [ 0, 0 ]
            },
            dva_t {
                uint64_t [2] dva_word = [ 0, 0 ]
            },
        ]
        uint64_t blk_prop = 0
        uint64_t [2] blk_pad = [ 0, 0 ]
        uint64_t blk_phys_birth = 0
        uint64_t blk_birth = 0
        uint64_t blk_fill = 0
        zio_cksum_t blk_cksum = {
            uint64_t [4] zc_word = [ 0, 0, 0, 0 ]
        }
    }
    void *io_data = 0
    void *io_orig_data = 0
    uint64_t io_size = 0
    uint64_t io_orig_size = 0
    vdev_t *io_vd = 0
    void *io_vsd = 0
    const zio_vsd_ops_t *io_vsd_ops = 0
    uint64_t io_offset = 0
    uint64_t io_deadline = 0
    avl_node_t io_offset_node = {
        struct avl_node *[2] avl_child = [ 0, 0 ]
        uintptr_t avl_pcb = 0
    }
    avl_node_t io_deadline_node = {
        struct avl_node *[2] avl_child = [ 0, 0 ]
        uintptr_t avl_pcb = 0
    }
    avl_tree_t *io_vdev_tree = 0
    enum zio_flag io_flags = 0x140 (ZIO_FLAG_{CANFAIL|CONFIG_WRITER})
    enum zio_stage io_stage = 0x80000 (ZIO_STAGE_CHECKSUM_VERIFY)
    enum zio_stage io_pipeline = 0x108000 (ZIO_STAGE_{READY|DONE})
    enum zio_flag io_orig_flags = 0x140 (ZIO_FLAG_{CANFAIL|CONFIG_WRITER})
    enum zio_stage io_orig_stage = 0x1 (ZIO_STAGE_OPEN)
    enum zio_stage io_orig_pipeline = 0x108000 (ZIO_STAGE_{READY|DONE})
    int io_error = 0
    int [4] io_child_error = [ 0, 0, 0, 0 ]
    unsigned long [4][2] io_children = [
        unsigned long [2] [ 0, 0x4 ]
        unsigned long [2] [ 0, 0 ]
        unsigned long [2] [ 0, 0 ]
        unsigned long [2] [ 0, 0 ]
    ]
    uint64_t io_child_count = 0x4
    uint64_t io_parent_count = 0
    uint64_t *io_stall = 0xffffff0918ca4f60
    zio_t *io_gang_leader = 0
    zio_gang_node_t *io_gang_tree = 0
    void *io_executor = 0xffffff003da48c40
    void *io_waiter = 0xffffff003da48c40
    kmutex_t io_lock = {
        void *[1] _opaque = [ 0 ]
    }
    kcondvar_t io_cv = {
        ushort_t _opaque = 0x1
    }
    zio_cksum_report_t *io_cksum_report = 0
    uint64_t io_ena = 0
}
> ffffff0918ca4cd0::print -t struct zio io_waiter
void *io_waiter = 0xffffff003da48c40
> ::walk thread ! grep ffffff003da48c40
mdb: cannot access bash: No such file or directory
> 0xffffff003da48c40::findstack
stack pointer for thread ffffff003da48c40: ffffff003da48920
[ ffffff003da48920 _resume_from_idle+0xf1() ]
  ffffff003da48950 swtch+0x145()
  ffffff003da48980 cv_wait+0x61()
  ffffff003da489c0 zio_wait+0x5d()
  ffffff003da48a40 vdev_uberblock_sync_list+0x163()
  ffffff003da48ad0 vdev_config_sync+0x129()
  ffffff003da48b80 spa_sync+0x5cd()
  ffffff003da48c20 txg_sync_thread+0x247()
  ffffff003da48c30 thread_start+8()
> 

I can (and am keen to) provide SSH access to the box so people can test.

I'm also in a position to quickly compile and test patches against the kernel (I've got a build box which churns out builds in ~30 mins)

Cheers,

Alasdair


More information about the Developer mailing list