[illumos-Developer] Thread stuck in zio_wait(), holding a mutex many threads want...

[Gordon Ross]

Anyone recognize this?  We see a thread stuck in zio_wait(),
holding a mutex many other threads want...

Note this is older ZFS code, not latest illumos.
Pointers to already fixed bugs appreciated.

Here's the thread that owns the mutex everyone is waiting for,
and the ZIO object it's working on.

 > ffffff002eaebc60 ::findstack -v
 stack pointer for thread ffffff002eaebc60: ffffff002eaeb5b0
  ffffff002eaeb5e0 swtch+0x145()
  ffffff002eaeb610 cv_wait+0x61(ffffff07a3fd8ca8, ffffff07a3fd8ca0)
  ffffff002eaeb650 zio_wait+0x5d(ffffff07a3fd89a0)
  ffffff002eaeb6b0 dbuf_read+0x1e8(ffffff079f7b9aa0, 0, a)
  ffffff002eaeb730 dmu_buf_hold+0xac(ffffff0774a379c0, ffffffffffffffff, 0, 0,
  ffffff002eaeb768, 1)
  ffffff002eaeb7d0 zap_lockdir+0x6d(ffffff0774a379c0, ffffffffffffffff, 0, 1, 1
  , 0, ffffff002eaeb7f8)
  ffffff002eaeb860 zap_lookup_norm+0x55(ffffff0774a379c0, ffffffffffffffff,
  ffffff002eaeb940, 8, 1, ffffff002eaeb8d8, 0, 0, 0, 0)
  ffffff002eaeb8c0 zap_lookup+0x2d(ffffff0774a379c0, ffffffffffffffff,
  ffffff002eaeb940, 8, 1, ffffff002eaeb8d8)
  ffffff002eaeb930 zap_increment+0x64(ffffff0774a379c0, ffffffffffffffff,
  ffffff002eaeb940, fffffffc001e6a00, ffffff079c6f5470)
  ffffff002eaeb9b0 zap_increment_int+0x68(ffffff0774a379c0, ffffffffffffffff,
  ea61, fffffffc001e6a00, ffffff079c6f5470)
  ffffff002eaeba10 do_userquota_update+0x69(ffffff0774a379c0, 3ffe19400, 3, ea61
  , ea61, 1, ffffff079c6f5470)
  ffffff002eaeba70 dmu_objset_do_userquota_updates+0xde(ffffff0774a379c0,
  ffffff079c6f5470)
  ffffff002eaebaf0 dsl_pool_sync+0x112(ffffff073f046900, 2508)
  ffffff002eaebba0 spa_sync+0x37b(ffffff06fca7aac0, 2508)
  ffffff002eaebc40 txg_sync_thread+0x247(ffffff073f046900)
  ffffff002eaebc50 thread_start+8()


 ffffff07a3fd89a0 ::zio -cr
 ADDRESS                                  TYPE  STAGE            WAITER

 ffffff07a3fd89a0                         NULL  CHECKSUM_VERIFY
ffffff002eaebc60
  ffffff079fabd058                        READ  VDEV_IO_START    -
   ffffff07aa936650                       READ  VDEV_IO_DONE     -
    ffffff0755b4c378                      READ  VDEV_IO_START    -

Thanks!