[illumos-Advocates] RTI 175 zfs vdev cache consumes excessive memory
Gordon Ross
gordon.w.ross at gmail.com
Fri Apr 22 06:51:04 PDT 2011
Looks good to me. (though it looks like you got approval already?)
On Fri, Apr 22, 2011 at 12:37 AM, Garrett D'Amore <garrett at nexenta.com> wrote:
> This is a trivial change, already reviewed by the ZFS list -- we
> basically disable the vdev cache by setting the size to zero. (Diff
> below.)
>
> garrett at thinkpad{8}> hg outgoing -v
> running ssh anonhg at hg.illumos.org "hg -R illumos-gate serve --stdio"
> comparing with ssh://anonhg@hg.illumos.org/illumos-gate
> searching for changes
>
> changeset: 13343:5211e1f2192b
> tag: tip
> user: Garrett D'Amore <garrett at nexenta.com>
> date: Thu Apr 21 21:34:04 2011 -0700
>
> description:
> 175 zfs vdev cache consumes excessive memory
> Reviewed by: George Wilson <george.wilson at delphix.com>
> Reviewed by: Eric Schrock <eric.schrock at delphix.com>
>
> modified:
> usr/src/uts/common/fs/zfs/vdev_cache.c
>
>
> hg pbchk:
>
> garrett at thinkpad{9}> hg pbchk
> Copyright check:
> usr/src/uts/common/fs/zfs/vdev_cache.c: no copyright claim for current
> year found
>
> C style check:
>
> Header format check:
>
> Java style check:
>
> Mapfile comment check:
>
> File permission check:
>
> Keywords check:
>
> Comments check:
>
> Checking for new tags:
>
> Checking for multiple heads (or branches):
>
> Checking for branch changes:
>
> Checking for uncommitted changes:
>
> Checking for merges:
>
>
> Testing: We (Nexenta) have been running with this change for some time
> now in some large deployments. As well, I've verified this on my own
> laptop. Additionally, the tunable is zero on Solaris 11, so this has
> been well exercised broadly.
>
> Note that in the nightly build, the complaints are the result of my
> naive copying of the closed tree, which broke links and symlinks,
> resulting in some things being improperly populated in the proto area.
> I'm not going to rebuild though, since it doesn't relate to this
> particular change.
>
> garrett at thinkpad{10}> cat log/log.2011-04-21.03:05/mail_msg
>
> ==== Nightly distributed build started: Thu Apr 21 00:57:19 PDT 2011
> ====
> ==== Nightly distributed build completed: Thu Apr 21 03:05:06 PDT 2011
> ====
>
> ==== Total build time ====
>
> real 2:07:46
>
> ==== Build environment ====
>
> /usr/bin/uname
> SunOS thinkpad 5.11 qlc2322 i86pc i386 i86pc
>
> /opt/SUNWspro/bin/dmake
> dmake: Sun Distributed Make 7.8 SunOS_i386 Patch 126504-01 2007/07/19
> number of concurrent jobs = 10
>
> 32-bit compiler
> /opt/onbld/bin/i386/cw -_cc
> cw version 1.29
> primary: /opt/SUNWspro/bin/cc
> cc: Sun C 5.9 SunOS_i386 Patch 124868-10 2009/04/30
> shadow: /usr/sfw/bin/gcc
> gcc (GCC) 3.4.3 (csl-sol210-3_4-20050802)
>
> 64-bit compiler
> /opt/onbld/bin/i386/cw -_cc
> cw version 1.29
> primary: /opt/SUNWspro/bin/cc
> cc: Sun C 5.9 SunOS_i386 Patch 124868-10 2009/04/30
> shadow: /usr/sfw/bin/gcc
> gcc (GCC) 3.4.3 (csl-sol210-3_4-20050802)
>
> /usr/java/bin/javac
> java full version "1.6.0_21-b06"
>
> /usr/ccs/bin/as
> as: Sun Compiler Common 12 SunOS_i386 snv_121 08/03/2009
>
> /usr/ccs/bin/ld
> ld: Software Generation Utilities - Solaris Link Editors: 5.11-1.1726
>
> Build project: group.staff
> Build taskid: 169
>
> ==== Nightly argument issues ====
>
>
> ==== Build version ====
>
> zfs-vdev
>
> ==== Make clobber ERRORS ====
>
>
> ==== Make tools clobber ERRORS ====
>
>
> ==== Tools build errors ====
>
>
> ==== Build errors (DEBUG) ====
>
>
> ==== Build warnings (DEBUG) ====
>
>
> ==== Elapsed build time (DEBUG) ====
>
> real 1:16:26.6
> user 6:41:40.6
> sys 1:21:38.0
>
> ==== Build noise differences (DEBUG) ====
>
>
> ==== package build errors (DEBUG) ====
>
>
> ==== Validating manifests against proto area ====
>
> Entries present in proto area but not manifests:
> dir group=group mode=0755 owner=owner
> path=usr/lib/locale/POSIX/LC_COLLATE
> dir group=group mode=0755 owner=owner
> path=usr/lib/locale/POSIX/LC_CTYPE
> dir group=group mode=0755 owner=owner
> path=usr/lib/locale/POSIX/LC_MESSAGES
> dir group=group mode=0755 owner=owner
> path=usr/lib/locale/POSIX/LC_MONETARY
> dir group=group mode=0755 owner=owner
> path=usr/lib/locale/POSIX/LC_NUMERIC
> dir group=group mode=0755 owner=owner path=usr/lib/locale/POSIX/LC_TIME
> file usr/lib/locale/POSIX/locale_description group=group mode=0444
> owner=owner path=usr/lib/locale/POSIX/locale_description
>
> Entries that differ between manifests and proto area:
> manifests hardlink path=kernel/strmod/amd64/sdpib
> target=kernel/drv/amd64/sdpib
> proto area file kernel/strmod/amd64/sdpib group=group mode=0755
> owner=owner path=kernel/strmod/amd64/sdpib
> manifests hardlink path=kernel/strmod/sdpib target=kernel/drv/sdpib
> proto area file kernel/strmod/sdpib group=group mode=0755
> owner=owner path=kernel/strmod/sdpib
> manifests hardlink
> path=platform/i86pc/kernel/cpu/amd64/cpu_ms.GenuineIntel.6.47
> target=platform/i86pc/kernel/cpu/amd64/cpu_ms.GenuineIntel.6.46
> proto area file
> platform/i86pc/kernel/cpu/amd64/cpu_ms.GenuineIntel.6.47 group=group
> mode=0755 owner=owner
> path=platform/i86pc/kernel/cpu/amd64/cpu_ms.GenuineIntel.6.47
> manifests hardlink
> path=platform/i86pc/kernel/cpu/cpu_ms.GenuineIntel.6.47
> target=platform/i86pc/kernel/cpu/cpu_ms.GenuineIntel.6.46
> proto area file platform/i86pc/kernel/cpu/cpu_ms.GenuineIntel.6.47
> group=group mode=0755 owner=owner
> path=platform/i86pc/kernel/cpu/cpu_ms.GenuineIntel.6.47
> manifests link path=usr/lib/fwflash/verify/ses-LSILOGIC.so
> target=ses-SUN.so
> proto area file usr/lib/fwflash/verify/ses-LSILOGIC.so group=group
> mode=0755 owner=owner path=usr/lib/fwflash/verify/ses-LSILOGIC.so
> manifests link path=usr/lib/fwflash/verify/sgen-LSILOGIC.so
> target=ses-SUN.so
> proto area file usr/lib/fwflash/verify/sgen-LSILOGIC.so group=group
> mode=0755 owner=owner path=usr/lib/fwflash/verify/sgen-LSILOGIC.so
> manifests link path=usr/lib/fwflash/verify/sgen-SUN.so
> target=ses-SUN.so
> proto area file usr/lib/fwflash/verify/sgen-SUN.so group=group
> mode=0755 owner=owner path=usr/lib/fwflash/verify/sgen-SUN.so
> manifests link path=usr/lib/locale/POSIX target=C
> proto area dir group=group mode=0755 owner=owner
> path=usr/lib/locale/POSIX
>
>
> ==== Check ELF runtime attributes ====
>
>
> ==== Diff ELF runtime attributes (since last build) ====
>
>
> ==== 'dmake lint' of src ERRORS ====
>
>
> ==== Elapsed time of 'dmake lint' of src ====
>
> real 35:58.2
> user 1:35:25.6
> sys 1:25:37.6
>
> ==== lint warnings src ====
>
>
> ==== lint noise differences src ====
>
>
> ==== cstyle/hdrchk errors ====
>
>
> ==== Find core files ====
>
>
> ==== Check lists of files ====
>
>
> ==== Impact on file permissions ====
>
>
>
> diff -r af0a1d7f121d -r 5211e1f2192b
> usr/src/uts/common/fs/zfs/vdev_cache.c
> --- a/usr/src/uts/common/fs/zfs/vdev_cache.c Wed Apr 20 19:50:50 2011
> -0400
> +++ b/usr/src/uts/common/fs/zfs/vdev_cache.c Thu Apr 21 21:34:04 2011
> -0700
> @@ -71,9 +71,16 @@
> * 1<<zfs_vdev_cache_bshift byte reads by the vdev_cache (aka software
> * track buffer). At most zfs_vdev_cache_size bytes will be kept in
> each
> * vdev's vdev_cache.
> + *
> + * TODO: Note that with the current ZFS code, it turns out that the
> + * vdev cache is not helpful, and in some cases actually harmful. It
> + * is better if we disable this. Once some time has passed, we should
> + * actually remove this to simplify the code. For now we just disable
> + * it by setting the zfs_vdev_cache_size to zero. Note that Solaris 11
> + * has made these same changes.
> */
> int zfs_vdev_cache_max = 1<<14; /* 16KB */
> -int zfs_vdev_cache_size = 10ULL << 20; /* 10MB */
> +int zfs_vdev_cache_size = 0;
> int zfs_vdev_cache_bshift = 16;
>
> #define VCBS (1 << zfs_vdev_cache_bshift) /* 64KB */
>
>
>
> _______________________________________________
> Advocates mailing list
> Advocates at lists.illumos.org
> http://lists.illumos.org/m/listinfo/advocates
>
More information about the Advocates
mailing list