[illumos-Developer] Webrev for bug 323: Need fix for glob() resource exhaustion

[Gary Mills]

This is for illumos bug 323: Need fix for glob() resource exhaustion.
It fixes the FTP server portion of the bug report.  My webrev is at:

    http://cr.illumos.org/view/2oi3hoic/illumos323/

Note that the original glob.c contains 521 assorted cstyle violations.
I didn't fix any of these but added 23 more.  I'd have to reformat the
entire file to do this correctly, greatly expanding my fix.

Ideas for my fix are from the BSD patch.  I also used the same
resource limits.  These limits are severe but not normally exceeded.
Resources used in normal operation and those imposed by the exploits
are quite different, easily distinguished by the limits.

I did all my testing with a small program that links with glob.c,
calling ftpglob() in the same manner as the FTP server does.  I tested
each limit separately to ensure that each was effective.  When many
paths are matched, ftpglob() will now report `Out of memory'.  With no
paths matched, it will now report `Arguments too long'.  These are
existing error strings, not new ones, so that translations will not be
affected.

Without my fixes, ftpglob() is vulnerable to both exploits listed in
the bug report.  With them, it handles both of them by terminating
with an error.  This is an interim fix that resolves the DOS
vulnerabilities.  Replacing the FTP server with a modern one that's
better maintained is a better solution.

-- 
-Gary Mills-        -Unix Group-        -Computer and Network Services-