[illumos-Developer] tr webrev

[Joerg Schilling]

"Garrett D'Amore" <garrett at nexenta.com> wrote:

> /usr/bin/tr was integrated with very little testing... basic
> functionality testing that I performed. I didn't perform exhaustive
> testing for two reasons:
>
> a) I thought I had heard that FreeBSD tr was already reasonably tested.

See the classification that I send before.

If only a single part is changed and the rest of the software could be called
tested and bug-free, tests for the changed part are easier if you know what to 
test.

You did run an untested (ported) FreeBSD tr on top of an untested i18n 
implementation. It turned out later, that the i18n implementation did not set 
up any locale except for "C" if the official POSIX procedures were used and for 
this reason tr with any non-US-ASCII char did not work. The reason to integrate 
the FreeBSD tr was however to achive i18n support, so tr needs to work with 
non-US-ASCII characters.


> b) What we had at the time was *completely* unusable.  We did not even
> have access to a closed tr.   This was preventing illumos from hosting
> itself for compilation.

Allowing to self host Illumos is an unrelated issue.


> As far as testing procedures, I'm happy to work to develop more
> resilient and robust testing plans.

If you do a small change that is _fully_ understood (i.e. it is provable that 
the change has no side effects), you could do limited testing of the affected 
features only.

Replacing i18n and tr at the same time is a really big change that cannot be 
fully understood and for this rason, extensive testing is needed.

An example for a change that cannot be fully understood is the hardening I 
applied in 2007 to the "hsfs" code against panics caused by defective 
filesystems. This is something important as optical media comes from unknown 
sources and you cannot asume anything. I had to run a random byte 
destroy / mount / find -ls test for two months together with a friend in order 
to find the last forgotten test against inconststent data in a 
ISO-9660/Rock-Ridge filesystem. Fortunately, this is something that _can_ be 
tested by using a random test method.

Unless we have an automated test for tr that would allow to test everything 
(which is not possible), we could create a "testing" source gate and encourage 
people to do testing for a longer period of time until we could move the tested 
bits into the stable gate. After all POSIX related Closed source code was 
replaced by OSS, I am willing to get a free copy of the POSIX conformance test 
tools in order to be able to run at least standardized test (even though the 
coverage of the POSIX test suite does not seem to be good).

I did just finish the first SchilliX distro release based on Illumos (after 
fixing an inherited bug from Oracle with missing dependencies to the 
"svc:/network/ipsec/ipsecalgs:default" service.

		ftp://ftp.berlios.de/pub/schillix/

The name if this distro is "0.7.1i" (i for Illumos), see the DVD image:
SchilliX-0.7.1i.iso.xz

As this offers an easy to reproduce Illumos based platform for testing, I 
recommend to run tests in this platform, even though it still contains the 
FreeBSD tr that does not support -c and -C.

As this is the first reproducable and only Illumos based distro, this is also 
the right platform for independently working on the self-hosting problem. There 
is no unknown code on the distro so everything we need to add in order to 
achieve self hosting is really needed.

> I'm not happy to integrate any change that has not had the basic minimum
> level of testing that consists of "perform a full nightly build and boot
> a system with your change in place".  I *did* perform this level of
> testing with my tr replacement, btw.

You did not do a testing of all features and the complete functionality.

You _may_ have tested the following (if you did use _all_ scripts in the OS):

	tr -s '/:' '__'
	tr -s ' '
	tr '.' ' '
	tr '[A-Z]' '[a-z]'
	tr , '\012'
	tr -d ' '
	tr / '\001'
	tr '\001'
	tr -d '\012'
	tr / ' '

As it is most unlikely that you used any of the scripts that call tr (did you 
call "shotdown" and watch the messages?), there is a high probability that tr 
was not tested at all by what you call a "boot and system test"

I however did test the full set of all features for my fix to the keyboard 
layout bug.

For the fix to the dependency bug in 
/lib/svc/manifest/network/ipsec/ipsecalgs.xml, it can now be said that the 
problem was fully understood (the problem was the missing device/socket setup
before calling "/usr/sbin/ipsecalgs -s" and the fix proved that calling the 
command after device/socket setup caused the problem to disappear.

The problem in /lib/svc/method/keymap was also fully understood and my tests 
did even test the complete functionality.

What we see here, that it is obvious that the official Sun/Oracle procedures 
for testing before doing a putback are not sufficient and that we need better
testing procedures. If we don't work on this problem, the quality of Illumos
that was already degraded by Sun in the past 2 years would further degrade.

usr/src/cmd/kbd/keymap (installed as /lib/svc/method/keymap) was not tested by 
Sun and did work by accident only. After Sun did introduce an incompatible 
change in in /etc/default/kbd in Spring, /lib/svc/method/keymap failed for 
non-US keyboards as this change did discover a bug that existed for a long time
already.

/lib/svc/manifest/network/ipsec/ipsecalgs.xml was not tested by Sun/Oracle and 
for this reason, a dependency to device/socket setup was mssing and caused 
SchilliX to fail at a rate > 95%.

The integration of OSS i18n and tr is important and needed but it is not ready
and it has degraded the current Illumos to a pre-alpha state.

I am willing to share my testing know how....

In hope that we will be able to approach the well known product quality for
Solaris on Illumos soon again.


Jörg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily