[illumos-Developer] networking question - local address when virtual IP aliases exist?

[Dan McDonald]

On Mon, Jun 27, 2011 at 02:28:47PM -0400, Gordon Ross wrote:
> I'm seeing something odd when a system has virtual NICs, i.e. an "IP alias"
> in addition to the system's primary IP address.  This situation often exists
> with cluster setups where all cluster nodes have a "physical" IP that never
> changes or moves, plus some set of "virtual" IP addresses that may move
> to any physical host based on cluster fail-over logic, etc.
> 
> I don't have all the details yet, but it appears that outbound connections
> are sometimes using the "physical" IP address as the local address,
> and other times using a "virtual" IP as the local address.  Who cares?
> 
> Well, when the SMB service makes a connection to an AD server, the
> AD server appears to decide we're an "impostor" trying to use some
> other machine's AD account, and denies our logon request.
> 
> So...  Other than making the application bind the local endpoint using
> an explicit IP address, is there any way to mark some interfaces as
> more or less preferred for anonymous (local) IP bind requests?

You can use IFF_DEPRECATED on the less-important addresses.

This link:

http://src.illumos.org/source/s?refs=ipif_select_source_v4&project=illumos-gate
points to the source-address selection for IPv4.  This is where these
addresses get decided.  Look at the comment block:

/*
 * Pick a source address based on the destination ill and an optional setsrc
 * address.
 * The result is stored in srcp. If generation is set, then put the source
 * generation number there before we look for the source address (to avoid
 * missing changes in the set of source addresses.
 * If flagsp is set, then us it to pass back ipif_flags.
 *
 * If the caller wants to cache the returned source address and detect when
 * that might be stale, the caller should pass in a generation argument,
 * which the caller can later compare against ips_src_generation
 *
 * The precedence order for selecting an IPv4 source address is:
 *  - RTF_SETSRC on the offlink ire always wins.
 *  - If usrsrc is set, swap the ill to be the usesrc one.
 *  - If IPMP is used on the ill, select a random address from the most
 *    preferred ones below:
 * 1. If onlink destination, same subnet and not deprecated, not ALL_ZONES
 * 2. Not deprecated, not ALL_ZONES
 * 3. If onlink destination, same subnet and not deprecated, ALL_ZONES
 * 4. Not deprecated, ALL_ZONES
 * 5. If onlink destination, same subnet and deprecated
 * 6. Deprecated.
 *
 * We have lower preference for ALL_ZONES IP addresses,
 * as they pose problems with unlabeled destinations.
 *
 * Note that when multiple IP addresses match e.g., #1 we pick
 * the first one if IPMP is not in use. With IPMP we randomize.
 */

So there's factors to consider:

	- Any ALL_ZONES IP addresses in a zoned system?

	- Any use of IPMP?

	- As you pointed on IRC, is the desired local address the first one
          in a list (e.g. e1000g0:0)?

Someone else may need to chime in here in case I'm missing something, but
please watch out for those.  A simple test running connect() with unspecified
local addresses and differing remotes can help shake things out as well.

Dan