CDDL HEADER START The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License. You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License. When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner] CDDL HEADER END Copyright 2010 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Implementation Overview for the NetWork AutoMagic daemon John Beck, Renee Danson, Michael Hunter, Alan Maguire, Kacheong Poon, Garima Tripathi, Jan Xie, Anurag Maskey [Structure and some content shamelessly stolen from Peter Memishian's dhcpagent architecture overview.] INTRODUCTION ============ Details about the NWAM requirements, architecture, and design are available via the NWAM opensolaris project at http://opensolaris.org/os/project/nwam. The point of this document is to place details relevant to somebody attempting to understand the implementation close to the source code. THE BASICS ========== SOURCE FILE ORGANIZATION ======================= event sources: dlpi_events.c routing_events.c sysevent_events.c object-specific event handlers: enm.c known_wlans.c loc.c ncp.c ncu_ip.c ncu_phys.c legacy config upgrade llp.c generic code: objects.c events.c conditions.c logging.c util.c nwam door requests: door_if.c entry point: main.c OVERVIEW ======== Here we discuss the essential objects and subtle aspects of the NWAM daemon implementation. Note that there is of course much more that is not discussed here, but after this overview you should be able to fend for yourself in the source code. Events and Objects ================== Events come to NWAM from a variety of different sources asyncronously. o routing socket o dlpi o sysevents o doors Routing sockets and dlpi (DL_NOTE_LINK_UP|DOWN events) are handled by dedicated threads. Sysevents and doors are both seen as callbacks into the process proper and will often post their results to the main event queue. All event sources post events onto the main event queue. In addition state changes of objects and door requests (requesting current state or a change of state, specification of a WiFi key etc) can lead to additional events. We have daemon-internal events (object initialization, periodic state checks) which are simply enqueued on the event queue, and external events which are both enqueued on the event queue and sent to registered listeners (via nwam_event_send()). So the structure of the daemon is a set of threads that drive event generation. Events are posted either directly onto the event queue or are delayed by posting onto the pending event queue. SIGALARMs are set for the event delay, and when the SIGALARM is received pending events that have expired are moved onto the event queue proper. Delayed enqueueing is useful for periodic checks. Decisions to change conditions based upon object state changes are delayed until after bursts of events. This is achieved by marking a flag when it is deemed checking is necessary and then the next time the queue is empty performing those checks. A typical event profile will be one event (e.g. a link down) causing a flurry of other events (e.g. related interface down). By waiting until all the consequences of the initial event have been carried out to make higher level decisions we implicitly debounce those higher level decisions. At the moment queue quiet actually means that the queue has been quiet for some short period of time (.1s). Typically the flurry of events we want to work through are internally generated and are back to back in the queue. We wait a bit longer in case there are reprucussions from what we do that cause external events to be posted on us. We are not interested in waiting for longer term things to happen but merely to catch immediate changes. When running, the daemon will consist of a number of threads: o the event handling thread: a thread blocking until events appear on the event queue, processing each event in order. Events that require time-consuming processing are spawned in worker threads (e.g. WiFi connect, DHCP requests etc). o door request threads: the door infrastructure manages server threads which process synchronous NWAM client requests (e.g. get state of an object, connect to a specific WLAN, initiate a scan on a link etc). o various wifi/IP threads: threads which do asynchronous work such as DHCP requests, WLAN scans etc that cannot hold up event processing in the main event handling thread. o routing socket threads: process routing socket messages of interest (address additons/deletions) and package them as NWAM messages. o dlpi threads: used to monitor for DL_NOTE_LINK messages on links The daemon is structured around a set of objects representing NCPs[1], NCUs[2], ENMs[3] and known WLANs and a set of state machines which consume events which act on those objects. Object lists are maintained for each object type, and these contain both a libnwam handle (to allow reading the object directly) and an optional object data pointer which can point to state information used to configure the object. Events can be associated with specific objects (e.g. link up), or associated with no object in particular (e.g. shutdown). Each object type registers a set of event handler functions with the event framework such that when an event occurs, the appropriate handler for the object type is used. The event handlers are usually called nwamd_handle_*_event(). [1] NCP Network Configuration Profile; the set of link- and IP-layer configuration units which collectively specify how a system should be connected to the network [2] NCU Network Configuration Unit; the individual components of an NCP [3] ENM External Network Modifiers; user executable scripts often used to configure a VPN Doors and External Events ========================= The command interface to nwamd is thread a door at NWAM_DOOR (/etc/svc/volatile/nwam/nwam_door). This door allows external program to send messages to nwamd. The way doors work is to provide a mechanism for another process to execute code in your process space. This looks like a CSPish send/receive/reply in that the receiving process provide a syncronization point (via door_create(3C)), the calling process uses that syncronization point to rendezvous with and provide arguments (via door_call(3C), and then the receive process reply (via door_return(3C))) passing back data as required. The OS makes it such that the memory used to pass data via door_call(3C) is mapped into the receiving process which can write back into it and then transparently have it mapped back to the calling process. As well as handling internal events of interest, the daemon also needs to send events of interest (link up/down, WLAN scan/connect results etc) to (possibly) multiple NWAM client listeners. This is done via System V message queues. On registering for events via a libnwam door request into the daemon (nwam_events_register()), a per-client (identified by pid) message queue file is created. The daemon sends messages to all listeners by examining the list of message queue files (allowing registration to be robust across daemon restarts) and sending events to each listener. This is done via the libnwam function nwam_event_send() which hides the IPC mechanism from the daemon. Objects ======= Four object lists are maintained within the daemon - one each for the configuration objects libnwam manages. i.e.: o ENMs o locations o known WLANs o NCUs of the current active NCP Objects have an associated libnwam handle and an optional data field (which is used for NCUs only). Locking is straightforward - nwamd_object_init() will initialize an object of a particular type in the appropriate object list, returning it with the object lock held. When it is no longer needed, nwamd_object_unlock() should be called on the object. To retrieve an existing object, nwamd_object_find() should be called - again this returns the object in a locked state. nwamd_object_lock() is deliberately not exposed outside of objects.c, since object locking is implicit in the above creation/retrieval functions. An object is removed from the object list (with handle destroyed) via nwamd_object_fini() - the object data (if any) is returned from this call to allow deallocation. Object state ============ nwamd deals with 3 broad types of object that need to maintain internal state: NCUs, ENMs and locations (known WLANs are configuration objects but don't have a state beyond simply being present). NWAM objects all share a basic set of states: State Description ===== =========== uninitialized object representation not present on system or in nwamd initialized object representation present in system and in nwamd disabled disabled manually offline external conditions are not satisfied offline* external conditions are satisfied, trying to move online online* external conditions no longer satisfied, trying to move offline online conditions satisfied and configured maintenance error occurred in applying configuration These deliberately mimic SMF states. The states of interest are offline, offline* and online. An object (link/interface NCU, ENM or location) should only move online when its conditions are satisfied _and_ its configuration has been successfully applied. This occurs when an ENM method has run or a link is up, or an interface has at least one address assigned. To understand the distinction between offline and offline*, consider the case where a link is of prioritized activation, and either is a lower priority group - and hence inactive (due to cable being unplugged or inability to connect to wifi) - or a higher priority group - and hence active. In general, we want to distinguish between two cases: 1) when we are actively configuring the link with a view to moving online (offline*), as would be the case when the link's priority group is active. 2) when external policy-based conditions prevent a link from being active. offline should be used for such cases. Links in priority groups above and below the currently-active group will be offline, since policy precludes them from activating (as less-prioritized links). So we see that offline and offline* can thus be used to distinguish between cases that have the potentiality to move online (offline*) from a policy perspective - i.e. conditions on the location allow it, or link prioritization allows it - and cases where external conditions dictate that it should not (offline). Once an object reaches offline*, its configuration processes should kick in. This is where auxiliary state is useful, as it allows us to distinguish between various states in that configuration process. For example, a link can be waiting for WLAN selection or key data, or an interface can be waiting for DHCP response. This auxiliary state can then also be used diagnostically by libnwam consumers to determine the current status of a link, interface, ENM etc. WiFi links present a problem however. On the one hand, we want them to be inactive when they are not part of the current priority grouping, while on the other we want to watch out for new WLANs appearing in scan data if the WiFi link is of a higher priority than the currently-selected group. The reason we watch out for these is they represent the potential to change priority grouping to a more preferred group. To accommodate this, WiFi links of the same or lower (more preferred) priority group will always be trying to connect (and thus be offline* if they cannot). It might appear unnecessary to have a separate state value/machine for auxiliary state - why can't we simply add the auxiliary state machine to the global object state machine? Part of the answer is that there are times we need to run through the same configuration state machine when the global object state is different - in paticular either offline* or online. Consider WiFi - we want to do periodic scans to find a "better" WLAN - we can easily do this by running back through the link state machine of auxiliary states, but we want to stay online while we do it, since we are still connected (if the WLAN disconnects of course we go to LINK_DOWN and offline). Another reason we wish to separate the more general states (offline, online etc) from the more specific ones (WIFI_NEED_SELECTION etc) is to ensure that the representation of configuration objects closely matches the way SMF works. For an NCU physical link, the following link-specific auxiliary states are used: Auxiliary state Description =============== =========== LINK_WIFI_SCANNING Scan in progress LINK_WIFI_NEED_SELECTION Need user to specify WLAN LINK_WIFI_NEED_KEY Need user to specify a WLAN key for selection LINK_WIFI_CONNECTING Connecting to current selection A WiFI link differs from a wired one in that it always has the potential to be available - it just depends if visited WLANs are in range. So such links - if they are higher in the priority grouping than the currently-active priority group - should always be able to scan, as they are always "trying" to be activated. Wired links that do not support DL_NOTE_LINK_UP/DOWN are problematic, since we have to simply assume a cable is plugged in. If an IP NCU is activated above such a link, and that NCU uses DHCP, a timeout will be triggered eventually (user-configurable via the nwamd/ncu_wait_time SMF property of the network/physical:nwam instance) which will cause us to give up on the link. For an IP interface NCU, the following auxiliary states are suggested. Auxiliary state Description =============== =========== NWAM_AUX_STATE_IF_WAITING_FOR_ADDR Waiting for an address to be assigned NWAM_AUX_STATE_IF_DHCP_TIMED_OUT DHCP timed out on interface A link can have multiple logical interfaces plumbed on it consisting of a mix of static and DHCP-acquired addresses. This means that we need to decide how to aggregate the state of these logical interfaces into the NCU state. The concept of "up" we use here does not correspond to IFF_UP or IFF_RUNNING, but rather when we get (via getting RTM_NEWADDR events with non-zero addresses) at least one address assigned to the link. We use this concept of up as it represents the potential for network communication - e.g. after assigning a static address, if the location specifies nameserver etc, it is possible to communicate over the network. One important edge case here is that when DHCP information comes in, we need to reassess location activation conditions and possibly change or reapply the current location. The problem is that if we have a static/DHCP mix, and if we rely on the IP interface's notion of "up" to trigger location activation, we will likely first apply the location when the static address has been assigned and before the DHCP information has been returned (which may include nameserver info). So the solution is that on getting an RTM_NEWADDR, we check if the (logical) interface associated is DHCP, and even if the interface NCU is already up, we reassess location activation. This will lead to a reapplication of the current location or possibly a location switch. In order to move through the various states, a generic API is supplied nwam_error_t nwamd_object_set_state(nwamd_object_t obj, nwamd_state_t state, nwamd_aux_state_t aux_state); This function creates an OBJECT_STATE event containing the new state/aux_state and enqueues it in the event queue. Each object registers its own handler for this event, and in response to the current state/aux state and desired aux state it responds appropriately in the event handling thread, spawning other threads to carry out actions as appropriate. The object state event is then sent to any registered listeners. So for NCUs, we define a handle_object_state() function to run the state machine for the NCU object. Link state and NCP policy ========================= NCPs can be either: o prioritized: where the constituent link NCUs specify priority group numbers (where lower are more favoured) and grouping types. These are used to allow link NCUs to be either grouped separately (exclusive) or together (shared or all). o manual: their activation is governed by the value of their enabled property. o a combination of the above. IP interface NCUs interit their activation from the links below them, so an IP interface NCU will be active if its underlying link is (assuming it hasn't been disabled). At startup, and at regular intervals (often triggered by NWAM events), the NCP policy needs to be reassessed. There are a number of causes for NCP policy to be reassessed - o a periodic check of link state that occurs every N seconds o a link goes from offline(*) to online (cable plug/wifi connect) o a link goes from online to offline (cable unplug/wifi disconnect). Any of these should cause the link selecton algorithm to rerun. The link selection algorithm works as follows: Starting from the lowest priority grouping value, assess all links in that priority group. The current priority-group is considered failed if: o "exclusive" NCUs exist and none are offline*/online, o "shared" NCUs exist and none are offline*/online, o "all" NCUs exist and all are not offline*/online, o no NCUs are offline*/online. We do not invalidate a link that is offline* since its configuration is in progress. This has the unfortunate side-effect that wired links that do not do DL_NOTE_LINK_UP/DOWN will never fail. If such links wish to be skipped, their priority group value should be increased (prioritizing wireless links). One a priority group has been selected, all links in groups above _and_ below it need to be moved offline. Location Activation =================== A basic set of system-supplied locations are supplied - NoNet and Automatic. nwamd will apply the NoNet location until such a time as an interface NCU is online, at which point it will switch to the Automatic location. If a user-supplied location is supplied, and it is either manually enabled or its conditions are satisfied, it will be preferred and activated instead. Only one location can be active at once since each location has its own specification of nameservices etc. ENM Activation ============== ENMs are either manual or conditional in activation and will be activated if they are enabled (manual) or if the conditions are met (conditional). Multiple ENMs can be active at once.