From owner-freebsd-xen@FreeBSD.ORG Thu Sep 1 17:34:48 2011 Return-Path: Delivered-To: freebsd-xen@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9FB861065672 for ; Thu, 1 Sep 2011 17:34:48 +0000 (UTC) (envelope-from gibbs@scsiguy.com) Received: from aslan.scsiguy.com (www.scsiguy.com [70.89.174.89]) by mx1.freebsd.org (Postfix) with ESMTP id 6A97B8FC08 for ; Thu, 1 Sep 2011 17:34:48 +0000 (UTC) Received: from Justins-MacBook-Pro.local (207-225-98-3.dia.static.qwest.net [207.225.98.3]) (authenticated bits=0) by aslan.scsiguy.com (8.14.4/8.14.4) with ESMTP id p81GubFc067589 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Thu, 1 Sep 2011 10:56:37 -0600 (MDT) (envelope-from gibbs@scsiguy.com) Message-ID: <4E5FB8F0.9090204@scsiguy.com> Date: Thu, 01 Sep 2011 10:55:12 -0600 From: "Justin T. Gibbs" User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:6.0) Gecko/20110812 Thunderbird/6.0 MIME-Version: 1.0 To: "freebsd-xen@freebsd.org" Content-Type: multipart/mixed; boundary="------------020503020206070006020408" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (aslan.scsiguy.com [70.89.174.89]); Thu, 01 Sep 2011 10:56:37 -0600 (MDT) Cc: Subject: [CFT/CFR] Improved Xen Suspend/Resume Support X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Sep 2011 17:34:48 -0000 This is a multi-part message in MIME format. --------------020503020206070006020408 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit This work was sponsored by BQ Internet Corporation and they have successfully migrated 100+ FreeBSD VMs with these changes (slightly modified for use in 8.x). I'd like to have feedback from other developers and FreeBSD Xen users before submitting this and/or asking for approval to push them into 9.0. Thanks, Justin --------------020503020206070006020408 Content-Type: text/plain; x-mac-type="0"; x-mac-creator="0"; name="xen_suspend.diffs" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="xen_suspend.diffs" ========================================================================== Change 501527 by justing@justing-ns1 on 2011/08/14 14:12:39 Properly handle suspend/resume events in the Xen device framework. sys/xen/xenbus/xenbusb.c: o In xenbusb_resume(), publish the state transition of the resuming device into XenbusStateIntiailising so that the remote peer can see it. Recording the state locally is not sufficient to trigger a re-connect sequence. o In xenbusb_resume(), defer new-bus resume processing until after the remote peer's XenStore address has been updated. The drivers may need to refer to this information during resume processing. sys/xen/xenbus/xenbusb_back.c: sys/xen/xenbus/xenbusb_front.c: Register xenbusb_resume() rather than bus_generic_resume() as the handler for device_resume events. sys/xen/xenstore/xenstore.c: o Fix grammer in a comment. o In xs_suspend(), pass suspend events on to the child devices (e.g. xenbusb_front/back, that are attached to the XenStore. Affected files ... ... //depot/SpectraBSD/head/sys/xen/xenbus/xenbusb.c#9 edit ... //depot/SpectraBSD/head/sys/xen/xenbus/xenbusb_back.c#7 edit ... //depot/SpectraBSD/head/sys/xen/xenbus/xenbusb_front.c#6 edit ... //depot/SpectraBSD/head/sys/xen/xenstore/xenstore.c#5 edit Differences ... ==== //depot/SpectraBSD/head/sys/xen/xenbus/xenbusb.c#9 (text) ==== @@ -773,7 +773,7 @@ ivars = device_get_ivars(kids[i]); xs_unregister_watch(&ivars->xd_otherend_watch); - ivars->xd_state = XenbusStateInitialising; + xenbus_set_state(kids[i], XenbusStateInitialising); /* * Find the new backend details and @@ -783,16 +783,16 @@ if (error) return (error); - DEVICE_RESUME(kids[i]); - statepath = malloc(ivars->xd_otherend_path_len + strlen("/state") + 1, M_XENBUS, M_WAITOK); sprintf(statepath, "%s/state", ivars->xd_otherend_path); free(ivars->xd_otherend_watch.node, M_XENBUS); ivars->xd_otherend_watch.node = statepath; + + DEVICE_RESUME(kids[i]); + xs_register_watch(&ivars->xd_otherend_watch); - #if 0 /* * Can't do this yet since we are running in ==== //depot/SpectraBSD/head/sys/xen/xenbus/xenbusb_back.c#7 (text) ==== @@ -292,7 +292,7 @@ DEVMETHOD(device_detach, bus_generic_detach), DEVMETHOD(device_shutdown, bus_generic_shutdown), DEVMETHOD(device_suspend, bus_generic_suspend), - DEVMETHOD(device_resume, bus_generic_resume), + DEVMETHOD(device_resume, xenbusb_resume), /* Bus Interface */ DEVMETHOD(bus_print_child, xenbusb_print_child), ==== //depot/SpectraBSD/head/sys/xen/xenbus/xenbusb_front.c#6 (text) ==== @@ -171,7 +171,7 @@ DEVMETHOD(device_detach, bus_generic_detach), DEVMETHOD(device_shutdown, bus_generic_shutdown), DEVMETHOD(device_suspend, bus_generic_suspend), - DEVMETHOD(device_resume, bus_generic_resume), + DEVMETHOD(device_resume, xenbusb_resume), /* Bus Interface */ DEVMETHOD(bus_print_child, xenbusb_print_child), ==== //depot/SpectraBSD/head/sys/xen/xenstore/xenstore.c#5 (text) ==== @@ -721,8 +721,8 @@ /* * The count of transactions drops if we attempted * to end a transaction (even if that attempt fails - * in error), we receive a transaction end acknowledgement - * or if our attempt to begin a transactionfails. + * in error), we receive a transaction end acknowledgement, + * or if our attempt to begin a transaction fails. */ if (request_msg_type == XS_TRANSACTION_END || (request_reply_error == 0 && reply_msg_type == XS_TRANSACTION_END) @@ -1194,9 +1194,15 @@ * all transactions and individual requests have completed. */ static int -xs_suspend(device_t dev __unused) +xs_suspend(device_t dev) { + int error; + /* Suspend child Xen devices. */ + error = bus_generic_suspend(dev); + if (error != 0) + return (error); + sx_xlock(&xs.suspend_mutex); sx_xlock(&xs.request_mutex); @@ -1227,6 +1233,9 @@ sx_xunlock(&xs.suspend_mutex); + /* Resume child Xen devices. */ + bus_generic_resume(dev); + return (0); } ========================================================================== Change 501528 by justing@justing-ns1 on 2011/08/14 14:16:19 sys/dev/xen/control/control.c: Fix locking violations in Xen HVM suspend processing and have it perform similar actions to those performed during an ACPI triggered suspend. Affected files ... ... //depot/SpectraBSD/head/sys/dev/xen/control/control.c#7 edit Differences ... ==== //depot/SpectraBSD/head/sys/dev/xen/control/control.c#7 (text) ==== @@ -115,6 +115,7 @@ #include #include #include +#include #include #include #include @@ -201,6 +202,8 @@ int i, j, k, fpp; unsigned long max_pfn, start_info_mfn; + EVENTHANDLER_INVOKE(power_suspend); + #ifdef SMP struct thread *td; cpuset_t map; @@ -221,7 +224,13 @@ stop_cpus(map); #endif + /* + * Be sure to hold Giant across DEVICE_SUSPEND/RESUME since non-MPSAFE + * drivers need this. + */ + mtx_lock(&Giant); if (DEVICE_SUSPEND(root_bus) != 0) { + mtx_unlock(&Giant); printf("xen_suspend: device_suspend failed\n"); #ifdef SMP if (!CPU_EMPTY(&map)) @@ -229,6 +238,7 @@ #endif return; } + mtx_unlock(&Giant); local_irq_disable(); @@ -283,11 +293,14 @@ vcpu_prepare(i); #endif + /* * Only resume xenbus /after/ we've prepared our VCPUs; otherwise * the VCPU hotplug callback can race with our vcpu_prepare */ + mtx_lock(&Giant); DEVICE_RESUME(root_bus); + mtx_unlock(&Giant); #ifdef SMP thread_lock(curthread); @@ -296,6 +309,7 @@ if (!CPU_EMPTY(&map)) restart_cpus(map); #endif + EVENTHANDLER_INVOKE(power_resume); } static void @@ -322,39 +336,47 @@ { int suspend_cancelled; + EVENTHANDLER_INVOKE(power_suspend); + + /* + * Be sure to hold Giant across DEVICE_SUSPEND/RESUME since non-MPSAFE + * drivers need this. + */ + mtx_lock(&Giant); if (DEVICE_SUSPEND(root_bus)) { + mtx_unlock(&Giant); printf("xen_suspend: device_suspend failed\n"); return; } - - /* - * Make sure we don't change cpus or switch to some other - * thread. for the duration. - */ - critical_enter(); + mtx_unlock(&Giant); /* * Prevent any races with evtchn_interrupt() handler. */ + disable_intr(); irq_suspend(); - disable_intr(); suspend_cancelled = HYPERVISOR_suspend(0); - if (!suspend_cancelled) + if (suspend_cancelled) + irq_resume(); + else xenpci_resume(); /* * Re-enable interrupts and put the scheduler back to normal. */ enable_intr(); - critical_exit(); /* * FreeBSD really needs to add DEVICE_SUSPEND_CANCEL or * similar. */ + mtx_lock(&Giant); if (!suspend_cancelled) DEVICE_RESUME(root_bus); + mtx_unlock(&Giant); + + EVENTHANDLER_INVOKE(power_resume); } #endif ========================================================================== Change 501529 by justing@justing-ns1 on 2011/08/14 14:33:36 Add suspend/resume support to the Xen blkfront driver. sys/dev/xen/blkfront/block.h: sys/dev/xen/blkfront/blkfront.c: Remove now unused blkif_vdev_t from the blkfront soft. sys/dev/xen/blkfront/blkfront.c: o In blkfront_suspend(), indicate the desire to suspend by changing the softc connected state to SUSPENDED, and then wait for any I/O pending on the remote peer to drain. Cancel suspend processing if I/O does not drain within 30 seconds. o Enable and update blkfront_resume(). Since I/O is drained prior to the suspension of the VM, the complicated recovery process performed by other Xen blkfront implementations is avoided. We simply tear down the connection to our old peer, and then re-connect. o In blkif_initialize(), fix a resource leak and botched return if we cannot allocate shadow memory for our requests. o In blkfront_backend_changed(), correct our response to the XenbusStateInitialised state. This state indicates that our backend peer has published sufficient data for blkfront to publish ring information and other XenStore data, not that a connection can occur. Blkfront now will only perform connection processing in response to the XenbusStateConnected state. This corrects an issue where blkfront connected before the backend was ready during resume processing. Affected files ... ... //depot/SpectraBSD/head/sys/dev/xen/blkfront/blkfront.c#12 edit ... //depot/SpectraBSD/head/sys/dev/xen/blkfront/block.h#8 edit Differences ... ==== //depot/SpectraBSD/head/sys/dev/xen/blkfront/blkfront.c#12 (text) ==== @@ -77,11 +77,8 @@ static int setup_blkring(struct xb_softc *); static void blkif_int(void *); static void blkfront_initialize(struct xb_softc *); -#if 0 -static void blkif_recover(struct xb_softc *); -#endif static int blkif_completion(struct xb_command *); -static void blkif_free(struct xb_softc *, int); +static void blkif_free(struct xb_softc *); static void blkif_queue_cb(void *, bus_dma_segment_t *, int, int); MALLOC_DEFINE(M_XENBLOCKFRONT, "xbd", "Xen Block Front driver data"); @@ -452,9 +449,6 @@ sc->vdevice = vdevice; sc->connected = BLKIF_STATE_DISCONNECTED; - /* Front end dir is a number, which is used as the id. */ - sc->handle = strtoul(strrchr(xenbus_get_node(dev),'/')+1, NULL, 0); - /* Wait for backend device to publish its protocol capabilities. */ xenbus_set_state(dev, XenbusStateInitialising); @@ -465,29 +459,40 @@ blkfront_suspend(device_t dev) { struct xb_softc *sc = device_get_softc(dev); + int retval; + int saved_state; /* Prevent new requests being issued until we fix things up. */ mtx_lock(&sc->xb_io_lock); + saved_state = sc->connected; sc->connected = BLKIF_STATE_SUSPENDED; + + /* Wait for outstanding I/O to drain. */ + retval = 0; + while (TAILQ_EMPTY(&sc->cm_busy) == 0) { + if (msleep(&sc->cm_busy, &sc->xb_io_lock, + PRIBIO, "blkf_susp", 30 * hz) == EWOULDBLOCK) { + retval = EBUSY; + break; + } + } mtx_unlock(&sc->xb_io_lock); - return (0); + if (retval != 0) + sc->connected = saved_state; + + return (retval); } static int blkfront_resume(device_t dev) { -#if 0 struct xb_softc *sc = device_get_softc(dev); DPRINTK("blkfront_resume: %s\n", xenbus_get_node(dev)); -/* XXX This can't work!!! */ - blkif_free(sc, 1); + blkif_free(sc); blkfront_initialize(sc); - if (sc->connected == BLKIF_STATE_SUSPENDED) - blkif_recover(sc); -#endif return (0); } @@ -499,8 +504,10 @@ int error; int i; - if (xenbus_get_state(sc->xb_dev) != XenbusStateInitialising) - return; + if (xenbus_get_state(sc->xb_dev) != XenbusStateInitialising) { + /* Initialization has already been performed. */ + return; + } /* * Protocol defaults valid even if negotiation for a @@ -593,8 +600,10 @@ sc->shadow = malloc(sizeof(*sc->shadow) * sc->max_requests, M_XENBLOCKFRONT, M_NOWAIT|M_ZERO); if (sc->shadow == NULL) { + bus_dma_tag_destroy(sc->xb_io_dmat); xenbus_dev_fatal(sc->xb_dev, error, "Cannot allocate request structures\n"); + return; } for (i = 0; i < sc->max_requests; i++) { @@ -755,10 +764,10 @@ break; case XenbusStateInitWait: + case XenbusStateInitialised: blkfront_initialize(sc); break; - case XenbusStateInitialised: case XenbusStateConnected: blkfront_initialize(sc); blkfront_connect(sc); @@ -775,7 +784,7 @@ } /* -** Invoked when the backend is finally 'ready' (and has told produced +** Invoked when the backend is finally 'ready' (and has published ** the details about the physical device - #sectors, size, etc). */ static void @@ -809,13 +818,15 @@ if (!err || feature_barrier) sc->xb_flags |= XB_BARRIER; - device_printf(dev, "%juMB <%s> at %s", - (uintmax_t) sectors / (1048576 / sector_size), - device_get_desc(dev), - xenbus_get_node(dev)); - bus_print_child_footer(device_get_parent(dev), dev); + if (sc->xb_disk == NULL) { + device_printf(dev, "%juMB <%s> at %s", + (uintmax_t) sectors / (1048576 / sector_size), + device_get_desc(dev), + xenbus_get_node(dev)); + bus_print_child_footer(device_get_parent(dev), dev); - xlvbd_add(sc, sectors, sc->vdevice, binfo, sector_size); + xlvbd_add(sc, sectors, sc->vdevice, binfo, sector_size); + } (void)xenbus_set_state(dev, XenbusStateConnected); @@ -825,7 +836,6 @@ xb_startio(sc); sc->xb_flags |= XB_READY; mtx_unlock(&sc->xb_io_lock); - } /** @@ -859,7 +869,7 @@ DPRINTK("blkfront_remove: %s removed\n", xenbus_get_node(dev)); - blkif_free(sc, 0); + blkif_free(sc); mtx_destroy(&sc->xb_io_lock); return 0; @@ -1140,6 +1150,9 @@ mtx_assert(&sc->xb_io_lock, MA_OWNED); + if (sc->connected != BLKIF_STATE_CONNECTED) + return; + while (RING_FREE_REQUESTS(&sc->ring) >= sc->max_request_blocks) { if (sc->xb_flags & XB_FROZEN) break; @@ -1174,7 +1187,7 @@ mtx_lock(&sc->xb_io_lock); - if (unlikely(sc->connected != BLKIF_STATE_CONNECTED)) { + if (unlikely(sc->connected == BLKIF_STATE_DISCONNECTED)) { mtx_unlock(&sc->xb_io_lock); return; } @@ -1232,19 +1245,21 @@ xb_startio(sc); + if (unlikely(sc->connected == BLKIF_STATE_SUSPENDED)) + wakeup(&sc->cm_busy); + mtx_unlock(&sc->xb_io_lock); } static void -blkif_free(struct xb_softc *sc, int suspend) +blkif_free(struct xb_softc *sc) { uint8_t *sring_page_ptr; int i; /* Prevent new requests being issued until we fix things up. */ mtx_lock(&sc->xb_io_lock); - sc->connected = suspend ? - BLKIF_STATE_SUSPENDED : BLKIF_STATE_DISCONNECTED; + sc->connected = BLKIF_STATE_DISCONNECTED; mtx_unlock(&sc->xb_io_lock); /* Free resources associated with old device channel. */ @@ -1276,6 +1291,12 @@ } free(sc->shadow, M_XENBLOCKFRONT); sc->shadow = NULL; + + bus_dma_tag_destroy(sc->xb_io_dmat); + + xb_initq_free(sc); + xb_initq_ready(sc); + xb_initq_complete(sc); } if (sc->irq) { @@ -1292,21 +1313,6 @@ return (BLKIF_SEGS_TO_BLOCKS(s->nseg)); } -#if 0 -static void -blkif_recover(struct xb_softc *sc) -{ - /* - * XXX The whole concept of not quiescing and completing all i/o - * during suspend, and then hoping to recover and replay the - * resulting abandoned I/O during resume, is laughable. At best, - * it invalidates the i/o ordering rules required by just about - * every filesystem, and at worst it'll corrupt data. The code - * has been removed until further notice. - */ -} -#endif - /* ** Driver registration ** */ static device_method_t blkfront_methods[] = { /* Device interface */ ==== //depot/SpectraBSD/head/sys/dev/xen/blkfront/block.h#8 (text) ==== @@ -142,7 +142,6 @@ #define XB_READY (1 << 2) /* Is ready */ #define XB_FROZEN (1 << 3) /* Waiting for resources */ int vdevice; - blkif_vdev_t handle; int connected; u_int ring_pages; uint32_t max_requests; ========================================================================== Change 503342 by justing@justing-ns1 on 2011/09/01 10:35:49 Correct suspend/resume support in the Netfront driver. sys/dev/xen/netfront/netfront.c: o Implement netfront_suspend(), a specialized suspend handler for the netfront driver. This routine simply disables the carrier so the driver is idle during system suspend processing. o Fix a leak when re-initializing LRO during a link reset. o In netif_release_tx_bufs(), when cleaning up the grant references for our TX ring, use gnttab_end_foreign_access_ref instead of attempting to grant the page again. o In netif_release_tx_bufs(), we do not track mbufs associated with mbuf chains, but instead just free each mbuf directly. Use m_free(), not m_freem(), to avoid double frees of mbufs. o Refactor some code to enhance clarity. Affected files ... ... //depot/SpectraBSD/head/sys/dev/xen/netfront/netfront.c#20 edit Differences ... ==== //depot/SpectraBSD/head/sys/dev/xen/netfront/netfront.c#20 (text) ==== @@ -159,6 +159,7 @@ static void xn_ifinit_locked(struct netfront_info *); static void xn_ifinit(void *); static void xn_stop(struct netfront_info *); +static int xn_configure_lro(struct netfront_info *np); #ifdef notyet static void xn_watchdog(struct ifnet *); #endif @@ -174,7 +175,7 @@ static int create_netdev(device_t dev); static void netif_disconnect_backend(struct netfront_info *info); static int setup_device(device_t dev, struct netfront_info *info); -static void end_access(int ref, void *page); +static void free_ring(int *ref, void *ring_ptr_ref); static int xn_ifmedia_upd(struct ifnet *ifp); static void xn_ifmedia_sts(struct ifnet *ifp, struct ifmediareq *ifmr); @@ -463,6 +464,18 @@ return 0; } +static int +netfront_suspend(device_t dev) +{ + struct netfront_info *info = device_get_softc(dev); + + XN_RX_LOCK(info); + XN_TX_LOCK(info); + netfront_carrier_off(info); + XN_TX_UNLOCK(info); + XN_RX_UNLOCK(info); + return (0); +} /** * We are reconnecting to the backend, due to a suspend/resume, or a backend @@ -749,10 +762,7 @@ */ if (((uintptr_t)m) <= NET_TX_RING_SIZE) continue; - gnttab_grant_foreign_access_ref(np->grant_tx_ref[i], - xenbus_get_otherend_id(np->xbdev), - virt_to_mfn(mtod(m, vm_offset_t)), - GNTMAP_readonly); + gnttab_end_foreign_access_ref(np->grant_tx_ref[i]); gnttab_release_grant_reference(&np->gref_tx_head, np->grant_tx_ref[i]); np->grant_tx_ref[i] = GRANT_REF_INVALID; @@ -761,7 +771,7 @@ if (np->xn_cdata.xn_tx_chain_cnt < 0) { panic("netif_release_tx_bufs: tx_chain_cnt must be >= 0"); } - m_freem(m); + m_free(m); } } @@ -1914,6 +1924,7 @@ netif_release_tx_bufs(np); /* Step 2: Rebuild the RX buffer freelist and the RX ring itself. */ + xn_configure_lro(np); for (requeue_idx = 0, i = 0; i < NET_RX_RING_SIZE; i++) { struct mbuf *m; u_long pfn; @@ -1978,6 +1989,30 @@ #endif } +static int +xn_configure_lro(struct netfront_info *np) +{ + int err; + + err = 0; +#if __FreeBSD_version >= 700000 + if ((np->xn_ifp->if_capabilities & IFCAP_LRO) != 0) + tcp_lro_free(&np->xn_lro); + np->xn_ifp->if_capabilities &= ~IFCAP_LRO; + if (xn_enable_lro) { + err = tcp_lro_init(&np->xn_lro); + if (err) { + device_printf(np->xbdev, "LRO initialization failed\n"); + } else { + np->xn_lro.ifp = np->xn_ifp; + np->xn_ifp->if_capabilities |= IFCAP_LRO; + } + } + np->xn_ifp->if_capenable = np->xn_ifp->if_capabilities; +#endif + return (err); +} + /** Create a network device. * @param handle device handle */ @@ -2057,17 +2092,9 @@ ifp->if_capabilities = IFCAP_HWCSUM; #if __FreeBSD_version >= 700000 ifp->if_capabilities |= IFCAP_TSO4; - if (xn_enable_lro) { - int err = tcp_lro_init(&np->xn_lro); - if (err) { - device_printf(dev, "LRO initialization failed\n"); - goto exit; - } - np->xn_lro.ifp = ifp; - ifp->if_capabilities |= IFCAP_LRO; - } #endif ifp->if_capenable = ifp->if_capabilities; + xn_configure_lro(np); ether_ifattach(ifp, np->mac); callout_init(&np->xn_stat_ch, CALLOUT_MPSAFE); @@ -2133,12 +2160,8 @@ XN_TX_UNLOCK(info); XN_RX_UNLOCK(info); - end_access(info->tx_ring_ref, info->tx.sring); - end_access(info->rx_ring_ref, info->rx.sring); - info->tx_ring_ref = GRANT_REF_INVALID; - info->rx_ring_ref = GRANT_REF_INVALID; - info->tx.sring = NULL; - info->rx.sring = NULL; + free_ring(&info->tx_ring_ref, &info->tx.sring); + free_ring(&info->rx_ring_ref, &info->rx.sring); if (info->irq) unbind_from_irqhandler(info->irq); @@ -2146,12 +2169,17 @@ info->irq = 0; } - static void -end_access(int ref, void *page) +free_ring(int *ref, void *ring_ptr_ref) { - if (ref != GRANT_REF_INVALID) - gnttab_end_foreign_access(ref, page); + void **ring_ptr_ptr = ring_ptr_ref; + + if (*ref != GRANT_REF_INVALID) { + /* This API frees the associated storage. */ + gnttab_end_foreign_access(*ref, *ring_ptr_ptr); + *ref = GRANT_REF_INVALID; + } + *ring_ptr_ptr = NULL; } static int @@ -2174,7 +2202,7 @@ DEVMETHOD(device_attach, netfront_attach), DEVMETHOD(device_detach, netfront_detach), DEVMETHOD(device_shutdown, bus_generic_shutdown), - DEVMETHOD(device_suspend, bus_generic_suspend), + DEVMETHOD(device_suspend, netfront_suspend), DEVMETHOD(device_resume, netfront_resume), /* Xenbus interface */ --------------020503020206070006020408--