From owner-freebsd-hackers@freebsd.org Sat Jan 4 22:35:35 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C8BA11DC6A1 for ; Sat, 4 Jan 2020 22:35:35 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47qxS670NKz4jHP for ; Sat, 4 Jan 2020 22:35:34 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.15.2) with ESMTPS id 004MZXUv020756 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 4 Jan 2020 23:35:33 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1578177333; bh=pd9TR5hHjJ4XRSgnt7/SvB4C6TocI48LHBAXECbtNkY=; h=Date:From:To:Subject; b=QgimCu92faMxMTuxQzSvydZ+JXf/gi0PByJA0aL14imDcaWb2CRxEoWWtc2k12WZP ftn0ZqKJ1BRhklz54WP48honp9aG8rW5pfKZnmHf259YDkLf04NbWzUInbc+2Majp2 k+DGfNGdoVMIjisGEfnBiguKcn6iMj1GcXNYKD5c= Received: from localhost (puchar-wojtek@localhost) by puchar.net (8.15.2/8.15.2/Submit) with ESMTP id 004MZXgh020753 for ; Sat, 4 Jan 2020 23:35:33 +0100 (CET) (envelope-from puchar-wojtek@puchar.net) Date: Sat, 4 Jan 2020 23:35:33 +0100 (CET) From: Wojciech Puchar To: freebsd-hackers@freebsd.org Subject: processes are killed because of out of swap space Message-ID: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-Rspamd-Queue-Id: 47qxS670NKz4jHP X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=fail (rsa verify failed) header.d=puchar.net header.s=default header.b=QgimCu92; dmarc=none; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net X-Spamd-Result: default: False [-4.47 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_REJECT(1.00)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DMARC_NA(0.00)[puchar.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[puchar.net:-]; RCVD_IN_DNSWL_NONE(0.00)[90.144.1.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.17)[ip: (-8.38), ipnet: 194.1.144.0/24(-4.19), asn: 43476(-3.35), country: PL(0.07)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Jan 2020 22:35:35 -0000 when i try to use more virtual memory (tested by putting files to tmpfs /tmp). like that pid 16977 (bhyve), jid 0, uid 0, was killed: out of swap space the problem is that it's less than 10GB swap used while i have 120GB available. before processed begin to be killed system stalls for a while. any idea what to check? below my sysctl -a vm vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) =============================================== Processes: (RUNQ: 2 Disk Wait: 13 Page Wait: 0 Sleep: 614) Virtual Memory: (Total: 35115848K Active: 34336216K) Real Memory: (Total: 17559404K Active: 17498736K) Shared Virtual Memory: (Total: 819960K Active: 231808K) Shared Real Memory: (Total: 117712K Active: 59860K) Free Memory: 707048K vm.loadavg: { 1,56 1,61 1,82 } vm.v_free_min: 51326 vm.v_free_target: 173333 vm.v_free_reserved: 10657 vm.v_inactive_target: 259999 vm.v_pageout_free_min: 34 vm.swap_enabled: 1 vm.overcommit: 1 vm.kvm_free: 2160297242624 vm.kvm_size: 2199023251456 vm.pmap.pdpe.demotions: 3 vm.pmap.pde.promotions: 60330 vm.pmap.pde.p_failures: 42765 vm.pmap.pde.mappings: 14905 vm.pmap.pde.demotions: 15645 vm.pmap.allow_2m_x_ept: 1 vm.pmap.pcid_save_cnt: 142000859 vm.pmap.pti: 1 vm.pmap.invpcid_works: 1 vm.pmap.pcid_enabled: 1 vm.pmap.pg_ps_enabled: 1 vm.pmap.pat_works: 1 vm.idlezero_enable: 0 vm.swap_idle_threshold2: 360 vm.swap_idle_threshold1: 180 vm.swap_idle_enabled: 1 vm.reserv.reclaimed: 10817 vm.reserv.partpopq: LEVEL SIZE NUMBER -1: 0K, 0 vm.reserv.fullpop: 380 vm.reserv.freed: 97537 vm.reserv.broken: 11002 vm.default_policy: rr vm.ndomains: 1 vm.phys_segs: SEGMENT 0: start: 0x10000 end: 0x96000 domain: 0 free list: 0xffffffff80d5bf10 SEGMENT 1: start: 0x100000 end: 0x200000 domain: 0 free list: 0xffffffff80d5bf10 SEGMENT 2: start: 0x200000 end: 0x1000000 domain: 0 free list: 0xffffffff80d5bf10 SEGMENT 3: start: 0x1000000 end: 0x1372000 domain: 0 free list: 0xffffffff80d5bca0 SEGMENT 4: start: 0x1378000 end: 0x13a3000 domain: 0 free list: 0xffffffff80d5bca0 SEGMENT 5: start: 0x1400000 end: 0x791c9000 domain: 0 free list: 0xffffffff80d5bca0 SEGMENT 6: start: 0x100000000 end: 0x849e72000 domain: 0 free list: 0xffffffff80d5bca0 vm.phys_free: DOMAIN 0: FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 ( 16384K) | 0 | 0 11 ( 8192K) | 0 | 0 10 ( 4096K) | 0 | 0 9 ( 2048K) | 0 | 0 8 ( 1024K) | 0 | 0 7 ( 512K) | 0 | 0 6 ( 256K) | 52 | 1 5 ( 128K) | 462 | 6 4 ( 64K) | 2152 | 9 3 ( 32K) | 9950 | 36 2 ( 16K) | 10623 | 125 1 ( 8K) | 182 | 121 0 ( 4K) | 0 | 3 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 -- -- -- -- -- -- 12 ( 16384K) | 0 | 0 11 ( 8192K) | 0 | 0 10 ( 4096K) | 0 | 0 9 ( 2048K) | 0 | 0 8 ( 1024K) | 0 | 0 7 ( 512K) | 0 | 0 6 ( 256K) | 0 | 0 5 ( 128K) | 0 | 0 4 ( 64K) | 8 | 0 3 ( 32K) | 9 | 0 2 ( 16K) | 15 | 0 1 ( 8K) | 18 | 0 0 ( 4K) | 21 | 0 vm.max_wired: 2659648 vm.background_launder_max: 20480 vm.background_launder_rate: 4096 vm.background_launder_target: 12200 vm.act_scan_laundry_weight: 3 vm.pageout_oom_seq: 12 vm.pageout_lock_miss: 0 vm.disable_swapspace_pageouts: 0 vm.lowmem_period: 1 vm.pageout_update_period: 600 vm.pageout_wakeup_thresh: 56452 vm.panic_on_oom: 0 vm.page_blacklist: vm.tryrelock_restart: 31748 vm.boot_pages: 64 vm.old_msync: 0 vm.mincore_mapped: 1 vm.old_mlock: 0 vm.stats.object.bypasses: 18723 vm.stats.object.collapses: 78824 vm.stats.misc.cnt_prezero: 0 vm.stats.misc.zero_page_count: 325 vm.stats.vm.v_kthreadpages: 0 vm.stats.vm.v_rforkpages: 100 vm.stats.vm.v_vforkpages: 52157 vm.stats.vm.v_forkpages: 1018587 vm.stats.vm.v_kthreads: 103 vm.stats.vm.v_rforks: 2 vm.stats.vm.v_vforks: 1484 vm.stats.vm.v_forks: 18704 vm.stats.vm.v_interrupt_free_min: 2 vm.stats.vm.v_pageout_free_min: 34 vm.stats.vm.v_cache_count: 0 vm.stats.vm.v_laundry_count: 841044 vm.stats.vm.v_inactive_count: 2695784 vm.stats.vm.v_inactive_target: 259999 vm.stats.vm.v_active_count: 3536963 vm.stats.vm.v_wire_count: 883581 vm.stats.vm.v_free_count: 176747 vm.stats.vm.v_free_min: 51326 vm.stats.vm.v_free_target: 173333 vm.stats.vm.v_free_reserved: 10657 vm.stats.vm.v_page_count: 8134081 vm.stats.vm.v_page_size: 4096 vm.stats.vm.v_tfree: 714905915 vm.stats.vm.v_pfree: 38188448 vm.stats.vm.v_dfree: 139195941 vm.stats.vm.v_tcached: 0 vm.stats.vm.v_pdshortfalls: 114 vm.stats.vm.v_pdpages: 381445429 vm.stats.vm.v_pdwakeups: 1093 vm.stats.vm.v_reactivated: 1829047 vm.stats.vm.v_intrans: 1165647 vm.stats.vm.v_vnodepgsout: 780 vm.stats.vm.v_vnodepgsin: 407218 vm.stats.vm.v_vnodeout: 383 vm.stats.vm.v_vnodein: 74309 vm.stats.vm.v_swappgsout: 11838348 vm.stats.vm.v_swappgsin: 2406191 vm.stats.vm.v_swapout: 825054 vm.stats.vm.v_swapin: 1142018 vm.stats.vm.v_ozfod: 1020 vm.stats.vm.v_zfod: 33027903 vm.stats.vm.v_cow_optim: 2572 vm.stats.vm.v_cow_faults: 2530416 vm.stats.vm.v_io_faults: 310268 vm.stats.vm.v_vm_faults: 44268076 vm.stats.sys.v_soft: 5808251 vm.stats.sys.v_intr: 8974021 vm.stats.sys.v_syscall: 251679096 vm.stats.sys.v_trap: 272835752 vm.stats.sys.v_swtch: 260966303 vm.v_free_severe: 30991 vm.max_kernel_address: 18446744073709547520 vm.min_kernel_address: 18446741874686296064 vm.kstacks: 1130 vm.kstack_cache_size: 128 vm.zone_warnings: 1 vm.zone_count: 120 vm.nswapdev: 1 vm.dmmax: 32 vm.swap_fragmentation: Free space on device gpt/swap.eli: number of maximal free ranges: 20935 largest free range: 20512000 average maximal free range size: 1418 number of maximal free ranges of different sizes: count | size range ----- | ---------- 12501 | 1 2117 | 2 1473 | 3 to 4 789 | 5 to 7 658 | 8 to 12 593 | 13 to 20 906 | 21 to 33 298 | 34 to 54 298 | 55 to 88 240 | 89 to 143 191 | 144 to 232 176 | 233 to 376 137 | 377 to 609 92 | 610 to 986 105 | 987 to 1596 91 | 1597 to 2583 76 | 2584 to 4180 51 | 4181 to 6764 37 | 6765 to 10945 33 | 10946 to 17710 27 | 17711 to 28656 10 | 28657 to 46367 14 | 46368 to 75024 9 | 75025 to 121392 3 | 121393 to 196417 2 | 196418 to 317810 4 | 317811 to 514228 3 | 514229 to 832039 1 | 20512000 vm.swap_async_max: 4 vm.swap_maxpages: 65072752 vm.swzone: 553118392 vm.swap_reserved: 11839164416 vm.swap_total: 128849018880 vm.phys_pager_cluster: 1024 vm.kmem_map_free: 32650514432 vm.kmem_map_size: 666681344 vm.kmem_size_scale: 1 vm.kmem_size_max: 1319413950874 vm.kmem_size_min: 0 vm.kmem_zmax: 65536 vm.kmem_size: 33317195776 vm.md_malloc_wait: 0