Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 24 Jul 2014 18:36:34 -0500
From:      Larry Rosenman <ler@lerctr.org>
To:        Mark Martinec <Mark.Martinec+freebsd@ijs.si>, Allan Jude <allanjude@freebsd.org>
Cc:        Freebsd fs <freebsd-fs@freebsd.org>, freebsd-current@freebsd.org
Subject:   Re: zfs send/recv: STILL invalid Backup Stream
Message-ID:  <fbf874048b94066aabc4e656e2d3157d@thebighonker.lerctr.org>
In-Reply-To: <d5bda7f9ad7f86ec35ab6ba5571aa8bc@thebighonker.lerctr.org>
References:  <62315eb454a95db636b7764aad3c0f9b@thebighonker.lerctr.org> <53D1448C.40908@freebsd.org> <d0612ef9399b4e95bd09e5271c91505b@thebighonker.lerctr.org> <eefa10e173e2e23a88c85c38221a1c22@thebighonker.lerctr.org> <53D15438.6040105@freebsd.org> <0a9cd451c3b4304d2b9d899fcb3decc3@thebighonker.lerctr.org> <cc635a377e817cb57d27650ed3e558f4@thebighonker.lerctr.org> <3d2aac84d962a703fbf56a864ba5f19c@mailbox.ijs.si> <c93ef94a8e9e076b86a65f4986d0e30d@thebighonker.lerctr.org> <9eef3e7e964cb33fd163cc2f9300f326@mailbox.ijs.si> <d5bda7f9ad7f86ec35ab6ba5571aa8bc@thebighonker.lerctr.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On 2014-07-24 15:44, Larry Rosenman wrote:
> On 2014-07-24 15:26, Mark Martinec wrote:
>> 2014-07-24 21:57, Larry Rosenman wrote:
>> 
>>>>>> Sending zroot/home@zxfer_26699_20140724135840 to 
>>>>>> zroot/backups/TBH/zroot/home.
>>>>> Write failed: Cannot allocate memory
>>>>   ====================================
>>>>> cannot receive new filesystem stream: invalid backup stream
>>>>> Error when zfs send/receiving.
>>>>> borg.lerctr.org /home/ler #
>>>>> 
>>>>> well that's different.......
>>>> 
>>>> Sounds familiar, check my posting of today and links therein:
>>>>   
>>>> http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039347.html
>> 
>>> I'm not using netgraph to the best of my knowledge....
>> 
>> Check to make sure:
>>   vmstat -z | fgrep NetGraph
>> 
>>> and the only fails on the SENDING host are:
>>> 8 Bucket:                64,      0,      41,    3555,  257774,  11,  
>>>  0
>>> 12 Bucket:               96,      0,      96,    2569,  123653,   0,  
>>>  0
>>> 16 Bucket:              128,      0,   17195,     506,  215573,   0,  
>>>  0
>>> 32 Bucket:              256,      0,     340,    4670,  900638,  50,  
>>>  0
>>> 64 Bucket:              512,      0,   10691,     365,  
>>> 546888,185232,
>>> 0
>>> 128 Bucket:            1024,      0,    3563,     905,  348419,   0,  
>>>  0
>>> 256 Bucket:            2048,      0,    2872,     162,  249995,59834,
>>> 0
>>> vmem btag:               56,      0,  192811,   51500,  502264,1723,  
>>>  0
>> 
>> 
>> Adam Vande More gave other suggestions on that thread from 2011:
>> 
>>   
>> http://lists.freebsd.org/pipermail/freebsd-emulation/2011-July/008971.html
>> 
>> 
>> Mark
> 
> 
> thebighonker.lerctr.org /home/ler $ vmstat -z|fgrep NetGraph
> thebighonker.lerctr.org /home/ler $
> 
> This on physical hardware, FWIW.
> 
> thebighonker.lerctr.org /home/ler $ vmstat -z
> ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL 
> SLEEP
> 
> UMA Kegs:               384,      0,     202,       8,     202,   0,   
> 0
> UMA Zones:             1664,      0,     202,       0,     202,   0,   
> 0
> UMA Slabs:               80,      0,  361152,   38448, 1561588,   0,   
> 0
> UMA RCntSlabs:           88,      0,    4360,       5,    4360,   0,   
> 0
> UMA Hash:               256,      0,       8,      82,      82,   0,   
> 0
> 4 Bucket:                32,      0,    4816,   33684, 2974043,   0,   
> 0
> 6 Bucket:                48,      0,    6118,    2514,  624434,   0,   
> 0
> 8 Bucket:                64,      0,     161,    3435,  265443,  11,   
> 0
> 12 Bucket:               96,      0,     177,    2488,  130226,   0,   
> 0
> 16 Bucket:              128,      0,   20301,     531,  224788,   0,   
> 0
> 32 Bucket:              256,      0,     433,    4577,  945047,  50,   
> 0
> 64 Bucket:              512,      0,   11137,     575,  579209,211292,
> 0
> 128 Bucket:            1024,      0,    3509,     959,  369839,   0,   
> 0
> 256 Bucket:            2048,      0,    2847,     187,  269044,59834,
> 0
> vmem btag:               56,      0,  209393,   34918,  557576,1723,   
> 0
> VM OBJECT:              256,      0,   91384,     791, 3480613,   0,   
> 0
> RADIX NODE:             144,      0,  227133,   37683,16029821,   0,   
> 0
> MAP:                    240,      0,       3,      61,       3,   0,   
> 0
> KMAP ENTRY:             128,      0,      10,     269,      10,   0,   
> 0
> MAP ENTRY:              128,      0,   12666,   23604, 9357932,   0,   
> 0
> VMSPACE:                448,      0,     108,     306,   75357,   0,   
> 0
> fakepg:                 104,      0,       0,       0,       0,   0,   
> 0
> mt_zone:               4112,      0,     368,       0,     368,   0,   
> 0
> 16:                      16,      0,  267839,    2488,45160196,   0,   
> 0
> 32:                      32,      0,  148139,    1486,39174301,   0,   
> 0
> 64:                      64,      0,  291630,  504078,40379996,   0,   
> 0
> 128:                    128,      0,  182409,   35490,63455920,   0,   
> 0
> 256:                    256,      0,   89368,  230912,35841036,   0,   
> 0
> 512:                    512,      0,     741,    3987,24215854,   0,   
> 0
> 1024:                  1024,      0,    1371,    1385, 1726440,   0,   
> 0
> 2048:                  2048,      0,     277,     417,10240207,   0,   
> 0
> 4096:                  4096,      0,   38306,   17186,  913638,   0,   
> 0
> SLEEPQUEUE:              80,      0,     613,     658,     613,   0,   
> 0
> 64 pcpu:                  8,      0,    1884,    1188,    1884,   0,   
> 0
> Files:                   80,      0,     829,    1571, 4279893,   0,   
> 0
> rl_entry:                40,      0,     209,    1891,     209,   0,   
> 0
> TURNSTILE:              136,      0,     613,     307,     613,   0,   
> 0
> umtx pi:                 96,      0,       0,       0,       0,   0,   
> 0
> MAC labels:              40,      0,       0,       0,       0,   0,   
> 0
> PROC:                  1208,      0,     131,     121,   76219,   0,   
> 0
> THREAD:                1168,      0,     519,      93,    2039,   0,   
> 0
> cpuset:                  72,      0,     270,     280,     426,   0,   
> 0
> cyclic_id_cache:         64,      0,       0,       0,       0,   0,   
> 0
> audit_record:          1248,      0,       0,       0,       0,   0,   
> 0
> mbuf_packet:            256, 6519810,    4081,    3256, 4910227,   0,
> 0
> mbuf:                   256, 6519810,    1052,    3251,93444045,   0,
> 0
> mbuf_cluster:          2048, 1018718,    7337,      13,    7337,   0,
> 0
> mbuf_jumbo_page:       4096, 509359,      12,     673,15015378,   0,   
> 0
> mbuf_jumbo_9k:         9216, 150921,       0,       0,       0,   0,   
> 0
> mbuf_jumbo_16k:       16384,  84893,       0,       0,       0,   0,   
> 0
> mbuf_ext_refcnt:          4,      0,       0,    2510,   85094,   0,   
> 0
> dtrace_state_cache:    4096,      0,       0,       0,       0,   0,   
> 0
> g_bio:                  248,      0,       0,    1600,29986631,   0,   
> 0
> ttyinq:                 160,      0,     180,     570,    2100,   0,   
> 0
> DMAR_MAP_ENTRY:         120,      0,       0,       0,       0,   0,   
> 0
> ttyoutq:                256,      0,      95,     640,    1093,   0,   
> 0
> ata_request:            336,      0,       0,     242,   33344,   0,   
> 0
> vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   
> 0
> cryptop:                 88,      0,       0,       0,       0,   0,   
> 0
> cryptodesc:              72,      0,       0,       0,       0,   0,   
> 0
> FPU_save_area:          512,      0,       0,       0,       0,   0,   
> 0
> taskq_zone:              48,      0,       1,    3070,  194184,   0,   
> 0
> taskq_zone:              48,      0,       0,       0,       0,   0,   
> 0
> VNODE:                  472,      0,  109527,   66161, 2663226,   0,   
> 0
> VNODEPOLL:              112,      0,       8,     762,      10,   0,   
> 0
> BUF TRIE:               144,      0,       0,  105354,       0,   0,   
> 0
> S VFS Cache:            108,      0,  114973,   60552, 2293717,   0,   
> 0
> STS VFS Cache:          148,      0,       0,       0,       0,   0,   
> 0
> L VFS Cache:            328,      0,     468,  145536,  800279,   0,   
> 0
> LTS VFS Cache:          368,      0,       0,       0,       0,   0,   
> 0
> NAMEI:                 1024,      0,       0,     156,23968745,   0,   
> 0
> DIRHASH:               1024,      0,       0,       0,       0,   0,   
> 0
> NCLNODE:                528,      0,       0,       0,       0,   0,   
> 0
> Mountpoints:            816,      0,      11,      19,      11,   0,   
> 0
> range_seg_cache:         64,      0,   58377,    7157, 3500156,   0,   
> 0
> zio_cache:              920,      0,       1,   42591,104595517,   0,
> 0
> zio_link_cache:          48,      0,       0,   44073,72482492,   0,   
> 0
> zio_buf_512:            512,      0,  552060,  193836, 1557459,   0,   
> 0
> zio_data_buf_512:       512,      0,  114904,    2056,  526943,   0,   
> 0
> zio_buf_1024:          1024,      0,    4246,      58,   56140,   0,   
> 0
> zio_data_buf_1024:     1024,      0,   51117,    1255,  211875,   0,   
> 0
> zio_buf_1536:          1536,      0,    1957,      59,   72179,   0,   
> 0
> zio_data_buf_1536:     1536,      0,   22199,     719,  127817,   0,   
> 0
> zio_buf_2048:          2048,      0,     691,      43,  217643,   0,   
> 0
> zio_data_buf_2048:     2048,      0,   13419,    1147,   87149,   0,   
> 0
> zio_buf_2560:          2560,      0,     455,      20,   16265,   0,   
> 0
> zio_data_buf_2560:     2560,      0,    9335,     493,   73967,   0,   
> 0
> zio_buf_3072:          3072,      0,     245,      16,    4658,   0,   
> 0
> zio_data_buf_3072:     3072,      0,    6940,     512,   53932,   0,   
> 0
> zio_buf_3584:          3584,      0,     248,      10,   38558,   0,   
> 0
> zio_data_buf_3584:     3584,      0,    5076,     352,   43162,   0,   
> 0
> zio_buf_4096:          4096,      0,     464,     729,14217630,   0,   
> 0
> zio_data_buf_4096:     4096,      0,    3857,     288,   41192,   0,   
> 0
> zio_buf_5120:          5120,      0,     167,       9,    6442,   0,   
> 0
> zio_data_buf_5120:     5120,      0,    6025,     461,   51297,   0,   
> 0
> zio_buf_6144:          6144,      0,     115,       6,    3196,   0,   
> 0
> zio_data_buf_6144:     6144,      0,    4453,     309,   38753,   0,   
> 0
> zio_buf_7168:          7168,      0,      85,      12,   36945,   0,   
> 0
> zio_data_buf_7168:     7168,      0,    3373,     273,   29007,   0,   
> 0
> zio_buf_8192:          8192,      0,      44,      39, 1373975,   0,   
> 0
> zio_data_buf_8192:     8192,      0,    2543,     205,   28736,   0,   
> 0
> zio_buf_10240:        10240,      0,      66,       7,    2534,   0,   
> 0
> zio_data_buf_10240:   10240,      0,    3986,     275,   39568,   0,   
> 0
> zio_buf_12288:        12288,      0,      71,      15,  654156,   0,   
> 0
> zio_data_buf_12288:   12288,      0,    2937,     209,   37077,   0,   
> 0
> zio_buf_14336:        14336,      0,      32,       9,    3164,   0,   
> 0
> zio_data_buf_14336:   14336,      0,    2236,     194,   23959,   0,   
> 0
> zio_buf_16384:        16384,      0,   72627,     226, 2784131,   0,   
> 0
> zio_data_buf_16384:   16384,      0,    1778,     155,   22372,   0,   
> 0
> zio_buf_20480:        20480,      0,     210,      14,  598092,   0,   
> 0
> zio_data_buf_20480:   20480,      0,    2724,     216,   29228,   0,   
> 0
> zio_buf_24576:        24576,      0,      18,      14,  270172,   0,   
> 0
> zio_data_buf_24576:   24576,      0,    2101,     173,   21241,   0,   
> 0
> zio_buf_28672:        28672,      0,      10,      54,  503029,   0,   
> 0
> zio_data_buf_28672:   28672,      0,    1406,     113,   18667,   0,   
> 0
> zio_buf_32768:        32768,      0,      10,      11,   77998,   0,   
> 0
> zio_data_buf_32768:   32768,      0,    1088,      96,   14116,   0,   
> 0
> zio_buf_36864:        36864,      0,       6,      11,   79009,   0,   
> 0
> zio_data_buf_36864:   36864,      0,     871,      87,   10066,   0,   
> 0
> zio_buf_40960:        40960,      0,       8,      10,   80622,   0,   
> 0
> zio_data_buf_40960:   40960,      0,     785,      57,    8149,   0,   
> 0
> zio_buf_45056:        45056,      0,      50,      10,   70143,   0,   
> 0
> zio_data_buf_45056:   45056,      0,     659,      61,    6889,   0,   
> 0
> zio_buf_49152:        49152,      0,     190,       9,   73508,   0,   
> 0
> zio_data_buf_49152:   49152,      0,     512,      42,    8420,   0,   
> 0
> zio_buf_53248:        53248,      0,      25,      10,   99339,   0,   
> 0
> zio_data_buf_53248:   53248,      0,     465,      30,    4838,   0,   
> 0
> zio_buf_57344:        57344,      0,       2,      15,   95818,   0,   
> 0
> zio_data_buf_57344:   57344,      0,     337,      33,    4753,   0,   
> 0
> zio_buf_61440:        61440,      0,       1,      23,   98972,   0,   
> 0
> zio_data_buf_61440:   61440,      0,     294,      33,    3792,   0,   
> 0
> zio_buf_65536:        65536,      0,       1,       9,  114664,   0,   
> 0
> zio_data_buf_65536:   65536,      0,     291,      29,    5152,   0,   
> 0
> zio_buf_69632:        69632,      0,       2,      10,   90765,   0,   
> 0
> zio_data_buf_69632:   69632,      0,     291,      22,    3555,   0,   
> 0
> zio_buf_73728:        73728,      0,       1,      12,   87316,   0,   
> 0
> zio_data_buf_73728:   73728,      0,     219,      23,    3361,   0,   
> 0
> zio_buf_77824:        77824,      0,       2,       9,  101758,   0,   
> 0
> zio_data_buf_77824:   77824,      0,     196,      25,    2907,   0,   
> 0
> zio_buf_81920:        81920,      0,       0,      15,  129645,   0,   
> 0
> zio_data_buf_81920:   81920,      0,     187,      18,    3578,   0,   
> 0
> zio_buf_86016:        86016,      0,       1,       7,   99241,   0,   
> 0
> zio_data_buf_86016:   86016,      0,     180,      22,   26051,   0,   
> 0
> zio_buf_90112:        90112,      0,       1,      12,   51491,   0,   
> 0
> zio_data_buf_90112:   90112,      0,     163,      18,    2438,   0,   
> 0
> zio_buf_94208:        94208,      0,       0,       9,   51043,   0,   
> 0
> zio_data_buf_94208:   94208,      0,     149,      12,    2288,   0,   
> 0
> zio_buf_98304:        98304,      0,       2,       8,   50322,   0,   
> 0
> zio_data_buf_98304:   98304,      0,     170,      25,    3002,   0,   
> 0
> zio_buf_102400:      102400,      0,       2,      10,   44642,   0,   
> 0
> zio_data_buf_102400: 102400,      0,     145,      19,    2017,   0,   
> 0
> zio_buf_106496:      106496,      0,       0,      15,  161697,   0,   
> 0
> zio_data_buf_106496: 106496,      0,     161,      16,    1944,   0,   
> 0
> zio_buf_110592:      110592,      0,       0,      19,  146777,   0,   
> 0
> zio_data_buf_110592: 110592,      0,      97,      15,    1636,   0,   
> 0
> zio_buf_114688:      114688,      0,       2,       8,   51698,   0,   
> 0
> zio_data_buf_114688: 114688,      0,     115,      20,    2691,   0,   
> 0
> zio_buf_118784:      118784,      0,       0,       8,   62915,   0,   
> 0
> zio_data_buf_118784: 118784,      0,      96,      11,    1440,   0,   
> 0
> zio_buf_122880:      122880,      0,       0,      12,   85924,   0,   
> 0
> zio_data_buf_122880: 122880,      0,      73,      17,    1510,   0,   
> 0
> zio_buf_126976:      126976,      0,       1,      11,   86133,   0,   
> 0
> zio_data_buf_126976: 126976,      0,      80,       9,    1351,   0,   
> 0
> zio_buf_131072:      131072,      0,      43,      43,  548063,   0,   
> 0
> zio_data_buf_131072: 131072,      0,   67545,       2,  646567,   0,   
> 0
> lz4_ctx:              16384,      0,       0,      16, 2249605,   0,   
> 0
> sa_cache:                80,      0,  109455,   66445, 2663034,   0,   
> 0
> dnode_t:                744,      0,  501793,  259152,  873895,   0,   
> 0
> dmu_buf_impl_t:         224,      0,  635273,  343604, 3042828,   0,   
> 0
> arc_buf_hdr_t:          216,      0,  643049,  141229, 3158205,   0,   
> 0
> arc_buf_t:               72,      0,  481443,  119872, 3705908,   0,   
> 0
> zil_lwb_cache:          192,      0,       8,    5072,   66838,   0,   
> 0
> zfs_znode_cache:        368,      0,  109455,   66105, 2663034,   0,   
> 0
> pipe:                   744,      0,     113,     177,   66135,   0,   
> 0
> procdesc:               128,      0,       0,       0,       0,   0,   
> 0
> ksiginfo:               112,      0,     237,    2528, 8311580,   0,   
> 0
> itimer:                 352,      0,       1,      32,       1,   0,   
> 0
> KNOTE:                  128,      0,     376,    1391,17307485,   0,   
> 0
> socket:                 696, 523340,     258,     302,  668458,   0,   
> 0
> unpcb:                  240, 523344,     151,     649,  130961,   0,   
> 0
> ipq:                     56,  31879,       0,    2059,     317,   0,   
> 0
> udp_inpcb:              392, 523340,      19,     421,  469503,   0,   
> 0
> udpcb:                   16, 523586,      19,    2993,  469503,   0,   
> 0
> tcp_inpcb:              392, 523340,     100,     370,   67983,   0,   
> 0
> tcpcb:                 1024, 523340,      78,     234,   67983,   0,   
> 0
> tcptw:                   88,  27810,      22,    1013,   22066,   0,   
> 0
> syncache:               160,  15375,       0,     675,   45270,   0,   
> 0
> hostcache:              136,  15370,     255,     499,    3892,   0,   
> 0
> tcpreass:                40,  63700,       0,    2600,    7616,   0,   
> 0
> sackhole:                32,      0,       0,    1875,    7184,   0,   
> 0
> sctp_ep:               1408, 523340,       0,       0,       0,   0,   
> 0
> sctp_asoc:             2352,  40000,       0,       0,       0,   0,   
> 0
> sctp_laddr:              48,  80012,       0,    1328,       8,   0,   
> 0
> sctp_raddr:             728,  80000,       0,       0,       0,   0,   
> 0
> sctp_chunk:             136, 400026,       0,       0,       0,   0,   
> 0
> sctp_readq:             104, 400026,       0,       0,       0,   0,   
> 0
> sctp_stream_msg_out:    104, 400026,       0,       0,       0,   0,   
> 0
> sctp_asconf:             40, 400000,       0,       0,       0,   0,   
> 0
> sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   
> 0
> udplite_inpcb:          392, 523340,       0,       0,       0,   0,   
> 0
> ripcb:                  392, 523340,       2,      58,       3,   0,   
> 0
> rtentry:                200,      0,      32,     408,      34,   0,   
> 0
> selfd:                   56,      0,     333,    2223,29658712,   0,   
> 0
> SWAPMETA:               288, 2037438,       0,       0,       0,   0,
> 0
> 
> thebighonker.lerctr.org /home/ler

and I ran the script again....  It got further:
18:20:52   25.1G   zroot/home@zxfer_26699_20140724135840
18:20:53   25.1G   zroot/home@zxfer_26699_20140724135840
18:20:54   25.1G   zroot/home@zxfer_26699_20140724135840
18:20:55   25.1G   zroot/home@zxfer_26699_20140724135840
18:20:56   25.1G   zroot/home@zxfer_26699_20140724135840
18:20:57   25.1G   zroot/home@zxfer_26699_20140724135840
18:20:58   25.1G   zroot/home@zxfer_26699_20140724135840
Connection to home.lerctr.org closed by remote host.
warning: cannot send 'zroot/home@zxfer_26699_20140724135840': Broken 
pipe
TIME        SENT   SNAPSHOT
warning: cannot send 'zroot/home@2014-07-24': Broken pipe
cannot open 'zroot/backups/TBH2/home': dataset does not exist
borg.lerctr.org /home/ler/bin #

BUT we still fail.

#!/bin/sh
DATE=`date "+%Y-%m-%d"`
#DATE2=2013-03-24
#DATE2=`date -v "-1d" "+%Y-%m-%d"`
# snap the source
ssh root@tbh.lerctr.org zfs snapshot -r zroot@${DATE}
# zfs copy the source to here.
ssh root@tbh.lerctr.org "zfs send  -v -R zroot@${DATE} | \
      ssh home.lerctr.org \"zfs recv -F -u -v -d zroot/backups/TBH2\""
# make sure we NEVER allow the backup stuff to automount.
/sbin/zfs list -H -t filesystem -r zroot/backups/TBH2| \
     awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh
borg.lerctr.org /home/ler/bin #

this was to a FRESH zroot/backups/TBH2


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c)     E-Mail: ler@lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?fbf874048b94066aabc4e656e2d3157d>