Logical Domains for CoolThreads Servers - reproducible control domain hang

I configured a T2000 as described in the beginner's guide (<a href="http://www.sun.com/blueprints/0207/820-0832.pdf">http://www.sun.com/blueprints/0207/820-0832.pdf</a>) with the exception of the device allocated for the root disk. For that I came up with my own variant of <a href="http://unixconsole.blogspot.com/2007/04/time-to-build-guest-domain.html">http://unixconsole.blogspot.com/2007/04/time-to-build-guest-domain.html</a>.

My variant of using a file involved creating the file with mkfile on a zfs file system. That is...

<pre>
zpool create zfs mirror c1t0d0s4 c1t1d0s4
zfs create zfs/ldoms
zfs set compress=on zfs/ldoms
mkfile 32G /zfs/ldoms/root.img
</pre>

As I install Solaris in the ldom, the server (control domain) dies after extracting a few hundred megabytes of a flash archive. I have traced this down to it running out of memory.

Here's "vmstat 4" output on the control domain console:

<pre>
...
0 0 0 8636536 24776 1 70 0 0 0 0 0 0 0 0 0 1954 311 2244 0 23 76
0 0 0 8645096 22280 1 36 0 0 0 0 0 0 0 0 0 5957 370 9827 0 19 80
0 0 0 8649008 17720 1 40 0 296 313 0 44 0 0 0 0 9975 361 15877 0 24 75
0 0 0 8651104 15944 1 52 0 807 1671 0 700 0 0 0 0 10725 347 17545 0 26 74
0 0 0 8650800 18376 0 60 0 88 239 0 127 0 0 0 0 9816 391 15545 1 33 67
0 0 0 8640432 15936 0 76 0 497 3025 0 3874 0 0 0 0 11367 432 17975 0 35 65
0 0 0 8642968 17032 1 59 0 452 2028 0 842 0 0 0 0 10266 363 16127 0 27 73
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m2 m1 in sy cs us sy id
0 0 0 8644768 15744 0 56 0 387 1298 0 126 0 0 0 0 10170 330 16355 0 24 75
0 0 0 8652504 18368 1 113 0 372 2462 0 273 0 0 0 0 11171 321 18613 0 35 65
0 0 0 8652832 15720 1 134 0 411 6081 0 738 0 0 0 0 11541 332 18979 0 34 66
0 0 0 8652232 14312 1 94 0 413 1806 0 7775 0 0 0 0 10718 358 18271 0 38 62
0 0 0 8647360 12592 18 133 9 555 5176 0 17490 1 0 1 0 10394 320 16970 1 37 63
0 0 0 8645248 14408 2 73 22 486 5039 0 3111 2 1 1 0 11749 383 18336 0 40 59
2 0 43 8641800 2784 1 148 99 1070 1517 0 53982 19 9 9 5 8316 356 14226 0 43 57
0 0 116 8647032 800 1 42 127 134 312 3688 76207 14 7 7 1 2153 114 3726 0 29 71
</pre>

At this point the server froze. Note that 116 processes were swapped and the "de" column is 3688. Very bad news.

My initial thoughts were that I was running into some of the low-memory problems known to happen with the ZFS arc. This does not seem to be the case. According to mdb, the arc size is around 60 MB:

<pre>
# mdb unix.3 vmcore.3
...

arc::print -td size

uint64_t size = 0t61455360
</pre>

The control domain is S10 11/06 118833-36 those required for ldoms + many others. The ldom is in the process of being installed is booted from a S10 11/06 netinstall image (118833-33).

Are you trying to jumpstart the control domain or a guest domain?

This would be the guest domain.

How much memory did you allocate for the control/service domain? Also, what are the bindings for your control and guest domain?

I allocated 1 GB for the control domain and 4 GB for the service domain. Here's my control domain bindings...

<pre>
# ldm list-bindings primary
Name: primary
State: active
Flags: transition,control,vio service
OS:
Util: 0.7%
Uptime: 7d 19h 25m
Vcpu: 4
vid pid util strand
0 0 1.3% 100%
1 1 0.6% 100%
2 2 1.1% 100%
3 3 0.0% 100%
Mau: 1
mau cpuset (0, 1, 2, 3)
Memory: 1G
real-addr phys-addr size
0x4000000 0x4000000 1G
Vars: auto-boot-on-error?=true
boot-device=truncated so as to not mess up the forum formatting
error-reset-recovery=sync
reboot-command=boot
IO: pci@780 (bus_a)
pci@7c0 (bus_b)
Vldc: primary-vldc0
(HV Control channel)]
[LDC: 0x1]
[LDom primary (Domain Services channel)]
[LDC: 0x3]
[LDom primary (FMA Services channel)]
[LDC: 0xf]
[LDom lab-test-prod-01 (Domain Services channel)]
Vldc: primary-vldc3
(SP channel)]
(SP channel)]
(SP channel)]
(SP channel)]
(SP channel)]
(SP channel)]
(SP channel)]
Vds: primary-vds0
vdsdev: root device=/zfs/ldomroots/2007Q1.img
vdsdev: vol1 device=/dev/md/dsk/d1000
[LDom lab-test-prod-01, dev-name: root]
[LDC: 0xc]
Vcc: primary-vcc0
[LDC: 0xe]
[LDom lab-test-prod-01, group: lab-test-prod-01, port: 5000]
port-range=5000-5100
Vsw: primary-vsw0
mac-addr=0:14:4f:f8:ec:5e
net-dev=e1000g0
[LDC: 0xb]
[LDom lab-test-prod-01, name: prod0, mac-addr:0x144ffb5821]
mode=prog,promisc
Vldcc: vldcc1 [FMA Services]
service: ldmfma
service: primary-vldc0 @ primary
[LDC: 0x4]
Vldcc: vldcc2 [SP Channel]
service: spfma
Vldcc: vldcc0 [Domain Services]
service: primary-vldc0 @ primary
[LDC: 0x2]
Vldcc: hvctl [Hypervisor Control]
service: primary-vldc0 @ primary
[LDC: 0x0]
Vcons: SP
</pre>

I have since removed the binding for vol1@primary-vds0 - it was a leftover from trying to use soft partitions and failing there too. Retrying the jumpstart after removing this led to a similar hang.

During this latest hang I was also running "prstat -s size -c" on the console (no user processes were growing) and I had run ::kmastat in mdb. ::kmastat did not finish but I did get...

<pre>

::kmastat

cache buf buf buf memory alloc alloc
name size in use total in use succeed fail

------

---------

kmem_magazine_1 16 216 812 32768 55452 0
kmem_magazine_3 32 447 4572 294912 24764 0
kmem_magazine_7 64 556 11508 1122304 73180 0
kmem_magazine_15 128 497 5796 1130496 43293 0
kmem_magazine_31 256 61 1050 344064 11462 0
kmem_magazine_47 384 55 252 114688 1183 0
kmem_magazine_63 512 125 392 229376 4858 0
kmem_magazine_95 768 8 54 49152 262 0
kmem_magazine_143 1152 0 0 0 0 0
kmem_slab_cache 56 24574 28381 2301952 1478117 0
kmem_bufctl_cache 24 1138 4056 196608 50160 0
kmem_bufctl_audit_cache 128 794790 848318 131121152 4436827 0
kmem_va_8192 8192 15539 17696 144965632 20509 0
kmem_va_16384 16384 249 288 4718592 2751 0
kmem_va_24576 24576 3900 3910 102498304 5064 0
kmem_va_32768 32768 5 16 524288 870 0
kmem_va_40960 40960 10 18 786432 1974 0
kmem_va_49152 49152 6 10 524288 727 0
kmem_va_57344 57344 19 36 2359296 1967 0
kmem_va_65536 65536 323 328 21495808 1029 0
kmem_alloc_8 8 9880 15360 491520 29384145 0
kmem_alloc_16 16 49180 51612 2072576 106580076 0
kmem_alloc_24 24 5007 17170 827392 3624861 0
kmem_alloc_32 32 23743 38690 2170880 13937187 0
kmem_alloc_40 40 6650 35456 2269184 2437686 0
kmem_alloc_48 48 51877 101022 7323648 1337683 0
kmem_alloc_56 56 3754 53550 4300800 1905113 0
kmem_alloc_64 64 2097 29504 3776512 11461812 0
kmem_alloc_80 80 4665 28080 2949120 737718 0
kmem_alloc_96 96 8837 14076 1695744 2032261 0
kmem_alloc_112 112 24855 26340 3596288 108921 0
kmem_alloc_128 128 1338 2562 499712 1920174 0
kmem_alloc_160 160 911 1848 344064 4933550 0
kmem_alloc_192 192 270 320 81920 1559636 0
kmem_alloc_224 224 550 825 204800 176907 0
kmem_alloc_256 256 454 750 245760 8445251 0
kmem_alloc_320 320 262 378 147456 392254 0
kmem_alloc_384 384 41240 41598 18931712 6028188 0
kmem_alloc_448 448 63 96 49152 2461252 0
kmem_alloc_512 512 353 924 540672 1260259 0
kmem_alloc_640 640 96 792 589824 9092211 0
kmem_alloc_768 768 23 45 40960 82825 0
kmem_alloc_896 896 17 48 49152 342 0
kmem_alloc_1152 1152 610 650 819200 2869973 0
kmem_alloc_1344 1344 31 88 131072 8301 0
kmem_alloc_1600 1600 7 27 49152 93881 0
kmem_alloc_2048 2048 8305 8327 18604032 55053 0
kmem_alloc_2688 2688 24686 24704 75890688 213546 0
kmem_alloc_4096 4096 76 76 622592 29475 0
kmem_alloc_8192 8192 2715 2715 22241280 364803 0
kmem_alloc_12288 12288 19 19 311296 227 0
kmem_alloc_16384 16384 15 15 245760 39360 0
streams_mblk 64 26144 27285 2629632 506588253 0
streams_dblk_16 128 74 126 24576 4623201 0
streams_dblk_80 192 292 1088 278528 293518870 0
streams_dblk_144 256 2 300 98304 23652413 0
streams_dblk_208 320 173 420 163840 1359658 0
streams_dblk_272 384 1 90 40960 720328 0
streams_dblk_336 448 0 64 32768 975862 0
streams_dblk_528 640 0 44 32768 1667752 0
streams_dblk_1040 1152 1 65 81920 177681 0
streams_dblk_1488 1600 0 18 32768 554 0
streams_dblk_1936 2048 2 44 98304 21604 0
streams_dblk_2576 2688 0 192 589824 50430016 0
streams_dblk_3920 4032 0 6 24576 366 0
streams_dblk_8192 112 0 42 8192 12182 0
streams_dblk_12112 12224 0 0 0 213 0
streams_dblk_16384 112 0 42 8192 1 0
streams_dblk_20304 20416 0 0 0 0 0
streams_dblk_24576 112 0 0 0 0 0
streams_dblk_28496 28608 0 0 0 0 0
streams_dblk_32768 112 0 42 8192 3 0
streams_dblk_36688 36800 0 0 0 3 0
streams_dblk_40960 112 0 0 0 0 0
streams_dblk_44880 44992 0 0 0 0 0
streams_dblk_49152 112 0 42 8192 1 0
streams_dblk_53072 53184 0 0 0 0 0
streams_dblk_57344 112 0 0 0 0 0
streams_dblk_61264 61376 0 0 0 0 0
streams_dblk_65536 112 0 42 8192 2 0
streams_dblk_69456 69568 0 0 0 461530 0
streams_dblk_73728 112 0 0 0 0 0
streams_dblk_esb 112 25600 25620 4997120 47599192 0
streams_fthdr 264 0 0 0 0 0
streams_ftblk 232 0 0 0 0 0
multidata 248 0 0 0 0 0
multidata_pdslab 7112 0 0 0 0 0
multidata_pattbl 32 0 0 0 0 0
log_cons_cache 48 13 113 8192 222 0
taskq_ent_cache 56 1837 2346 188416 14881 0
taskq_cache 216 89 102 24576 114 0
id32_cache 32 1 128 8192 27 0
bp_map_8192 8192 0 0 0 23414 0
bp_map_16384 16384 0 0 0 8 0
bp_map_24576 24576 0 0 0 2 0
bp_map_32768 32768 0 0 0 5 0
bp_map_40960 40960 0 0 0 0 0
bp_map_49152 49152 0 0 0 0 0
bp_map_57344 57344 0 0 0 0 0
bp_map_65536 65536 0 0 0 0 0
memseg_cache 112 0 0 0 0 0
mod_hash_entries 24 265 680 32768 1347218 0
ipp_mod 304 0 0 0 0 0
ipp_action 368 0 0 0 0 0
ipp_packet 64 0 0 0 0 0
sfmmuid_cache 256 83 116 32768 151749 0
sfmmu_tsbinfo_cache 64 84 186 16384 298864 0
sfmmu_tsb8k_cache 8192 0 0 0 0 0
sfmmu_tsb_cache 8192 38 40 655360 153370 0
sfmmu8_cache 312 17465 46152 15753216 6532821 0
sfmmu1_cache 88 1732 7632 868352 1314219 0
pa_hment_cache 64 62 184 16384 153318 0
ism_blk_cache 272 0 0 0 0 0
ism_ment_cache 32 0 0 0 0 0
seg_cache 72 4633 5610 540672 5752734 0
dev_info_node_cache 480 204 256 131072 815 0
segkp_8192 8192 69 80 655360 31359 0
segkp_16384 16384 0 0 0 0 0
segkp_24576 24576 0 0 0 0 0
segkp_32768 32768 628 672 22020096 6666 0
segkp_40960 40960 0 0 0 0 0
umem_np_8192 8192 0 0 0 12390 0
umem_np_16384 16384 0 0 0 6189 0
umem_np_24576 24576 0 0 0 0 0
umem_np_32768 32768 0 0 0 6133 0
umem_np_40960 40960 0 0 0 0 0
umem_np_49152 49152 0 0 0 0 0
umem_np_57344 57344 0 0 0 0 0
umem_np_65536 65536 0 0 0 6083 0
thread_cache 792 236 279 253952 95083 0
lwp_cache 904 236 280 286720 5424 0
turnstile_cache 64 633 837 73728 113165 0
tslabel_cache 48 2 113 8192 2 0
cred_cache 172 72 120 24576 12132 0
rctl_cache 40 1183 1408 90112 982809 0
rctl_val_cache 64 2371 2604 229376 2088132 0
task_cache 104 45 128 16384 6611 0
cyclic_id_cache 64 3 93 8192 3 0
dnlc_space_cache 24 0 680 32768 2891 0
vn_cache 240 9502 83325 27303936 6157311 0
vsk_anchor_cache 40 24 128 8192 58 0
file_cache 56 590 714 57344 2542492 0
stream_head_cache 400 253 304 131072 199439 0
queue_cache 656 584 648 442368 292448 0
syncq_cache 160 16 44 8192 250 0
qband_cache 64 2 93 8192 2 0
linkinfo_cache 48 13 113 8192 23 0
ciputctrl_cache 1024 0 0 0 0 0
serializer_cache 64 24 93 8192 750 0
as_cache 216 82 136 32768 146548 0
marker_cache 128 0 53 8192 77719 0
anon_cache 48 17448 18156 1458176 7107774 0
anonmap_cache 48 2825 3503 253952 3126820 0
segvn_cache 104 4633 5184 663552 5624635 0
flk_edges 48 0 113 8192 2 0
fdb_cache 104 0 0 0 0 0
timer_cache 136 2 51 8192 3 0
physio_buf_cache 248 0 30 8192 31 0
snode_cache 152 353 414 73728 633687 0
ufs_inode_cache 368 8333 75240 30818304 1362996 0
directio_buf_cache 272 0 0 0 0 0
lufs_save 24 0 170 8192 39162 0
lufs_bufs 256 0 29 8192 44473 0
lufs_mapentry_cache 112 6 1020 139264 1447748 0
px1_dvma_8192 8192 16 24 196608 4888902 0
mpt0_cache 480 6 112 57344 2930331 0
dv_node_cache 120 81 448 65536 1634 0
clnt_clts_endpnt_cache 88 0 73 8192 2 0
md_stripe_parent 96 0 136 16384 2178767 0
md_stripe_child 312 0 72 24576 2470849 0
md_mirror_parent 160 0 88 16384 1486941 0
md_mirror_child 304 0 96 32768 2376152 0
md_mirror_wow 16440 0 0 0 10 0
md_softpart_parent 88 0 73 8192 103 0
md_softpart_child 304 0 24 8192 115 0
ldc_memhdl_cache 48 1028 1130 81920 2782 0
ldc_memseg_cache 64 515 558 49152 28868 0
mac_impl_cache 752 5 10 8192 5 0
dls_cache 168 8 84 16384 2575 0
soft_ring_cache 176 0 0 0 0 0
dls_vlan_cache 48 9 113 8192 13 0
dls_link_cache 624 5 12 8192 5 0
dld_ctl_1 1 0 0 0 0 0
dld_str_cache 256 10 29 8192 3576 0
px0_dvma_8192 8192 0 0 0 0 0
kcf_sreq_cache 48 0 64 8192 244 0
kcf_areq_cache 272 0 0 0 0 0
kcf_context_cache 88 0 0 0 0 0
ipsec_actions 72 2 85 8192 212 0
ipsec_selectors 72 2 85 8192 2 0
ipsec_policy 72 4 85 8192 4 0
ipsec_info 304 0 24 8192 13578 0
ip_minor_arena_1 1 93 128 128 40881 0
ipcl_conn_cache 480 60 80 40960 39661 0
ipcl_tcpconn_cache 1664 34 54 98304 1696 0
ire_cache 352 36 63 24576 53773 0
tcp_timercache 88 48 146 16384 2611 0
tcp_sack_info_cache 80 10 78 8192 1057 0
tcp_iphc_cache 120 34 112 16384 1446 0
squeue_cache 136 4 42 8192 4 0
sctp_conn_cache 2232 1 7 16384 1 0
sctp_faddr_cache 168 0 0 0 0 0
sctp_set_cache 24 0 0 0 0 0
sctp_ftsn_set_cache 16 0 0 0 0 0
ire_gw_secattr_cache 32 0 0 0 0 0
sctpsock 616 0 0 0 0 0
sctp_assoc 64 0 0 0 0 0
socktpi_cache 456 54 68 32768 32220 0
socktpi_unix_cache 456 3 34 16384 411 0
udp_cache 408 46 72 32768 39422 0
process_cache 3048 84 104 319488 70100 0
exacct_object_cache 40 0 128 8192 7881318 0
fctl_cache 112 0 60 8192 20 0
tl_cache 432 35 68 32768 929 0
keysock_1 1 0 0 0 0 0
spdsock_1 1 0 0 0 9 0
fnode_cache 176 7 36 8192 319 0
pipe_cache 320 42 92 32768 74459 0
fp2_cache 728 1 10 8192 32 0
fp0_cache 728 1 10 8192 18 0
fcp2_cache 1168 0 39 49152 124848 0
fcp0_cache 1168 0 39 49152 137641 0
ncp_ds_cache 80 0 0 0 0 0
ncp_mactl_cache 24 0 0 0 0 0
ncp_mabuf_cache 2048 0 0 0 0 0
kssl_cache 1560 0 0 0 0 0
namefs_inodes_1 1 21 64 64 21 0
port_cache 80 7 78 8192 32 0
qif_head_cache 264 10 28 8192 3575 0
ip_minor_1 1 0 0 0 0 0
ar_minor_1 1 0 0 0 0 0
lnode_cache 32 2 146 8192 2 0
icmp_minor_1 1 0 0 0 0 0
zio_buf_512 512 14 45 24576 1652 0
zio_buf_1024 1024 5 40 40960 6987 0
zio_buf_1536 1536 0 10 16384 111 0
zio_buf_2048 2048 0 16 32768 967 0
zio_buf_2560 2560 0 15 40960 359 0
zio_buf_3072 3072 0 16 49152 1054 0
zio_buf_3584 3584 0 9 32768 1910 0
zio_buf_4096 4096 5 14 57344 939 0
zio_buf_5120 5120 0 16 81920 7230 0
zio_buf_6144 6144 0 8 49152 4837 0
zio_buf_7168 7168 0 8 57344 4333 0
zio_buf_8192 8192 0 6 49152 3797 0
zio_buf_10240 10240 0 8 81920 27250 0
zio_buf_12288 12288 0 6 73728 5937 0
zio_buf_14336 14336 0 40 573440 15750 0
zio_buf_16384 16384 162 163 2670592 4816144 0
zio_buf_20480 20480 0 6 122880 8855 0
zio_buf_24576 24576 0 5 122880 6919 0
zio_buf_28672 28672 0 8 229376 5719 0
zio_buf_32768 32768 0 5 163840 6477 0
zio_buf_40960 40960 0 3 122880 9672 0
zio_buf_49152 49152 0 4 196608 5027 0
zio_buf_57344 57344 0 5 286720 3872 0
zio_buf_65536 65536 322 324 21233664 28824 0
zio_buf_73728 73728 0 4 294912 5806 0
zio_buf_81920 81920 0 4 327680 3513 0
zio_buf_90112 90112 0 3 270336 2458 0
zio_buf_98304 98304 0 4 393216 2458 0
zio_buf_106496 106496 0 5 532480 1827 0
zio_buf_114688 114688 1 7 802816 29960 0
zio_buf_122880 122880 0 5 614400 182 0
zio_buf_131072 131072 378 385 50462720 3375481 0
dmu_buf_impl_t 328 558 2070 737280 8124993 0
dnode_t 640 32 36 24576 56 0
arc_buf_hdr_t 128 719 2862 442368 8108237 0
arc_buf_t 40 550 1408 90112 8120313 0
zil_lwb_cache 216 1 34 8192 5 0
zfs_znode_cache 192 4 37 8192 4 0
pty_map 56 4 102 8192 59 0
crypto_session_cache 96 0 0 0 0 0
md_raid_parent 120 0 0 0 0 0
md_raid_child 1040 0 0 0 0 0
md_raid_cbufs 376 0 0 0 0 0
md_trans_parent 80 0 0 0 0 0
md_trans_child 248 0 0 0 0 0
authkern_cache 72 0 85 8192 12637 0
authloopback_cache 72 0 85 8192 5945 0
authdes_cache_handle 80 0 0 0 0 0
rnode_cache 648 11 72 49152 428 0
nfs_access_cache 56 0 102 8192 201 0
client_handle_cache 32 1 146 8192 15 0
rnode4_cache 960 0 0 0 0 0
svnode_cache 40 0 0 0 0 0
nfs4_access_cache 56 0 0 0 0 0
client_handle4_cache 32 0 0 0 0 0
nfs4_ace4vals_cache 48 0 0 0 0 0
nfs4_ace4_list_cache 264 0 0 0 0 0
NFS_idmap_cache 48 0 0 0 0 0
lm_vnode 184 0 0 0 0 0
lm_xprt 32 0 0 0 0 0
lm_sysid 160 1 44 8192 2 0
lm_client 128 0 53 8192 2 0
lm_async 32 0 0 0 0 0
lm_sleep 96 1 68 8192 1 0
lm_config 80 1 78 8192 1 0
dtrace_state_cache 4096 0 0 0 0 0
sd2_cache 56 0 0 0 0 0
hsfs_hsnode_cache 192 1 37 8192 1 0

------

---------

Total [static] 270336 248401 0
Total [hat_memload] 15753216 6532821 0
Total [kmem_msb] 136937472 6179558 0
Total [kmem_firewall] 23420928 895621 0
Total [kmem_va] 277872640 34891 0
Total [kmem_default] 303939584 1236802702 0
Total [bp_map] 0 23429 0
Total [kmem_tsb_default] 655360 153370 0
Total [hat_memload1] 868352 1314219 0
Total [umem_np] 0 30795 0
Total [id32] 8192 27 0
Total [segkp] 22675456 38025 0
Total [px1_dvma] 196608 4888902 0
Total [ip_minor_arena] 128 40881 0
Total [spdsock] 0 9 0
Total [namefs_inodes] 64 21 0

------

---------

vmem memory memory memory alloc alloc
name in use total import succeed fail

---------

-----
</pre>

Hmm.. I have not run into this issue and can't reproduce it. What version of Solaris are you using? Also, I don't see the bindings for your guest domain.

It looks like the control domain and the service domain are one and the same (the domain 'primary' contains both bus_a and bus_b). This domain has only 1G of memory so I think you are running into '6447701 ZFS hangs when iSCSI Target attempts to initialize its backing store' which is fixed in OpenSolaris build 49 and will be in the next update release of S10.
 
In the meantime a workaround is to ensure that there is at least 4G of memory in the control/service domain - aka 'primary'.

participate

Sun Forums

Welcome

Forums Statistics

About Sun forums