[TargetCLI ZFS / ESXi] Onverklaarbare crashes

Pagina: 1
Acties:

Acties:
  • 0 Henk 'm!

  • FireDrunk
  • Registratie: November 2002
  • Laatst online: 30-09 12:37
Allereerst mijn setup:

iSCSI host:
Xeon E5-2560v3
64GB RAM
4 NIC's met 4 ip's in 4 verschillende subnets
LUN 0: 2 * 128GB SSD's in een ZFS Mirror
LUN 1: 4 * 256GB SSD's in 2 ZFS Mirrors (RAID10 like dus)

iSCSI Client:

ESXi 6.0u1 host
Intel i5-3570
32GB RAM
4 NIC's

Mijn iSCSI Configuratie:

[root@NAS thijs]# targetcli
targetcli shell version 2.1.fb42
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 2]
  | | o- 840PRO_LUN0 ................................................... [/dev/zvol/VM_TIER_1/zvol0 (110.0GiB) write-thru activated]
  | | o- MX100_LUN0 .................................................... [/dev/zvol/VM_TIER_2/zvol0 (400.0GiB) write-thru activated]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2003-01.org.linux-iscsi.nas.x8664:sn.80c34394d007 ............................................................. [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 5]
  |     | o- iqn.1998-01.com.vmware:vh01-38673156 ................................................................. [Mapped LUNs: 2]
  |     | | o- mapped_lun0 ........................................................................... [lun0 block/840PRO_LUN0 (rw)]
  |     | | o- mapped_lun1 ............................................................................ [lun1 block/MX100_LUN0 (rw)]
  |     | o- iqn.1998-01.com.vmware:vh02-33d4caa1 ................................................................. [Mapped LUNs: 2]
  |     | | o- mapped_lun0 ........................................................................... [lun0 block/840PRO_LUN0 (rw)]
  |     | | o- mapped_lun1 ............................................................................ [lun1 block/MX100_LUN0 (rw)]
  |     | o- iqn.1998-01.com.vmware:vh03-7d421107 ................................................................. [Mapped LUNs: 2]
  |     | | o- mapped_lun0 ........................................................................... [lun0 block/840PRO_LUN0 (rw)]
  |     | | o- mapped_lun1 ............................................................................ [lun1 block/MX100_LUN0 (rw)]
  |     | o- iqn.1998-01.com.vmware:vh04-77a89664 ................................................................. [Mapped LUNs: 2]
  |     | | o- mapped_lun0 ........................................................................... [lun0 block/840PRO_LUN0 (rw)]
  |     | | o- mapped_lun1 ............................................................................ [lun1 block/MX100_LUN0 (rw)]
  |     | o- iqn.1998-01.com.vmware:vh05-5e200351 ................................................................. [Mapped LUNs: 2]
  |     |   o- mapped_lun0 ........................................................................... [lun0 block/840PRO_LUN0 (rw)]
  |     |   o- mapped_lun1 ............................................................................ [lun1 block/MX100_LUN0 (rw)]
  |     o- luns .......................................................................................................... [LUNs: 2]
  |     | o- lun0 .................................................................. [block/840PRO_LUN0 (/dev/zvol/VM_TIER_1/zvol0)]
  |     | o- lun1 ................................................................... [block/MX100_LUN0 (/dev/zvol/VM_TIER_2/zvol0)]
  |     o- portals .................................................................................................... [Portals: 4]
  |       o- 10.0.1.1:3260 .................................................................................................... [OK]
  |       o- 10.0.2.1:3260 .................................................................................................... [OK]
  |       o- 10.0.3.1:3260 .................................................................................................... [OK]
  |       o- 10.0.4.1:3260 .................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]
  o- vhost ............................................................................................................ [Targets: 0]

VH01 t/m VH04 zijn momenteel niet in gebruik

De ESXi host is voorzien van multipathing naar de 4 verschillende ip's van de iSCSI host.
Alles werkt perfect, en ik satureer bij benchmarks de 4 links, so far so good.

Na verloop van tijd (vooral als ik meerdere dingen tegelijk doe) crasht de hele ESXi host omdat de iSCSI verbinding wegvalt.

Laatste stuk van de logging:

code:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
[< 1572.267304>] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[<    0.002795>] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[< 1800.179389>] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[<    0.002613>] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[<  881.961722>] iSCSI Login timeout on Network Portal 10.0.1.1:3260
[<   44.421173>] iSCSI Login timeout on Network Portal 10.0.4.1:3260
[<    0.000099>] Unexpected ret: -32 send data 48
[<    0.000035>] tx_data returned -32, expecting 48.
[<    0.000031>] iSCSI Login negotiation failed.
[<   15.041596>] iSCSI Login timeout on Network Portal 10.0.4.1:3260
[<    0.000100>] Unexpected ret: -32 send data 48
[<    0.000035>] tx_data returned -32, expecting 48.
[<    0.000031>] iSCSI Login negotiation failed.
[<   15.041565>] iSCSI Login timeout on Network Portal 10.0.4.1:3260
[<    0.000098>] Unexpected ret: -32 send data 48
[<    0.000037>] tx_data returned -32, expecting 48.
[<    0.000031>] iSCSI Login negotiation failed.
[<   15.041591>] iSCSI Login timeout on Network Portal 10.0.4.1:3260
[<    0.000095>] Unexpected ret: -32 send data 48
[<    0.000034>] tx_data returned -32, expecting 48.
[<    0.000028>] iSCSI Login negotiation failed.
[<    0.319868>] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[<    0.000098>] Unexpected ret: -32 send data 48
[<    0.000035>] tx_data returned -32, expecting 48.
[<    0.000031>] iSCSI Login negotiation failed.
[<   14.721562>] iSCSI Login timeout on Network Portal 10.0.4.1:3260
[<    0.000099>] Unexpected ret: -32 send data 48
[<    0.000036>] tx_data returned -32, expecting 48.
[<    0.000031>] iSCSI Login negotiation failed.
[<    0.319842>] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[<    0.000094>] Unexpected ret: -32 send data 48
[<    0.000035>] tx_data returned -32, expecting 48.
[<    0.000031>] iSCSI Login negotiation failed.
[<   14.721560>] iSCSI Login timeout on Network Portal 10.0.4.1:3260
[<    0.000068>] Unexpected ret: -32 send data 48
[<    0.000029>] tx_data returned -32, expecting 48.
[<    0.000029>] iSCSI Login negotiation failed.
[<    0.319880>] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[<    0.000088>] Unexpected ret: -32 send data 48
[<    0.000035>] tx_data returned -32, expecting 48.
[<    0.000031>] iSCSI Login negotiation failed.
[<   14.465585>] iSCSI Login timeout on Network Portal 10.0.2.1:3260
[<    0.001118>] Unexpected ret: -32 send data 48
[<    0.001098>] tx_data returned -32, expecting 48.
[<    0.001093>] iSCSI Login negotiation failed.
[<    0.252709>] iSCSI Login timeout on Network Portal 10.0.4.1:3260
[<    0.001125>] Unexpected ret: -32 send data 48
[<    0.001087>] tx_data returned -32, expecting 48.
[<    0.001062>] iSCSI Login negotiation failed.
[<    0.316713>] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[<    0.001132>] Unexpected ret: -32 send data 48
[<    0.001135>] tx_data returned -32, expecting 48.
[<    0.001098>] iSCSI Login negotiation failed.
[<   14.462361>] iSCSI Login timeout on Network Portal 10.0.2.1:3260
[<    0.001176>] Unexpected ret: -32 send data 48
[<    0.001145>] tx_data returned -32, expecting 48.
[<    0.001120>] iSCSI Login negotiation failed.
[<    0.252573>] iSCSI Login timeout on Network Portal 10.0.4.1:3260
[<    0.001171>] Unexpected ret: -32 send data 48
[<    0.001197>] tx_data returned -32, expecting 48.
[<    0.001126>] iSCSI Login negotiation failed.
[<    0.316538>] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[<    0.001173>] Unexpected ret: -32 send data 48
[<    0.001146>] tx_data returned -32, expecting 48.
[<    0.001090>] iSCSI Login negotiation failed.
[<   14.462262>] iSCSI Login timeout on Network Portal 10.0.2.1:3260
[<    0.001177>] Unexpected ret: -32 send data 48
[<    0.001150>] tx_data returned -32, expecting 48.
[<    0.001065>] iSCSI Login negotiation failed.
[<    0.252641>] iSCSI Login timeout on Network Portal 10.0.4.1:3260
[<    0.001169>] Unexpected ret: -32 send data 48
[<    0.001153>] tx_data returned -32, expecting 48.
[<    0.001129>] iSCSI Login negotiation failed.
[<    0.316606>] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[<    0.001161>] Unexpected ret: -32 send data 48
[<    0.001124>] tx_data returned -32, expecting 48.
[<    0.001127>] iSCSI Login negotiation failed.
[<    7.878775>] Unexpected ret: -32 send data 360
[<    0.000808>] tx_data returned -32, expecting 360.
[<    0.000861>] iSCSI Login negotiation failed.
[<    0.001140>] Unexpected ret: -32 send data 360
[<    0.001132>] tx_data returned -32, expecting 360.
[<    0.001160>] iSCSI Login negotiation failed.
[<    0.001335>] Unexpected ret: -32 send data 360
[<    0.000748>] tx_data returned -32, expecting 360.
[<    0.000792>] iSCSI Login negotiation failed.
[<    0.000955>] Unexpected ret: -32 send data 360
[<    0.000959>] tx_data returned -32, expecting 360.
[<    0.001017>] iSCSI Login negotiation failed.
[<    0.001176>] Unexpected ret: -32 send data 360
[<    0.000834>] tx_data returned -32, expecting 360.
[<    0.001099>] iSCSI Login negotiation failed.
[<    0.001349>] Unexpected ret: -32 send data 360
[<    0.001124>] tx_data returned -32, expecting 360.
[<    0.001099>] iSCSI Login negotiation failed.
[<    0.001168>] Unexpected ret: -32 send data 360
[<    0.000714>] tx_data returned -32, expecting 360.
[<    0.000735>] iSCSI Login negotiation failed.
[<    0.000897>] Unexpected ret: -32 send data 360
[<    0.000929>] tx_data returned -32, expecting 360.
[<    0.000922>] iSCSI Login negotiation failed.
[<    0.001130>] Unexpected ret: -32 send data 360
[<    0.001082>] tx_data returned -32, expecting 360.
[<    0.001080>] iSCSI Login negotiation failed.
[<    0.001220>] Unexpected ret: -32 send data 360
[<    0.000983>] tx_data returned -32, expecting 360.
[<    0.000953>] iSCSI Login negotiation failed.
[<    0.001181>] Unexpected ret: -32 send data 360
[<    0.001003>] tx_data returned -32, expecting 360.
[<    0.000967>] iSCSI Login negotiation failed.
[<    0.001249>] Unexpected ret: -32 send data 360
[<    0.000961>] tx_data returned -32, expecting 360.
[<    0.000959>] iSCSI Login negotiation failed.
[<    0.001185>] Unexpected ret: -32 send data 360
[<    0.000952>] tx_data returned -32, expecting 360.
[<    0.000994>] iSCSI Login negotiation failed.
[<    0.001331>] Unexpected ret: -32 send data 360
[<    0.000931>] tx_data returned -32, expecting 360.
[<    0.000919>] iSCSI Login negotiation failed.
[<    0.001125>] Unexpected ret: -32 send data 360
[<    0.000713>] tx_data returned -32, expecting 360.
[<    0.000575>] iSCSI Login negotiation failed.
[<    0.000778>] Unexpected ret: -32 send data 360
[<    0.000944>] tx_data returned -32, expecting 360.
[<    0.000880>] iSCSI Login negotiation failed.
[<    0.000970>] Unexpected ret: -32 send data 360
[<    0.000744>] tx_data returned -32, expecting 360.
[<    0.000718>] iSCSI Login negotiation failed.
[<    0.000908>] Unexpected ret: -32 send data 360
[<    0.000562>] tx_data returned -32, expecting 360.
[<    0.000561>] iSCSI Login negotiation failed.
[<    0.000736>] Unexpected ret: -32 send data 360
[<    0.000733>] tx_data returned -32, expecting 360.
[<    0.000755>] iSCSI Login negotiation failed.
[<    0.000883>] Unexpected ret: -32 send data 360
[<    0.000588>] tx_data returned -32, expecting 360.
[<    0.000571>] iSCSI Login negotiation failed.
[<    0.000696>] Unexpected ret: -32 send data 360
[<    0.000586>] tx_data returned -32, expecting 360.
[<    0.000542>] iSCSI Login negotiation failed.
[<    0.000712>] Unexpected ret: -32 send data 360
[<    0.000554>] tx_data returned -32, expecting 360.
[<    0.000512>] iSCSI Login negotiation failed.
[<    0.000678>] Unexpected ret: -32 send data 360
[<    0.000753>] tx_data returned -32, expecting 360.
[<    0.000746>] iSCSI Login negotiation failed.
[<    0.000874>] Unexpected ret: -32 send data 360
[<    0.000699>] tx_data returned -32, expecting 360.
[<    0.000711>] iSCSI Login negotiation failed.
[<    0.000947>] Unexpected ret: -32 send data 360
[<    0.000862>] tx_data returned -32, expecting 360.
[<    0.000855>] iSCSI Login negotiation failed.
[<    0.000954>] Unexpected ret: -32 send data 360
[<    0.000558>] tx_data returned -32, expecting 360.
[<    0.000564>] iSCSI Login negotiation failed.
[<    0.001085>] Unexpected ret: -32 send data 360
[<    0.000648>] tx_data returned -32, expecting 360.
[<    0.000653>] iSCSI Login negotiation failed.
[<    0.000762>] Unexpected ret: -32 send data 360
[<    0.000674>] tx_data returned -32, expecting 360.
[<    0.000666>] iSCSI Login negotiation failed.
[<    0.000808>] Unexpected ret: -32 send data 360
[<    0.000538>] tx_data returned -32, expecting 360.
[<    0.000573>] iSCSI Login negotiation failed.
[<    0.000637>] Unexpected ret: -32 send data 360
[<    0.000521>] tx_data returned -32, expecting 360.
[<    0.000549>] iSCSI Login negotiation failed.
[<    0.000636>] Unexpected ret: -32 send data 360
[<    0.000658>] tx_data returned -32, expecting 360.
[<    0.000671>] iSCSI Login negotiation failed.
[<    0.000828>] Unexpected ret: -32 send data 360
[<    0.000620>] tx_data returned -32, expecting 360.
[<    0.000665>] iSCSI Login negotiation failed.
[<    0.000754>] Unexpected ret: -32 send data 360
[<    0.000633>] tx_data returned -32, expecting 360.
[<    0.000678>] iSCSI Login negotiation failed.
[<    0.000737>] Unexpected ret: -32 send data 360
[<    0.000615>] tx_data returned -32, expecting 360.
[<    0.000692>] iSCSI Login negotiation failed.
[<    0.000757>] Unexpected ret: -32 send data 360
[<    0.000610>] tx_data returned -32, expecting 360.
[<    0.000657>] iSCSI Login negotiation failed.
[<    0.000746>] Unexpected ret: -32 send data 360
[<    0.000656>] tx_data returned -32, expecting 360.
[<    0.000619>] iSCSI Login negotiation failed.
[<    0.000706>] Unexpected ret: -32 send data 360
[<    0.000617>] tx_data returned -32, expecting 360.
[<    0.000592>] iSCSI Login negotiation failed.
[<    0.000705>] Unexpected ret: -32 send data 360
[<    0.000583>] tx_data returned -32, expecting 360.
[<    0.000600>] iSCSI Login negotiation failed.
[<    0.000723>] Unexpected ret: -32 send data 360
[<    0.000642>] tx_data returned -32, expecting 360.
[<    0.000613>] iSCSI Login negotiation failed.
[<   57.346452>] Detected MISCOMPARE for addr: ffff880fbf1f6000 buf: ffff8808ca09c200
[<    0.000006>] Target/iblock: Send MISCOMPARE check condition and sense
[<   14.003937>] Detected MISCOMPARE for addr: ffff880dfd7ee000 buf: ffff8808ca09cc00
[<    0.000009>] Target/iblock: Send MISCOMPARE check condition and sense
[<    0.000032>] Detected MISCOMPARE for addr: ffff880faa2fd000 buf: ffff8808ca09e600
[<    0.000003>] Target/iblock: Send MISCOMPARE check condition and sense
[<    0.000138>] Detected MISCOMPARE for addr: ffff880ced7b3000 buf: ffff8808ca09dc00
[<    0.000003>] Target/iblock: Send MISCOMPARE check condition and sense
[<    0.011295>] Detected MISCOMPARE for addr: ffff880b79e22000 buf: ffff880ec9cf4c00
[<    0.000005>] Target/iblock: Send MISCOMPARE check condition and sense
[<    0.003822>] Detected MISCOMPARE for addr: ffff880c6520a000 buf: ffff880f5ddc7200
[<    0.000004>] Target/iblock: Send MISCOMPARE check condition and sense
[<  250.175975>] Detected MISCOMPARE for addr: ffff880d827ae000 buf: ffff880e8fb9d400
[<    0.000006>] Target/iblock: Send MISCOMPARE check condition and sense
[<    0.000023>] Detected MISCOMPARE for addr: ffff880af1f30000 buf: ffff880e8fb9d400
[<    0.000002>] Target/iblock: Send MISCOMPARE check condition and sense
[<   97.963759>] Detected MISCOMPARE for addr: ffff880e2862c000 buf: ffff880edef75e00
[<    0.000018>] Target/iblock: Send MISCOMPARE check condition and sense
[<    0.000058>] Detected MISCOMPARE for addr: ffff880e9e616000 buf: ffff880edef75e00
[<    0.000002>] Target/iblock: Send MISCOMPARE check condition and sense
[<  114.546646>] Detected MISCOMPARE for addr: ffff880f860ea000 buf: ffff880edef75600
[<    0.000006>] Target/iblock: Send MISCOMPARE check condition and sense
[<    0.000017>] Detected MISCOMPARE for addr: ffff880cd8ff6000 buf: ffff880edef75600
[<    0.000003>] Target/iblock: Send MISCOMPARE check condition and sense
[<  328.102750>] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[<   13.584736>] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[<  216.688079>] Exiting Time2Retain handler because session_reinstatement=1
[<  298.662261>] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000002
[<    0.001314>] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000003
[<   27.133730>] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000004
[<    0.001217>] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000005
[<   27.146405>] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000006
[<    0.001295>] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000007
[<    0.001189>] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000008


Laatste stuk logging op ESXi host:
code:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
2016-06-10T14:09:16.960Z cpu3:33234)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.0.2.50:50453 R: 10.0.2.1:3260]
2016-06-10T14:09:16.960Z cpu1:33245)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6001405acd066c278cc4d64a27245fd6" state in doubt; requested fast path state update...
2016-06-10T14:09:16.960Z cpu1:33245)ScsiDeviceIO: 2613: Cmd(0x439d806d1540) 0x89, CmdSN 0x256f7 from world 32801 to dev "naa.6001405acd066c278cc4d64a27245fd6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
2016-06-10T14:09:16.960Z cpu3:45608)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x89 (0x439d806a92c0, 32778) to dev "naa.60014059765db72798947069da81e525" on path "vmhba38:C1:T0:L1" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-06-10T14:09:16.960Z cpu3:45608)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014059765db72798947069da81e525" state in doubt; requested fast path state update...
2016-06-10T14:09:16.960Z cpu3:45608)ScsiDeviceIO: 2651: Cmd(0x439d806a92c0) 0x89, CmdSN 0x244c2 from world 32778 to dev "naa.60014059765db72798947069da81e525" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-06-10T14:09:16.960Z cpu1:33245)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x28 (0x439d806b2c00, 34678) to dev "naa.60014059765db72798947069da81e525" on path "vmhba38:C1:T0:L1" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-06-10T14:09:16.960Z cpu1:33245)ScsiDeviceIO: 2613: Cmd(0x439d806b2c00) 0x28, CmdSN 0x244c1 from world 34678 to dev "naa.60014059765db72798947069da81e525" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-06-10T14:09:16.960Z cpu1:33245)ScsiDeviceIO: 2613: Cmd(0x439d806e4900) 0x28, CmdSN 0x8e from world 45781 to dev "naa.60014059765db72798947069da81e525" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-06-10T14:09:16.960Z cpu3:33234)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba38:CH:1 T:0 L:1 : Task mgmt "Abort Task" with itt=0x940 (refITT=0x93f) timed out.
2016-06-10T14:09:16.986Z cpu1:36957)NMP: nmp_ThrottleLogForDevice:3231: last error status from device naa.60014059765db72798947069da81e525 repeated 10 times
2016-06-10T14:09:17.032Z cpu0:33295)NMP: nmp_ThrottleLogForDevice:3231: last error status from device naa.60014059765db72798947069da81e525 repeated 20 times
2016-06-10T14:09:17.125Z cpu1:36957)NMP: nmp_ThrottleLogForDevice:3231: last error status from device naa.60014059765db72798947069da81e525 repeated 40 times
2016-06-10T14:09:17.305Z cpu0:32851)NMP: nmp_ThrottleLogForDevice:3231: last error status from device naa.60014059765db72798947069da81e525 repeated 80 times
2016-06-10T14:09:17.355Z cpu1:45226)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6001405acd066c278cc4d64a27245fd6" state in doubt; requested fast path state update...
2016-06-10T14:09:17.355Z cpu1:45226)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014059765db72798947069da81e525" state in doubt; requested fast path state update...
2016-06-10T14:09:17.667Z cpu1:36956)NMP: nmp_ThrottleLogForDevice:3231: last error status from device naa.60014059765db72798947069da81e525 repeated 160 times
2016-06-10T14:09:18.351Z cpu1:32782)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014059765db72798947069da81e525" state in doubt; requested fast path state update...
2016-06-10T14:09:18.365Z cpu1:32892)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6001405acd066c278cc4d64a27245fd6" state in doubt; requested fast path state update...
2016-06-10T14:09:18.413Z cpu0:32787)ScsiDeviceIO: 2651: Cmd(0x439d806b1700) 0x89, CmdSN 0x244c3 from world 32778 to dev "naa.60014059765db72798947069da81e525" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-06-10T14:09:18.423Z cpu1:32788)ScsiDeviceIO: 2595: Cmd(0x439d806b2c00) 0x28, CmdSN 0x244c1 from world 34678 to dev "naa.60014059765db72798947069da81e525" failed H:0x8 D:0x0 P:0x0
2016-06-10T14:09:18.423Z cpu1:32788)ScsiDeviceIO: 2595: Cmd(0x439d806e4900) 0x28, CmdSN 0x8e from world 45781 to dev "naa.60014059765db72798947069da81e525" failed H:0x8 D:0x0 P:0x0
2016-06-10T14:09:18.423Z cpu1:32788)ScsiDeviceIO: 2595: Cmd(0x439d80699100) 0x28, CmdSN 0x8d from world 45779 to dev "naa.60014059765db72798947069da81e525" failed H:0x8 D:0x0 P:0x0
2016-06-10T14:09:18.423Z cpu1:32788)ScsiDeviceIO: 2595: Cmd(0x439d806c9300) 0x28, CmdSN 0x84 from world 45780 to dev "naa.60014059765db72798947069da81e525" failed H:0x8 D:0x0 P:0x0
2016-06-10T14:09:18.423Z cpu1:32788)ScsiDeviceIO: 2595: Cmd(0x439d806c1e80) 0x28, CmdSN 0x8b from world 32876 to dev "naa.60014059765db72798947069da81e525" failed H:0x8 D:0x0 P:0x0
2016-06-10T14:09:18.515Z cpu3:32803)NMP: nmp_ThrottleLogForDevice:3231: last error status from device naa.60014059765db72798947069da81e525 repeated 320 times
2016-06-10T14:09:18.515Z cpu3:32803)ScsiDeviceIO: 2613: Cmd(0x439d806c8b80) 0x28, CmdSN 0x244c4 from world 32803 to dev "naa.60014059765db72798947069da81e525" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-06-10T14:09:19.356Z cpu3:32782)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014059765db72798947069da81e525" state in doubt; requested fast path state update...
2016-06-10T14:09:19.356Z cpu1:33013)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6001405acd066c278cc4d64a27245fd6" state in doubt; requested fast path state update...
2016-06-10T14:09:19.414Z cpu1:32788)ScsiDeviceIO: 2651: Cmd(0x439d806d1540) 0x89, CmdSN 0x256f7 from world 32801 to dev "naa.6001405acd066c278cc4d64a27245fd6" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x20 0x0.
2016-06-10T14:09:19.415Z cpu1:34689 opID=1450f3db)HBX: 2802: 'VM_TIER_1': HB at offset 3215360 - Waiting for timed out HB:
2016-06-10T14:09:19.415Z cpu1:34689 opID=1450f3db)  [HB state abcdef02 offset 3215360 gen 1013 stampUS 11709584366 uuid 575a93cb-b4a2bff7-ffba-f4ce46a74f78 jrnl <FB 17800> drv 14.61 lockImpl 4]
2016-06-10T14:09:19.415Z cpu3:32802)ScsiDeviceIO: 2613: Cmd(0x439d806a95c0) 0x28, CmdSN 0x256f8 from world 32802 to dev "naa.6001405acd066c278cc4d64a27245fd6" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-06-10T14:09:19.728Z cpu0:33234)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x430403ecf880 network resource pool netsched.pools.persist.iscsi associated
2016-06-10T14:09:19.728Z cpu0:33234)iscsi_vmk: iscsivmk_ConnNetRegister: socket 0x430403ecf880 network tracker id 16768 tracker.iSCSI.10.0.2.1 associated
2016-06-10T14:09:19.980Z cpu0:33234)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba38:CH:1 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2016-06-10T14:09:19.980Z cpu0:33234)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn.2003-01.org.linux-iscsi.nas.x8664:sn.80c34394d007 TPGT: 1 TSIH: 0]
2016-06-10T14:09:19.980Z cpu0:33234)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: 10.0.2.50:16421 R: 10.0.2.1:3260]
2016-06-10T14:09:19.990Z cpu3:32803)HBX: 276: 'VM_TIER_2': HB at offset 3215360 - Reclaimed heartbeat [Timeout]:
2016-06-10T14:09:19.990Z cpu3:32803)  [HB state abcdef02 offset 3215360 gen 1051 stampUS 13918638678 uuid 575a93cb-b4a2bff7-ffba-f4ce46a74f78 jrnl <FB 7400> drv 14.61 lockImpl 4]
2016-06-10T14:09:21.351Z cpu3:33139)NMP: nmp_ResetDeviceLogThrottling:3349: last error status from device naa.6001405acd066c278cc4d64a27245fd6 repeated 940 times


Ik zou niet weten waar het in zou kunnen zitten, kan iemand mij op weg helpen, met waar ik het in moet zoeken?

EDIT:
Als ik de sense data (0x05, 0x20, 0x00) op zoek (http://www.t10.org/lists/asc-num.txt) krijg ik dit resultaat:
20h/00h  DZTPROMAEBKVF  INVALID COMMAND OPERATION CODE


Iemand die dit kan bevestigen? Debug ik dit op de goede manier? Want dit zegt mij vrij weinig dan dat ESXi iets probeert wat TargetCLI niet ondersteund.

Even niets...


Acties:
  • 0 Henk 'm!

  • Oid
  • Registratie: November 2002
  • Niet online

Oid

misschien lees ik er over heen; maar welke OS draait de iSCSI host?
Als je een tweede host koppelt disconnect die dan ook?

Acties:
  • 0 Henk 'm!

  • FireDrunk
  • Registratie: November 2002
  • Laatst online: 30-09 12:37
Host is Fedora 23, tweede guest kan ik ook eens doen, dat is een goed idee.

Even niets...


Acties:
  • 0 Henk 'm!

  • CAPSLOCK2000
  • Registratie: Februari 2003
  • Laatst online: 12:22

CAPSLOCK2000

zie teletekst pagina 888

Misschien ten overvloede, maar als je het nog niet gedaan hebt, lees dan de VMWare documentatie, dat geeft een beetje inzicht in wat die logs betekenen.

https://kb.vmware.com/sel...playKC&externalId=1029039
https://kb.vmware.com/sel...splayKC&externalId=289902

In mijn ervaring zijn iSCSI-problemen meestal problemen met het onderliggende netwerk. iSCSI stelt hogere eisen dan de meeste andere protocollen en heeft het eerste last als er iets mis gaat. Ik zou dus goed opletten of ik tekenen zie van netwerkproblemen. Bij verzadigde links kunnen kleine verstoringen al grote gevolgen hebben.

This post is warranted for the full amount you paid me for it.


Acties:
  • 0 Henk 'm!

  • FireDrunk
  • Registratie: November 2002
  • Laatst online: 30-09 12:37
Ondertussen eens getest met de vier andere hosts (die een tijd uit stonden, wegens lage utilization :) )
Nu dus 4 hosts er bij, welke allemaal met dual gigabit en dezelfde multipathing configuratie aangesloten zitten.

4 VM"s tegelijk het commando:
dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync

uit laten voeren, en alles werkt keurig. De iSCSI host verstookt rustig 35k sync IOPS bij de eerste run, en bij de tweede run al een stuk minder (dedup & compressie staan aan).

Ik begin te vermoeden dat het aan die ene 5e host ligt, maar wat vreemd is, is dat nadat alles gecrasht is, en ik de VH05 herstart, niet alles goed opkomt, tot ik de iSCSI host een trap verkoop. Alsof er iets in de Target kernel module mis zit. (maar 1 van de 2 lun's is te mounten).

Na een herstart van zowel de ESXi host als de NAS werkt alles weer naar behoren.

Ga nu eens benchen met alles hosts tegelijk. Geen verschil, alles blijft gewoon werken.
Ik heb 1 miscompare gezien:

Hele log tijdens de benchmarks:
code:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
[Jun12 11:54] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002150] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 12:11] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[  +0.006632] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[Jun12 12:13] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.003469] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 12:43] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002813] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 13:13] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002650] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 13:43] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002220] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 14:13] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002715] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 14:43] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002774] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 15:13] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002740] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 15:43] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002897] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 16:08] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[  +0.010220] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[  +2.208165] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[  +0.007377] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[  +0.523336] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[  +0.016124] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[  +0.856159] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[  +0.007237] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[Jun12 16:10] iSCSI/iqn.1998-01.com.vmware:vh01-38673156: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.003927] iSCSI/iqn.1998-01.com.vmware:vh01-38673156: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +1.706604] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.003176] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.571767] iSCSI/iqn.1998-01.com.vmware:vh03-7d421107: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.064704] iSCSI/iqn.1998-01.com.vmware:vh03-7d421107: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +1.012375] iSCSI/iqn.1998-01.com.vmware:vh02-33d4caa1: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.058412] iSCSI/iqn.1998-01.com.vmware:vh02-33d4caa1: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 16:13] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002447] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 16:35] Detected MISCOMPARE for addr: ffff880d3ca66000 buf: ffff880d8b8ae000
[  +0.000003] Target/iblock: Send MISCOMPARE check condition and sense
[Jun12 16:40] iSCSI/iqn.1998-01.com.vmware:vh01-38673156: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002440] iSCSI/iqn.1998-01.com.vmware:vh01-38673156: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +1.843464] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002511] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.508433] iSCSI/iqn.1998-01.com.vmware:vh03-7d421107: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.011870] iSCSI/iqn.1998-01.com.vmware:vh03-7d421107: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +1.156248] iSCSI/iqn.1998-01.com.vmware:vh02-33d4caa1: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002230] iSCSI/iqn.1998-01.com.vmware:vh02-33d4caa1: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 16:43] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002690] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 17:10] iSCSI/iqn.1998-01.com.vmware:vh01-38673156: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002551] iSCSI/iqn.1998-01.com.vmware:vh01-38673156: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +1.868107] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002250] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.503394] iSCSI/iqn.1998-01.com.vmware:vh03-7d421107: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.011830] iSCSI/iqn.1998-01.com.vmware:vh03-7d421107: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +1.180650] iSCSI/iqn.1998-01.com.vmware:vh02-33d4caa1: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002347] iSCSI/iqn.1998-01.com.vmware:vh02-33d4caa1: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[Jun12 17:12] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[  +0.002826] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.


0x85 is te negeren, dat is het ATA PASSTHROUGH commando wat ESXi probeert te doen, maar dat is geen blocking issue.

[ Voor 78% gewijzigd door FireDrunk op 12-06-2016 17:20 ]

Even niets...


Acties:
  • 0 Henk 'm!

  • CAPSLOCK2000
  • Registratie: Februari 2003
  • Laatst online: 12:22

CAPSLOCK2000

zie teletekst pagina 888

Doe je iSCSI helemaal in software (daar lijkt het op) of heb je nog iets van hardware-acceleratie?
Draait alles op dezelfde firmware-versie?

This post is warranted for the full amount you paid me for it.


Acties:
  • 0 Henk 'm!

  • FireDrunk
  • Registratie: November 2002
  • Laatst online: 30-09 12:37
Alles draait op software (Software iSCSI Adapter in ESXi, en Linux Target Kernel module).

Niet dat ik denk dat dat uitmaakt, alle grote SAN/NASsen draaien ook gewoon software :)

Even niets...


Acties:
  • 0 Henk 'm!

  • CAPSLOCK2000
  • Registratie: Februari 2003
  • Laatst online: 12:22

CAPSLOCK2000

zie teletekst pagina 888

Ik probeerde niet te zeggen dat hardware beter is, doe mij ook maar software, al is het maar omdat je dan geen firmwareversies hoeft bij te houden ;)

This post is warranted for the full amount you paid me for it.


Acties:
  • 0 Henk 'm!

  • FireDrunk
  • Registratie: November 2002
  • Laatst online: 30-09 12:37
Dat is zeker waar :)

Even niets...


Acties:
  • 0 Henk 'm!

  • FireDrunk
  • Registratie: November 2002
  • Laatst online: 30-09 12:37
Zojuist na lange tijd weer eens een flinke crash. Ik was een nieuwe VM aan het uitrollen dmv salt-cloud.
Het ging dus om het klonen van een template denk ik.

Excerpt van de logging:

code:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
[641126.250580] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[641126.309406] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[641127.581723] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[641128.799561] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[641129.077172] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[641129.252925] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[641129.315783] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[641130.634042] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[641569.205059] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[641569.207460] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[642237.433318] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[642237.435764] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[643369.055021] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[643369.057559] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[644037.297890] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[644037.300381] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[645168.905684] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[645168.909038] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[645664.467719] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[645664.475155] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[645743.117417] iSCSI/iqn.1998-01.com.vmware:vh03-7d421107: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[645743.454730] iSCSI/iqn.1998-01.com.vmware:vh03-7d421107: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[645837.162331] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[645837.164743] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[646968.755167] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[646968.757782] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[647637.027768] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[647637.030280] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[648768.601658] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[648768.604903] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[649436.892807] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[649436.895202] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[650568.451309] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[650568.454001] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[651236.757817] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[651236.760309] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[652368.299454] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[652368.301754] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[653036.621895] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[653036.624461] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[654168.147883] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[654168.150556] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[654836.485784] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[654836.488203] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[655967.996121] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[655967.998731] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[656636.349993] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[656636.352272] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[657767.843735] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[657767.846441] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[658436.214618] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[658436.216915] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[659567.692579] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[659567.695229] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[660236.078033] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[660236.080374] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[661367.539941] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[661367.542626] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[662035.943373] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[662035.945974] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[663167.388698] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[663167.391503] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[663835.806966] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[663835.809460] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[664967.236109] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[664967.238665] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[665635.675034] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[665635.677499] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[666767.085134] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[666767.087714] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[667435.544775] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[667435.547243] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[668566.934008] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[668566.936692] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[669235.412742] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[669235.415258] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[670366.783066] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[670366.785813] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[671035.280284] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[671035.282805] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[672166.632238] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[672166.634848] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[672835.149621] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[672835.152005] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[673966.480363] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[673966.483044] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[674635.017737] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[674635.020267] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[675766.330073] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[675766.333652] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[676434.887084] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[676434.889540] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[677566.181654] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[677566.185211] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[678234.755893] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[678234.758438] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[679366.040098] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[679366.042580] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[680034.625694] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[680034.628346] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[681165.887473] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[681165.890189] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[681834.495276] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[681834.497773] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[682965.734718] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[682965.737190] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[683634.364115] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[683634.366607] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[684765.582930] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[684765.585634] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[685434.233656] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[685434.236126] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[686565.430793] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[686565.433134] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[687234.101843] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[687234.104395] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[688365.280641] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[688365.283472] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[689033.969918] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[689033.972569] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[690165.131013] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[690165.133836] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[690833.838149] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[690833.840629] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[691241.533680] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[691241.540911] MODE SENSE: unimplemented page/subpage: 0x1c/0x02
[691320.166363] iSCSI/iqn.1998-01.com.vmware:vh03-7d421107: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[691320.260535] iSCSI/iqn.1998-01.com.vmware:vh03-7d421107: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[691964.978533] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[691964.980889] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[692633.708373] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[692633.710706] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[693764.865251] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[693764.868085] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[694433.576588] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[694433.579054] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[695564.715990] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[695564.718700] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[696233.444738] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[696233.447187] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[697364.565473] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[697364.568243] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[698033.314141] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[698033.316511] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[699164.415168] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[699164.420605] iSCSI/iqn.1998-01.com.vmware:vh05-5e200351: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[699833.183748] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[699833.186275] iSCSI/iqn.1998-01.com.vmware:vh04-77a89664: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
[700161.329821] iSCSI Login timeout on Network Portal 10.0.4.1:3260
[700166.797871] ABORT_TASK: Found referenced iSCSI task_tag: 7010
[700166.797877] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 7010
[700170.449201] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[700170.449291] iSCSI Login negotiation failed.
[700187.983994] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[700187.984126] iSCSI Login negotiation failed.
[700205.518805] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[700205.518924] iSCSI Login negotiation failed.
[700219.597844] iSCSI Login timeout on Network Portal 10.0.1.1:3260
[700219.597945] Unexpected ret: -32 send data 48
[700219.597980] tx_data returned -32, expecting 48.
[700219.598010] iSCSI Login negotiation failed.
[700223.053621] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[700223.053715] iSCSI Login negotiation failed.
[700234.636808] iSCSI Login timeout on Network Portal 10.0.1.1:3260
[700234.636912] Unexpected ret: -32 send data 48
[700234.636948] tx_data returned -32, expecting 48.
[700234.636979] iSCSI Login negotiation failed.
[700240.716386] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[700240.716503] iSCSI Login negotiation failed.
[700249.675786] iSCSI Login timeout on Network Portal 10.0.1.1:3260
[700249.676632] Unexpected ret: -32 send data 48
[700249.677436] tx_data returned -32, expecting 48.
[700249.678213] iSCSI Login negotiation failed.
[700258.251214] iSCSI Login timeout on Network Portal 10.0.3.1:3260
[700258.252058] iSCSI Login negotiation failed.
[700263.196673] Unexpected ret: -32 send data 360
[700263.197400] tx_data returned -32, expecting 360.
[700263.198231] iSCSI Login negotiation failed.
[700263.199297] Unexpected ret: -32 send data 360
[700263.200120] tx_data returned -32, expecting 360.
[700263.200972] iSCSI Login negotiation failed.
[700263.202082] Unexpected ret: -32 send data 360
[700263.202953] tx_data returned -32, expecting 360.
[700263.203898] iSCSI Login negotiation failed.
[700263.205399] Unexpected ret: -32 send data 360
[700263.206220] tx_data returned -32, expecting 360.
[700263.207068] iSCSI Login negotiation failed.
[700263.208178] Unexpected ret: -32 send data 360
[700263.209049] tx_data returned -32, expecting 360.
[700263.209965] iSCSI Login negotiation failed.
[700263.211471] Unexpected ret: -32 send data 360
[700263.212383] tx_data returned -32, expecting 360.
[700263.213353] iSCSI Login negotiation failed.
[700263.214551] Unexpected ret: -32 send data 360
[700263.215497] tx_data returned -32, expecting 360.
[700263.216470] iSCSI Login negotiation failed.
[700263.217812] Unexpected ret: -32 send data 360
[700263.218952] tx_data returned -32, expecting 360.
[700263.220080] iSCSI Login negotiation failed.
[700263.221441] Unexpected ret: -32 send data 360
[700263.222435] tx_data returned -32, expecting 360.
[700263.223371] iSCSI Login negotiation failed.
[700263.224574] Unexpected ret: -32 send data 360
[700263.225627] tx_data returned -32, expecting 360.
[700263.226643] iSCSI Login negotiation failed.
[700263.228331] Unexpected ret: -32 send data 360
[700263.229452] tx_data returned -32, expecting 360.
[700263.230658] iSCSI Login negotiation failed.
[700263.231907] Unexpected ret: -32 send data 360
[700263.232766] tx_data returned -32, expecting 360.
[700263.233742] iSCSI Login negotiation failed.
[700263.234922] Unexpected ret: -32 send data 360
[700263.235949] tx_data returned -32, expecting 360.
[700263.236940] iSCSI Login negotiation failed.
[700263.238484] Unexpected ret: -32 send data 360
[700263.239488] tx_data returned -32, expecting 360.
[700263.240446] iSCSI Login negotiation failed.
[700263.241581] Unexpected ret: -32 send data 360
[700263.242474] tx_data returned -32, expecting 360.
[700263.243402] iSCSI Login negotiation failed.
[700263.244506] Unexpected ret: -32 send data 360
[700263.245374] tx_data returned -32, expecting 360.
[700263.246240] iSCSI Login negotiation failed.
[700263.247375] Unexpected ret: -32 send data 360
[700263.248190] tx_data returned -32, expecting 360.
[700263.249035] iSCSI Login negotiation failed.
[700263.250220] Unexpected ret: -32 send data 360
[700263.251048] tx_data returned -32, expecting 360.
[700263.251867] iSCSI Login negotiation failed.
[700263.253140] Unexpected ret: -32 send data 360
[700263.253994] tx_data returned -32, expecting 360.
[700263.254840] iSCSI Login negotiation failed.
[700263.256027] Unexpected ret: -32 send data 360
[700263.256776] tx_data returned -32, expecting 360.
[700263.257498] iSCSI Login negotiation failed.


Ik snap er echt niks van :'(

[ Voor 100% gewijzigd door FireDrunk op 12-07-2016 12:29 ]

Even niets...


Acties:
  • 0 Henk 'm!

  • FireDrunk
  • Registratie: November 2002
  • Laatst online: 30-09 12:37
*subtiel schopje*

Even met wat kennissen overlegd, en misschien wordt het tijd om een test te doen met FreeNAS op dezelfde (SSD) pool. Op zich sluit je dan het hele netwerk en ZFS gedeelte uit, en test je vooral een ander iSCSI stack.

Even niets...


Acties:
  • 0 Henk 'm!

  • RvV
  • Registratie: Juli 2000
  • Laatst online: 07:50

RvV

Ben je hier nog verder mee gekomen?

Ik heb een Synology iSCSI target, 10Gbe >> 10Gbe switch >> Windows 2012R2 server met 10Gbe NIC en heb op de meest vreemde momenten last van het wegvallen van de iSCSI schijf binnen Windows.
Je kunt bijvoorbeeld een uur lang met 1GB/s benchmarken zonder problemen, en 's-nachts zonder load knalt hij er ineens een minuut uit.


Windows 2012 R2 server,
system log:
code:
1
2
3
4
5
6
12:41:17: Initiator sent a task management command to reset the target. The target name is given in the dump data.
12:41:37: Target failed to respond in time to a Task Management request.
12:41:37: Target did not respond in time for a SCSI request. The CDB is given in the dump data.
12:41:37: Can not Reset the Target or LUN. Will attempt session recovery.
12:41:40: Target returned an invalid login response packet. The login response packet is given in the dump data.
12:42:09: A connection to the target was lost, but Initiator successfully reconnected to the target. Dump data contains the target name.


Synology iscsi target log
code:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
/var/log/messages:
Aug 16 12:41:47 SYNOCUC1 kernel: [352330.247603] Unexpected ret: -32 send data 48


/var/log/iscsi.log
Aug 16 12:41:40 SYNOCUC1 kernel: [352322.956810] iSCSI: The number(1) of current sessions will exceed the maximum number(1).
Aug 16 12:41:40 SYNOCUC1 kernel: [352322.964891] iSCSI: iSCSI Login negotiation failed.
Aug 16 12:41:48 SYNOCUC1 kernel: [352330.894594] iSCSI: The number(1) of current sessions will exceed the maximum number(1).
Aug 16 12:41:48 SYNOCUC1 kernel: [352330.902675] iSCSI: iSCSI Login negotiation failed.
Aug 16 12:41:56 SYNOCUC1 kernel: [352338.893385] iSCSI: The number(1) of current sessions will exceed the maximum number(1).
Aug 16 12:41:56 SYNOCUC1 kernel: [352338.901468] iSCSI: iSCSI Login negotiation failed.
Aug 16 12:42:04 SYNOCUC1 kernel: [352346.892222] iSCSI: The number(1) of current sessions will exceed the maximum number(1).
Aug 16 12:42:04 SYNOCUC1 kernel: [352346.900303] iSCSI: iSCSI Login negotiation failed.
Aug 16 12:42:08 SYNOCUC1 kernel: [352350.833999] iSCSI: Close - I[iqn.1991-05.com.microsoft:backup2.loc][1.1.2.1:58195], T[iqn.2000-01.com.synology:SYNOCUC1.Target-1.3eb93bd3a0]
Aug 16 12:42:09 SYNOCUC1 kernel: [352351.867347] iSCSI: Login - I[iqn.1991-05.com.microsoft:backup2.loc][1.1.2.1:63469], T[iqn.2000-01.com.synology:SYNOCUC1.Target-1.3eb93bd3a0][1.1.2.100:3260], P[iSCSI/TCP]


Op een andere locatie heb ik zo'n zelfde setup draaien zonder problemen. Het enige verschil is dat er daar geen 10Gbe switch tussen zit en de Windows server rechtstreeks zit aangesloten op het Synology iSCSI target.
Binnenkort die switch maar eens een reboot geven...

Y'24


Acties:
  • 0 Henk 'm!

  • FireDrunk
  • Registratie: November 2002
  • Laatst online: 30-09 12:37
Ik heb geen definitieve oorzaak gevonden, maar ik werd recentelijk getriggerd dat er een bug zit in de Intel 1000/PT driver code onder ESXi 6.0U2. Bij hoge load kan de netwerkkaart driver crashen.

Dat verklaart waarom ik geen specifieke foutmeldingen kreeg op de iSCSI host.

Even niets...

Pagina: 1