Vastloper Debian 8

[DPLuS]

Vannacht hing mijn virtuele machine.
Het is een Debian 8 machine op een Vmware ESXi v6 server.

Nu zag ik dit in mijn syslog, maar ik begrijp niet wat er nu gebeurd is.
Geen memory meer vrij?
Of hing de kernel gewoon?

Oct 6 02:13:59 xenon postfix/cleanup[28068]: EDAA5CC1E03: message-id=<[e-mail]>
Oct 6 02:14:21 xenon postfix/master[1290]: warning: unix_trigger_event: read timeout for service public/pickup
Oct 6 02:14:25 xenon kernel: [1168911.244615] apache2 invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
Oct 6 02:14:25 xenon kernel: [1168911.244621] apache2 cpuset=/ mems_allowed=0
Oct 6 02:14:25 xenon kernel: [1168911.244631] CPU: 3 PID: 26285 Comm: apache2 Not tainted 3.16.0-4-amd64 #1 Debian 3.16.43-2+deb8u5
Oct 6 02:14:25 xenon kernel: [1168911.244633] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
Oct 6 02:14:25 xenon kernel: [1168911.244635] 0000000000000000 ffffffff81514291 ffff88000ac80390 0000000000000000
Oct 6 02:14:25 xenon kernel: [1168911.244638] ffffffff81511e69 0000000000000000 ffffffff810d6f6f 0000000000000001
Oct 6 02:14:25 xenon kernel: [1168911.244640] ffffffff81518c4e 0000000000000200 ffffffff81068a53 ffffffff810c44e4
Oct 6 02:14:25 xenon kernel: [1168911.244643] Call Trace:
Oct 6 02:14:25 xenon kernel: [1168911.244661] [<ffffffff81514291>] ? dump_stack+0x5d/0x78
Oct 6 02:14:25 xenon kernel: [1168911.244665] [<ffffffff81511e69>] ? dump_header+0x76/0x1e8
Oct 6 02:14:25 xenon kernel: [1168911.244670] [<ffffffff810d6f6f>] ? smp_call_function_single+0x5f/0xa0
Oct 6 02:14:25 xenon kernel: [1168911.244674] [<ffffffff81518c4e>] ? mutex_lock+0xe/0x2a
Oct 6 02:14:25 xenon kernel: [1168911.244679] [<ffffffff81068a53>] ? put_online_cpus+0x23/0x80
Oct 6 02:14:25 xenon kernel: [1168911.244683] [<ffffffff810c44e4>] ? rcu_oom_notify+0xc4/0xe0
Oct 6 02:14:25 xenon kernel: [1168911.244688] [<ffffffff8115431c>] ? do_try_to_free_pages+0x4ac/0x520
Oct 6 02:14:25 xenon kernel: [1168911.244692] [<ffffffff81142ddd>] ? oom_kill_process+0x21d/0x370
Oct 6 02:14:25 xenon kernel: [1168911.244694] [<ffffffff8114299d>] ? find_lock_task_mm+0x3d/0x90
Oct 6 02:14:25 xenon kernel: [1168911.244697] [<ffffffff81143543>] ? out_of_memory+0x473/0x4b0
Oct 6 02:14:25 xenon kernel: [1168911.244700] [<ffffffff8114940f>] ? __alloc_pages_nodemask+0x9ef/0xb50
Oct 6 02:14:25 xenon kernel: [1168911.244703] [<ffffffff8118aa68>] ? alloc_pages_vma+0x98/0x160
Oct 6 02:14:25 xenon kernel: [1168911.244707] [<ffffffff8117d26c>] ? read_swap_cache_async+0xec/0x160
Oct 6 02:14:25 xenon kernel: [1168911.244709] [<ffffffff8113f040>] ? sleep_on_page+0x10/0x10
Oct 6 02:14:25 xenon kernel: [1168911.244711] [<ffffffff8117d42b>] ? swapin_readahead+0x14b/0x1b0
Oct 6 02:14:25 xenon kernel: [1168911.244714] [<ffffffff8117dfc3>] ? swap_entry_free+0x153/0x390
Oct 6 02:14:25 xenon kernel: [1168911.244719] [<ffffffff812b38ce>] ? radix_tree_lookup_slot+0xe/0x30
Oct 6 02:14:25 xenon kernel: [1168911.244721] [<ffffffff8113f349>] ? find_get_entry+0x19/0x90
Oct 6 02:14:25 xenon kernel: [1168911.244724] [<ffffffff81140156>] ? pagecache_get_page_fixed+0x26/0x1b0
Oct 6 02:14:25 xenon kernel: [1168911.244728] [<ffffffff8116c773>] ? handle_mm_fault+0xcb3/0x1150
Oct 6 02:14:25 xenon kernel: [1168911.244731] [<ffffffff810582c7>] ? __do_page_fault+0x177/0x4f0
Oct 6 02:14:25 xenon kernel: [1168911.244735] [<ffffffff812bbdd8>] ? lockref_put_or_lock+0x48/0x80
Oct 6 02:14:25 xenon kernel: [1168911.244738] [<ffffffff811c0d6f>] ? dput+0x1f/0x170
Oct 6 02:14:25 xenon kernel: [1168911.244741] [<ffffffff811ac901>] ? __fput+0x161/0x1d0
Oct 6 02:14:25 xenon kernel: [1168911.244745] [<ffffffff810867a1>] ? task_work_run+0x91/0xb0
Oct 6 02:14:25 xenon kernel: [1168911.244749] [<ffffffff8151c4a8>] ? page_fault+0x28/0x30
Oct 6 02:14:25 xenon kernel: [1168911.244750] Mem-Info:
Oct 6 02:14:25 xenon kernel: [1168911.244752] Node 0 DMA per-cpu:
Oct 6 02:14:25 xenon kernel: [1168911.244755] CPU 0: hi: 0, btch: 1 usd: 0
Oct 6 02:14:25 xenon kernel: [1168911.244756] CPU 1: hi: 0, btch: 1 usd: 0
Oct 6 02:14:25 xenon kernel: [1168911.244757] CPU 2: hi: 0, btch: 1 usd: 0
Oct 6 02:14:25 xenon kernel: [1168911.244759] CPU 3: hi: 0, btch: 1 usd: 0
Oct 6 02:14:25 xenon kernel: [1168911.244760] Node 0 DMA32 per-cpu:
Oct 6 02:14:25 xenon kernel: [1168911.244761] CPU 0: hi: 186, btch: 31 usd: 168
Oct 6 02:14:25 xenon kernel: [1168911.244763] CPU 1: hi: 186, btch: 31 usd: 173
Oct 6 02:14:25 xenon kernel: [1168911.244764] CPU 2: hi: 186, btch: 31 usd: 167
Oct 6 02:14:25 xenon kernel: [1168911.244765] CPU 3: hi: 186, btch: 31 usd: 175
Oct 6 02:14:25 xenon kernel: [1168911.244766] Node 0 Normal per-cpu:
Oct 6 02:14:25 xenon kernel: [1168911.244767] CPU 0: hi: 186, btch: 31 usd: 39
Oct 6 02:14:25 xenon kernel: [1168911.244769] CPU 1: hi: 186, btch: 31 usd: 152
Oct 6 02:14:25 xenon kernel: [1168911.244770] CPU 2: hi: 186, btch: 31 usd: 155
Oct 6 02:14:25 xenon kernel: [1168911.244771] CPU 3: hi: 186, btch: 31 usd: 175
Oct 6 02:14:25 xenon kernel: [1168911.244775] active_anon:651981 inactive_anon:240186 isolated_anon:248
Oct 6 02:14:25 xenon kernel: [1168911.244775] active_file:2817 inactive_file:2424 isolated_file:232
Oct 6 02:14:25 xenon kernel: [1168911.244775] unevictable:0 dirty:0 writeback:171 unstable:0
Oct 6 02:14:25 xenon kernel: [1168911.244775] free:22749 slab_reclaimable:5787 slab_unreclaimable:17506
Oct 6 02:14:25 xenon kernel: [1168911.244775] mapped:4515 shmem:776 pagetables:44511 bounce:0
Oct 6 02:14:25 xenon kernel: [1168911.244775] free_cma:0
Oct 6 02:14:25 xenon kernel: [1168911.244779] Node 0 DMA free:15904kB min:264kB low:328kB high:396kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 6 02:14:25 xenon kernel: [1168911.244784] lowmem_reserve[]: 0 2980 3947 3947
Oct 6 02:14:25 xenon kernel: [1168911.244787] Node 0 DMA32 free:57720kB min:50832kB low:63540kB high:76248kB active_anon:2196196kB inactive_anon:548720kB active_file:7668kB inactive_file:6984kB unevictable:0kB isolated(anon):768kB isolated(file):580kB present:3129280kB managed:3054792kB mlocked:4kB dirty:0kB writeback:436kB mapped:11344kB shmem:2492kB slab_reclaimable:17160kB slab_unreclaimable:43140kB kernel_stack:7808kB pagetables:115076kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:75450 all_unreclaimable? yes
Oct 6 02:14:25 xenon kernel: [1168911.244792] lowmem_reserve[]: 0 0 966 966
Oct 6 02:14:25 xenon kernel: [1168911.244794] Node 0 Normal free:17372kB min:16484kB low:20604kB high:24724kB active_anon:411728kB inactive_anon:412024kB active_file:3600kB inactive_file:2712kB unevictable:0kB isolated(anon):224kB isolated(file):348kB present:1048576kB managed:990036kB mlocked:0kB dirty:0kB writeback:248kB mapped:6716kB shmem:612kB slab_reclaimable:5984kB slab_unreclaimable:26884kB kernel_stack:3760kB pagetables:62968kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:11660 all_unreclaimable? yes
Oct 6 02:14:25 xenon kernel: [1168911.244799] lowmem_reserve[]: 0 0 0 0
Oct 6 02:14:25 xenon kernel: [1168911.244801] Node 0 DMA: 2*4kB (UE) 1*8kB (E) 1*16kB (E) 2*32kB (UE) 3*64kB (UE) 2*128kB (UE) 2*256kB (UE) 1*512kB (E) 2*1024kB (UE) 2*2048kB (ER) 2*4096kB (M) = 15904kB
Oct 6 02:14:25 xenon kernel: [1168911.244812] Node 0 DMA32: 1999*4kB (UEM) 4189*8kB (UEM) 519*16kB (UEM) 105*32kB (UEMR) 12*64kB (UER) 4*128kB (ER) 2*256kB (ER) 0*512kB 1*1024kB (R) 1*2048kB (R) 0*4096kB = 58036kB
Oct 6 02:14:25 xenon kernel: [1168911.244822] Node 0 Normal: 1880*4kB (UEM) 801*8kB (UEM) 85*16kB (UEM) 15*32kB (UEMR) 8*64kB (UER) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 17432kB
Oct 6 02:14:25 xenon kernel: [1168911.244832] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 6 02:14:25 xenon kernel: [1168911.244834] 7034 total pagecache pages
Oct 6 02:14:25 xenon kernel: [1168911.244835] 698 pages in swap cache
Oct 6 02:14:25 xenon kernel: [1168911.244837] Swap cache stats: add 7522125, delete 7521427, find 10531875/12956486
Oct 6 02:14:25 xenon kernel: [1168911.244838] Free swap = 0kB
Oct 6 02:14:25 xenon kernel: [1168911.244839] Total swap = 8388604kB
Oct 6 02:14:25 xenon kernel: [1168911.244840] 1048462 pages RAM
Oct 6 02:14:25 xenon kernel: [1168911.244841] 0 pages HighMem/MovableOnly
Oct 6 02:14:25 xenon kernel: [1168911.244842] 14635 pages reserved
Oct 6 02:14:25 xenon kernel: [1168911.244843] 0 pages hwpoisoned
Oct 6 02:14:25 xenon kernel: [1168911.244844] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Oct 6 02:14:25 xenon kernel: [1168911.244852] [ 191] 0 191 10408 73 22 340 -1000 systemd-udevd
Oct 6 02:14:25 xenon kernel: [1168911.244854] [ 193] 0 193 10388 194 24 574 0 systemd-journal
Oct 6 02:14:25 xenon kernel: [1168911.244857] [ 655] 0 655 13796 29 32 139 -1000 sshd
Oct 6 02:14:25 xenon kernel: [1168911.244859] [ 656] 0 656 25069 81 43 132 0 vmtoolsd
Oct 6 02:14:25 xenon kernel: [1168911.244861] [ 658] 0 658 6876 77 18 49 0 cron
Oct 6 02:14:25 xenon kernel: [1168911.244863] [ 659] 106 659 84424 0 29 2843 0 memcached
Oct 6 02:14:25 xenon kernel: [1168911.244866] [ 660] 114 660 24618 65 50 992 0 freshclam
Oct 6 02:14:25 xenon kernel: [1168911.244868] [ 663] 0 663 4964 17 15 50 0 systemd-logind
Oct 6 02:14:25 xenon kernel: [1168911.244870] [ 672] 0 672 4431 15 14 68 0 dovecot
Oct 6 02:14:25 xenon kernel: [1168911.244872] [ 675] 109 675 10530 0 26 103 -900 dbus-daemon
Oct 6 02:14:25 xenon kernel: [1168911.244875] [ 687] 107 687 64930 441 122 4755 0 redis-server
Oct 6 02:14:25 xenon kernel: [1168911.244877] [ 701] 0 701 64668 55 28 398 0 rsyslogd
Oct 6 02:14:25 xenon kernel: [1168911.244879] [ 702] 0 702 1064 65 8 36 0 acpid
Oct 6 02:14:25 xenon kernel: [1168911.244882] [ 704] 114 704 216724 61 347 137772 0 clamd
Oct 6 02:14:25 xenon kernel: [1168911.244884] [ 720] 0 720 3604 63 13 38 0 agetty
Oct 6 02:14:25 xenon kernel: [1168911.244886] [ 807] 0 807 5462 63 15 116 0 mysqld_safe
Oct 6 02:14:25 xenon kernel: [1168911.244888] [ 939] 113 939 8346 33 21 121 0 ntpd
Oct 6 02:14:25 xenon kernel: [1168911.244890] [ 940] 110 940 2311 0 9 60 0 anvil
Oct 6 02:14:25 xenon kernel: [1168911.244892] [ 941] 0 941 2343 51 12 57 0 log
Oct 6 02:14:25 xenon kernel: [1168911.244895] [ 997] 108 997 197835 3494 248 64655 0 mysqld
Oct 6 02:14:25 xenon kernel: [1168911.244897] [ 998] 0 998 5929 72 15 54 0 logger
Oct 6 02:14:25 xenon kernel: [1168911.244899] [ 1002] 112 1002 21379 41 41 246 0 opendkim
Oct 6 02:14:25 xenon kernel: [1168911.244901] [ 1004] 112 1004 99242 5 52 1547 0 opendkim
Oct 6 02:14:25 xenon kernel: [1168911.244903] [ 1022] 0 1022 177056 595 42 1363 0 fail2ban-server
Oct 6 02:14:25 xenon kernel: [1168911.244906] [ 1048] 0 1048 119691 749 171 2330 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244908] [ 1290] 0 1290 9042 28 21 116 0 master
Oct 6 02:14:25 xenon kernel: [1168911.244910] [ 1292] 105 1292 9599 26 24 134 0 qmgr
Oct 6 02:14:25 xenon kernel: [1168911.244912] [ 1295] 0 1295 22342 203 48 5747 0 miniserv.pl
Oct 6 02:14:25 xenon kernel: [1168911.244914] [ 1782] 105 1782 10680 37 26 176 0 tlsmgr
Oct 6 02:14:25 xenon kernel: [1168911.244917] [19783] 33 19783 153622 4672 230 13227 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244919] [19795] 33 19795 157674 4976 238 17075 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244921] [19796] 33 19796 155946 4660 234 15370 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244923] [19797] 33 19797 159940 4943 242 19068 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244925] [19806] 33 19806 152542 4389 228 12275 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244928] [19829] 33 19829 157207 4936 237 16438 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244930] [22711] 33 22711 155426 4522 233 15117 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244932] [22746] 33 22746 156429 6115 235 14481 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244934] [25280] 105 25280 9558 18 23 123 0 pickup
Oct 6 02:14:25 xenon kernel: [1168911.244936] [25425] 33 25425 138257 3308 233 18037 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244938] [25440] 33 25440 158331 5047 239 17276 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244940] [26197] 33 26197 135696 3967 202 15074 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244943] [26198] 33 26198 135504 3985 201 14769 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244945] [26199] 33 26199 135834 4773 202 14361 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244947] [26200] 33 26200 135765 4794 202 14308 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244949] [26201] 33 26201 125788 4210 180 4978 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244951] [26202] 33 26202 137178 4259 206 16158 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244953] [26203] 33 26203 135107 5619 201 12989 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244979] [26204] 33 26204 127161 3733 186 6834 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244981] [26205] 33 26205 134269 4962 199 12621 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244984] [26206] 33 26206 127414 6071 180 4745 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244986] [26207] 33 26207 134653 6680 200 11244 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244988] [26208] 33 26208 127414 4144 180 6560 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244990] [26209] 33 26209 137114 6406 206 13946 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244992] [26210] 33 26210 126585 4282 179 5703 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244994] [26211] 33 26211 135504 4380 201 14517 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244996] [26212] 33 26212 131624 6058 194 8900 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.244998] [26213] 33 26213 137040 4608 205 15732 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245001] [26214] 33 26214 135171 5708 201 12898 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245003] [26215] 33 26215 137178 4646 206 15768 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245005] [26216] 33 26216 134201 5302 199 12364 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245007] [26218] 33 26218 135691 4300 202 14686 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245009] [26219] 33 26219 127414 4748 180 6111 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245012] [26220] 33 26220 125753 5439 177 3750 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245014] [26221] 33 26221 137168 7610 206 12833 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245016] [26222] 33 26222 137040 7579 205 12753 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245018] [26223] 33 26223 137079 3882 205 16502 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245020] [26224] 33 26224 127414 4942 180 5886 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245022] [26226] 33 26226 125753 3839 177 5393 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245024] [26227] 33 26227 133420 6070 197 10905 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245027] [26230] 33 26230 135443 3998 201 14892 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245029] [26232] 33 26232 135107 6688 201 11823 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245031] [26238] 33 26238 136457 5302 203 14491 0 apache2
Oct 6 02:14:25 xenon kernel: [1168911.245033] [26239] 33 26239 135501 5851 201 13078 0 apache2

[Kees]

Je swap zat helemaal vol, en apache roept: apache2 invoked oom-killer:

Free swap = 0kB
Total swap = 8388604kB

Gezien de lijst met processen en hun geheugengebruik lijkt het mij niet dat een userland process uit zijn geheugen is gelopen. Heb je er toevallig iets als zfs op draaien?

[_JGC_]

4GB geheugen, 8GB swap en een paar geheugenvreters (clamav, mysql, maar ook fail2ban en opendkim zijn niet licht). Vervolgens Apache prefork met waarschijnlijk mod_php ingebouwd en te hoge MaxClients en als dat een beetje druk wordt graaft ie zich in de swap en sterft de boel af.