Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Server Crash after SWAP is full #2803

Closed
monetdb-team opened this issue Nov 30, 2020 · 0 comments
Closed

Server Crash after SWAP is full #2803

monetdb-team opened this issue Nov 30, 2020 · 0 comments
Labels
bug Something isn't working major MAL/M5
Milestone
Ancient Release

Comments

@monetdb-team
Copy link

Date: 2011-04-11 16:52:54 +0200
From: Vladimir Vlach <>
To: MonetDB5 devs <>
Version: -- development
CC: @mlkersten, risc

Last updated: 2015-10-02 08:59:26 +0200

Comment 15731

Date: 2011-04-11 16:52:54 +0200
From: Vladimir Vlach <>

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16
Build Identifier:

I have a table with 3000+ columns and about 20.000 rows. To fill the table with test data I execute following SQL query in Python followed by COMMIT. Once I update 30-40 tables the server will crash. Once there is no more available swap space the server will crash.

Sample Script loop:

for i in range(3000):
sql = "UPDATE reportdata_2010 SET column_float_type_%s=RAND() WHERE column_float_type_%s IS NULL" % (i, i)
cursorUpdate.execute(sql)
connection.commit()

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
root@monetdb:~ uname -a
Linux monetdb 2.6.32-5-amd64 1 SMP Mon Mar 7 21:35:22 UTC 2011 x86_64 GNU/Linux

root@monetdb:~ free
total used free shared buffers cached
Mem: 3803280 542080 3261200 0 61628 212028
-/+ buffers/cache: 268424 3534856
Swap: 8835708 0 8835708

Reproducible: Always

Log file error:

Apr 8 09:28:25 monetdb kernel: [106708.744036] mserver5[6398]: segfault at 28 ip 00007f53cddfa0c8 sp 00007f531652da80 error 4 in lib_sql.so[7f53cdd5e000+146000]
Apr 8 12:10:20 monetdb kernel: [116423.607958] mserver5 invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
Apr 8 12:10:20 monetdb kernel: [116423.607963] mserver5 cpuset=/ mems_allowed=0
Apr 8 12:10:20 monetdb kernel: [116423.607966] Pid: 7030, comm: mserver5 Not tainted 2.6.32-5-amd64 1
Apr 8 12:10:20 monetdb kernel: [116423.607968] Call Trace:
Apr 8 12:10:20 monetdb kernel: [116423.607975] [] ? oom_kill_process+0x7f/0x23f
Apr 8 12:10:20 monetdb kernel: [116423.607978] [] ? __out_of_memory+0x12a/0x141
Apr 8 12:10:20 monetdb kernel: [116423.607980] [] ? out_of_memory+0x140/0x172
Apr 8 12:10:20 monetdb kernel: [116423.607984] [] ? __alloc_pages_nodemask+0x4e5/0x5f4
Apr 8 12:10:20 monetdb kernel: [116423.607988] [] ? ____pagevec_lru_add+0x160/0x176
Apr 8 12:10:20 monetdb kernel: [116423.607992] [] ? handle_mm_fault+0x27a/0x80f
Apr 8 12:10:20 monetdb kernel: [116423.607995] [] ? do_brk+0x227/0x307
Apr 8 12:10:20 monetdb kernel: [116423.607999] [] ? do_page_fault+0x2e0/0x2fc
Apr 8 12:10:20 monetdb kernel: [116423.608019] [] ? page_fault+0x25/0x30
Apr 8 12:10:20 monetdb kernel: [116423.608022] Mem-Info:
Apr 8 12:10:20 monetdb kernel: [116423.608024] Node 0 DMA per-cpu:
Apr 8 12:10:20 monetdb kernel: [116423.608027] CPU 0: hi: 0, btch: 1 usd: 0
Apr 8 12:10:20 monetdb kernel: [116423.608029] CPU 1: hi: 0, btch: 1 usd: 0
Apr 8 12:10:20 monetdb kernel: [116423.608031] Node 0 DMA32 per-cpu:
Apr 8 12:10:20 monetdb kernel: [116423.608033] CPU 0: hi: 186, btch: 31 usd: 30
Apr 8 12:10:20 monetdb kernel: [116423.608035] CPU 1: hi: 186, btch: 31 usd: 1
Apr 8 12:10:20 monetdb kernel: [116423.608037] Node 0 Normal per-cpu:
Apr 8 12:10:20 monetdb kernel: [116423.608039] CPU 0: hi: 186, btch: 31 usd: 30
Apr 8 12:10:20 monetdb kernel: [116423.608041] CPU 1: hi: 186, btch: 31 usd: 0
Apr 8 12:10:20 monetdb kernel: [116423.608045] active_anon:666438 inactive_anon:258874 isolated_anon:32
Apr 8 12:10:20 monetdb kernel: [116423.608046] active_file:24 inactive_file:79 isolated_file:0
Apr 8 12:10:20 monetdb kernel: [116423.608047] unevictable:0 dirty:0 writeback:648 unstable:0
Apr 8 12:10:20 monetdb kernel: [116423.608047] free:6690 slab_reclaimable:1478 slab_unreclaimable:2055
Apr 8 12:10:20 monetdb kernel: [116423.608048] mapped:46 shmem:1 pagetables:7198 bounce:0
Apr 8 12:10:20 monetdb kernel: [116423.608050] Node 0 DMA free:15024kB min:28kB low:32kB high:40kB active_anon:308kB inactive_anon:508kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15264kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:4kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 8 12:10:20 monetdb kernel: [116423.608058] lowmem_reserve[]: 0 2743 3753 3753
Apr 8 12:10:20 monetdb kernel: [116423.608061] Node 0 DMA32 free:9736kB min:5716kB low:7144kB high:8572kB active_anon:2170028kB inactive_anon:540708kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2809760kB mlocked:0kB dirty:0kB writeback:1320kB mapped:120kB shmem:4kB slab_reclaimable:1464kB slab_unreclaimable:1028kB kernel_stack:64kB pagetables:15200kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 8 12:10:20 monetdb kernel: [116423.608069] lowmem_reserve[]: 0 0 1010 1010
Apr 8 12:10:20 monetdb kernel: [116423.608071] Node 0 Normal free:2000kB min:2104kB low:2628kB high:3156kB active_anon:495416kB inactive_anon:494408kB active_file:96kB inactive_file:188kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:1034240kB mlocked:0kB dirty:0kB writeback:1272kB mapped:64kB shmem:0kB slab_reclaimable:4448kB slab_unreclaimable:7192kB kernel_stack:1000kB pagetables:13588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:266 all_unreclaimable? no
Apr 8 12:10:20 monetdb kernel: [116423.608078] lowmem_reserve[]: 0 0 0 0
Apr 8 12:10:20 monetdb kernel: [116423.608080] Node 0 DMA: 24kB 38kB 316kB 132kB 364kB 3128kB 0256kB 0512kB 21024kB 22048kB 24096kB = 15024kB
Apr 8 12:10:20 monetdb kernel: [116423.608086] Node 0 DMA32: 14224kB 68kB 416kB 732kB 364kB 2128kB 1256kB 2512kB 21024kB 02048kB 04096kB = 9800kB
Apr 8 12:10:20 monetdb kernel: [116423.608092] Node 0 Normal: 1044kB 08kB 116kB 932kB 964kB 4128kB 1256kB 0512kB 01024kB 02048kB 0*4096kB = 2064kB
Apr 8 12:10:20 monetdb kernel: [116423.608098] 5028 total pagecache pages

Apr 8 12:10:20 monetdb kernel: [116423.608099] 4888 pages in swap cache
Apr 8 12:10:20 monetdb kernel: [116423.608101] Swap cache stats: add 9115342, delete 9110454, find 1415731/1861016
Apr 8 12:10:20 monetdb kernel: [116423.608102] Free swap = 0kB
Apr 8 12:10:20 monetdb kernel: [116423.608103] Total swap = 7427064kB
Apr 8 12:10:20 monetdb kernel: [116423.621726] 983024 pages RAM
Apr 8 12:10:20 monetdb kernel: [116423.621728] 32204 pages reserved
Apr 8 12:10:20 monetdb kernel: [116423.621729] 457 pages shared
Apr 8 12:10:20 monetdb kernel: [116423.621730] 943744 pages non-shared
Apr 8 12:10:20 monetdb kernel: [116423.662539] mserver5 invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Apr 8 12:10:20 monetdb kernel: [116423.662542] mserver5 cpuset=/ mems_allowed=0
Apr 8 12:10:20 monetdb kernel: [116423.662544] Pid: 6922, comm: mserver5 Not tainted 2.6.32-5-amd64 1
Apr 8 12:10:20 monetdb kernel: [116423.662546] Call Trace:
Apr 8 12:10:20 monetdb kernel: [116423.662550] [] ? oom_kill_process+0x7f/0x23f
Apr 8 12:10:20 monetdb kernel: [116423.662553] [] ? __out_of_memory+0x12a/0x141
Apr 8 12:10:20 monetdb kernel: [116423.662555] [] ? out_of_memory+0x140/0x172
Apr 8 12:10:20 monetdb kernel: [116423.662558] [] ? __bitmap_weight+0x3a/0x7e
Apr 8 12:10:20 monetdb kernel: [116423.662561] [] ? __alloc_pages_nodemask+0x4e5/0x5f4
Apr 8 12:10:20 monetdb kernel: [116423.662565] [] ? read_swap_cache_async+0x5d/0xf3
Apr 8 12:10:20 monetdb kernel: [116423.662567] [] ? swapin_readahead+0x57/0x98
Apr 8 12:10:20 monetdb kernel: [116423.662571] [] ? handle_mm_fault+0x47f/0x80f
Apr 8 12:10:20 monetdb kernel: [116423.662574] [] ? finish_task_switch+0x3a/0xaf
Apr 8 12:10:20 monetdb kernel: [116423.662577] [] ? thread_return+0x79/0xe0
Apr 8 12:10:20 monetdb kernel: [116423.662580] [] ? do_page_fault+0x2e0/0x2fc
Apr 8 12:10:20 monetdb kernel: [116423.662583] [] ? page_fault+0x25/0x30
Apr 8 12:10:20 monetdb kernel: [116423.662586] [] ? __get_user_8+0x1c/0x23
Apr 8 12:10:20 monetdb kernel: [116423.662589] [] ? exit_robust_list+0x32/0x131
Apr 8 12:10:20 monetdb kernel: [116423.662592] [] ? mm_release+0x20/0xe2
Apr 8 12:10:20 monetdb kernel: [116423.662594] [] ? exit_mm+0x1c/0x10d
Apr 8 12:10:20 monetdb kernel: [116423.662596] [] ? do_exit+0x1f8/0x6c6
Apr 8 12:10:20 monetdb kernel: [116423.662598] [] ? futex_wake+0xd6/0xe7
Apr 8 12:10:20 monetdb kernel: [116423.662600] [] ? do_group_exit+0x76/0x9d
Apr 8 12:10:20 monetdb kernel: [116423.662603] [] ? get_signal_to_deliver+0x310/0x339
Apr 8 12:10:20 monetdb kernel: [116423.662607] [] ? do_notify_resume+0x87/0x73f
Apr 8 12:10:20 monetdb kernel: [116423.662609] [] ? thread_return+0x79/0xe0
Apr 8 12:10:20 monetdb kernel: [116423.662611] [] ? sys_futex+0x113/0x131
Apr 8 12:10:20 monetdb kernel: [116423.662613] [] ? int_signal+0x12/0x17
Apr 8 12:10:20 monetdb kernel: [116423.662615] Mem-Info:
Apr 8 12:10:20 monetdb kernel: [116423.662615] Node 0 DMA per-cpu:
Apr 8 12:10:20 monetdb kernel: [116423.662617] CPU 0: hi: 0, btch: 1 usd: 0
Apr 8 12:10:20 monetdb kernel: [116423.662619] CPU 1: hi: 0, btch: 1 usd: 0
Apr 8 12:10:20 monetdb kernel: [116423.662620] Node 0 DMA32 per-cpu:
Apr 8 12:10:20 monetdb kernel: [116423.662621] CPU 0: hi: 186, btch: 31 usd: 0
Apr 8 12:10:20 monetdb kernel: [116423.662623] CPU 1: hi: 186, btch: 31 usd: 1
Apr 8 12:10:20 monetdb kernel: [116423.662624] Node 0 Normal per-cpu:
Apr 8 12:10:20 monetdb kernel: [116423.662625] CPU 0: hi: 186, btch: 31 usd: 22
Apr 8 12:10:20 monetdb kernel: [116423.662627] CPU 1: hi: 186, btch: 31 usd: 0
Apr 8 12:10:20 monetdb kernel: [116423.662630] active_anon:666342 inactive_anon:259002 isolated_anon:0
Apr 8 12:10:20 monetdb kernel: [116423.662630] active_file:51 inactive_file:15 isolated_file:0
Apr 8 12:10:20 monetdb kernel: [116423.662631] unevictable:0 dirty:0 writeback:648 unstable:0
Apr 8 12:10:20 monetdb kernel: [116423.662632] free:6776 slab_reclaimable:1478 slab_unreclaimable:2038
Apr 8 12:10:20 monetdb kernel: [116423.662633] mapped:46 shmem:1 pagetables:7198 bounce:0
Apr 8 12:10:20 monetdb kernel: [116423.662634] Node 0 DMA free:15024kB min:28kB low:32kB high:40kB active_anon:308kB inactive_anon:508kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15264kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:4kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 8 12:10:20 monetdb kernel: [116423.662641] lowmem_reserve[]: 0 2743 3753 3753

root@monetdb:~ vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 2602224 76336 239256 0 203 8 243 158 206 1 0 98 1
1 0 0 2598496 76380 241692 0 0 0 320 38647 76214 51 5 44 0
1 0 0 2596496 76412 243752 0 0 0 257 38755 76387 50 6 44 0
1 0 0 2594512 76440 245752 0 0 0 290 38688 76201 51 5 44 0
1 0 0 2571432 76476 248188 0 0 0 336 38482 75848 50 5 44 0
1 0 0 2567844 76508 250248 0 0 0 294 38700 76280 51 5 44 0
1 0 0 2395484 76512 250252 0 0 0 46 3423 6628 7 1 92 0
0 2 39932 26636 308 5260 3 4242 88 4491 13848 2136 38 6 40 16
0 2 448140 26712 540 5320 3418 40757 3845 40952 6728 1966 5 4 9 82
0 1 849688 29316 712 5756 462 40339 619 40541 7160 1428 6 3 8 84
0 2 1228196 26736 744 5668 464 37982 530 38123 6406 1268 5 3 10 82
0 2 1616368 26736 752 5748 859 39076 863 39182 6446 1217 6 2 17 75
0 2 2004984 26432 700 5888 613 39099 626 39282 6564 1225 6 3 10 81
0 2 2403996 26664 628 5992 518 40035 652 40156 6518 1183 6 3 8 84
0 2 2803644 26648 484 5616 796 40316 813 40425 7077 1282 5 3 12 79
0 2 3180608 26600 612 8180 341 37814 696 37941 4132 990 5 3 16 76
1 2 3572312 26580 840 8964 594 39409 724 39672 5881 1172 5 3 10 81
0 2 3944952 26608 904 10784 1162 38094 1604 38305 7163 1453 5 3 9 83
0 2 4342100 26664 888 9400 380 39814 550 39972 6021 1401 6 3 10 81
0 2 4743236 26804 792 8256 320 40365 328 40468 4851 1135 6 3 15 76
1 0 5121668 28904 784 8588 313 38278 367 38485 4957 1142 5 3 10 82
0 2 5518504 26600 652 7752 196 39798 226 39912 6945 1250 5 3 7 84
1 1 5910744 26888 600 7148 159 39323 162 39426 6093 1138 5 3 9 83
0 1 6318432 28072 736 8012 206 40849 321 41038 5771 1236 6 3 19 72
1 1 6695616 27164 820 8252 775 38271 825 38517 4023 1299 5 3 18 74
1 0 7076896 29796 828 8256 165 38199 182 38290 4361 1052 5 3 20 73
0 2 7475872 26736 824 8104 496 40144 497 40256 5465 1326 6 3 15 77
0 2 7883740 26564 820 7968 131 40856 131 40974 4820 1122 5 3 18 74
0 1 8265376 29096 844 8284 297 38312 297 38496 5043 1146 5 3 21 71
0 2 8670368 26748 844 7996 197 40567 198 40672 5109 1191 6 3 18 74 -< crash here
0 0 108860 3685160 392 2212 194 24481 517 24686 3248 1207 3 5 41 50

Comment 15732

Date: 2011-04-11 17:44:07 +0200
From: Vladimir Vlach <>

I actually discovered: This server will crash even when table is empty. It consumers more and more memory when I run many UPDATE statements on empty dataset.

UPDATE reportdata_2009 SET %s=RAND() WHERE %s IS NULL;

Comment 17911

Date: 2012-11-07 19:48:49 +0100
From: @grobian

thanks, we should test this

Comment 18160

Date: 2012-11-27 17:26:12 +0100
From: @mlkersten

Test has been added huge_table_update.Bug-2803
The test run does show eating the memory, it stabelizes after a few hundred
on 180K, but the system seems killed by OS after 1200 update runs.

Comment 19013

Date: 2013-08-16 21:27:01 +0200
From: @mlkersten

Multiple runs of the script against the same server on both FEB13-SP3 and default work without problems. Consider this bug as being closed.

Comment 21318

Date: 2015-10-02 08:59:26 +0200
From: MonetDB Mercurial Repository <>

Changeset 6753d241bce5 made by Stefan Manegold Stefan.Manegold@cwi.nl in the MonetDB repo, refers to this bug.

For complete details, see http//devmonetdborg/hg/MonetDB?cmd=changeset;node=6753d241bce5

Changeset description:

extended timeout for recently activated test huge_table_update.Bug-2803 on slow machines
@monetdb-team monetdb-team added bug Something isn't working major MAL/M5 labels Nov 30, 2020
@sjoerdmullender sjoerdmullender added this to the Ancient Release milestone Nov 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working major MAL/M5
Projects
None yet
Milestone
Ancient Release
Development

No branches or pull requests
2 participants
@sjoerdmullender @monetdb-team