Date: 2019-09-12 12:34:09 +0200
From: @swingbit
To: SQL devs <>
Version: 11.33.11 (Apr2019-SP1)
CC: @PedroTadim

Last updated: 2019-11-28 10:00:04 +0100

Comment 27275

Date: 2019-09-12 12:34:09 +0200
From: @swingbit

User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36
Build Identifier:

Thread 1 (Thread 0x7f189e1e6700 (LWP 329)):
0 0x00007f189f8a4e75 in raise () from /lib64/libc.so.6
1 0x00007f189f88f895 in abort () from /lib64/libc.so.6
2 0x00007f189f8e7d4f in __libc_message () from /lib64/libc.so.6
3 0x00007f189f8ee5fc in malloc_printerr () from /lib64/libc.so.6
4 0x00007f189f8effe8 in _int_free () from /lib64/libc.so.6
5 0x00007f189f8f23f6 in _int_realloc () from /lib64/libc.so.6
6 0x00007f189f8f35ab in realloc () from /lib64/libc.so.6
7 0x00007f18a057779a in GDKrealloc () from /opt/monetdb/lib/libbat.so.18
8 0x00007f189e873c6c in stack_push_frame () from /opt/monetdb/lib/monetdb5/lib_sql.so
9 0x00007f189e880d0b in rel_selects () from /opt/monetdb/lib/monetdb5/lib_sql.so
10 0x00007f189e890163 in schema_selects () from /opt/monetdb/lib/monetdb5/lib_sql.so
11 0x00007f189e89de6f in rel_schemas () from /opt/monetdb/lib/monetdb5/lib_sql.so
12 0x00007f189e87d628 in rel_with_query () from /opt/monetdb/lib/monetdb5/lib_sql.so
13 0x00007f189e880b7d in rel_selects () from /opt/monetdb/lib/monetdb5/lib_sql.so
14 0x00007f189e890163 in schema_selects () from /opt/monetdb/lib/monetdb5/lib_sql.so
15 0x00007f189e89de6f in rel_schemas () from /opt/monetdb/lib/monetdb5/lib_sql.so
16 0x00007f189e7c863d in sql_symbol2relation () from /opt/monetdb/lib/monetdb5/lib_sql.so
17 0x00007f189e7df3e8 in SQLparser () from /opt/monetdb/lib/monetdb5/lib_sql.so
18 0x00007f18a075a34c in runScenarioBody () from /opt/monetdb/lib/libmonetdb5.so.27
19 0x00007f18a075b17e in runScenario () from /opt/monetdb/lib/libmonetdb5.so.27
20 0x00007f18a075b6e2 in MSserveClient () from /opt/monetdb/lib/libmonetdb5.so.27
21 0x00007f18a075bd8d in MSscheduleClient () from /opt/monetdb/lib/libmonetdb5.so.27
--Type for more, q to quit, c to continue without paging--
22 0x00007f18a07e77e6 in doChallenge () from /opt/monetdb/lib/libmonetdb5.so.27
23 0x00007f18a057765d in THRstarter () from /opt/monetdb/lib/libbat.so.18
24 0x00007f18a05f0679 in thread_starter () from /opt/monetdb/lib/libbat.so.18
25 0x00007f189fa3b5a2 in start_thread () from /lib64/libpthread.so.0
26 0x00007f189f968303 in clone () from /lib64/libc.so.6

Reproducible: Always

Comment 27276

Date: 2019-09-12 12:36:42 +0200
From: @swingbit

(By mistake I saved this ticket before completing it)

The previous comment shows the backtrace taken from a core dump of a production server. Because it is a production server, there is no debug info. But hopefully the trace should help identifying the culprit.

Comment 27278

Date: 2019-09-12 14:09:55 +0200
From: @swingbit

With a bit of difficulty, I reproduced the issue with a debug compilation.
Here the trace again:

0 0x00007f3536c8fa6e in GDKrealloc (s=0x7f352c35aad0, size=9216) at gdk_utils.c:1844
1 0x00007f3534dad9bb in stack_set (sql=0x7f352c3cbc60, var=32, name=0x7f3534ee51b8 "SELECT", type=0x0, rel=0x0, t=0x0, wdef=0x0, exp=0x0, view=0, frame=1) at sql_mvc.c:1548
2 0x00007f3534dae886 in stack_push_frame (sql=0x7f352c3cbc60, name=0x7f3534ee51b8 "SELECT") at sql_mvc.c:1728
3 0x00007f3534dd73da in rel_selects (sql=0x7f352c3cbc60, s=0x7f352c5afe50) at rel_select.c:6481
4 0x00007f3534dd75b4 in schema_selects (sql=0x7f352c3cbc60, schema=0x7f352c18dd10, s=0x7f352c5afe50) at rel_select.c:6522
5 0x00007f3534de97bb in rel_create_view (sql=0x7f352c3cbc60, ss=0x0, qname=0x7f352c5afef0, column_spec=0x0, query=0x7f352c5afe50, check=0, persistent=0, replace=0) at rel_schema.c:1197
6 0x00007f3534df0477 in rel_schemas (sql=0x7f352c3cbc60, s=0x7f352c5afff0) at rel_schema.c:2718
7 0x00007f3534db5f8a in rel_semantic (sql=0x7f352c3cbc60, s=0x7f352c5afff0) at rel_semantic.c:164
8 0x00007f3534db79a3 in rel_with_query (sql=0x7f352c3cbc60, q=0x7f352c610f00) at rel_select.c:320
9 0x00007f3534dd739c in rel_selects (sql=0x7f352c3cbc60, s=0x7f352c610f00) at rel_select.c:6474
10 0x00007f3534dd75b4 in schema_selects (sql=0x7f352c3cbc60, schema=0x7f352c18dd10, s=0x7f352c610f00) at rel_select.c:6522
11 0x00007f3534de97bb in rel_create_view (sql=0x7f352c3cbc60, ss=0x0, qname=0x7f352c599b00, column_spec=0x0, query=0x7f352c610f00, check=0, persistent=1, replace=0) at rel_schema.c:1197
12 0x00007f3534df0477 in rel_schemas (sql=0x7f352c3cbc60, s=0x7f352c611000) at rel_schema.c:2718
13 0x00007f3534db5f8a in rel_semantic (sql=0x7f352c3cbc60, s=0x7f352c611000) at rel_semantic.c:164
14 0x00007f3534cdcdb4 in sql_symbol2relation (c=0x7f352c3cbc60, sym=0x7f352c611000) at sql.c:119
15 0x00007f3534cffe21 in SQLparser (c=0x7f35358fe380) at sql_scenario.c:1255
16 0x00007f3536f4927c in runPhase (c=0x7f35358fe380, phase=1) at mal_scenario.c:517
17 0x00007f3536f49381 in runScenarioBody (c=0x7f35358fe380, once=0) at mal_scenario.c:541
18 0x00007f3536f49649 in runScenario (c=0x7f35358fe380, once=0) at mal_scenario.c:580
19 0x00007f3536f4b3c6 in MSserveClient (c=0x7f35358fe380) at mal_session.c:525
20 0x00007f3536f4ae8e in MSscheduleClient (command=0x7f352c107310 '\333' <repeats 127 times>, "۽\275\275\275\275\275\275\275!", challenge=0x7f353469be2b "hX56kdyN30", fin=0x7f352c3581a0, fout=0x7f35300050e0, protocol=PROTOCOL_9, blocksize=8190) at mal_session.c:403
21 0x00007f3536ff8b09 in doChallenge (data=0x7f3530004e90) at mal_mapi.c:271
22 0x00007f3536c8da0e in THRstarter (a=0x7f35300092b0) at gdk_utils.c:1419
23 0x00007f3536d07824 in thread_starter (arg=0x40dce10) at gdk_system.c:650
24 0x00007f3535fd95a2 in start_thread () from /lib64/libpthread.so.0
25 0x00007f3535f06303 in clone () from /lib64/libc.so.6

The failing realloc happens in stack_set().
The arguments of the realloc are:

(gdb) p *(struct sql_var *)s
$12 = {name = 0x7f352c587690 "\333\333\333\333\333\333\333\333\001\023", a = {isnull = 0, tpe = {type = 0x38ee980, digits = 32, scale = 0}, data = {val = {ival = 0, oval = 893353197568, shval = 0, btval = 0 '\000', fval = 0, pval = 0xd000000000, bval = 0,
sval = 0xd000000000 <error: Cannot access memory at address 0xd000000000>, dval = 4.4137512452077664e-312, lval = 893353197568, hval = 368934882367544229888}, len = 4, vtype = 5}, d = 0, varid = 0}, t = 0x0, rel = 0x0, wdef = 0x0, exp = 0x0, view = 0 '\000', frame = 0 '\000', visited = 0 '\000'}

(gdb) p size
$13 = 9216

Comment 27279

Date: 2019-09-12 15:10:30 +0200
From: @sjoerdmullender

Any chance you can share the way how to reproduce this?

Also, what are the values of the variables in stack frame 1?

Perhaps do "bt full".

Comment 27280

Date: 2019-09-12 17:24:08 +0200
From: @swingbit

In short, for both questions: not soon.

It is very hard to reproduce in an isolated example, for now. I could make it but it will require some digging.

Also, because it was on a production server, I had to rollback to Aug2018, so I can't inspect it anymore. To reproduce the error I will have to reproduce the same deployment with the same data in a sandbox. I hope to have time in the coming days.

Comment 27284

Date: 2019-09-13 10:42:02 +0200
From: @swingbit

Sjoerd, I reproduced the error, here is a "bt full", I don't know why it is slightly different. The original one failed with SIGABRT (also in debug mode), this one fails with SIGSEGV on memset (exact same code, just compiled on a different machine). Still, it is clearly the same bug:

Thread 39 "mserver5" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fe10e142700 (LWP 32382)]
0x00007fe13b73a987 in __memset_erms () from /lib64/libc.so.6
(gdb) bt full
0 0x00007fe13b73a987 in __memset_erms () from /lib64/libc.so.6
No symbol table info available.
1 0x00007fe13c60dde2 in GDKrealloc (s=0x7fe104152590, size=9216) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/gdk/gdk_utils.c:1848
nsize = 9216
asize = 140604411479728
osize = 140604411479728
os = 0x7fe1051ed7f0
PRETTY_FUNCTION = "GDKrealloc"
2 0x00007fe12d2bf120 in stack_set (sql=0x7fe10517b700, var=32, name=0x7fe12d3f86f0 "SELECT", type=0x0, rel=0x0, t=0x0, wdef=0x0, exp=0x0, view=0, frame=1) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/sql_mvc.c:1548
_ptr = 0x7fe104152590
_size = 9216
_res = 0x7fe10508d0f0
v = 0x7fe10508e040
nvars = 0x7fe0f007f790
nextsize = 64
func = "stack_set"
3 0x00007fe12d2bffeb in stack_push_frame (sql=0x7fe10517b700, name=0x7fe12d3f86f0 "SELECT") at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/sql_mvc.c:1728
res = 0x7fe10508d0f0
4 0x00007fe12d2e8bd7 in rel_selects (sql=0x7fe10517b700, s=0x7fe10508f260) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/rel_select.c:6481
ek = {type = 0 '\000', card = 4 '\004', reduce = 1 '\001'}
sn = 0x7fe10508f260
ret = 0x0
5 0x00007fe12d2e8db1 in schema_selects (sql=0x7fe10517b700, schema=0x7fe0f007f180, s=0x7fe10508f260) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/rel_select.c:6522
res = 0x7fe104fa1040
os = 0x7fe0f007f180
6 0x00007fe12d2fb015 in rel_create_view (sql=0x7fe10517b700, ss=0x0, qname=0x7fe10508f300, column_spec=0x0, query=0x7fe10508f260, check=0, persistent=0, replace=0) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/rel_schema.c:1197
sq = 0x0
q = 0x7fe104005be0 "-- -1\n-- \ncreate view s0_ifthenelse_1_result_result as with q0_x0 as (select 0 as a1, a2, prob from (select paramname as a1, value as a2, prob from params_str where paramname = 's0_keyword') as t__x7)"...
name = 0x7fe10508e340 '\333' <repeats 199 times>, <incomplete sequence \333>...
sname = 0x0
s = 0x7fe0f007f180
t = 0x0
instantiate = 1
deps = 0
create = 0
base = 0x7fe12d3fa7d9 "CREATE"
func = "rel_create_view"
7 0x00007fe12d301cd1 in rel_schemas (sql=0x7fe10517b700, s=0x7fe10508f400) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/rel_schema.c:2718
l = 0x7fe10508f2e0
ret = 0x0
PRETTY_FUNCTION = "rel_schemas"
8 0x00007fe12d2c76ef in rel_semantic (sql=0x7fe10517b700, s=0x7fe10508f400) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/rel_semantic.c:164
No locals.
9 0x00007fe12d2c91a0 in rel_with_query (sql=0x7fe10517b700, q=0x7fe1051c4e30) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/rel_select.c:320
sym = 0x7fe10508f400
dn = 0x7fe10508f340
name = 0x7fe10508e340 '\333' <repeats 199 times>, <incomplete sequence \333>...
nrel = 0x7fe1051ed8b0
d = 0x7fe10508f420
next = 0x7fe1051c4d50
rel = 0x7fe13c8042a8 <GDKthreads+168>
PRETTY_FUNCTION = "rel_with_query"
--Type for more, q to quit, c to continue without paging--
10 0x00007fe12d2e8b99 in rel_selects (sql=0x7fe10517b700, s=0x7fe1051c4e30) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/rel_select.c:6474
ret = 0x0
11 0x00007fe12d2e8db1 in schema_selects (sql=0x7fe10517b700, schema=0x7fe0f007f180, s=0x7fe1051c4e30) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/rel_select.c:6522
res = 0x0
os = 0x7fe0f007f180
12 0x00007fe12d2fb015 in rel_create_view (sql=0x7fe10517b700, ss=0x0, qname=0x7fe105078f10, column_spec=0x0, query=0x7fe1051c4e30, check=0, persistent=1, replace=0) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/rel_schema.c:1197
sq = 0x0
q = 0x7fe104005be0 "-- -1\n-- \ncreate view s0_ifthenelse_1_result_result as with q0_x0 as (select 0 as a1, a2, prob from (select paramname as a1, value as a2, prob from params_str where paramname = 's0_keyword') as t__x7)"...
name = 0x7fe105078ef0 '\333' <repeats 199 times>, <incomplete sequence \333>...
sname = 0x0
s = 0x7fe0f007f180
t = 0x0
instantiate = 0
deps = 0
create = 1
base = 0x7fe12d3fa7d9 "CREATE"
func = "rel_create_view"
13 0x00007fe12d301cd1 in rel_schemas (sql=0x7fe10517b700, s=0x7fe1051c4f30) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/rel_schema.c:2718
l = 0x7fe1051c4e50
ret = 0x0
PRETTY_FUNCTION = "rel_schemas"
14 0x00007fe12d2c76ef in rel_semantic (sql=0x7fe10517b700, s=0x7fe1051c4f30) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/server/rel_semantic.c:164
No locals.
15 0x00007fe12d1ede38 in sql_symbol2relation (c=0x7fe10517b700, sym=0x7fe1051c4f30) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/backends/monet5/sql.c:119
r = 0x0
16 0x00007fe12d210f4f in SQLparser (c=0x7fe12dbff380) at /home/roberto/MonetDBServer/MonetDB.Spinque_Apr2019/src/sql/backends/monet5/sql_scenario.c:1255
r = 0x7fe10e1419c0
in = 0x7fe1044881e0
out = 0x7fe108002b90
msg = 0x0
be = 0x7fe104485420
m = 0x7fe10517b700
oldvtop = 1
oldstop = 1
pstatus = 0
err = 0
opt = 0
q = 0x7fe1050a8f80 '\333' <repeats 199 times>, <incomplete sequence \333>...
func = "SQLparser"
PRETTY_FUNCTION = "SQLparser"

(gdb) p *sql
$7 = {errstr = '\333' <repeats 8191 times>..., sa = 0xdbdbdbdbdbdbdbdb, qc = 0xdbdbdbdbdbdbdbdb, clientid = -606348325, scanner = {rs = 0xdbdbdbdbdbdbdbdb, ws = 0xdbdbdbdbdbdbdbdb, log = 0xdbdbdbdbdbdbdbdb, yynext = -606348325, yylast = -606348325, yysval = -606348325, yyval = -606348325, yycur = -606348325,
yybak = -37 '\333', as = -606348325, key = -606348325, started = -606348325, mode = (LINE_N | unknown: 3688618970), schema = 0xdbdbdbdbdbdbdbdb <error: Cannot access memory at address 0xdbdbdbdbdbdbdbdb>, errstr = 0xdbdbdbdbdbdbdbdb <error: Cannot access memory at address 0xdbdbdbdbdbdbdbdb>},
sqs = 0xdbdbdbdbdbdbdbdb, params = 0xdbdbdbdbdbdbdbdb, forward = 0xdbdbdbdbdbdbdbdb, vars = 0xdbdbdbdbdbdbdbdb, topvars = -606348325, sizevars = -606348325, frame = -606348325, use_views = -606348325, args = 0xdbdbdbdbdbdbdbdb, argc = -606348325, argmax = -606348325, sym = 0xdbdbdbdbdbdbdbdb,
no_mitosis = -606348325, user_id = -606348325, role_id = -606348325, last_id = -2604246222170760229, rowcnt = -2604246222170760229, timezone = -606348325, cache = -606348325, caching = -606348325, reply_size = -606348325, sizeheader = 219, debug = -606348325, Topt = -2604246222170760229, emode = -37 '\333',
emod = -37 '\333', session = 0xdbdbdbdbdbdbdbdb, type = -606348325, pushdown = -606348325, label = -606348325, remote = -606348325, cascade_action = 0xdbdbdbdbdbdbdbdb, opt_stats = {-606348325, -606348325, -606348325, -606348325, -606348325, -606348325, -606348325, -606348325}, result_id = -606348325,
results = 0xdbdbdbdbdbdbdbdb}

Comment 27289

Date: 2019-09-16 17:34:04 +0200
From: @swingbit

Created attachment 633
SQL script to reproduce the bug

Attached file: bug6757.sql (application/sql, 37476 bytes)
Description: SQL script to reproduce the bug

Comment 27290

Date: 2019-09-16 17:35:51 +0200
From: @swingbit

Added script to reproduce the crash.
No data needed, just the schema.

Comment 27291

Date: 2019-09-16 18:47:25 +0200
From: MonetDB Mercurial Repository <>

Changeset b87bd0e2f8f8 made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see https//devmonetdborg/hg/MonetDB?cmd=changeset;node=b87bd0e2f8f8

Changeset description:

Don't overwrite m->vars and m->sizevars with stale values.
This fixes bug #6757.

Comment 27294

Date: 2019-09-16 19:59:42 +0200
From: MonetDB Mercurial Repository <>

Changeset 302e9c2c813d made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see https//devmonetdborg/hg/MonetDB?cmd=changeset;node=302e9c2c813d

Changeset description:

Test for bug #6757.