Date: 2013-02-12 19:45:54 +0100
From: Ashish Kumar Singh <>
To: SQL devs <>
Version: 11.15.1 (Feb2013)
CC: ashishk, @mlkersten, @njnes, @drstmane

Last updated: 2013-03-07 12:41:22 +0100

Comment 18467

Date: 2013-02-12 19:45:54 +0100
From: Ashish Kumar Singh <>

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17
Build Identifier:

We got below when a monet db compiled against sles 10 SP4 was run on SLES 11 SP2. Any suggestion filing as a bug as don't have access to my registered email any poionter wil help!

cat /var/log/messages | grep mserver Feb 12 14:54:17 blrec3vm6 kernel: mserver5[26559]: segfault at 2000 ip 00007fa105edb4d5 sp 00007fa0ffffee30 error 6 in lib_sql.so[7fa105dd8000+140000] Feb 12 15:03:03 blrec3vm6 kernel: mserver5[32124]: segfault at 28 ip 00007fc0eb916e95 sp 00007fc0e78d4f20 error 6 in libmonetdb5.so.13[7fc0eb467000+63b000]
Feb 12 15:17:38 blrec3vm6 kernel: mserver5[12463]: segfault at 28 ip 00007f6b7b1dae95 sp 00007f6b77198f20 error 6 in libmonetdb5.so.13[7f6b7ad2b000+63b000]
Feb 12 15:36:29 blrec3vm6 kernel: mserver5[15897]: segfault at 8 ip 00007f9e9858baf1 sp 00007f9e941d6420 error 4 in libbat.so.8[7f9e9853e000+28a000] Feb 12 16:09:41 blrec3vm6 kernel: mserver5[17883]: segfault at 18 ip 00007fabed454708 sp 00007fabec74cde0 error 4 in lib_sql.so[7fabed359000+140000] Feb 12 17:40:43 blrec3vm6 kernel: mserver5[24264]: segfault at 0 ip 00007fa921c7876e sp 00007fa91fb63d10 error 6 in libc-2.9.so[7fa921b93000+14f000] Feb 12 17:44:18 blrec3vm6 kernel: mserver5[26067]: segfault at 2000 ip 00007fb5048144d5 sp 00007fb503d05e30 error 6 in lib_sql.so[7fb504711000+140000] Feb 12 18:52:54 blrec3vm6 kernel: mserver5[28356]: segfault at 2000 ip 00007f0c3a93b4d5 sp 00007f0c39e2ce30 error 6 in lib_sql.so[7f0c3a838000+140000]

Reproducible: Always

Comment 18473

Date: 2013-02-13 17:26:43 +0100
From: Ashish Kumar Singh <>

Hi Guys,

This I got with latest candidate build also, any suggestions when this can happen?

Feb 13 14:16:36 blrec3vm30 kernel: mserver5[12299]: segfault at 28 ip 00007f712a08de95 sp 00007f71260cbf20 error 6 in libmonetdb5.so.13[7f7129bde000+63b000]
Feb 13 14:38:39 blrec3vm30 kernel: mserver5[11872]: segfault at 20 ip 00007fb85c420e1d sp 00007fb85ae72810 error 4 in lib_sql.so[7fb85c325000+140000]
Feb 13 14:52:21 blrec3vm30 kernel: mserver5[11876]: segfault at 28 ip 00007feb7af57e95 sp 00007feb76f95f20 error 6 in libmonetdb5.so.13[7feb7aaa8000+63b000]
Feb 13 15:28:19 blrec3vm30 kernel: mserver5[3541]: segfault at 20 ip 00007fbc8ec56932 sp 00007fbc8dcad020 error 4 in lib_sql.so[7fbc8eb5b000+140000]
Feb 13 15:30:43 blrec3vm30 kernel: mserver5[8618]: segfault at 20 ip 00007f604502cf75 sp 00007f6043cdd760 error 6 in lib_sql.so[7f6044fad000+140000]
Feb 13 16:18:42 blrec3vm30 kernel: mserver5[8674]: segfault at 28 ip 00007f4b011d4e95 sp 00007f4afd212f20 error 6 in libmonetdb5.so.13[7f4b00d25000+63b000]
Feb 13 16:38:33 blrec3vm30 kernel: mserver5[8994]: segfault at 28 ip 00007f45352cce95 sp 00007f453130af20 error 6 in libmonetdb5.so.13[7f4534e1d000+63b000]
Feb 13 17:27:08 blrec3vm30 kernel: mserver5[9300]: segfault at 28 ip 00007fc87c115e95 sp 00007fc878153f20 error 6 in libmonetdb5.so.13[7fc87bc66000+63b000]
Feb 13 17:47:06 blrec3vm30 kernel: mserver5[9463]: segfault at 28 ip 00007fd5fd88ce95 sp 00007fd5f98caf20 error 6 in libmonetdb5.so.13[7fd5fd3dd000+63b000]
Feb 13 21:18:21 blrec3vm30 kernel: mserver5[8465]: segfault at 158a5000 ip 00007fd8175cb703 sp 00007fd80fbfc700 error 4 in libbat.so.9[7fd81757d000+2ee000]
Feb 13 21:35:13 blrec3vm30 kernel: mserver5[9599]: segfault at 7f2ab304b000 ip 00007f2abbcd5703 sp 00007f2ab5129700 error 4 in libbat.so.9[7f2abbc87000+2ee000]
Feb 13 21:35:52 blrec3vm30 kernel: mserver5[11622]: segfault at 5c2e000 ip 00007fa96787d703 sp 00007fa95fe48700 error 4 in libbat.so.9[7fa96782f000+2ee000]
Feb 13 21:38:01 blrec3vm30 kernel: mserver5[18981]: segfault at 9012000 ip 00007ff8db76f703 sp 00007ff8d3b39700 error 4 in libbat.so.9[7ff8db721000+2ee000]

Comment 18474

Date: 2013-02-13 17:40:24 +0100
From: @mlkersten

Hi Ashish

No, for sensible information you should use compilation with

--enable-assert --enable-debug --disable-optimize --enable-strict

Followed by a backtrace of the threads, e.g.

thr apply all where

regards, Martin

Comment 18475

Date: 2013-02-14 10:00:03 +0100
From: Ashish Kumar Singh <>

Guys,

When I was comiling with these options as suggested by you it is giving me error below any pointers please ?

/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -I. -I../common/options -I./../common/options -I../common/stream -I./../common/stream -I../common/utils -I./../common/utils -DLIBGDK -g -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wmissing-include-dirs -D_REENTRANT -c -o libbat_la-gdk_utils.lo test -f 'gdk_utils.c' || echo './'gdk_utils.c
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I. -I../common/options -I./../common/options -I../common/stream -I./../common/stream -I../common/utils -I./../common/utils -DLIBGDK -g -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wmissing-include-dirs -D_REENTRANT -c gdk_utils.c -fPIC -DPIC -o .libs/libbat_la-gdk_utils.o
cc1: warnings being treated as errors
gdk_utils.c: In function 'GDKmallocmax':
gdk_utils.c:689: warning: value computed is not used
gdk_utils.c: In function 'GDKfree_':
gdk_utils.c:750: warning: value computed is not used
gdk_utils.c: In function 'GDKreallocmax':
gdk_utils.c:812: warning: value computed is not used
gdk_utils.c:813: warning: value computed is not used
gdk_utils.c: In function 'GDKmmap':
gdk_utils.c:863: warning: value computed is not used
gdk_utils.c: In function 'GDKmunmap':
gdk_utils.c:877: warning: value computed is not used
make[3]: *** [libbat_la-gdk_utils.lo] Error 1
make[3]: Leaving directory /opt/MonetDB-11.15.1/gdk' make[2]: *** [all] Error 2 make[2]: Leaving directory /opt/MonetDB-11.15.1/gdk'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/opt/MonetDB-11.15.1'
make: *** [all] Error 2
[root@localhost MonetDB-11.15.1]

Comment 18476

Date: 2013-02-14 10:19:04 +0100
From: @drstmane

a)
which exact code version of MonetDB are we talking about? HG changeset? tarball of when from where?

b)
omit --enable-strict

Comment 18477

Date: 2013-02-14 10:21:01 +0100
From: @drstmane

Moreover, with out knowing at what your doing (e.g., which kind of workload you run on what kind of data) that does/might trigger the segfault, we cannot say much about why they happen ...

Comment 18478

Date: 2013-02-14 10:33:31 +0100
From: Ashish Kumar Singh <>

removing enable strict did the trick.

Comment 18482

Date: 2013-02-18 14:03:17 +0100
From: Ashish Kumar Singh <>

Hi ,

I have created core dump files with backtrack which shall be helpful for you.

Core was generated by `/usr/local/bin/mserver5 --dbpath=/opt/ashishtest/testfarm/pbsworksdb --set mero'.
Program terminated with signal 6, Aborted.
0 0x0000003722430285 in raise () from /lib64/libc.so.6
(gdb) thr app all bt

Thread 9 (Thread 0x2b73390a7f90 (LWP 7635)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b73383175bb in MT_sleep_ms (ms=5000) at gdk_posix.c:1003
2 0x0000000000403596 in main (argc=20, av=0x7ffff60d0d98) at mserver5.c:626

Thread 8 (Thread 7636):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b73383175bb in MT_sleep_ms (ms=50) at gdk_posix.c:1003
2 0x00002b73382446ee in GDKvmtrim (limit=0x2b733895c0d8) at gdk_utils.c:918
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 7 (Thread 7637):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b73383175bb in MT_sleep_ms (ms=1000) at gdk_posix.c:1003
2 0x00002b7337be68ea in profilerHeartbeat (dummy=0x0) at mal_profiler.c:1431
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 6 (Thread 7638):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b7337d81a39 in SERVERlistenThread (Sock=0xf339a70) at mal_mapi.c:209
---Type to continue, or q to quit---
2 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
3 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 5 (Thread 7640):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b73383175bb in MT_sleep_ms (ms=50) at gdk_posix.c:1003
2 0x00002aaaab153b30 in store_manager () at store.c:1533
3 0x00002aaaab0eb4ae in mvc_logmanager () at sql_mvc.c:149
4 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
5 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 4 (Thread 7796):
0 0x0000003722c0d9eb in read () from /lib64/libpthread.so.0
1 0x00002b7338c94959 in socket_read (s=0xf1f0d60, buf=0x411e4ec6, elmsize=2,
cnt=1) at stream.c:1455
2 0x00002b7338c97b1d in mnstr_readSht (s=0xf1f0d60, val=0x411e4ec6)
at stream.c:2867
3 0x00002b7338c96eef in bs_read (ss=0xf1e0f40, buf=0xf682760, elmsize=1,
cnt=1048320) at stream.c:2556
4 0x00002b7338c98ad5 in bstream_read (s=0xf1c0190, size=1048320)
at stream.c:3122
5 0x00002b7338c98e4e in bstream_next (s=0xf1c0190) at stream.c:3189
6 0x00002aaaab04e584 in SQLreader (c=0xe030b20) at sql_scenario.c:1216
---Type to continue, or q to quit---
7 0x00002b7337bf4d2a in runPhase (c=0xe030b20, phase=0) at mal_scenario.c:522
8 0x00002b7337bf4e07 in runScenarioBody (c=0xe030b20) at mal_scenario.c:552
9 0x00002b7337bf4fd1 in runScenario (c=0xe030b20) at mal_scenario.c:586
10 0x00002b7337bf5e5a in MSserveClient (dummy=0xe030b20) at mal_session.c:431
11 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
12 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 3 (Thread 7797):
0 0x0000003722c0cd91 in sem_wait () from /lib64/libpthread.so.0
1 0x00002b7337bcf5f1 in q_dequeue (q=0xe112dd0) at mal_dataflow.c:196
2 0x00002b7337bcf7f5 in DFLOWworker (t=0x2b73380b4f60) at mal_dataflow.c:257
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 2 (Thread 7798):
0 0x0000003722c0cd91 in sem_wait () from /lib64/libpthread.so.0
1 0x00002b7337bcf5f1 in q_dequeue (q=0xe112dd0) at mal_dataflow.c:196
2 0x00002b7337bcf7f5 in DFLOWworker (t=0x2b73380b4f68) at mal_dataflow.c:257
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x41a93940 (LWP 9131)):
0 0x0000003722430285 in raise () from /lib64/libc.so.6
---Type to continue, or q to quit---
1 0x0000003722431d30 in abort () from /lib64/libc.so.6
2 0x0000003722429706 in __assert_fail () from /lib64/libc.so.6
3 0x00002aaaab162b73 in delta_append_val (bat=0xfa398e0, i=0xf2ab300)
at bat_storage.c:319
4 0x00002aaaab162def in append_col (tr=0xfa3a200, c=0xfc03890, i=0xf2ab300,
tpe=11) at bat_storage.c:343
5 0x00002aaaab084ae4 in mvc_append_wrap (cntxt=0xe030838, mb=0xfd31980,
stk=0xf2ab170, pci=0xfca2b80) at sql.mx:3261
6 0x00002b7337bcc737 in runMALsequence (cntxt=0xe030838, mb=0xfd31980,
startpc=1, stoppc=0, stk=0xf2ab170, env=0x0, pcicaller=0x0)
at mal_interpreter.c:641
7 0x00002b7337bcbc4a in callMAL (cntxt=0xe030838, mb=0xfd31980,
env=0x41a92e48, argv=0xf439ca0, debug=0 '\000') at mal_interpreter.c:469
8 0x00002aaaab0507b7 in SQLexecutePrepared (c=0xe030838, be=0xfa30ca0,
q=0xfbc19a0) at sql_scenario.c:1773
9 0x00002aaaab050b27 in SQLengineIntern (c=0xe030838, be=0xfa30ca0)
at sql_scenario.c:1840
10 0x00002aaaab050fd8 in SQLengine (c=0xe030838) at sql_scenario.c:1941
11 0x00002b7337bf4d2a in runPhase (c=0xe030838, phase=4) at mal_scenario.c:522
12 0x00002b7337bf4ee8 in runScenarioBody (c=0xe030838) at mal_scenario.c:566
13 0x00002b7337bf4fd1 in runScenario (c=0xe030838) at mal_scenario.c:586
14 0x00002b7337bf5e5a in MSserveClient (dummy=0xe030838) at mal_session.c:431
15 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
---Type to continue, or q to quit---
16 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Comment 18483

Date: 2013-02-18 14:03:55 +0100
From: Ashish Kumar Singh <>

Created attachment 180
trace 1

Attached file: core-1 (application/octet-stream, 5483 bytes)
Description: trace 1

Comment 18484

Date: 2013-02-18 14:04:07 +0100
From: Ashish Kumar Singh <>

Created attachment 181
trace 2

Attached file: core-2 (application/octet-stream, 4021 bytes)
Description: trace 2

Comment 18485

Date: 2013-02-18 14:06:45 +0100
From: Ashish Kumar Singh <>

A description of our use case is :

One of our application component does following activities:

1. Insertion of data
2. Updation of data when updates are available
3. Deletion of data when record is stale from business perspective
4. Reading of data to calculate some derived matrices

Above can happen in parallel.

Please guide this crash issue is very critical for our successful usage of monetdb and is kind of showstopper for us.

Comment 18486

Date: 2013-02-18 14:09:49 +0100
From: Ashish Kumar Singh <>

This is using feb candidate build from http://dev.monetdb.org/downloads/testing/sources/Latest/

Comment 18487

Date: 2013-02-18 17:07:27 +0100
From: Ashish Kumar Singh <>

Just wondering if anyone can have a look on the issue , and can suggest some workarounds / fixes??

Comment 18488

Date: 2013-02-18 17:16:03 +0100
From: @sjoerdmullender

Does this still happen with the latest nightly build?
You can either get the latest version from Mercurial repository (hg clone -u Feb2013 http://dev.monetdb.org/hg/MonetDB) or you can get a pre-bootstrapped tar ball from the nightly testing (http://monetdb.cwi.nl/testweb/web/status.php, go to the highest finished run on the Feb2013 branch, click on the link on the first line that says XX files, then download the tar ball).

Comment 18489

Date: 2013-02-18 17:18:56 +0100
From: Ashish Kumar Singh <>

I took one posted in latest testing area, which was placed on 12 th of this month, are you suggesting something like this was fixed after that?

Comment 18490

Date: 2013-02-18 18:06:45 +0100
From: Ashish Kumar Singh <>

Latest is not building from feb branch!

Comment 18491

Date: 2013-02-18 18:24:03 +0100
From: @sjoerdmullender

You took the sources of the Feb2013 release candidate. After the release candidate was built there have been fixes to the Feb2013 branch from which it was built. What I want to know is whether one of those fixes fixed your issue already (in which case I may decide to build a new relase candidate).

I hope this is clear.

Comment 18492

Date: 2013-02-18 18:33:38 +0100
From: Ashish Kumar Singh <>

Hi ,

Yes this is clear , when I am trying to compile latest from February 2013 dev branch it is giving me errors below:

I am doing :

1. bootstrap
2. ./configure --enable-assert --enable-debug --disable-optimize
3. make

Change list I took is f82d192b0bf7

12th Feb release candidate was building fine though.

-I../common/stream -I./../common/stream -I../common/utils -I./../common/utils -DLIBGDK -g -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wmissing-include-dirs -D_REENTRANT -c gdk_search.c -fPIC -DPIC -o .libs/libbat_la-gdk_search.o
cc1: warnings being treated as errors
gdk_search.c: In function 'BAThash':
gdk_search.c:239: warning: value computed is not used
gdk_search.c:239: warning: value computed is not used
gdk_search.c:239: warning: value computed is not used
make[3]: *** [libbat_la-gdk_search.lo] Error 1
make[3]: Leaving directory /opt/MonetDB-f82d192b0bf7/gdk' make[2]: *** [all] Error 2 make[2]: Leaving directory /opt/MonetDB-f82d192b0bf7/gdk'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/opt/MonetDB-f82d192b0bf7'
make: *** [all] Error 2
[root@localhost MonetDB-f82d192b0bf7]

Comment 18493

Date: 2013-02-18 18:51:26 +0100
From: @sjoerdmullender

(In reply to comment 17)

Hi ,

Yes this is clear , when I am trying to compile latest from February 2013 dev
branch it is giving me errors below:

I am doing :

1. bootstrap
2. ./configure --enable-assert --enable-debug --disable-optimize
3. make

Change list I took is f82d192b0bf7

That is not the correct branch. Please use the command I gave you:

hg clone -u Feb2013 http://dev.monetdb.org/hg/MonetDB

(Or clean up everything that was created by this attempt and then do:
hg update Feb2013
./bootstrap
./configure ...
etc.)

Comment 18494

Date: 2013-02-18 18:56:46 +0100
From: @grobian

(In reply to comment 13)

Does this still happen with the latest nightly build?
You can either get the latest version from Mercurial repository (hg clone -u
Feb2013 http://dev.monetdb.org/hg/MonetDB) or you can get a pre-bootstrapped
tar ball from the nightly testing
(http://monetdb.cwi.nl/testweb/web/status.php, go to the highest finished run
on the Feb2013 branch, click on the link on the first line that says XX files,
then download the tar ball).

or just
http://monetdb.cwi.nl/testweb/web/MonetDB-Feb2013-latest.tar.bz2

Comment 18495

Date: 2013-02-18 19:15:32 +0100
From: Ashish Kumar Singh <>

Thanks! now tests are running will let you know about outcome.

Comment 18496

Date: 2013-02-18 19:27:36 +0100
From: Ashish Kumar Singh <>

No Luck, Houston we have a problem!!

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x407f8940 (LWP 1865)]
0x0000003722430285 in raise () from /lib64/libc.so.6
(gdb) thr app all bt

Thread 46 (Thread 0x407f8940 (LWP 1865)):
0 0x0000003722430285 in raise () from /lib64/libc.so.6
1 0x0000003722431d30 in abort () from /lib64/libc.so.6
2 0x0000003722429706 in __assert_fail () from /lib64/libc.so.6
3 0x00002aaaab16516f in tr_update_delta (tr=0x15fcdef0, obat=0x16093910,
cbat=0x161ddee0, snapshot_minsize=1024) at bat_storage.c:1441
4 0x00002aaaab165d0f in update_table (tr=0x15fcdef0, ft=0x163a4dd0,
tt=0x16084930) at bat_storage.c:1563
5 0x00002aaaab1567dd in rollforward_update_table (tr=0x15fcdef0,
ft=0x163a4dd0, tt=0x16084930, mode=3) at store.c:2671
6 0x00002aaaab155789 in rollforward_changeset_updates (tr=0x15fcdef0,
fs=0x16383d00, ts=0x15ff0270, b=0x15ff0250,
rollforward_updates=0x2aaaab15644d <rollforward_update_table>,
rollforward_creates=0x2aaaab155d62 <rollforward_create_table>,
rollforward_deletes=0x2aaaab156248 <rollforward_drop_table>,
fd=0x2aaaab1568b7 <conditional_table_dup>, mode=3) at store.c:2284
7 0x00002aaaab156c1f in rollforward_update_schema (tr=0x15fcdef0,
fs=0x16383ce0, ts=0x15ff0250, mode=3) at store.c:2746
8 0x00002aaaab155789 in rollforward_changeset_updates (tr=0x15fcdef0,
fs=0x15fcdf20, ts=0x161c6720, b=0x161c66f0,
rollforward_updates=0x2aaaab1569a8 <rollforward_update_schema>,
rollforward_creates=0x2aaaab156403 <rollforward_create_schema>,
---Type to continue, or q to quit---
rollforward_deletes=0x2aaaab156397 <rollforward_drop_schema>,
fd=0x2aaaab154ebd <schema_dup>, mode=3) at store.c:2284
9 0x00002aaaab156dc1 in rollforward_trans (tr=0x15fcdef0, mode=3)
at store.c:2768
10 0x00002aaaab1581f4 in sql_trans_commit (tr=0x15fcdef0) at store.c:3170
11 0x00002aaaab0ea742 in mvc_commit (m=0x1643d390, chain=0, name=0x0)
at sql_mvc.c:267
12 0x00002aaaab04d0da in SQLautocommit (c=0x14dec838, m=0x1643d390)
at sql_scenario.c:1116
13 0x00002aaaab04d314 in SQLreader (c=0x14dec838) at sql_scenario.c:1204
14 0x00002af73ca6cad2 in runPhase (c=0x14dec838, phase=0)
at mal_scenario.c:522
15 0x00002af73ca6cbaf in runScenarioBody (c=0x14dec838) at mal_scenario.c:552
16 0x00002af73ca6cd79 in runScenario (c=0x14dec838) at mal_scenario.c:586
17 0x00002af73ca6dc02 in MSserveClient (dummy=0x14dec838) at mal_session.c:431
18 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
19 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x41edf940 (LWP 1130)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002af73d18a27f in MT_sleep_ms (ms=50) at gdk_posix.c:1003
2 0x00002af73d0b73b2 in GDKvmtrim (limit=0x2af73d7ce358) at gdk_utils.c:918
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
---Type to continue, or q to quit---
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x40312940 (LWP 1131)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002af73d18a27f in MT_sleep_ms (ms=1000) at gdk_posix.c:1003
2 0x00002af73ca5e692 in profilerHeartbeat (dummy=0x0) at mal_profiler.c:1431
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x420e0940 (LWP 1132)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002af73cbf97e1 in SERVERlistenThread (Sock=0x160f5a70)
at mal_mapi.c:209
2 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
3 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x4194b940 (LWP 1133)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002af73d18a27f in MT_sleep_ms (ms=50) at gdk_posix.c:1003
2 0x00002aaaab152a90 in store_manager () at store.c:1533
3 0x00002aaaab0ea222 in mvc_logmanager () at sql_mvc.c:149
4 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
5 0x00000037224d3c1d in clone () from /lib64/libc.so.6
---Type to continue, or q to quit---

Thread 3 (Thread 0x41631940 (LWP 1215)):
0 0x0000003722c0cd91 in sem_wait () from /lib64/libpthread.so.0
1 0x00002af73ca473cd in q_dequeue (q=0x14eb4010) at mal_dataflow.c:196
2 0x00002af73ca475d1 in DFLOWworker (t=0x2af73cf2bf40) at mal_dataflow.c:257
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x422e1940 (LWP 1216)):
0 0x0000003722c0cd91 in sem_wait () from /lib64/libpthread.so.0
1 0x00002af73ca473cd in q_dequeue (q=0x14eb4010) at mal_dataflow.c:196
2 0x00002af73ca475d1 in DFLOWworker (t=0x2af73cf2bf48) at mal_dataflow.c:257
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2af73df16f30 (LWP 1129)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002af73d18a27f in MT_sleep_ms (ms=5000) at gdk_posix.c:1003
2 0x0000000000403596 in main (argc=20, av=0x7fffaff24338) at mserver5.c:626
(gdb)

Comment 18497

Date: 2013-02-18 19:37:49 +0100
From: @njnes

did your last run start with a clean database or some older (and therefor possibly corrupt) db? Just to exclude this case..

Comment 18498

Date: 2013-02-18 19:47:05 +0100
From: Ashish Kumar Singh <>

That was with OLD database:

Below is the crash with a fresh DB farm :

(gdb) thr app all bt

Thread 40 (Thread 0x4201a940 (LWP 4668)):
0 0x0000003722430285 in raise () from /lib64/libc.so.6
1 0x0000003722431d30 in abort () from /lib64/libc.so.6
2 0x0000003722429706 in __assert_fail () from /lib64/libc.so.6
3 0x00002aaaab161b40 in delta_append_val (bat=0x1d14e310, i=0x1e2e3350) at bat_storage.c:327
4 0x00002aaaab161dbc in append_col (tr=0x1e0e3ed0, c=0x1e61f150, i=0x1e2e3350, tpe=11) at bat_storage.c:351
5 0x00002aaaab08389c in mvc_append_wrap (cntxt=0x1cd07768, mb=0x1d31f040, stk=0x1e2e31c0, pci=0x1e063620) at sql.mx:3261
6 0x00002b502b61e515 in runMALsequence (cntxt=0x1cd07768, mb=0x1d31f040, startpc=1, stoppc=0, stk=0x1e2e31c0, env=0x0, pcicaller=0x0)
at mal_interpreter.c:642
7 0x00002b502b61da4e in callMAL (cntxt=0x1cd07768, mb=0x1d31f040, env=0x42019e48, argv=0x1cddd9a0, debug=0 '\000') at mal_interpreter.c:469
8 0x00002aaaab04f666 in SQLexecutePrepared (c=0x1cd07768, be=0x1ce5a000, q=0x1df76380) at sql_scenario.c:1779
9 0x00002aaaab04f9d6 in SQLengineIntern (c=0x1cd07768, be=0x1ce5a000) at sql_scenario.c:1846
10 0x00002aaaab04fe98 in SQLengine (c=0x1cd07768) at sql_scenario.c:1947
11 0x00002b502b646ad2 in runPhase (c=0x1cd07768, phase=4) at mal_scenario.c:522
12 0x00002b502b646c90 in runScenarioBody (c=0x1cd07768) at mal_scenario.c:566
13 0x00002b502b646d79 in runScenario (c=0x1cd07768) at mal_scenario.c:586
14 0x00002b502b647c02 in MSserveClient (dummy=0x1cd07768) at mal_session.c:431
15 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
16 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x40cac940 (LWP 4190)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b502bd6427f in MT_sleep_ms (ms=50) at gdk_posix.c:1003
2 0x00002b502bc913b2 in GDKvmtrim (limit=0x2b502c3a8358) at gdk_utils.c:918
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x41e19940 (LWP 4191)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b502bd6427f in MT_sleep_ms (ms=1000) at gdk_posix.c:1003
2 0x00002b502b638692 in profilerHeartbeat (dummy=0x0) at mal_profiler.c:1431
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x40a0a940 (LWP 4192)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b502b7d37e1 in SERVERlistenThread (Sock=0x1e010e30) at mal_mapi.c:209
2 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
3 0x00000037224d3c1d in clone () from /lib64/libc.so.6
---Type to continue, or q to quit---

Thread 4 (Thread 0x417e4940 (LWP 4193)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b502bd6427f in MT_sleep_ms (ms=50) at gdk_posix.c:1003
2 0x00002aaaab152a90 in store_manager () at store.c:1533
3 0x00002aaaab0ea222 in mvc_logmanager () at sql_mvc.c:149
4 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
5 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x41bb3940 (LWP 4227)):
0 0x0000003722c0cd91 in sem_wait () from /lib64/libpthread.so.0
1 0x00002b502b6213cd in q_dequeue (q=0x1ce63820) at mal_dataflow.c:196
2 0x00002b502b6215d1 in DFLOWworker (t=0x2b502bb05f40) at mal_dataflow.c:257
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x4221b940 (LWP 4228)):
0 0x0000003722c0cd91 in sem_wait () from /lib64/libpthread.so.0
1 0x00002b502b6213cd in q_dequeue (q=0x1ce63820) at mal_dataflow.c:196
2 0x00002b502b6215d1 in DFLOWworker (t=0x2b502bb05f48) at mal_dataflow.c:257
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2b502caf0f30 (LWP 4189)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b502bd6427f in MT_sleep_ms (ms=5000) at gdk_posix.c:1003
2 0x0000000000403596 in main (argc=20, av=0x7ffff9ec17e8) at mserver5.c:626

Comment 18499

Date: 2013-02-18 20:03:17 +0100
From: @drstmane

Ashish,

in your debugger, once the assertion happens, could you please go to the very thread that it happened in (thread 40 in your latest trace), there "up" to the function where the assertion was triggered ("delta_append_val()"), execute the following print commands, and share their output:

p c
p *c
p *c->U
p *c->P
p bat
p *bat
p bat->ibase

Thanks!

Comment 18500

Date: 2013-02-18 20:12:04 +0100
From: Ashish Kumar Singh <>

HI Neil,

That session got closed , and a new crash is showing different log: Do you want me to run same commands and share output?

0 0x0000003722430285 in raise () from /lib64/libc.so.6
1 0x0000003722431d30 in abort () from /lib64/libc.so.6
2 0x0000003722429706 in __assert_fail () from /lib64/libc.so.6
3 0x00002aaaab16516f in tr_update_delta (tr=0x1446fb30, obat=0x13ef8410,
cbat=0x13eba370, snapshot_minsize=1024) at bat_storage.c:1441
4 0x00002aaaab165d0f in update_table (tr=0x1446fb30, ft=0x14945ef0,
tt=0x1444c3c0) at bat_storage.c:1563
5 0x00002aaaab1567dd in rollforward_update_table (tr=0x1446fb30,
ft=0x14945ef0, tt=0x1444c3c0, mode=3) at store.c:2671
6 0x00002aaaab155789 in rollforward_changeset_updates (tr=0x1446fb30,
fs=0x14932f00, ts=0x14095200, b=0x140951e0,
rollforward_updates=0x2aaaab15644d <rollforward_update_table>,
rollforward_creates=0x2aaaab155d62 <rollforward_create_table>,
rollforward_deletes=0x2aaaab156248 <rollforward_drop_table>,
fd=0x2aaaab1568b7 <conditional_table_dup>, mode=3) at store.c:2284
7 0x00002aaaab156c1f in rollforward_update_schema (tr=0x1446fb30,
fs=0x14932ee0, ts=0x140951e0, mode=3) at store.c:2746
8 0x00002aaaab155789 in rollforward_changeset_updates (tr=0x1446fb30,
fs=0x1446fb60, ts=0x14151360, b=0x14151330,
rollforward_updates=0x2aaaab1569a8 <rollforward_update_schema>,
rollforward_creates=0x2aaaab156403 <rollforward_create_schema>,
---Type to continue, or q to quit---
rollforward_deletes=0x2aaaab156397 <rollforward_drop_schema>,
fd=0x2aaaab154ebd <schema_dup>, mode=3) at store.c:2284
9 0x00002aaaab156dc1 in rollforward_trans (tr=0x1446fb30, mode=3)
at store.c:2768
10 0x00002aaaab1581f4 in sql_trans_commit (tr=0x1446fb30) at store.c:3170
11 0x00002aaaab0ea742 in mvc_commit (m=0x143a7190, chain=0, name=0x0)
at sql_mvc.c:267
12 0x00002aaaab04d0da in SQLautocommit (c=0x12dd90d0, m=0x143a7190)
at sql_scenario.c:1116
13 0x00002aaaab04d314 in SQLreader (c=0x12dd90d0) at sql_scenario.c:1204
14 0x00002b2a4729bad2 in runPhase (c=0x12dd90d0, phase=0)
at mal_scenario.c:522
15 0x00002b2a4729bbaf in runScenarioBody (c=0x12dd90d0) at mal_scenario.c:552
16 0x00002b2a4729bd79 in runScenario (c=0x12dd90d0) at mal_scenario.c:586
17 0x00002b2a4729cc02 in MSserveClient (dummy=0x12dd90d0) at mal_session.c:431
18 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
19 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 56 (Thread 0x40cea940 (LWP 7012)):
0 0x0000003722c0d9eb in read () from /lib64/libpthread.so.0
1 0x00002b2a48334073 in socket_read (s=0x146dea00, buf=0x40ce9ebe,
elmsize=2, cnt=1) at stream.c:1455
2 0x00002b2a483369f7 in mnstr_readSht (s=0x146dea00, val=0x40ce9ebe)
---Type to continue, or q to quit---
at stream.c:2867
3 0x00002b2a48335ff9 in bs_read (ss=0x1452eb20, buf=0x1424dbd0, elmsize=1,
cnt=1048320) at stream.c:2556
4 0x00002b2a48337779 in bstream_read (s=0x143a9960, size=1048320)
at stream.c:3122
5 0x00002b2a48337aac in bstream_next (s=0x143a9960) at stream.c:3189
6 0x00002aaaab04d3b5 in SQLreader (c=0x12dd8de8) at sql_scenario.c:1217
7 0x00002b2a4729bad2 in runPhase (c=0x12dd8de8, phase=0)
at mal_scenario.c:522
8 0x00002b2a4729bbaf in runScenarioBody (c=0x12dd8de8) at mal_scenario.c:552
9 0x00002b2a4729bd79 in runScenario (c=0x12dd8de8) at mal_scenario.c:586
10 0x00002b2a4729cc02 in MSserveClient (dummy=0x12dd8de8) at mal_session.c:431
11 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
12 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x4128d940 (LWP 6804)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b2a479b927f in MT_sleep_ms (ms=50) at gdk_posix.c:1003
2 0x00002b2a478e63b2 in GDKvmtrim (limit=0x2b2a47ffd358) at gdk_utils.c:918
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x41ad0940 (LWP 6805)):
---Type to continue, or q to quit---
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b2a479b927f in MT_sleep_ms (ms=1000) at gdk_posix.c:1003
2 0x00002b2a4728d692 in profilerHeartbeat (dummy=0x0) at mal_profiler.c:1431
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x41eed940 (LWP 6806)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b2a474287e1 in SERVERlistenThread (Sock=0x140e2030)
at mal_mapi.c:209
2 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
3 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x4148e940 (LWP 6807)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b2a479b927f in MT_sleep_ms (ms=50) at gdk_posix.c:1003
2 0x00002aaaab152a90 in store_manager () at store.c:1533
3 0x00002aaaab0ea222 in mvc_logmanager () at sql_mvc.c:149
4 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
5 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x408e8940 (LWP 6819)):
0 0x0000003722c0cd91 in sem_wait () from /lib64/libpthread.so.0
---Type to continue, or q to quit---
1 0x00002b2a472763cd in q_dequeue (q=0x143992d0) at mal_dataflow.c:196
2 0x00002b2a472765d1 in DFLOWworker (t=0x2b2a4775af40) at mal_dataflow.c:257
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x40ae9940 (LWP 6820)):
0 0x0000003722c0cd91 in sem_wait () from /lib64/libpthread.so.0
1 0x00002b2a472763cd in q_dequeue (q=0x143992d0) at mal_dataflow.c:196
2 0x00002b2a472765d1 in DFLOWworker (t=0x2b2a4775af48) at mal_dataflow.c:257
3 0x0000003722c0677d in start_thread () from /lib64/libpthread.so.0
4 0x00000037224d3c1d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2b2a48745f30 (LWP 6803)):
0 0x00000037224ccdb2 in select () from /lib64/libc.so.6
1 0x00002b2a479b927f in MT_sleep_ms (ms=5000) at gdk_posix.c:1003
2 0x0000000000403596 in main (argc=20, av=0x7fff8c0b68c8) at mserver5.c:626

Comment 18501

Date: 2013-02-18 20:15:41 +0100
From: @njnes

No need now. Lets first have a look at the java code you have..

Comment 18502

Date: 2013-02-18 20:17:01 +0100
From: Ashish Kumar Singh <>

Created attachment 182
Java Test Case Sources

Make Changes for DB connections in ConfigurationDBConnection.Java Class

Create Schema using schema file

Run com.altair.test.MonetDBLoadTestBasedOnQueueAndQueryFrequently class

Attached file: TestCaseSources.zip (application/x-zip-compressed, 470563 bytes)
Description: Java Test Case Sources

Comment 18503

Date: 2013-02-18 20:17:25 +0100
From: Ashish Kumar Singh <>

Created attachment 183
Schema File

Attached file: schema.txt (text/plain, 337 bytes)
Description: Schema File

Comment 18504

Date: 2013-02-18 20:18:10 +0100
From: Ashish Kumar Singh <>

(In reply to comment 26)

No need now. Lets first have a look at the java code you have..

Shared Let me know if you need an online skype meeting to get you up to speed with this.

Comment 18522

Date: 2013-02-19 13:45:45 +0100
From: Ashish Kumar Singh <>

Hi,

Any thoughts on this issue? Work arounds / possible time for fixes?

Regards,
Ashish

Comment 18528

Date: 2013-02-20 11:08:51 +0100
From: Ashish Kumar Singh <>

Hi team,

One of our appplication release is held waiting for this issue and we are pressed on time, we are ready to try out any work arounds if you can suggest which can help avoiding this issue as an intermediate step.

Any suggestions ill be helpful! Thanks in advance!

Regards,
Ashish

Comment 18529

Date: 2013-02-20 11:10:24 +0100
From: Ashish Kumar Singh <>

Corrected typo below

(In reply to comment 31)

Hi team,

One of our appplication release is held waiting for this issue and we are
pressed on time, we are ready to try out any work arounds if you can suggest
which can help avoiding this issue as an intermediate step.

Any suggestions will be helpful! Thanks in advance!

Regards,
Ashish

Comment 18530

Date: 2013-02-20 12:04:27 +0100
From: @mlkersten

Dear Ashish

I remind you that you are relying on the MonetDB as open-source project
without any commercial commitment imposed upon us. We deal with issues
as best as we can. The risks for deploying the code base rest at the user
as clearly identified in its license.
http://www.monetdb.org/Legal/MonetDBLicense

Thank you for reporting the issues. If fixes are required they will appear
regular bug-fix releases.

regards, Martin Kersten

Comment 18531

Date: 2013-02-20 14:27:23 +0100
From: Ashish Kumar Singh <>

Dear Martin,

I understand it. Your technology was surely very useful for us!
Regarding this bug I was interested to see if we can work around this particular issue or not?

Regards,
Ashish

Comment 18532

Date: 2013-02-21 08:28:15 +0100
From: Ashish Kumar Singh <>

Dear Niels and all,

I ran a slightly modified tests today which has not crashed so far in 10 hours, description is below:

1. removed use of sequences from schema in the table set default value for roe_id to 1, result is no crash.
2. removed use of sequences from schema in the table set default value for roe_id to 1, while inserting data I am using my java code to create an auto increment value and use it in insert statement,result no crash so far.

Observations:
Earlier we use to get crash with in 10-20 minutes, now for more than 10 hours no crash.
Insert performance seems to increase by avoiding sequences.

I hope this info might be useful in digging down the issue. Please share your thoughts, findings , suggestions on the same.

Regards,
Ashish

Comment 18533

Date: 2013-02-21 12:47:30 +0100
From: @njnes

Changeset ba7ad0186586 made by Niels Nes niels@cwi.nl in the MonetDB repo, refers to this bug.

For complete details, see http//devmonetdborg/hg/MonetDB?cmd=changeset;node=ba7ad0186586

Changeset description:

way more stricked table_validate. This will heart concurrency but solves
the continues story with inconsistent dbs.
See also bug #3233

Comment 18540

Date: 2013-02-22 08:10:35 +0100
From: Ashish Kumar Singh <>

Hi Niels,

Fix works great, straight 10 hours no crash so far. Will it be possible to have a February 2013 SP1 with this fix any time soon, I think it's important as it was leading to invalid database state/corruption?

Regards,
Ashish

Comment 18565

Date: 2013-02-25 16:12:41 +0100
From: @sjoerdmullender

I assume that the bug is now fixed.
I hope to be able to create an SP1 this week, but it depends on some other bugs that I still want to see fixed.

Comment 18591

Date: 2013-03-07 12:41:22 +0100
From: @sjoerdmullender

Feb2013-SP1 has been released.