Date: 2013-09-30 10:44:08 +0200
From: Christian Braun <>
To: SQL devs <>
Version: 11.15.15 (Feb2013-SP4)
CC: ajdamico, @hannesmuehleisen

Last updated: 2013-10-22 14:42:08 +0200

Comment 19212

Date: 2013-09-30 10:44:08 +0200
From: Christian Braun <>

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0
Build Identifier:

The error started after updating from Feb2013-SP3 to SP4.

COPY 879429 RECORDS INTO qwe.qwe FROM '/var/tmp/mito_fDop54' USING DELIMITERS '\t','\n';
ERROR = !value from line 1188 field 2 not inserted: !ERROR: HEAPextend: failed to extend to 17592186044416 for 60/34/603401.tail: MT_mremap() failed

Attached are the log files from the client and server. The failing field in the client.log is of type varchar(255).

2013-09-30 04:09:06 MSG merovingian[17461]: database 'db' (14996) was killed by signal SIGBUS
This crash happened after MonetDB filled up all available hard drive space (0.5TB). On restart the the space was released.

Reproducible: Always

Steps to Reproduce:

1.CREATE TABLE
2.run 50 times COPY 1000000 RECORDS INTO
3.COPY INTO fails with HEAPExtend

Comment 19213

Date: 2013-09-30 10:45:13 +0200
From: Christian Braun <>

Created attachment 236
client log file

Attached file: client.log (application/octet-stream, 1892 bytes)
Description: client log file

Comment 19214

Date: 2013-09-30 10:45:30 +0200
From: Christian Braun <>

Created attachment 237
server log file

Attached file: merovingian.log (application/octet-stream, 6060 bytes)
Description: server log file

Comment 19218

Date: 2013-09-30 12:35:30 +0200
From: @hannesmuehleisen

What is your table structure? Could you provide sample data?

Comment 19221

Date: 2013-09-30 15:02:09 +0200
From: Christian Braun <>

Created attachment 238
script to trigger to the bug

Attached file: import.sh (text/x-sh, 786 bytes)
Description: script to trigger to the bug

Comment 19222

Date: 2013-09-30 15:07:46 +0200
From: Christian Braun <>

The attached script creates the table and test data. After about 50 copy into statements the HEAPextend failure happens.

Comment 19223

Date: 2013-09-30 16:25:32 +0200
From: @sjoerdmullender

The problem is in the use of the heap cache.
The simple solution is to disable the heap cache by setting HEAP_CACHE_SIZE to 0 in gdk_heap.c and recompiling.

To elaborate on the problem, the query causes a heap to be used from the heap cache. This heap is used as the offset heap of a string bat. The string bat is appended to and because of that, the offset heap has to be upgraded to a wider offset heap. The function doing the upgrading doubles (or in the case we saw, quadrupled) the size of the heap. Afterwards the bat is freed and the heap given back to the cache. This cycle repeats a few times, and so the file in the heap cache grows exponentially. Eventually it grows so large that the system can't memory map it anymore and the server gives an error message (and sometime it crashes).

Comment 19225

Date: 2013-09-30 16:49:38 +0200
From: @hannesmuehleisen

*** Bug #3379 has been marked as a duplicate of this bug. ***

Comment 19226

Date: 2013-09-30 17:06:15 +0200
From: MonetDB Mercurial Repository <>

Changeset 49cc8f100cb4 made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see http//devmonetdborg/hg/MonetDB?cmd=changeset;node=49cc8f100cb4

Changeset description:

Get rid of the heap cache.
This fixes bug #3376 and relates to bug #3323.

In bug #3323, comment 15, I already remarked that using the heap cache
is actually slower on Linux than not using the heap cache.

In addition to this, using the heap cache can be very detrimental as
witnessed by bug #3376.  What happens there is that a new string bat is
created and then appended to.  The appended string bat is big enough
so that the offset heap needs to be memory mapped, so the heap cache
is used to find a usable heap.  In this case, there is only one file
there, so it is used.  BATappend then needs to widen the offset heap,
so it calls GDKupgradevarheap.  That function doubles (or in this
case, quadruples) the size of the heap it was given since it doesn't
know the required BAT capacity.  Then the BAT is freed and the heap
returned to the cache.  This cycle repeats a few times, and this
repetition causes the heap to grow exponentially, eventually leading
to the errors seen in the bug report.

Comment 19255

Date: 2013-10-08 11:28:32 +0200
From: @hannesmuehleisen

Fixed by removing the heap cache.