Date: 2013-02-15 17:50:19 +0100
From: Anthony Damico <>
To: SQL devs <>
Version: 11.15.7 (Feb2013-SP2)
CC: ajdamico, @njnes, @drstmane

Last updated: 2013-06-10 13:33:00 +0200

Comment 18479

Date: 2013-02-15 17:50:19 +0100
From: Anthony Damico <>

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0
Build Identifier:

I am running MonetDB5 Oct 2012 SP3 on my work Windows 7 desktop computer. My office has what I imagine is a pretty typical mix of networked machines, with a few more powerful servers and larger shared drives that everyone can connect to. (1) When I attempt to import a large CSV file into a MonetDB server on the local disk (C:\My Directory\MonetDB\dbfarm), the importation command runs as expected. (2) When I attempt to import the same CSV file to a MonetDB server stored on a networked drive, the import command fails. (3) Finally, when I remote desktop into the server that the networked drive is actually attached to, the importation succeeds again! Here's the error seen in scenario (2) --

GDKerror:!ERROR: GDKunlink(bat\13\1327.tail)
!OS: The process cannot access the file because it is being used by another process.
GDKunlink(bat\13\1327.theap)
!OS: The process cannot access the file because it is being used by another process.

So it seems pretty clear that something about windows networked drives is the culprit. This error does not occur on every file (larger files seem more prone to the error), but if an import command does not work on a file, it consistently does not work on that file. As an example, importing this large file works with methods (1) and (3) but not (2) --

http://downloads.cms.gov/BSAPUF/2008_BSA_Carrier_Line_Items_PUF_1.zip

I tried disabling the windows firewall and using different server ports, but I still hit the above error.

I spoke with Hannes about it briefly, he said: "The issue is probably related to the Windows mmap() implementation when talking to a SMB filesystem. It is far from certain that there is something we can do about this, but in either case, we'd like to know about it."

Thanks!!!

Reproducible: Always

Steps to Reproduce:

These are the three different scenarios described in "Details" -- (1) and (3) work, (2) failed.

(1) - local disk - works -----

fetch next block: start at:196
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
Database: MonetDB v11.13.9 (Oct2012-SP3), 'bsa'
closing result set
Type \q to quit, ? for a list of available commands
auto commit mode: on
mapi_query:46:SET TIME ZONE INTERVAL '-05:00' HOUR TO MINUTE
fetch next block: start at:198
got next block: length:3
text:&3

got complete block:
text:&3

read_line:&3
allocating new result set
fetch next block: start at:201
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
closing result set
sql>create table mytable (
mapi_query_part:23:create table mytable (

fetch next block: start at:203
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:206
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_id VARCHAR(255),
mapi_query_part:27: car_line_id VARCHAR(255),

fetch next block: start at:208
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:211
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> bene_sex_ident_cd INTEGER,
mapi_query_part:28: bene_sex_ident_cd INTEGER,

fetch next block: start at:213
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:216
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> bene_age_cat_cd INTEGER,
mapi_query_part:26: bene_age_cat_cd INTEGER,

fetch next block: start at:218
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:221
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_icd9_dgns_cd VARCHAR(255),
mapi_query_part:37: car_line_icd9_dgns_cd VARCHAR(255),

fetch next block: start at:223
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:226
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_hcpcs_cd VARCHAR(255),
mapi_query_part:33: car_line_hcpcs_cd VARCHAR(255),

fetch next block: start at:228
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:231
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_betos_cd VARCHAR(255),
mapi_query_part:33: car_line_betos_cd VARCHAR(255),

fetch next block: start at:233
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:236
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_srvc_cnt INTEGER,
mapi_query_part:28: car_line_srvc_cnt INTEGER,

fetch next block: start at:238
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:241
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_prvdr_type_cd INTEGER,
mapi_query_part:33: car_line_prvdr_type_cd INTEGER,

fetch next block: start at:243
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:246
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_cms_type_srvc_cd VARCHAR(255),
mapi_query_part:41: car_line_cms_type_srvc_cd VARCHAR(255),

fetch next block: start at:248
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:251
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_place_of_srvc_cd INTEGER,
mapi_query_part:36: car_line_place_of_srvc_cd INTEGER,

fetch next block: start at:253
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:256
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_hcpcs_pmt_amt INTEGER
mapi_query_part:27: car_hcpcs_pmt_amt INTEGER

fetch next block: start at:258
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:261
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more>) ;
mapi_query_part:4:) ;

fetch next block: start at:263
got next block: length:3
text:&3

got complete block:
text:&3

read_line:&3
allocating new result set
fetch next block: start at:266
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
operation successful (118.294ms)
closing result set
sql>
fetch next block: start at:268
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
sql>COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Directory\BSAPUF\2008
\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n','"' NULL AS '
' ;
mapi_query_part:160:COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Direc
tory\BSAPUF\2008\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n
','"' NULL AS '' ;

fetch next block: start at:270
got next block: length:14
text:&2 9676439 -1

got complete block:
text:&2 9676439 -1

read_line:&2 9676439 -1
allocating new result set
fetch next block: start at:284
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
9676439 affected rows (3m 26s)
closing result set
sql>

(2) - networked disk - fails -----

read_line:☺
more> car_line_id VARCHAR(255),
mapi_query_part:27: car_line_id VARCHAR(255),

fetch next block: start at:208
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:211
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> bene_sex_ident_cd INTEGER,
mapi_query_part:28: bene_sex_ident_cd INTEGER,

fetch next block: start at:213
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:216
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> bene_age_cat_cd INTEGER,
mapi_query_part:26: bene_age_cat_cd INTEGER,

fetch next block: start at:218
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:221
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_icd9_dgns_cd VARCHAR(255),
mapi_query_part:37: car_line_icd9_dgns_cd VARCHAR(255),

fetch next block: start at:223
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:226
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_hcpcs_cd VARCHAR(255),
mapi_query_part:33: car_line_hcpcs_cd VARCHAR(255),

fetch next block: start at:228
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:231
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_betos_cd VARCHAR(255),
mapi_query_part:33: car_line_betos_cd VARCHAR(255),

fetch next block: start at:233
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:236
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_srvc_cnt INTEGER,
mapi_query_part:28: car_line_srvc_cnt INTEGER,

fetch next block: start at:238
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:241
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_prvdr_type_cd INTEGER,
mapi_query_part:33: car_line_prvdr_type_cd INTEGER,

fetch next block: start at:243
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:246
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_cms_type_srvc_cd VARCHAR(255),
mapi_query_part:41: car_line_cms_type_srvc_cd VARCHAR(255),

fetch next block: start at:248
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:251
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_place_of_srvc_cd INTEGER,
mapi_query_part:36: car_line_place_of_srvc_cd INTEGER,

fetch next block: start at:253
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:256
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_hcpcs_pmt_amt INTEGER
mapi_query_part:27: car_hcpcs_pmt_amt INTEGER

fetch next block: start at:258
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:261
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more>) ;
mapi_query_part:4:) ;

fetch next block: start at:263
got next block: length:3
text:&3

got complete block:
text:&3

read_line:&3
allocating new result set
fetch next block: start at:266
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
operation successful (289.145ms)
closing result set
sql>
fetch next block: start at:268
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
sql>COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Directory\BSAPUF\2008
\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n','"' NULL AS '
' ;
mapi_query_part:160:COPY 9676440 offset 2 records INTO mytable FROM 'C:\My Direc
tory\BSAPUF\2008\2008_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n
','"' NULL AS '' ;

fetch next block: start at:270
got next block: length:252
text:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!GDKunlink(bat\13\1327.theap)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!

got complete block:
text:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!GDKunlink(bat\13\1327.theap)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!

read_line:!GDKerror:!ERROR: GDKunlink(bat\13\1327.tail)
allocating new result set
got complete block:
text:!!OS: The process cannot access the file because it is being used by anothe
r process.
!GDKunlink(bat\13\1327.theap)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!

read_line:!!OS: The process cannot access the file because it is being used by a
nother process.
got complete block:
text:!GDKunlink(bat\13\1327.theap)
!!OS: The process cannot access the file because it is being used by another pro
cess.
!

read_line:!GDKunlink(bat\13\1327.theap)
got complete block:
text:!!OS: The process cannot access the file because it is being used by anothe
r process.
!

read_line:!!OS: The process cannot access the file because it is being used by a
nother process.
got complete block:
text:!

read_line:!
fetch next block: start at:522
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
GDKerror:!ERROR: GDKunlink(bat\13\1327.tail)
!OS: The process cannot access the file because it is being used by another proc
ess.
GDKunlink(bat\13\1327.theap)
!OS: The process cannot access the file because it is being used by another proc
ess.

closing result set
sql>

(3) - remote desktop into server so networked disk becomes local disk - works -----

read_line:[ "monet_release", "Oct2012-SP3" ]
fetch next block: start at:196
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
Database: MonetDB v11.13.9 (Oct2012-SP3), 'bsa'
closing result set
Type \q to quit, ? for a list of available commands
auto commit mode: on
mapi_query:46:SET TIME ZONE INTERVAL '-05:00' HOUR TO MINUTE
fetch next block: start at:198
got next block: length:3
text:&3

got complete block:
text:&3

read_line:&3
allocating new result set
fetch next block: start at:201
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
closing result set
sql>create table mytable3 (
mapi_query_part:24:create table mytable3 (

fetch next block: start at:203
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:206
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_id VARCHAR(255),
mapi_query_part:27: car_line_id VARCHAR(255),

fetch next block: start at:208
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:211
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> bene_sex_ident_cd INTEGER,
mapi_query_part:28: bene_sex_ident_cd INTEGER,

fetch next block: start at:213
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:216
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> bene_age_cat_cd INTEGER,
mapi_query_part:26: bene_age_cat_cd INTEGER,

fetch next block: start at:218
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:221
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_icd9_dgns_cd VARCHAR(255),
mapi_query_part:37: car_line_icd9_dgns_cd VARCHAR(255),

fetch next block: start at:223
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:226
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_hcpcs_cd VARCHAR(255),
mapi_query_part:33: car_line_hcpcs_cd VARCHAR(255),

fetch next block: start at:228
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:231
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_betos_cd VARCHAR(255),
mapi_query_part:33: car_line_betos_cd VARCHAR(255),

fetch next block: start at:233
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:236
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_srvc_cnt INTEGER,
mapi_query_part:28: car_line_srvc_cnt INTEGER,

fetch next block: start at:238
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:241
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_prvdr_type_cd INTEGER,
mapi_query_part:33: car_line_prvdr_type_cd INTEGER,

fetch next block: start at:243
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:246
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_cms_type_srvc_cd VARCHAR(255),
mapi_query_part:41: car_line_cms_type_srvc_cd VARCHAR(255),

fetch next block: start at:248
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:251
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_line_place_of_srvc_cd INTEGER,
mapi_query_part:36: car_line_place_of_srvc_cd INTEGER,

fetch next block: start at:253
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:256
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more> car_hcpcs_pmt_amt INTEGER
mapi_query_part:27: car_hcpcs_pmt_amt INTEGER

fetch next block: start at:258
got next block: length:3
text:☺☻

got complete block:
text:☺☻

read_line:☺☻
fetch next block: start at:261
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
more>) ;
mapi_query_part:4:) ;

fetch next block: start at:263
got next block: length:3
text:&3

got complete block:
text:&3

read_line:&3
allocating new result set
fetch next block: start at:266
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
operation successful (276.500ms)
closing result set
sql>
fetch next block: start at:268
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
sql>COPY 9676440 offset 2 records INTO mytable3 FROM 'v:\temp\2008_BSA_Carrier_L
ine_Items_PUF_1.csv' USING DELIMITERS ',','\n','"' NULL AS '' ;
mapi_query_part:141:COPY 9676440 offset 2 records INTO mytable3 FROM 'v:\temp\20
08_BSA_Carrier_Line_Items_PUF_1.csv' USING DELIMITERS ',','\n','"' NULL AS '' ;

fetch next block: start at:270
got next block: length:14
text:&2 9676439 -1

got complete block:
text:&2 9676439 -1

read_line:&2 9676439 -1
allocating new result set
fetch next block: start at:284
got next block: length:0
text:
got complete block:
text:☺

read_line:☺
9676439 affected rows (2m 14s)
closing result set
sql>

Actual Results:

GDKerror:!ERROR: GDKunlink(bat\13\1327.tail)
!OS: The process cannot access the file because it is being used by another process.
GDKunlink(bat\13\1327.theap)
!OS: The process cannot access the file because it is being used by another process.

Expected Results:

9676439 affected rows (2m 14s)
closing result set

MonetDB 5 server v11.13.9 "Oct2012-SP3"
Serving database 'bsa', using 2 threads
Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked
Found 15.873 GiB available main-memory.
Copyright (c) 1993-July 2008 CWI.
Copyright (c) August 2008-2013 MonetDB B.V., all rights reserved
Visit http://www.monetdb.org/ for further information
Listening for connection requests on mapi:monetdb://127.0.0.1:50001/
MonetDB/JAQL module loaded
MonetDB/SQL module loaded

Comment 18566

Date: 2013-02-25 16:19:57 +0100
From: @sjoerdmullender

There may be several reasons why you would get this error.
One is indeed the windows implementation of memory mapped files over network drives. We have not experimented with this, so I can't say whether it really is an issue.
Another issue is the Windows indexing service. Normally the database is stored inside the %APPDATA% folder which is not indexed, but if your data is on another system and in a folder that the indexing service on that system indexes, it may well happen that the indexing service is working on a file when MonetDB decides that it wants to delete the file. I have seen things like this happen in the past on a local drive. So, please make sure the database is not getting indexed.

Having said this, we have not really tried the server with data on remote systems. And honestly, we don't recommend it. It would be better to run the server where the data is. I do understand the use case, though.

Comment 18807

Date: 2013-06-10 09:12:42 +0200
From: @njnes

could you verify if the index service was running.

Comment 18808

Date: 2013-06-10 13:26:42 +0200
From: Anthony Damico <>

it was, but we turned it off, and had the same problem. i understand this is a low-priority for your team, but i do not believe this problem has been resolved :)

Comment 18809

Date: 2013-06-10 13:33:00 +0200
From: @drstmane

Any virus scanner or any other background job accessing files might be another candidate that might cause such problems ...