Date: 2018-09-08 08:52:24 +0200
From: @sjoerdmullender
To: SQL devs <>
Version: 11.31.7 (Aug2018)

Last updated: 2018-10-17 10:07:10 +0200

Comment 26610

Date: 2018-09-08 08:52:24 +0200
From: @sjoerdmullender

Created attachment 607
Plam from which the excerpt was taken (produced from within gdb)

There is a pretty serious race condition that sometimes gets triggered by the storagemodel test.

The problem is in the plan for the storagemdelinit function in 75_storagemdel.sql. Here is a relevant excerpt from the plan:

 X_628=<tmp_1247>[278]:bat[:lng] := sql.bind(X_586=0:int, "sys":str, "storagemodelinput":str, "distinct":str, 1:int);
 X_631=<tmp_1247>[278]:bat[:lng] := sql.delta(X_627=<sql_empty_lng_bat>[0]:bat[:lng], X_629=<sql_empty_oid_bat>[0]:bat[:oid], X_630=<sql_empty_lng_bat>[0]:bat[:lng], X_628=<tmp_1247>[278]:bat[:lng]);
 X_683=<tmp_556>[98]:bat[:lng] := algebra.projectionpath(C_676=<tmp_1454>[98]:bat[:oid], C_590=<tmp_1456>[278]:bat[:oid], X_631=<tmp_1247> 278]:bat[:lng]);

[...]

 algebra.projection(C_590=<tmp_1456>[278]:bat[:oid], X_631=<tmp_1247>[278]:bat[:lng]);

[...]

 sql.update(X_586=0:int, "sys":str, "storagemodelinput":str, "distinct":str, C_690=<tmp_557>[98]:bat[:oid], X_682=<tmp_1445>[98]:bat[:lng]);

[...]

 language.pass(X_631=<tmp_1247>[278]:bat[:lng]);

The problem is as follows. The first blurb creates a new BAT, here tmp_1247, which holds the values for the "distinct" column of the sys.storagemodelinput table after it was filled in by the call to sys.storage() in the storagemodelinit() function. All values are equal at this point (i.e. sorted and revsorted).

Some time later, this BAT is used in a call to algebra.projection(). (Actually, when running it, I normally see a call to algebra.projectionpath in which it is the last (third) argument. This is relevant because of the sorted flag.)

And here is the race: just a few lines further, there is a call to sql.update of that same column. Since the only other instructions between the projection(path) and the update are other projections and an unrelated aggr.count() call, the projection may happen simultaneous with the update. When this happens (and it does happen sometimes!) the projection produces a BAT that it thinks is sorted (because the inputs were sorted when the projection started) but the second input is changed while running and so the output isn't in fact sorted.

I guess the language.pass call should be moved to in front of the call to sql.update, but where does that happen? (I haven't yet looked.)

Attached file: m (text/plain, 70602 bytes)
Description: Plam from which the excerpt was taken (produced from within gdb)

Comment 26611

Date: 2018-09-08 09:28:40 +0200
From: MonetDB Mercurial Repository <>

Changeset 2b9a71c4600e made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see https//devmonetdborg/hg/MonetDB?cmd=changeset;node=2b9a71c4600e

Changeset description:

Force a dataflow breakpoint before sql update instructions.
This fixes bug #6641.