Monet DB giving wrong value from a table when a value is queried after insertion/ Data Loss #3222
Milestone
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Date: 2013-01-15 15:15:18 +0100
From: Ashish Kumar Singh <>
To: SQL devs <>
Version: 11.13.5 (Oct2012-SP1)
CC: hcb, @njnes, @drstmane
Last updated: 2013-02-19 13:23:51 +0100
Comment 18334
Date: 2013-01-15 15:15:18 +0100
From: Ashish Kumar Singh <>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17
Build Identifier:
We are facing one strange behavior with Monetdb. We have a simple table called test with following structure:
CREATE SEQUENCE "testtable_seq" as bigint START WITH 1;
CREATE TABLE testtable
(dim_id bigint DEFAULT NEXT VALUE FOR "testtable_seq",
dim_type varchar(100) NOT NULL,
dim_value varchar(100) NOT NULL,
CONSTRAINT unique_dimension UNIQUE(dim_type,dim_value) ,
CONSTRAINT pk_testtable PRIMARY KEY (dim_id));
We have a program which reads and writes to this table along with other tables using JDBC.
We are facing below problems interminently:
select dim_id,dim_type,dim_value from testtable where dim_type='user' and dim_value=’user1’ we are getting back ID for some other value present in testable .
Need your help in debugging..
Please suggest , we are using Oct SP1 build.
Reproducible: Always
Steps to Reproduce:
It is happening for random values from the value set being inserted.
Comment 18338
Date: 2013-01-15 18:49:08 +0100
From: @njnes
similar bug was fixed recently, ie could you try to upgrade to Oct2012-sp3
Comment 18339
Date: 2013-01-15 18:51:12 +0100
From: Ashish Kumar Singh <>
Can you please let me know issue id of that similar bug? when SP3 is slated for release?
Thanks a bunch!
Regards,
Ashish
Comment 18340
Date: 2013-01-15 19:05:09 +0100
From: @njnes
For SP3 I saw a pre-announcement within the group, ie should be soon.
There was no exact bug report for the similar report, ie was reported in house. But I have the checkin for you http://dev.monetdb.org/hg/MonetDB/rev/69da46562f4e?revcount=240
Comment 18341
Date: 2013-01-15 21:33:35 +0100
From: Ashish Kumar Singh <>
Great! Can i get somewhere list of bugs which are fixed in sp3 that will help me a lot.
Thanks!
Comment 18345
Date: 2013-01-17 18:03:55 +0100
From: Ashish Kumar Singh <>
FYA, with SP3 also we saw an issue of values not coming back correctly after insertion.
Comment 18356
Date: 2013-01-21 17:21:36 +0100
From: Ashish Kumar Singh <>
Guys any update on this?
Comment 18388
Date: 2013-01-23 09:07:02 +0100
From: Ashish Kumar Singh <>
Created attachment 174
Java program to reproduce the issue
Java program to reproduce the issue
Comment 18389
Date: 2013-01-23 09:07:29 +0100
From: Ashish Kumar Singh <>
Created attachment 175
SQL schema
SQL schema
Comment 18390
Date: 2013-01-23 09:08:21 +0100
From: Ashish Kumar Singh <>
Adding test case to reproduce Duplicate records in database table test.
Comment 18391
Date: 2013-01-23 09:12:40 +0100
From: Ashish Kumar Singh <>
Keep on checking for duplicate IDs using query :
select roe_id ,count() from test group by roe_id having count() > 1
Regards,
Ashish
Comment 18392
Date: 2013-01-23 16:26:48 +0100
From: @drstmane
Please bear with us --- we also have "real work" to do ...
Being a Java illiterate , I quickly tried the following, but it does not seem to work; maybe some more detailed instructions or easier to use script would be helpful ...
$ mclient /tmp/Scema.txt
operation successful
operation successful
$ diff -du /tmp/monetdbIssue/src/com/altair/test/ConfigurationDBConnection.java{
,}2013-01-22 15:59:56.000000000 +0100--- /tmp/monetdbIssue/src/com/altair/test/ConfigurationDBConnection.java
+++ /tmp/monetdbIssue/src/com/altair/test/ConfigurationDBConnection.java 2013-01-23 16:22:41.205961431 +0100
@@ -8,7 +8,7 @@
private String connectionString = "jdbc:monetdb://localhost:50000/demo";
private String password = "monetdb";
public ConfigurationDBConnection() {
$ java /tmp/monetdbIssue/bin/com/altair/test/MonetDBLoadTestBasedOnQueueAndQueryFrequently.class
Exception in thread "main" java.lang.NoClassDefFoundError: /tmp/monetdbIssue/bin/com/altair/test/MonetDBLoadTestBasedOnQueueAndQueryFrequently/class
Caused by: java.lang.ClassNotFoundException: .tmp.monetdbIssue.bin.com.altair.test.MonetDBLoadTestBasedOnQueueAndQueryFrequently.class
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: /tmp/monetdbIssue/bin/com/altair/test/MonetDBLoadTestBasedOnQueueAndQueryFrequently.class. Program will exit.
Comment 18393
Date: 2013-01-23 16:29:35 +0100
From: Ashish Kumar Singh <>
Oh my bad I didn't mentioned that i am giving you java sources :) Its an eclipse project so you can import it and run it from there.
Comment 18394
Date: 2013-01-23 16:39:32 +0100
From: @drstmane
I confess I hardly know what Eclipse is, let alone how to use it. Sorry ...
Comment 18395
Date: 2013-01-23 16:46:15 +0100
From: Ashish Kumar Singh <>
Okay let me put a complied jar for you for verifying this issue, you already have java sources when you need it. Hold on!
Comment 18396
Date: 2013-01-23 16:57:47 +0100
From: Ashish Kumar Singh <>
Do you have a JDK to compile the sources ? I can send you the instructions on how to compile and run it?
Comment 18397
Date: 2013-01-23 17:20:24 +0100
From: @drstmane
$ java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.5) (fedora-68.1.11.5.fc16-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
$ javac -version
javac 1.6.0_24
I/we would highly appreciate if your instructions were cast into, say, a shell script or alike.
Comment 18398
Date: 2013-01-23 17:38:28 +0100
From: Ashish Kumar Singh <>
Created attachment 176
Monet DB test case
Please follow below steps:
Windows: java -classpath monetdbissues.jar;log4j-1.2.14.jar;monetdb-jdbc-2.8.jar;DBConnection.properties com.altair.test.MonetDBLoadTestBasedOnQueueAndQueryFrequently
Linux: java -classpath monetdbissues.jar:log4j-1.2.14.jar:monetdb-jdbc-2.8.jar:DBConnection.properties com.altair.test.MonetDBLoadTestBasedOnQueueAndQueryFrequently
Comment 18399
Date: 2013-01-23 18:14:53 +0100
From: @njnes
just run it on the (to be released) feb2013 version and cannot reproduce the problem, ie will try on Oct-2012-sp3 now.
Comment 18400
Date: 2013-01-23 18:18:13 +0100
From: @njnes
your jar has a fixed connection uri in it, ie how could I override this?
Comment 18401
Date: 2013-01-23 18:46:01 +0100
From: Ashish Kumar Singh <>
Please update connection settings in DBConnection.properties file and run it.
Comment 18402
Date: 2013-01-23 18:47:01 +0100
From: Ashish Kumar Singh <>
Also please wait for some time 15-20 minutes ; then you will see duplicate records getting entered.
Comment 18403
Date: 2013-01-23 18:52:48 +0100
From: @drstmane
I run the following on a post Oct2012-SP3 Oct2012 branch:
$ cat Scema.txt ; echo
CREATE SEQUENCE "test_seq" as bigint START WITH 1;
CREATE TABLE test(roe_id bigint DEFAULT NEXT VALUE FOR "test_seq",
roe_type bigint NOT NULL,
node_event_id bigint NOT NULL,
dc_servername varchar(50),
roe_start_date_withtz varchar(50) ,
usr_id bigint,
customgroup_id bigint,
queuename_id bigint,
exitstatus_id bigint);
$ mclient Scema.txt
operation successful
operation successful
$ diff -du DBConnection.properties{
,}2013-01-23 21:34:26.000000000 +0100--- DBConnection.properties
+++ DBConnection.properties 2013-01-23 17:45:51.554648440 +0100
@@ -5,6 +5,6 @@
Usage of the software is only as explicitly permitted in the end user software license agreement.
Copyright notice does not imply publication.
DriverName=nl.cwi.monetdb.jdbc.MonetDriver
-ConnectionString=jdbc:monetdb://192.168.15.136:50000/pbsworksdb
+ConnectionString=jdbc:monetdb://localhost:50000/demo
username=monetdb
password=monetdb
$ ( date ; time java -classpath monetdbissues.jar:log4j-1.2.14.jar:monetdb-jdbc-2.8.jar:DBConnection.properties com.altair.test.MonetDBLoadTestBasedOnQueueAndQueryFrequently ; date ) &> MonetDBLoadTestBasedOnQueueAndQueryFrequently.log
has been running for an hour now, producing ~80 MB of output (error messages; see attached log).
checks look fine, though:
$ mclient -e /tmp/check.sql
select count() from test;
+-------+
| L1 |
+=======+
| 33100 |
+-------+
1 tuple
select count(roe_id) from test;
+-------+
| L1 |
+=======+
| 33100 |
+-------+
1 tuple
select count(distinct roe_id) from test;
+-------+
| L1 |
+=======+
| 33100 |
+-------+
1 tuple
select roe_id ,count() from test group by roe_id having count(*) > 1;
+--------+----+
| roe_id | L1 |
+========+====+
+--------+----+
0 tuples
Comment 18404
Date: 2013-01-23 18:53:23 +0100
From: @drstmane
Created attachment 177
output log
Comment 18405
Date: 2013-01-23 19:07:43 +0100
From: Ashish Kumar Singh <>
I am running it on Cent OS 5.8 64 bit? what else I can provide you to narrow down on this issue, It is very frequently reproducible on machines we are having.
Comment 18406
Date: 2013-01-23 19:13:08 +0100
From: Ashish Kumar Singh <>
Wow I just reviewed the logs you attached and here is my observation :
1.DB service went Down .
2. No suitable driver found for blah.... this is another monet bug.
For one hour number of records are way less than what I was seeing.
Sql State: 08001
Check whether database service is running or not
java.sql.SQLException: No suitable driver found for
at java.sql.DriverManager.getConnection(DriverManager.java:640)
at java.sql.DriverManager.getConnection(DriverManager.java:200)
at com.altair.test.ConfigurationDBConnection.getConnection(ConfigurationDBConnection.java:38)
at com.altair.test.JobInsertionThread.run(JobInsertionThread.java:71)
java.lang.NullPointerException
at com.altair.test.JobInsertionThread.run(JobInsertionThread.java:156)
Could not connect to the database
No suitable driver found for
Sql State: 08001
Check whether database service is running or not
java.sql.SQLException: No suitable driver found for
at java.sql.DriverManager.getConnection(DriverManager.java:640)
at java.sql.DriverManager.getConnection(DriverManager.java:200)
at com.altair.test.ConfigurationDBConnection.getConnection(ConfigurationDBConnection.java:38)
at com.altair.test.JobInsertionThread.run(JobInsertionThread.java:71)
java.lang.NullPointerException
at com.altair.test.JobInsertionThread.run(JobInsertionThread.java:156)
Could not connect to the database
Comment 18407
Date: 2013-01-23 19:31:26 +0100
From: Ashish Kumar Singh <>
Guys,
Thanks for looking in to it. Here are few proposals I would like to make to help you out on the issue I am facing and have it reproduced at your end:
zip Monet DB instance from my machine and send it to you with steps how to run it so you will be able to see final corrupt state of the system if you run it on a cent os operating system.
I can create a VM for you with Test program and Monet DB packaged with in it and make it available for you.
We can have a Web meeting to show you that.
Regards,
Ashish
Comment 18408
Date: 2013-01-23 21:55:05 +0100
From: @njnes
Just to make sure. Did you test SP3 starting with a fresh DB. Thats what we are doing now, and indeed sofar we cannot repeat your problem.
Comment 18409
Date: 2013-01-23 22:01:57 +0100
From: Ashish Kumar Singh <>
Yes we did , on more than one machines.
Comment 18410
Date: 2013-01-24 08:15:23 +0100
From: @njnes
it seems your test triggered an assert in my testing with oct2012-sp3. Sofar no
inconsistencies, but if compiled without asserts this may have caused this.
So will continue to search for the reason why the assert was triggered.
Comment 18411
Date: 2013-01-24 08:29:56 +0100
From: Ashish Kumar Singh <>
Thanks Neil , Can you guide how to compile with asserts! Why it is not mandatory setting for
Comment 18412
Date: 2013-01-24 08:30:24 +0100
From: Ashish Kumar Singh <>
---- monetdb compilation and is default?
Thanks Again
Comment 18413
Date: 2013-01-24 08:40:35 +0100
From: @njnes
For releases 'asserts' are disabled. This is normal and I would say expected. Asserts will have a big impact on the performance.
To enable asserts, you need to configure with --enable-asserts (and recompile, ie make clean; make install;).
Comment 18414
Date: 2013-01-24 11:01:24 +0100
From: Ashish Kumar Singh <>
Hi Neil ,
Will you be interested in getting a VM ware with my test and instance of monet db which produces this issue more frequently?
Regards,
Ashish
Comment 18415
Date: 2013-01-24 12:04:46 +0100
From: @drstmane
For the records:
My Oct2012-SP3++ run was with assertions enabled, but I did not see any assertions during the first hour. I'm now not at the console to check the server output; however, the server is still running, so there cannot have been any assertion so far ...
It seems that also the java program is still running (after ~18 hours!), but it seems to have stopped producing output (errors) after ~4 hours. Checks now say:
$ mclient -e check.sql
select count() from test;
+-------+
| L1 |
+=======+
| 99900 |
+-------+
1 tuple
select count(roe_id) from test;
+-------+
| L1 |
+=======+
| 99900 |
+-------+
1 tuple
select count(distinct roe_id) from test;
+-------+
| L1 |
+=======+
| 99900 |
+-------+
1 tuple
select roe_id ,count() from test group by roe_id having count(*) > 1;
+--------+----+
| roe_id | L1 |
+========+====+
+--------+----+
0 tuples
Comment 18416
Date: 2013-01-24 12:08:20 +0100
From: Ashish Kumar Singh <>
Okay so when you say Oct SP3++ , it is Feb candidate branch.
Comment 18417
Date: 2013-01-24 12:21:18 +0100
From: @drstmane
Nope, Oct2012-SP3++ is Oct2012 branch after Oct2012-SP3 release --- I don't think, though, there have been any relevant changes on that branch after the release.
Comment 18418
Date: 2013-01-24 12:33:34 +0100
From: Ashish Kumar Singh <>
So what shall be our next step you propose Neil and Stefan?
Comment 18419
Date: 2013-01-24 13:09:31 +0100
From: @drstmane
Well, maybe you could do as Niels suggested: build the Oct2012-SP2 release from scratch with assertions enabled (configure --enable-assert ...), run your experiment, and report whether it triggers any assertions. Please then also report in more detail what your program actually does (and how?), its output (if any) and the server output / logs.
My logs suggest that you program (in my case now running for more than 19 hours --- is it supposed to end sometime?) does lots of concurrent transactions. If so, I assume you are aware of MonetDB optimistic concurrency control and the consequences for your client application, right?
Comment 18420
Date: 2013-01-24 14:19:45 +0100
From: Ashish Kumar Singh <>
Yep that's correct , I am aware of consequences of optimistic concurrency control.
Only point is that it shall not end up in a db state where records are duplicated!
I will try one build with assertion enabled and update you .
Do you see by any chance any thirdparty libraries version can cause this issue?
Comment 18421
Date: 2013-01-24 14:30:57 +0100
From: @drstmane
FYI:
I just built Oct2012-SP1(!) from scratch with assertions enabled, and run your program. I produces the same java exceptions and errors as with Oct2012-SP3++ (just considerably more; >200 MB in one hour), but the server is running flawlessly, and the data looks fine:
$ mclient -e /tmp/check.sql
select * from test limit 1;
+--------+----------+---------------+---------------+-----------------------+--------+----------------+--------------+---------------+
| roe_id | roe_type | node_event_id | dc_servername | roe_start_date_withtz | usr_id | customgroup_id | queuename_id | exitstatus_id |
+========+==========+===============+===============+=======================+========+================+==============+===============+
| 1101 | 0 | 0 | s | s | 0 | 0 | 0 | 1 |
+--------+----------+---------------+---------------+-----------------------+--------+----------------+--------------+---------------+
1 tuple
select count() from test;
+-------+
| L1 |
+=======+
| 30661 |
+-------+
1 tuple
select count(roe_id) from test;
+-------+
| L1 |
+=======+
| 30661 |
+-------+
1 tuple
select count(distinct roe_id) from test;
+-------+
| L1 |
+=======+
| 30661 |
+-------+
1 tuple
select roe_id ,count() from test group by roe_id having count(*) > 1;
+--------+----+
| roe_id | L1 |
+========+====+
+--------+----+
0 tuples
Comment 18422
Date: 2013-01-24 14:33:15 +0100
From: @drstmane
I'm running under 64-bit Fedora 16 on an Intel Core i7-2600 CPU @ 3.40GHz (4 cores + hyperthreading = 8 HW threads)
Comment 18423
Date: 2013-01-24 14:36:09 +0100
From: Ashish Kumar Singh <>
I am running under Cent OS 5.8 64 bit , 2 core 4 GB ram . Client is windows client with 2 core 4 GB of ram if it helps. We saw this issue in RHEL and SLES 11 also .
Comment 18424
Date: 2013-01-24 14:45:57 +0100
From: @drstmane
So, how long is your programm supposed to run, and are those zillions of java
exceptions expected to appear on the console (i.e., not caught and handled by
your program)?
java.sql.SQLException: COMMIT: transaction is aborted because of concurency
conflicts, will ROLLBACK instead
at
nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnection.java:2535)
at
nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.processQuery(MonetConnection.java:2284)
at
nl.cwi.monetdb.jdbc.MonetStatement.internalExecute(MonetStatement.java:508)
at nl.cwi.monetdb.jdbc.MonetStatement.execute(MonetStatement.java:349)
at
nl.cwi.monetdb.jdbc.MonetStatement.executeUpdate(MonetStatement.java:544)
at
com.altair.test.UpdateFrequentlyThread.run(UpdateFrequentlyThread.java:24)
Comment 18425
Date: 2013-01-24 14:50:41 +0100
From: Ashish Kumar Singh <>
Okay after 100000 records are inserted it shall stop inserting new data, it doesn't exits as of now by default my bad :(.
Those exceptions i am printing intentionally as my observation was during concurrency conflicts only records are getting duplicated in DB.
Now I am running one assertion enabled build against Oct SP3. Where to look for logs after assertion fails?
Comment 18426
Date: 2013-01-24 14:55:10 +0100
From: @drstmane
An assertion stops the server. If you start mserver5 by hand, you'll see that on the console whether you started mserver5. If you use monetdbd, you'll need to check the merovingian log in you dbfarm.
Comment 18427
Date: 2013-01-24 15:00:17 +0100
From: @drstmane
After an hour and a half with Oct2012-SP1 I get
$ mclient -e /tmp/check.sql
select * from test limit 1;
+--------+----------+---------------+---------------+-----------------------+--------+----------------+--------------+---------------+
| roe_id | roe_type | node_event_id | dc_servername | roe_start_date_withtz | usr_id | customgroup_id | queuename_id | exitstatus_id |
+========+==========+===============+===============+=======================+========+================+==============+===============+
| 1101 | 0 | 0 | s | s | 0 | 0 | 0 | 1 |
+--------+----------+---------------+---------------+-----------------------+--------+----------------+--------------+---------------+
1 tuple
select count() from test;
+-------+
| L1 |
+=======+
| 46381 |
+-------+
1 tuple
select count(roe_id) from test;
+-------+
| L1 |
+=======+
| 46381 |
+-------+
1 tuple
select count(distinct roe_id) from test;
+-------+
| L1 |
+=======+
| 46360 |
+-------+
1 tuple
select roe_id ,count() from test group by roe_id having count(*) > 1;
+--------+------+
| roe_id | L1 |
+========+======+
| 482580 | 2 |
| 482581 | 2 |
| 482582 | 2 |
| 482583 | 2 |
| 482584 | 2 |
| 482585 | 2 |
| 482586 | 2 |
| 482587 | 2 |
| 482588 | 2 |
| 482589 | 2 |
| 482590 | 2 |
| 482591 | 2 |
| 482592 | 2 |
| 482593 | 2 |
| 482594 | 2 |
| 482595 | 2 |
| 482596 | 2 |
| 482597 | 2 |
| 482598 | 2 |
| 482599 | 2 |
| 482600 | 2 |
+--------+------+
21 tuples
So my conclusion so far: reproducible with Oct2012-SP1, but not with Oct2012-SP3++; will now try the Oct2012-SP3 release ...
Comment 18428
Date: 2013-01-24 17:16:29 +0100
From: Ashish Kumar Singh <>
On sp3 fresh compile i was able to reproduce it, if you are not able to reproduce it easily then running multiple instances on test program in paralllel helps! With two test client instances running this was more easily coming in my case!
Thanks guys for your help on debugging this issue with me!
Comment 18429
Date: 2013-01-24 18:30:00 +0100
From: @drstmane
Oct2012-SP3 with single client works fine for me; after 3 h:
$ mclient -e /tmp/check.sql
select * from test limit 1;
+--------+----------+---------------+---------------+-----------------------+--------+----------------+--------------+---------------+
| roe_id | roe_type | node_event_id | dc_servername | roe_start_date_withtz | usr_id | customgroup_id | queuename_id | exitstatus_id |
+========+==========+===============+===============+=======================+========+================+==============+===============+
| 1101 | 0 | 0 | s | s | 0 | 0 | 0 | 1 |
+--------+----------+---------------+---------------+-----------------------+--------+----------------+--------------+---------------+
1 tuple
select min(roe_id), max(roe_id), count(), count(roe_id), count(distinct roe_id) from test;
+------+---------+-------+-------+-------+
| L1 | L2 | L3 | L4 | L5 |
+======+=========+=======+=======+=======+
| 1101 | 1163900 | 99699 | 99699 | 99699 |
+------+---------+-------+-------+-------+
1 tuple
select roe_id ,count() from test group by roe_id having count(*) > 1;
+--------+----+
| roe_id | L1 |
+========+====+
+--------+----+
0 tuples
No time to test multiple clients soon. sorry.
Comment 18430
Date: 2013-01-24 18:50:17 +0100
From: @njnes
okay it triggered an assert again. I don't have a solution just jet, but one test you could do is to set MAX_SPARES (in sql/storage/store.c) to 1 (instead of 32).
Comment 18431
Date: 2013-01-24 19:01:52 +0100
From: @njnes
unfortunately MAXSPARES=1 doesn't help. Closing in on the problem a bit..
Comment 18432
Date: 2013-01-24 22:49:58 +0100
From: @njnes
found an issue. Probably a fix by tomorrow.
Comment 18433
Date: 2013-01-25 09:05:45 +0100
From: Ashish Kumar Singh <>
Kudos! Great thanks Neil !! I will wait to hear from you . If you need any other thing from my side I will be happy to help.
Comment 18434
Date: 2013-01-25 10:58:09 +0100
From: @njnes
Changeset 4abd55aed644 made by Niels Nes niels@cwi.nl in the MonetDB repo, refers to this bug.
For complete details, see http//devmonetdborg/hg/MonetDB?cmd=changeset;node=4abd55aed644
Changeset description:
Comment 18435
Date: 2013-01-25 11:02:52 +0100
From: @njnes
please test the fix, using one of the version SP3++, Feb2013 or default, all are only available via hg, ie sources
Comment 18436
Date: 2013-01-25 11:16:50 +0100
From: Ashish Kumar Singh <>
Let me try Neil!
Comment 18452
Date: 2013-01-30 08:36:13 +0100
From: @drstmane
Ashish,
any news on this one? Does it work for you, now?
Thanks!
Comment 18458
Date: 2013-02-10 13:28:44 +0100
From: @drstmane
Ashish,
any news on this one?
Does it work for you, now, and we can close this one as fixed?
Thanks!
Comment 18519
Date: 2013-02-19 13:23:51 +0100
From: @sjoerdmullender
Feb2013 has been released.
The text was updated successfully, but these errors were encountered: