No Socket Timeout: MonetDB Java client hangs forever on reads and consume threads when the database is crashed #2828

monetdb-team · 2020-11-30T11:43:19Z

Date: 2011-06-30 00:46:56 +0200
From: Philippe <<philippe.hanrigou>>
To: clients devs <>
Version: 11.3.3 (Apr2011-SP1) [obsolete]

Last updated: 2011-09-16 15:04:31 +0200

Comment 15860

Date: 2011-06-30 00:46:56 +0200
From: Philippe <<philippe.hanrigou>>

User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1
Build Identifier:

Due to another bug our MonetDB ends up with a crashed database on a regular basis. As a consequence we realized that in this situation our entire application server would go down even for workloads not related to MonetDB. This is because any request to MonetDB via Java (JDBC or not) would hand forever on a read (sample stack trace included below).

This is because java sockets have an infinite timeout by default. It would be great if you guys gave an option for the client API to customize the socket timeout at the driver level. It would provide better recovery and stability to any java client. For Example, here is a patch that solved the problem for us:

diff -r a3349ca90986 java/src/nl/cwi/monetdb/jdbc/MonetConnection.java
--- a/java/src/nl/cwi/monetdb/jdbc/MonetConnection.java Tue Jun 28 11:35:22 2011 -0700
+++ b/java/src/nl/cwi/monetdb/jdbc/MonetConnection.java Tue Jun 28 15:06:44 2011 -0700
@@ -118,6 +118,9 @@
/** Whether or not BLOB is mapped to BINARY within the driver */
private final boolean blobIsBinary;

/** SO_TIMEOUT for the underlying java.net.Socket */
private final int socketTimeoutInMillis;
/**
- Constructor of a Connection for MonetDB. At this moment the
- current implementation limits itself to storing the given host,
  @@ -149,7 +152,16 @@
  boolean debug = Boolean.valueOf(props.getProperty("debug")).booleanValue();
  String hash = props.getProperty("hash");
  blobIsBinary = Boolean.valueOf(props.getProperty("treat_blob_as_binary")).booleanValue();

int timeout = 0;
String timeoutStr = props.getProperty("socket_timeout");
if (null != timeoutStr) {
```
   try {
```

   timeout = Integer.parseInt(timeoutStr);

```
 } catch (NumberFormatException e) {
```

   addWarning("Invalid socket timeout " + timeoutStr + ", defaulting to 0");

```
   }
```
}
this.socketTimeoutInMillis = timeout;
// check input arguments
if (hostname == null || hostname.trim().equals(""))
throw new IllegalArgumentException("hostname should not be null or empty");
@@ -169,6 +181,7 @@
commandTempl = new String[3]; // pre, post, sep

server = new MapiSocket();
server.setSoTimeout(socketTimeoutInMillis);

if (hash != null) server.setHash(hash);
if (database != null) server.setDatabase(database);
diff -r a3349ca90986 java/src/nl/cwi/monetdb/mcl/net/MapiSocket.java
--- a/java/src/nl/cwi/monetdb/mcl/net/MapiSocket.java Tue Jun 28 11:35:22 2011 -0700
+++ b/java/src/nl/cwi/monetdb/mcl/net/MapiSocket.java Tue Jun 28 15:06:44 2011 -0700
@@ -119,6 +119,9 @@
/** A short in two bytes for holding the block size in bytes */
private byte[] blklen = new byte[2];
/** SO_TIMEOUT value for the underlying java.net.Socket */
private int socketTimeoutInMillis = 0;
/**
- Constructs a new MapiSocket.
  */
  @@ -203,7 +206,7 @@
- @throws MCLParseException if bogus data is received
- @throws MCLException if an MCL related error occurs
  */

public List connect(String host, int port, String user, String pass)

public List connect(String host, int port, String user, String pass)
throws IOException, MCLParseException, MCLException
{
// Wrap around the internal connect that needs to know if it
@@ -212,7 +215,7 @@
}

private List connect(String host, int port, String user, String pass,

```
 	boolean makeConnection) 
```

```
 boolean makeConnection)
 throws IOException, MCLParseException, MCLException
```
{
if (ttl-- <= 0)
@@ -223,6 +226,7 @@
// set nodelay, as it greatly speeds up small messages (like we
// often do)
con.setTcpNoDelay(true);

 con.setSoTimeout(socketTimeoutInMillis);

 	fromMonet = new BlockInputStream(con.getInputStream());
 	toMonet = new BlockOutputStream(con.getOutputStream());

@@ -584,6 +588,22 @@
}

/**

- Set the SO_TIMEOUT on the underlying Java socket using java.net.SocketsetSoTimeout.
- Useful to avoid consuming client threads blocked on read when the MonetDB database
- is in a crashed state.
- The option must be enabled prior to entering the blocking operation to have effect.
- @see http://download.oracle.com/javase/1.4.2/docs/api/java/net/Socket.htmlsetSoTimeout(int)
- @param timeout The specified timeout, in milliseconds. A timeout

           of zero is interpreted as an infinite timeout.

*/
public void setSoTimeout(int socketTimeoutInMillis) {
this.socketTimeoutInMillis = socketTimeoutInMillis;
}
/**
- Returns the mapi protocol version used by this socket. The
- protocol version depends on the server being used. Users of the
- MapiSocket should check this version to act appropriately.
  @@ -678,7 +698,7 @@
```
 // write the actual block
 out.write(block, 0, writePos);
```

 	if (debug) {
 		if (last) {
 			logTd("write final block: " + writePos + " bytes");

======== Sample Stack trace of a "hanging" read =============

"pool-4-thread-29" prio=10 tid=0x00007f3119335000 nid=0x3f4e sleeping[0x00007f3120aa6000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x00000007bd1c3590> (a java.io.BufferedInputStream)
at nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.read(MapiSocket.java:782)
at nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.readBlock(MapiSocket.java:829)
at nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.read(MapiSocket.java:901)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
- locked <0x00000007a33916a0> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.readLine(BufferedReader.java:299)
- locked <0x00000007a33916a0> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at nl.cwi.monetdb.mcl.io.BufferedMCLReader.readLine(BufferedMCLReader.java:109)
at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:239)
at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:211)
at nl.cwi.monetdb.jdbc.MonetConnection.(MonetConnection.java:203)
at nl.cwi.monetdb.jdbc.MonetDriver.connect(MonetDriver.java:159)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:154)
at clojure.contrib.sql.internal$get_connection.invoke(internal.clj:85)
at clojure.contrib.sql.internal$with_connection_STAR.invoke(internal.clj:102)

Reproducible: Always

Steps to Reproduce:

Crash a MonetDB database
Have a Java client try to access the database
Client hangs forever and consume threads (or at least a very long time) even if the database is recovered

Actual Results:

Thread blocked forever on a read

Expected Results:

Exception or error message after some configurable timeout option

Comment 15861

Date: 2011-06-30 08:18:04 +0200
From: @grobian

Your patch looks ok to me, apart from the indentation. Could you please use tabs, and attach your patch so that I can apply it? Thanks in advance!

Comment 15863

Date: 2011-06-30 20:43:31 +0200
From: Philippe <<philippe.hanrigou>>

Created attachment 62
Patch to add customizable socket timeout for the Java API

Cleaned up patch to use tabs

Attached file: monetdb-add-java-timeout.diff (text/plain, 4521 bytes)
Description: Patch to add customizable socket timeout for the Java API

Comment 15864

Date: 2011-06-30 21:31:18 +0200
From: @grobian

Thanks, I cannot fix this in the Apr2011 branch, since this is more a feature than a (bug)fix.

Comment 15865

Date: 2011-06-30 21:31:43 +0200
From: @grobian

Changeset d9f7d6285c63 made by Fabian Groffen fabian@cwi.nl in the MonetDB repo, refers to this bug.

For complete details, see http//devmonetdborg/hg/MonetDB?cmd=changeset;node=d9f7d6285c63

Changeset description:

JDBC: add so_timeout Driver property

On case the server locks up, the JDBC remains "hanging" too, waiting for
the server to send something.  When the so_timeout property is set, a
timeout for this can be set, such that the read eventually breaks, and
the caller (application) can continue.

Based on original patch by Philippe Hanrigou in bug #2828

Comment 16235

Date: 2011-09-16 15:04:31 +0200
From: @sjoerdmullender

The Aug2011 version has been released.

The text was updated successfully, but these errors were encountered:

monetdb-team added bug Something isn't working Client interfaces normal labels Nov 30, 2020

monetdb-team closed this as completed Nov 30, 2020

sjoerdmullender added this to the Ancient Release milestone Nov 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No Socket Timeout: MonetDB Java client hangs forever on reads and consume threads when the database is crashed #2828

No Socket Timeout: MonetDB Java client hangs forever on reads and consume threads when the database is crashed #2828

monetdb-team commented Nov 30, 2020

No Socket Timeout: MonetDB Java client hangs forever on reads and consume threads when the database is crashed #2828

No Socket Timeout: MonetDB Java client hangs forever on reads and consume threads when the database is crashed #2828

Comments

monetdb-team commented Nov 30, 2020

Comment 15860

Steps to Reproduce:

Actual Results:

Expected Results:

Comment 15861

Comment 15863

Comment 15864

Comment 15865

Comment 16235