Ticket #210 (closed defect: fixed)

Opened 10 years ago

Last modified 9 years ago

wave_serverV reports out of client threads on Windows and none free up, program never allows any new ones.

Reported by: paulf Owned by: tim
Priority: critical Milestone: All Platforms
Component: wave_serverV Version: 7.7
Keywords: Cc:

Description

We need a way to get out of this condition (besides restarting WSV):

20120531_UTC_00:00:03 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:06 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:09 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:12 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:15 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:18 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:21 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:24 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:27 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:30 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:33 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:36 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:39 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:42 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:45 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:48 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:51 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:54 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:00:57 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:01:00 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:01:03 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:01:06 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:01:09 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:01:13 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:01:16 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:01:19 Cannot accept client: out of server threads. Waiting.
20120531_UTC_00:01:22 Cannot accept client: out of server threads. Waiting.

There must be a degenerate condition that hangs up the threads and makes them inaccessible.

Change History

comment:1 Changed 9 years ago by paulf

Okay, This problem of using up WSV server threads was due to a poorly written client that in one instance didn't call the killMenu so it slowly eroded the WSV and used up all the threads. The connections persisted and the WSV never cleared them...ever.

I have a plan to fix this (as an option).

  1. start tracking how long since each server handling thread has responded to a client request.
  2. if max threads are busy, then go through the list of threads and free up the oldest response time one for this client to use.

And another optional setting would be to:

Timeout any WSV server thread that hasn't been accessed in N seconds by a client.

Any objections/suggestions?

This would be for 7.7 release, as it is too late to fix this now.

Paul

comment:2 Changed 9 years ago by paulf

  • Version changed from 7.6 to 7.7

comment:3 Changed 9 years ago by paulf

Steve Malone wrote to the ewdev mailing list, entering it in trac for posterity:

Paul,
	YEH!  This has been a persistent problem we have seen for years.  Since we always assumed that it was a client problem I have (from time to time) tried to figure out what client was doing it and get that one fixed, but always gave up and just restarted the waveservers.  They would usually run weeks to months before they ran out of threads so it always dropped off our priority list.
	I favor your second option with the 'N' being a configuration option.  For most of us a timeout of some 10s of seconds would probably be appropriate but there might be some cases where a longer timeout might be needed and thus an option would be good.
	Steve

comment:4 Changed 9 years ago by tim

  • Owner changed from somebody to tim

comment:5 Changed 9 years ago by tim

  • Status changed from new to closed
  • Resolution set to fixed

After digging in the code I found that the option Steve Malone asked for is already implemented as ClientTimeout?: http://www.earthwormcentral.org/documentation2/cmd/wave_serverV_cmd.html#ClientTimeout

When set this will timeout after ClientTimeout? milliseconds and the thread will be killed.

This is in the .d file enabled(but if it is commented out timeout is disabled), but comments suggest that it is not recommended.

ClientTimeout  60000 # Optional.  Not recommended feature but it does work.
                     # Timeout in MILLISECONDS for response from client.
                     # Threads that have not heard anything from their client
                     # in this period will exit.
                     # Comment out or set to -1 if you don't want to
                     # kill threads of silent clients.

If anyone is interested in the first option to go through the list of threads and kill the oldest please speak up and we will implement it, until then I am closing this ticket.

Note: See TracTickets for help on using tickets.