Ticket #543 (new defect)

Opened 5 years ago

ew2mseed gets stuck on gaps in wave servers, need StartTime to be set to work

Reported by: paulf Owned by: somebody
Priority: major Milestone: All Platforms
Component: ALL modules Version: 7.9
Keywords: Cc:

Description

At IRIS we have seen some of our ~25 ew2mseed processes occasionally get 'stuck' on gaps for some stations and stop pulling any data from the remote EW server for those stations even though a getmenu check against the remote EW server suggests there is data for the stations and time range. When the ew2mseed gets in this state I find that:

Not all the stations are affected - some will keep pulling data just fine. The stations that are affected all stop writing data to disk about the same minute, and the ew2mseed logs start logging messages like "<STA NET CHN>: requested time interval <2016Apr27 2315 49.40 - 2016Apr27 2317 31.80> is fully within a gap in the tank". These messages increment in time, always complaining that the request is in a gap (though as I said before getmenu suggests there is data to be had). I see this warning message is in the code at http://earthworm.isti.com/trac/earthworm/browser/trunk/src/archiving/ew2mseed/ws_requests.c?rev=3839. It looks like the code is trying to progress through a gap to data but never seems to reach it. A simple restart does not fix the issue. However by advancing the StartTime? in the config file and then doing a restart that does fix the issue and then all is well - except there is now a data gap from the onset of the bad state until StartIme?. Also, I have no idea how to chose StartTime? to minimize the gap - I have tried incrementing it a little but I always seem to have to jump to close to 'now' for the change to 'take'.

So the questions are: what does the ew2mseed complaint of a 'gap in the tank' mean, why does a getmenu show data on the server but ew2mseed doesn't pull it, is changing StartTime? the only way to fix this bad state, and if so how does one chose the optimal StartTime? to minimize the gap?

ew2mseed version appears to be - ew2mseed: r5878 2013-08-19 paulf $

Forrest

Note: See TracTickets for help on using tickets.