Jump to content


Attention: In order to reply to messages, create topics, have access to other features of the community you must sign up for an account.
Guest lance25

Unplanned Subway Service Changes

Recommended Posts

Posted (edited)
24 minutes ago, paulrivera said:

Isn't there a backup system in the RCC or did that fail also? (assuming a backup system actually exists...)

There is supposed to be a backup command center, but seemingly no one knows what its status is. Depending on where exactly the failure was tonight, could have been useful. 

Edited by RR503

Share this post


Link to post
Share on other sites

Posted (edited)
1 hour ago, RR503 said:

The sad part about all the ATS issues is that a well manned central control with decent physical infrastructure should actually be better at dealing with large scale disruptions than a bunch of decentralized towers. Everybody's in one room, every interlocking is staffed, all trains are visible. Issue is, of course, those two equivocations: staffing and infrastructure, neither of which MTA seems to understand how to manage. 

All trains are visible is a reach, the big meltdown events usually come with an event that eliminates the train identifications. ATS dump, there goes all the train IDs, power failure...all the track circuits just flipped and threw all the IDs, data overload...the entire model board just turns solid red, the IDs drop to whatever the non-described default setting is by the manufacturer and you can't look out the window to see what's coming you just have to constantly repeat "train at 32 ball identify yourself" over and over until hopefully people stop stepping on you so you can get the answer. Oh btw the phone is ringing, everyone in the field noticed the problem and is calling you so you can tell them what to do. Having worked both 'in the field' and in a major command center it isn't that its better or worse, in fact once you have a decent information system you and the person on the ground will get the same data at relatively the same time anyway. For busy events at my last job we actually took a team of Ops personnel down to the Airport and Dispatch the planes on the ground in real time face-to-face because playing information-tag was slowing down the operation. 

How many people does it take to deal with that situation? More people than the (MTA) is willing to staff on a regular basis or could even justify, so that's a dead end.

 

49 minutes ago, paulrivera said:

Isn't there a backup system in the RCC or did that fail also? (assuming a backup system actually exists...)

Can't say for sure with RCC but I will say the 'industry standard' answer is yes...depending on what you mean by back up. There is usually a secondary system that can function so you can do maintenance/software updates. But as far as a completely independent second line of control...I doubt that exists in most places, perhaps the most critical locations *might* have it on some Railroads. The problem with redundant systems is that they all feed to the same wires and hardware in the field, so unless you duplicate an interlocking or the signal wires there are a number of failure modes that will knock both systems out anyway. For example ATS receives the information from transponders in the field and sends that to the server rack at *location redacted* now if the pipeline to the server is fouled up anyone or anything drawing data from that server will get 'null' so you want to make a different storage server to hold the ID and control inputs? Good idea, lets put in another set of ATS wire, otherwise the downstream failure just ends up knocking out both of your expensive data servers.

Edited by Jsunflyguy
Finished sentence!
  • Upvote 1

Share this post


Link to post
Share on other sites
Posted (edited)
23 minutes ago, Jsunflyguy said:

All trains are visible is a reach, the big meltdown events usually come with an event that eliminates the train identifications. ATS dump, there goes all the train IDs, power failure...all the track circuits just flipped and threw all the IDs, data overload...the entire model board just turns solid red, the IDs drop to whatever the non-described default setting is by the manufacturer and you can't look out the window to see what's coming you just have to constantly repeat "train at 32 ball identify yourself" over and over until hopefully people stop stepping on you so you can get the answer. Oh btw the phone is ringing, everyone in the field noticed the problem and is calling you so you can tell them what to do. Having worked both 'in the field' and in a major command center it isn't that its better or worse, in fact once you have a decent information system you and the person on the ground will get the same data at relatively the same time anyway. For busy events at my last job we actually took a team of Ops personnel down to the Airport and Dispatch the planes on the ground in real time face-to-face because playing information-tag was slowing down the operation. 

IDs can get messed up during even normal operation conditions, but for non-signal events like 12-9s, BIEs, work train issues at critical locations, etc, you usually at least have all the basic track circuit data and maybe some IDs in one place to look at. That’s of course reliant on the backend infrastructure being something above garbage (not the case in NYC) and staffing levels allowing some flex capacity to handle such taxing events (as you say, a non starter at MTA), but provided both of those, it seems to be the experience elsewhere that having everyone in one room helps move things along.

Also, there are pretty few towers left in the system where you can physically see the traffic moving through your interlocking. Decentralized ops may give you some workload advantage, but that’s again a function of staffing levels. 

A question: RCC gets indication from the punchboxes on the A, right? 

Edited by RR503

Share this post


Link to post
Share on other sites

So my question is why isn’t there a procedural system in place to move trains to the next station if an outage is 10 minutes or longer?

Trains stuck in tunnels not moving is never a good look, but since these machines move forward, and the motorman has a window to see the red tails of the leader train, why can’t they “key-thru” or whatever the tech term is and platform at the next station, empty the train, and then do the same to go into the tunnel and sit so it’s follower(s) can do the same?

(But my bigger question is why (MTA) are reliant on one computer to run this instead of it being duplexed or triplexed so one can reboot without shutting the whole system down?)

Share this post


Link to post
Share on other sites
24 minutes ago, Deucey said:

So my question is why isn’t there a procedural system in place to move trains to the next station if an outage is 10 minutes or longer?

Trains stuck in tunnels not moving is never a good look, but since these machines move forward, and the motorman has a window to see the red tails of the leader train, why can’t they “key-thru” or whatever the tech term is and platform at the next station, empty the train, and then do the same to go into the tunnel and sit so it’s follower(s) can do the same?

(But my bigger question is why (MTA) are reliant on one computer to run this instead of it being duplexed or triplexed so one can reboot without shutting the whole system down?)

They almost always do something of the sort — key by until you can get a door panel in the platform. It just takes a while, requires coordination that’s usually provided by RCC/towers, and gets essentially impossible when you’re trying to bypass a non-functioning home signal.

Share this post


Link to post
Share on other sites
33 minutes ago, RR503 said:

They almost always do something of the sort — key by until you can get a door panel in the platform. It just takes a while, requires coordination that’s usually provided by RCC/towers, and gets essentially impossible when you’re trying to bypass a non-functioning home signal.

So is it because that yellow bar next to the signal is in the “up” position when a signal is red, and that bar trips the emergency brake?

Share this post


Link to post
Share on other sites

My question is what happens to express trains that are currently moving over a signal when ATC goes down? Do they go into BIE?

Share this post


Link to post
Share on other sites
5 hours ago, Deucey said:

So is it because that yellow bar next to the signal is in the “up” position when a signal is red, and that bar trips the emergency brake?

Yes, that’s the red signal issue. Automatics allow you to pass red signals by keying by, and you can pass homes if the tower/control center issues what’s known as a ‘call on,’ problem with the latter is that it requires there to be a tower/control center in control of the signals. You also may not want to pass a home if you’re not sure of what a switch beyond it is doing. 

Its important to remember that ATS is a signal control system and not a signal system. All the automatics keep working, and the interlocking signals are all still there — they’re just red. The problem is getting those controlled signals to be, well, controlled, rather than just pretty red lights in the tunnel. 

Share this post


Link to post
Share on other sites
Posted (edited)

I’m on a Queensboro Plaza-bound (N) via Brighton local. Apparently, there’s a switch problem at 36 Street.

Pictures: https://imgur.com/a/rHgwWa6

IQoihKE.jpg

boARxlp.jpg

Edited by CenSin
pictures
  • Upvote 1

Share this post


Link to post
Share on other sites
38 minutes ago, Mtatransit said:

Regarding yesterdays meltdown, someone on reddit posted this video.

Not gonna listen too long, but 2:00 in is telling. You've got a T/O asking to do the right thing to help his customers, which is wrong rail with caution into 3rd Ave to put a door panel in the station and allow passengers to deboard. RCC says that won't be allowed out of an abundance of caution. I understand their angle, but this is where they should respect and trust T/O judgment. At <5mph creeping, with other trains stationary, the risk isn't that high, and the reward (allowing however many hundreds or thousand-plus people to avoid sitting an hour and a half on a stopped train) is entirely worth it. 

  • Thumbs Up 3
  • Upvote 3

Share this post


Link to post
Share on other sites
Posted (edited)

It's evident the MTA is not prepared for this heat:

 

Multiple Impacts  Posted: 07/20/2019  4:54PM 

(D) and (F) trains are delayed in both directions because of switch problems and signal problems in Brooklyn

There's no (F) train service in either direction between Avenue X and Coney Island-Stillwell Av

Some southbound (F) trains will end at Church Av

Northbound (D) trains are running on the (N) line from Coney Island-Stillwell Av to 36 St (BKLYN)

 

Also today: (7) suspension (QBP-61st) , no 4th Avenue service due to a switch problem at 36th Street.

Edited by BM5 via Woodhaven

Share this post


Link to post
Share on other sites
Posted (edited)

Multiple Impacts

(A) train service has been restored in the Rockaways after a PSEG power outage on Long Island disrupted our service. 

There is still no Rockaway Park (S) service between Beach 116 St and Broad Channel

---------------------------------------------------------

Trains Rerouted

Some northbound (D) trains are running on the (N) line from Coney Island-Stillwell Av to 36 St (BKLYN) while we address switch problems at 9 Av

---------------------------------------------------------

Delays Posted: 7/20/19 6:47 PM

SIR trains are delayed in both directions because of a person struck by a train at Eltingville.

Edited by BM5 via Woodhaven

Share this post


Link to post
Share on other sites

Service is gone in the Rockaways again:

 

Part Suspended

Far Rockaway-bound (A) trains are ending at Howard Beach or Broad Channel and Rockaway Park (S) service is suspended while PSEG works to restore power in the area. 

Consider using Q22 or Q53-SBS bus service as an alternative.

Share this post


Link to post
Share on other sites
15 hours ago, RR503 said:

Yes, that’s the red signal issue. Automatics allow you to pass red signals by keying by, and you can pass homes if the tower/control center issues what’s known as a ‘call on,’ problem with the latter is that it requires there to be a tower/control center in control of the signals. You also may not want to pass a home if you’re not sure of what a switch beyond it is doing. 

Its important to remember that ATS is a signal control system and not a signal system. All the automatics keep working, and the interlocking signals are all still there — they’re just red. The problem is getting those controlled signals to be, well, controlled, rather than just pretty red lights in the tunnel. 

I guess the next question on this is why wasn’t a redundant system put in place? And I don’t mean a separate signal control system - I mean a less-manual mitigation system.

Pretty sure someone tweeted that they saw someone walking ahead of the train keying signals to green so it could platform. If the blocks are two football fields long, then we know that’s an operation that takes lots of times since no one’s running these blocks as fast as a footballer (ie 20 yd sprints x 33 yards in a block), plus the time it takes to key it.

So why wasn’t ATS built with a control to turn the lights red but lower these yellow arms when the system is down for 5-10 minutes, and then have two-way radio guidance to platforms for emergency evacs? (As an example.)

Did no one think of what to do when a reliable system eventually fails?

Share this post


Link to post
Share on other sites
56 minutes ago, Deucey said:

I guess the next question on this is why wasn’t a redundant system put in place? And I don’t mean a separate signal control system - I mean a less-manual mitigation system.

Pretty sure someone tweeted that they saw someone walking ahead of the train keying signals to green so it could platform. If the blocks are two football fields long, then we know that’s an operation that takes lots of times since no one’s running these blocks as fast as a footballer (ie 20 yd sprints x 33 yards in a block), plus the time it takes to key it.

So why wasn’t ATS built with a control to turn the lights red but lower these yellow arms when the system is down for 5-10 minutes, and then have two-way radio guidance to platforms for emergency evacs? (As an example.)

Did no one think of what to do when a reliable system eventually fails?

You're talking about entire stretches of track in a high-frequency network where the system failing effectively renders the entire thing dark territory.

You're also talking about radio communications in a system where catastrophic failure results in people basically entering panic mode and calling emergency services all at once, thus worsening the situation.

Share this post


Link to post
Share on other sites
19 minutes ago, Lex said:

You're talking about entire stretches of track in a high-frequency network where the system failing effectively renders the entire thing dark territory.

You're also talking about radio communications in a system where catastrophic failure results in people basically entering panic mode and calling emergency services all at once, thus worsening the situation.

That’s just one idea. There’s others.

My question is why is it (MTA) doesn’t have an “Oh Shit” backup plan?

After the IRT yesterday (Friday), and the Lex Line during the Cranberry Tube reconstruction (All Lex trains went to BG, and all service was stopped to where all trains were feet from each other in Downtown), and the West 4th St fire in 2015 that stopped all IND trains during evening rush, it’s evident that there needs to be some sort of plan other than ‘sit here and hope it all works out.’ 

Why isn’t there one?

Share this post


Link to post
Share on other sites
1 hour ago, Deucey said:

That’s just one idea. There’s others.

My question is why is it (MTA) doesn’t have an “Oh Shit” backup plan?

After the IRT yesterday (Friday), and the Lex Line during the Cranberry Tube reconstruction (All Lex trains went to BG, and all service was stopped to where all trains were feet from each other in Downtown), and the West 4th St fire in 2015 that stopped all IND trains during evening rush, it’s evident that there needs to be some sort of plan other than ‘sit here and hope it all works out.’ 

Why isn’t there one?

Did the second part of my response fall off your radar or something?

Let's say there's an explosion at 18th Street and 5th Avenue. If only one or two people call it in, emergency services will be able to dispatch in an effective manner and still be able to process other issues should they arise; if 30 do the same, emergency services will be swamped and unable to handle other issues.

Applied to a rail system, this concept is exacerbated while underground or on a guided path (to say nothing of both applying), as operators will still need instructions in order to safely move across limited vectors. If dispatch isn't able to see the trains before a particular distance is closed, it only becomes more difficult to move.

For clarity, I'm no employee. All I'm doing is considering my own experiences, how the system is set up (including the limitations), and concepts that others have stated (whether here or elsewhere).

Share this post


Link to post
Share on other sites
13 hours ago, Mtatransit said:

Regarding yesterdays meltdown, someone on reddit posted this video.

 

 

I was hoping someone got audio from the meltdown. Just goes to show how crap TA's Radio Comms are. In Dyre situations like this, communication is key. Kudos to the people at RCC though, must've been running around like headless chickens up there lol

 

  • Thanks 1

Share this post


Link to post
Share on other sites
6 hours ago, Lex said:

Did the second part of my response fall off your radar or something?

I deliberately ignored it because it had less to do with my question and more to do with you being upset I asked it.

Share this post


Link to post
Share on other sites
8 hours ago, Lex said:

You're talking about entire stretches of track in a high-frequency network where the system failing effectively renders the entire thing dark territory.

You're also talking about radio communications in a system where catastrophic failure results in people basically entering panic mode and calling emergency services all at once, thus worsening the situation.

This isn't true. ATS-A was specifically designed to be a non vital overlay over interlocking logic. When it fails, RCC loses control of interlockings, but someone in a master tower could easily take over provided the failure wasn't itself caused by some local interlocking defect, which, in these system-scale cases, it usually is not. 

9 hours ago, Deucey said:

I guess the next question on this is why wasn’t a redundant system put in place? And I don’t mean a separate signal control system - I mean a less-manual mitigation system.

Pretty sure someone tweeted that they saw someone walking ahead of the train keying signals to green so it could platform. If the blocks are two football fields long, then we know that’s an operation that takes lots of times since no one’s running these blocks as fast as a footballer (ie 20 yd sprints x 33 yards in a block), plus the time it takes to key it.

So why wasn’t ATS built with a control to turn the lights red but lower these yellow arms when the system is down for 5-10 minutes, and then have two-way radio guidance to platforms for emergency evacs? (As an example.)

Did no one think of what to do when a reliable system eventually fails?

The backup system is essentially the old towers and key bys. As @Jsunflyguy and I have discussed, it can be hard to immediately staff the master towers that underpin ATS, but if you do, then it's significantly easier to bring back service. There's (less) of a need to do these manual bypass ops. Until then, they rely on this sort of ad-hoc, location guided system to get at least one panel of each train into a station. 

To the point of the stop arms, though, when you are working to restore service after a failure, you can indeed hook stop arms down so that you don't have to repeat the process manually every time a train needs to pass. Sure as hell is not as efficient as just controlling the signals, but you in theory only would have to walk through a given area once. 

  • Upvote 2

Share this post


Link to post
Share on other sites

Suspended

There is no Franklin Shuttle (S) service in either direction while we work to correct signal problems at Prospect Park

See a station agent for a courtesy pass to continue your trip. 

Go to http://bt.mta.info/ to see real-time bus arrival info for the B43 or B48

Follow this link to see bus route options: http://bit.ly/2Yh8Q2u

Share this post


Link to post
Share on other sites

Multiple Impacts

2, 3, 4 and 5 train service between Manhattan and Brooklyn will be disrupted for tonight's rush hour because of a malfunctioning switch at Franklin Av that our crews will address later tonight.


Here's a summary of the service changes:

Southbound 4 trains will run on their normal route. 

Northbound 4 trains will run express from Crown Heights-Utica Av to Atlantic Av-Barclays Ctr. For service to Kingston Av or Nostrand Av, transfer at Franklin Av for a southbound 3 or 4 train.

Southbound 5 trains will end at Bowling Green. There is no 5 train service in either direction in Brooklyn.

2 trains and 3 trains are running on their normal route. 2 train service south of Franklin Av may be more limited than usual

Part Suspended

2 train service between Franklin Av and Flatbush Av-Brooklyn College is suspended because of multiple switch malfunctions near Franklin Av.

 

There is no 5 train service in Brooklyn. 

 

Take the B and Q trains making nearby station stops and transfer to the B41 bus at Prospect Park for service to Flatbush Av-Brooklyn College. 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.