<Kopachris> okay, so here's how our floor is wired:
<Kopachris> each slot machine communicates with a Slot Machine Interface Board (SMIB) inside the machine's cabinet using RS232
<Kopachris> multiple SMIBs are daisy-chained to each other using RS485, bank-in and bank-out on each SMIB electrically connected
<Kopachris> the first unit in this daisy chain of SMIBs (up to 31 of them) is then connected (again, using RS485, over Cat5 or Cat6 for this part) to a Data Collection Unit (DCU)
<Kopachris> multiple DCUs are then daisy-chained using RS485, with the first unit on the chain bein connected to a multiple serial-parallel adapter, which is plugged into the server
<Kopachris> something like 3-5 DCUs per serial port on the adapter
<Kopachris> now, since we can have up to 31 SMIBs (==31 slot machines) addressed on each DCU, we typically run RS485 (actual shielded RS485 cable) from one pod of slot machines to another one a few yards away
<Kopachris> today, we had issues with all the machines on DCUs 11 and 12. When I check the room with the servers and DCUs, I notice 10, 11, and 12 are daisy-chained together (so connected to the same port on the server)
<Kopachris> checking our database, I find DCU 10 doesn't have any slot machines connected to it
<Kopachris> okay, lets plug the server directly into DCU 11, completely bypassing DCU 10
<Kopachris> no change
<Kopachris> plug it into DCU 12, bypassing 11
<Kopachris> all the machines on DCU 12 come back up
<Kopachris> just to make sure, plug the server back into DCU 10 like it was, but daisy-chain 10 to 12, still bypassing 11
<Kopachris> everything on 12 still up
<Kopachris> okay, so the problem is with DCU 11 or the machines connected to it
<Kopachris> plug everything back the way it was, replace DCU 11, no change
<Kopachris> so then we start tearing apart the SMIB daisy chain connected to that DCU
<Kopachris> we think we have the problem isolated to one SMIB, so we go around it, taking it completely out of the daisy chain, and continue adding SMIBs to the daisy chain one by one
<Kopachris> it doesn't help that it was in intermittent issue to begin with - the SMIBs would lose communication with the server (not the DCU!) for a few minutes, then it'd be back for a few minutes, then down again, etc.
<Dis> sounds like a bad cable
<Kopachris> exactly
<Kopachris> so then we notice, "huh, the cable coming up from under the carpet on pod E-10 is a different color than the cable going into the carpet on pod E-08"
<Kopachris> so that means there's a splice somewhere between E-08 and E-10 (which is a no-no because it causes exactly this kind of problem)
<Dis> someone's in trouble!
<Kopachris> then we notice that the cable leaving E-10 is also a different cable than the one coming up under E-06! (DCU 11's daisy chain goes E-08-01 thru E-08-04, to E-10-03 thru E-10-02 (six machines), to E-06-01 thru E-06-06)
<Kopachris> so there's *another* fucking splice between E-10 and E-06
<Kopachris> and since it's Saturday on Super Bowl weekend, it's way too busy to start ripping up carpet and replacing cable
<Kopachris> so we just start RAM clearing and re-optioning SMIBs and hope that helps because I noticed in the logs that one of those machines sent an unknown command code earlier in the day, and thought maybe it's still trying to retransmit that if it's not getting a response
<Kopachris> that seemed to help a bit, it's more stable now than it was before
<Kopachris> now 22:00 rolls around and we've been working on it for six hours, the next guys come in to get ready for the 23:00 shift, and they tell us apparently last night on graveyard shift someone plugged a blower in on one of those rows to clean the machines and it popped the circuit breaker
<Kopachris> so now I'm thinking that might have fried some of the SMIBs
<Dis> that would've been a nice note to leave for your shift
<Kopachris> they did leave a note in their written shift report, but we didn't get to read it because day shift didn't mention it in their verbal pass-along so we've been fucking with it all day because "Fuck, we've got how many machines down??!?"
<Dis> fair enough
<Dis> they could've said "oh yeah, everything's fucked. Bye!"
<Kopachris> I'm guessing they didn't even know
<Kopachris> I was reading my email, hadn't read the shift report yet, when a slot service specialist (the ones who handle jackpots/hand-pays and really basic machine troubleshooting stuff, not technicians) calls over the radio that communication on E-08 is going up and down