A friend just reminded me of some of my antics when I took a computer job and moved to Ohio for just over a year. I had been laid off about 6 months prior from a tech support job in Ann Arbor, was paying the bills – barely – by running my own website “TheWild” when I was contacted by a recruiter.
A small, family owned printing and marketing company in mid-Ohio was interested in getting a new linux systems administrator for their small regional internet service provider. The money was a slight upgrade from what I had been making in tech support but I wasn’t really enthusiastic about the thought of moving to Ohio. But I agreed to an interview, and scheduled a trip down to Mansfield.
The company was reasonable and the people seemed nice but the ISP was just a small operation out back. Their system consisted of a single T1 line, 3 linux boxes, one Irix (SGI) computer for 3 of their larger customer websites and a couple of privately owned Macintoshes. They had apparently been having some trouble with their previous system administrator but they weren’t clear as to what the trouble was, so I didn’t inquire.
(An interesting aside, I stopped to visit an aunt who lived outside of Cleveland on my way home to stay the night. I hadn’t seen her but twice in my life but was interested in getting to know her. I rode my motorcycle down there and stayed the first night at a friend’s house in Dayton and we stayed up all night laughing our butts off doing stupid stuff. I do the interview and head for Cleveland but end up having a hard time finding her house. I’m going on no sleep in 2 days and someone is burning leaves in the vicinity of her neighborhood which is messing with my already frazzled eyes.
I spend close to two hours driving through this smoke trying to find her exact neighborhood, stopping a few times to call and clarify the directions. I finally show up at her place completely exhausted, eyes completely blood shot and considerably road beaten from the small 535 bike I had at that time which wasn’t really made for long trips like that.
I head home the next day to Ann Arbor, again pass out asleep and I am woken up by someone banging on my front door. It’s my father – who lives 200 miles away. He says simply “get in the car, we have to go to your Aunt’s.” Turns out she couldn’t find her – she claims – $30,000 engagement ring, she is accusing me of stealing it saying I showed up to her house ‘hopped up on drugs’ and is threatening to call the cops.
We drive all the way back to Cleveland, get essentially no where in offering to help look for the ring or trying to figure out where ‘she’ might have misplaced it. She finally agrees not to call police but we leave flustered. Apparently she found the ring a few days later – where she had put it and forgotten. She never bothered to call me, my parents or her sister to let anyone know and never apologized. Needless to say, I never went to visit my aunt again while living in Ohio.)
So I get home yet again after that mess, frazzled even further. A few days go by and the recruiter calls me telling me the company wants to make me an offer. Mind you I am not interested in moving to Ohio, but he asks me what I would want so he can present it to them. So I give him a list. I quote a price considerably higher than I would have done the same job for in Michigan, I tell him I will need at least a $1500 signing bonus to cover the cost of the move. They are to be responsible for my housing for the first 2 weeks and have information on available housing gathered for me when I get there. I tell him they have to check with the two nearby colleges and before I accept they have to assure me that at least one of them carries either a MIS or a BIS degree program locally. And I have to have at least joint decision power as to any new system purchasing decisions as well as some assurance that any purchasing suggestions I make are taken seriously.
Mind you, I am trying to be as difficult as possible because I don’t want to move. It was less than 24 hours and he calls me back to say “they agreed” with only one small caveat. They agreed that if they did the signing bonus, that I’d start at a lower rate of pay to be increased to the amount I asked for in 6 months if things were working out and they intended to keep me on. I couldn’t believe it and was kind of “called on my bluff” – and needed the money frankly – so I figured what the hell!
I packed up in the next few days and got a truck and headed for Ohio. Disappointments started rather quickly. Come to find out their idea of ‘taking care of my housing’ was having me sleep on someone’s couch. They also had no available information on local housing for me when I arrived, but instead had my supervisor drive me around to give me a ‘tour of the town’ pointing occasionally to apartment complexes as he drove about. By then I had already spent the a good deal of the signing bonus on the moving truck, rent on a storage facility, gas for the truck, etc. and needed to keep at least part of it available for security deposit and at least part of it for a first months rent if and when I found a place.
Alas, I am resourceful so I managed to find a ‘sleep room by the second night. It was slightly outside of town but held me down for the first week, but still ate away at another reasonable chunk more than I wanted to of my money reserve for a security deposit. The company meanwhile, turned out to be shoddier than it had seemed with just 3 small linux boxes running their primary ISP, web and email services respectively and a wall of USR modems zip-stripped to a piece of plywood to handle their incoming dial-up customers.
On my little ‘tour’ with the supervisor, I managed to learn some of my first hints as to the ‘problems’ with the prior system administrator. In short, the guy was a hack. In fact he wasn’t only a hack, he was a hacker. I quickly began to realize upon starting to look through the 3 linux systems that there was very little organization at all. The flavor of linux he was using at the time was slackware which at the time was all ‘tar ball’ installation files and an administration nightmare as far as time consumption.
The packages were all considerably out of date. Security seemed to be non-existent. The email was running a default sendmail distribution with a virtually unmodified rc file. They were handling their own DNS and his files had no heirarchical order – or even any hint of esthetically readable formatting whatsoever! The web server was at least a 3 major version old apache distribution with little or no cgi optimization. And there were extremely few administrative scripts available for handling day to day operations, instead he had preferred to do most functions by hand editing or scripting directly from the shell prompt to do things that needed to be done.
On the subject of security, I had inquired about it to the supervisor on that drive around town. He rolled his eyes and said something to the effect of “he knows all the hackers online. That was part of the problem. He spent more time in underground irc hacker chat rooms than he did working.” He then made some kind of joke suggesting that if anyone had ever tried to hack them, the former admin would have probably known who it was and hacked them back doing considerably more damage to their systems in response. This was their ‘security’.
But, like with most things, your security risk was only as high as your exposure. This was before wide scale port scanning and high speed internet lines. And I had a lot of items on my plate. Although I put the security issues near the top of my list, I assumed I would likely have a couple of weeks to get things up to speed security wise before anyone malicious in netland would even happen to notice the other guy was gone or that there were any issues.
Well, I don’t know if the other guy was pissed and told his buddies to look closer at the company machines or if I just happened to show up at the wrong time, but less than a week in, they got hacked. It took me less than 30 minutes to detect it and less than 3 to shut it down but by then considerable problems had been created. An exploit was used in an old plugin on the old apache server to pull the /etc/passwd file where all the encrypted password were stored some hours earlier. They had then used brute force methods to obtain a number of passwords, made even easier by the fact that they had no restrictions in place on customer generation of passwords to prevent the use of common words and phrases.
(Yet another aside: the initial web accesses to exploit the problem came in rapid order, like that of people ‘sharing notes’ upon discovering a system with the flaw. The addresses came from 5 different locations, two in the United states, one in Canada, one in the UK and one in Israel. I never got an exact confirmation, but some of the comments made by the agent who I ultimately was put in touch with by local police at the FBI Infragard division that covered Ohio lead me to believe that this wasn’t just an ordinary ‘script kiddie’ hacking, even though it essentially was. Something made me suspicious that the FBI had additional interest in this case but wasn’t letting on as to “why”.
It wasn’t too long after leaving Ohio that I heard Clifford Stoll speaking on some interview program and they referred back to his book “A Cuckoo’s Egg” and he referred to events similar to some of those that he described in that book that he had either been involved in or caught wind of since the book’s writing. The primary situation he described involved a ‘hacking ring’ that the FBI had recently caught that was based out of Israel and was funded by Hamas. They eventually attempted, and in some case had minor successes, at utilizing various known exploits or buffer overruns and brut force attacks against a number of US government and Israeli government systems on the internet. But, before they did this – and what brought the FBI and interpol’s attention to them – was the fact that they were slinking through IRC chat rooms and other ‘hacker’ forums giving tips and advice to young kids online as to how to hack into computer systems and monitoring their progress. i.e. they were getting these minor kids with a knack for computing to do their ‘dry runs’ for them and thus avoiding direct criminal liability in the process but getting real-world examples of whether or not these methods worked ‘before’ they eventually attempted them on their true targets.
As I say, I never did discover if there was any link, but I did have a single hit from an Israel net address. ALL of the other addresses had more than one connection to the system and each spent at least 5 minutes or more dinkering on our systems before I detected them and shut them down. The Israeli address simply logged in and logged right back out, as if to see that in fact things did work.)
The initial problem was that they made telnet available to their customers. I immediately shut the telnet access off on all the servers and that solved the gaping hole. A few other trojan type scripts were dropped in place, but looking at the history files for the shells on the detected accounts allowed me to quickly track down just about all the things that had been done in a matter of minutes. But then came the hard part.
I spent the better part of 15-18 hours a day for the next week working on not only my regular job functions but securing all the systems, generating new scripts to create more difficult-to-crack passwords, brute forcing the passwd file myself to generate a list of customers that would need to be contacted to update their passwords (in addition I suggested the company also strongly suggest everyone change their passwords regardless using yet another new tool that would restrict ‘weak’ passwords from being used)
I had to replace all the telnet access with ssh, contact all the customers that wanted to use telnet to let them know not only to use ssh now but ‘how’ it worked and how to set it up. I found there were a number of background cron jobs hiding hither and yon to either mirror information or to pass data that should have been secured. I found these because one of the hackers rather immediately uploaded and ran a ‘sniffer’ program and grabbed even more passwords using it through clear-channel passing of data. I had to change all those to ssh tunneled cron scripts instead.
I then replaced all the default shells on all the accounts that were allowed the access to prevent it from being able to execute certain commands such as the sniffer tools as well as change permissions system wide to prevent any of those accounts from being able to ‘see’ stuff they shouldn’t. I then shadowed all the passwd files so they weren’t accessible directly from those accounts either.
After I had the systems reasonably secured, I generated for myself a list of the ‘known’ compromised accounts that were either inactive or hadn’t been used by customers in some time. I did this because I then set up a false redirector in apache on our ISP bulletin board and a few other internet viewable services such that if anyone tried to log in from those accounts, they would be taken to a ‘mirror’ of the real service running instead on my PC where I could ‘watch’ what they were doing and trace their addresses.
While this was all going on, I was still doing my regular job functions, contacting local police and the FBI to arrange a criminal investigation, and was still readily making suggestions on systems to both my supervisor and the manager in the print shop out front. Also while this was all going on, the guy that administered the Macintoshes was coming and going regularly during regular business hours.
Before all was said and done, my little ‘false’ web service caught two mice. By the third day I got a call back from the FBI field operator saying that they finally managed to get past the clerk and were going to be going to the judge that afternoon to get a warrant to go after the most persistent of the US hackers I had been tracking from my trap. The other persistent one was in Scotland and he had heard from Interpol that Scotland yard was already executing a warrant. I had been talking with him on and off over those few days and giving him address information and updates on the activity attempts so I had an idea he would be calling with this information.
Because I knew he would be going after them soon, I closed my trap by shutting down the ‘fake’ website and removing the redirects and blocked off all the addresses seen on it from being able to do anything at all on any of our networks until the authorities caught up with them. But, because I had pulled down the ‘false front’ suddenly I noticed a taunt from one of them on the ‘real’ bulletin board. I checked and logged the address and promptly blocked it and up comes another taunt a few minutes later akin to ‘nice try, ha ha ha‘. I blocked that address and up comes another a few minutes after that.
So I look closer at the addresses. These aren’t dial-up or pooled addresses behind a ISP in some small town, these are company and business networks scattered all over the place. Yet it is obviously the same character who had been posting similar taunts on my ‘fake’ version of the company BBS. With a little more sniffing I was able to figure out what was going on. I temporarily shut down the accounts he was still using while I investigated further.
What the hacker had been doing was using a ‘hacker list’ of known, unsecured ‘proxy’ servers which basically work like a repeater for web traffic allowing him to access the website through the unsecured system. To our systems it looked like the traffic was coming from the unsecured proxy rather than from his address that was already blocked. But he made one mistake. As I checked the list of servers he was using, the last one he was on was in Michigan. With about 10 minutes of making calls to people I knew, I was able to find someone that knew someone that knew someone at the company that had an insecure proxy server. I had a name and a phone number of their system administrator and a “tell them I told you to call” to go along with it.
So I called the admin at the company that ran this hijacked proxy server, told him the situation, told him who had referred me to him and even told him how he could lock his proxy server down. He was more than happy to send me a complete log of all the activity that had been done from the address in question. Taunty boy made yet another mistake. After I completely locked him out, he forgot to set his proxy settings back. He then started surfing around the web looking for other potential exploits to use against me. I had a full laundry list, courtesy of his forgetfulness and my resourcefulness, to show me everything he was going to try next.
I started to have some fun. I checked through logs and found some of the things he had tried already but some he hadn’t. I set a few more traps for him just to send him some little taunts back while I continued to review the log. I also set alarms so I would know when he hit them, then promptly blocked the addresses off for the new proxies he was using. Not only was I taunting him, but I was using it to stop further attempts and further log his activity. By that point in time he must have been completely scratching his head and likely feeling he had been outwitted. Every time he tried an exploit, not only did it not work but it came back with a counter-taunt as though I was reading his mind.
Finally I got to the end of the log list of his activity. Mistake number 3 and the clincher! The big one! (and frankly I got lucky he was so stupid!) At the end of the Michigan company’s proxy log there was about a 2 hour break. Then a single hit to a public server for a company that offered free email services. Unfortunately for Mr. Bubrouski, this particular server required users to enter their full legal name for their email address, and only allowed you to access your email through a web interface – one that he was now accessing through the proxy server he forgot to turn off and for which I had the complete log of his activity. And the URL that you used to access said email included the first part of your address before the ‘@’ – his real life name!
I hit the web and in less than a few minutes had his home address which was under his ‘mothers’ name – it was just a kid. I also found his uncles’s name and address half way across the state. I found a few other tidbits of information on him just about the time the FBI officer called to let me know about the search warrant. When I told him the kid’s name and address he almost crapped. When I told him the kid’s mother’s name and where she worked, he didn’t even know that! When I told him where the kid went to school, he almost offered me a job!
(another interesting aside. When I told the FBI agents what I had done with ssh and all the automated ssh tunnels to secure the systems they were quite intrigued. What I didn’t tell them was that prior to this event, I had never even worked with ssh before and learned to use it in a matter of hours that first night. He asked if I could come to their next regional meeting which was in Columbus and would have agents from the entire east coast at it. I was now intrigued. Then I managed to figure out he expected me to give this talk for free. I told him to get stuffed!)
While this all was ongoing, as I mentioned, I was still making suggestions to my supervisor and manager. Among them I had suggested some ideas for getting new systems. I figured out a scheme by which if we added 2-4 new linux boxes, I could distribute process in such a fashion that no one server was solely responsible for any one function of the ISP. I could set each up with one primary, one secondary fallback mirrored and one standby inactive service each. That way if any one system either had a problem or needed to be upgraded or otherwise repaired, there were two other systems that could either manually or automatically be switched in to take it’s place with little or no downtime. Another suggestion I made was to being phasing out the plywood board and instead get an integrated unit with built in multi-modems all in one box. I made a handful of other suggestions as well. None seemed to illicit a response or even a reasonable acknowledgement.
I mentioned earlier also, the Mac guy. Somehow, as this was all ongoing and I’m busting my fucking ass to not only to do my normal job, do 2 months worth of work in 5 nights, fix things that were broke, make other people’s jobs of fixing the problems resulting from the ineptness of my predecessor so the entire thing doesn’t become a public relations nightmare, etc. and tracing down the hackers while sharing the info with the feds, Mr. Macintosh, an uppity lout with of British descent complete with accent, manages to convince the manager across the street that I’m not competent enough to be chief administrator. Provision #3 busted, I lose my administrative input on purchase decisions when they hire that asshole over me.
Alas, I felt like Dagney dealing with her brother James in Atlas Shrugged. He comes in and makes the same suggestions I had made previously that were ignored. But when he makes them, suddenly 3 new linux boxes show up, suddenly a new Cisco integrated multi-modem device shows up AND a second T1 line to feed it. Alas, I am a bit put off that they ignored me, but then jumped too when this British jackass suggested exactly the same things. But I still felt like (and emulated) Dagney thinking “who cares, at least they got done“.
Things settled back down and I was able to get back to 8 hour shifts so I checked in at the two local colleges. Lo-and-behold, they never checked for my degree program. Neither one offered it. Provision #4 shot all to hell! By this point in time, my money was running short and all the extra hours meant I had no time to look for a place. And I was still a good week away from my first paycheck. I kid you not, to make sure I would still have enough money to apartment look during the week, I checked out of the sleep room and slept on my couch for 4 days by sneaking into my storage bin after hours. This is what these incompetents had brought me to!! Welcome to f’king Ohio!
I had actually found a place by the time the first paycheck showed up, a small cabin that was for rent out on the west end of town back out in the woods. The people that owned it were a couple of Kentucky transplant hillbillies and their dad lived in a house just over the hill. Apparently this cabin was the ‘dad’s first house. It was a dump of a log cabin in the woods – ahhhhhhhh, it was just like living at deer camp year round! I was in heaven! (about the only problem with it was gramps over the hill still had so much ‘hill billy’ left in him that he refused to pay for trash pickup, instead burning all his non-compostable trash in a barrel in his front yard every few weeks stinking the whole area up to high heaven!)
The walls were all of 3” thick and poorly insulated, but they understood this and agreed to pay all the gas bills so who gave a crap? The windows were single pane so in the winter you didn’t dare sit too close to one, but they paid the electric too so I just got a small space heater for the living room for when I was watching TV. The roof was even thinner than the walls and when the walnut tree started dropping it’s nuts in the brisk winds of fall in that Ohio flatland, it sounded like world war III. But I had raccoons, turkeys, deer, flying squirrels, and all sorts of other critters hanging around the house regularly and I loved it! If you turned all the lights off on a cloudy night, you couldn’t see your hand in front of your face. (in other words, it was just far enough out of town, you had no city lighting to bother you – it was like living way out in the woods)
And the piesta resistance of the cabin was the closet opposite the front door that was near the center of the house and directly below the central peak of the roof. Or more specifically, what was ‘inside’ the closet. Open the closet and not only do you find a rod for hangers on which to hang your coats and a shelf above the rod, but there was a hole cut in the shelf before it had been put in place. Why? Because there was a TREE GROWING THROUGH IT. The the entire building had been constructed around it! You could only see the main trunk by opening the closet and the top had long since been cut off just above the roof to prevent the tree from out growing the cabin and starting to effect the construction, but it made for quite a conversation piece all the same!
(yet another aside: Mansfield, Ohio is in the center of the state. It’s the home of the state prison. I learned years later while watching my grandparents house after my grandfather passed away that the characteristics of Mansfield are consistent with the characteristics of Jackson – another city housing the state prison. The town consists of an eclectic mix of aging retired people of ‘old country’ values, low-life criminal or semi-criminal element possibly attributable to the locality of the prison – i.e. they are only living in town to be closer to loved ones on visiting day, and the outskirts entirely surrounded by rednecks, hillbillies and Amish or menonites.
Any ‘nightlife’ that was to be had required making over an hour drive north to Cleveland or the surrounding suburbs or an hour south to Columbus to hit the down-town college hotspots. At one such hotspot, I was minding my own business when I overheard some girls making jokes and – not really flirting, I made some crack back. The girl closest to me found it amusing and somehow realized I wasn’t being overbearing by making it but simply making conversation and started to talk with me. She soon reassured herself that I wasn’t the typical male asshole and the other girls eventually followed her lead. Turns out that they were all either workers at or a significant other of one of the workers from a nearby topless dance club somewhere just outside of Columbus. They were all lesbians and found most of the men at these clubs downright hilarious because “they always thought they were going to score and never could understand I am only being friendly with them to get better tips. When I’m done with work, I go home and the only thing going between my legs is her!” she said pointing to a girl blushing nearby.
They were a hoot to hang out with and they liked me because I didn’t need to be told ‘no’ every five minutes and helped them keep any clueless guy’s away, taking turns pretending I was their ‘boyfriend’ if for any reason their ‘girlfriend’ wasn’t there to do the job at the time and some guy was being overbearing. As a result, I hit that club a lot and this led to another interesting combination of events.
Shortly after moving down, I bought a new truck at a local car dealer. As with most used car dealers, I don’t think he told me the whole story. I came to suspect reasonably quickly that this one had been in some kind of accident before – a plain black, no frills Chevy ‘work truck’ with a 5 on the floor and a basic, stock engine and no frills when it came to just about any options. It even had the basic rubber flooring. Just fine for some guy that lives in a house built around a tree behind a bunch of hillbillies way back in the woods! But one of the things that led me to realize it may have been in an accident was the fact that it had a ‘lying gas gauge’ Or more specifically, the gas gauge had ‘shifted’ about 15 degrees in a clockwise direction such that the tank was ’empty’ when the needle read 1/4 tank.
The first night that I learned of this fact was the very night that the girls at the Columbus bar dared me to show up to a Halloween party in drag to hang out with them as ‘one of their “bitches”‘ at some big dance club that was throwing a huge Halloween affair. I found out about the lying gas gauge about 20 miles from no where smack dab in the middle of redneck and Amish country. Dressed up as a woman of all things! [images of Deliverance started to run through my mind as I pictured some kind of guy with 3 front teeth telling me “y’all gots ya some reeel purty lips thar. Bend over and squeel like a pig!”]
I assumed my odds of finding gas at any of the amish houses was low, so instead I went to one local farm house that was just up the hill and tried to stay outside in the fading daylight and the shadows of the trees on either side of the front door, just far enough away that the old woman that answered the door couldn’t get a good look at me. I spoke low with an androgynous voice so that if she ‘read’ me it wouldn’t sound like I was trying to fake it but if she “didn’t” I might get out without too much embarrassment either way. I asked simply if she had any gasoline, then if she could call a service station down the road about 10 miles if they could send a tow truck with gas.
She said she would but invited me in to wait while she called. I actually had a cell phone in the car, but the battery was already low and I didn’t want to use it unnecessarily and my main interest was just getting gas and getting on my way as quickly as possible! I went back to the car to wait for the tow truck, when another car pulls up behind me. I still do not know to this day if that guy read me or not, but it was some local country boy that had seen my hazard flashers and deduced I had run out of gas. I tried to stay distant from him in what was now near dark and tried to keep my words to simple, androgynous “thank yous” and short inquiries as to if I could pay him for the favor. He said no and even hinted at it being his honor to ‘help out a damsel in distress’ – SQUEEEEEL. – OK, unfair, he was very nice and polite and respected my ‘shyness’ for whatever reason it was – whether he read me or not. [I did look rather convincing though, taking the suggestion of my friends to keep it ‘casual’ looking as they wanted to see how well they too could pass me off. I mean, I was a skinny, long haired rock and roller type. I didn’t even need a wig to pull it off]
Needless to say, I never went to Columbus in ‘drag’ again and always made sure to fill my gas tank!)
The job itself turned out to be a mix of pure nightmare and pure screw off. Since no one else in the whole place was proficient with linux worth a damn, I still pretty much owned all the systems and could still coerce some administrative push and pull out of the sheer fact that no one knew what they were doing on those machines yet their business depended upon them. The nightmare side was made worse by the fact that they assumed that because ‘linux was free’ they never had to throw any money at it. As a result, I could never talk them into getting any serious upgrades.
Once such instance they actually approached me with an idea. We were doing system backups now all on one machine. We were now up to 7 servers and I was mirroring files through my ssh tunnels to that machine to make the back-ups easier, but it was still a process started manually and you had to also manually babysit the tapes. I ended up digging out all of their existing tapes and between backup schedules, I ran full read-write tests on the old ones and threw out any that were bad then set up a rotation schedule on all the tapes to make the process quicker. Running only incremental backups during the week then a full backup on a separate tape each week. I also ran a double backup at the ed of each month and moved the extra tape to the manager’s office across the road. But the process took forever.
They wanted to know if I could further automate it. I said “sure, but so much of my time is taken up with mundane day to day operations. You’d need to throw more responsibility on these young guys and keep them off my back for a week or more so I could write and test the scripts across all the machines.” They said they would think about it, but they were trying to find some way to get me to do it anyway – magically pull it out of my ass with no time spent or something. That or they wanted me to start on 18 hour work weeks again – yeah right! After you already screwed me on every contingency I placed on my coming? You wish!
So while they were ‘thinking about it’ I did some sniffing around. I ended up finding a piece of software that was gui based and had a number of the tools I would need already in it and even allowed installation of the secure tunnel ‘remote’ scripts on multiple machines using a single software license. It was perfect, not that expensive and could be set up in a matter of minutes rather than requiring weeks of scripting and testing. I showed it to them and they said “great, we’ll send over someone with the company credit card.”
(customer service aside: As with most jobs, my wide variety of skills with both computers and technologies of almost all types had me performing a ‘jack of all trades’ function for the ISP. Most of the employees generally helped out at some point doing just about any of the other tasks short of the really technical one’s like mine. I’d even taught a couple of short classes on some subjects while there and helped one of the guys across the street on a video presentation he was doing. But the one job that my supervisor did his best to never have me do was technical support. For one thing, I’d just left a job doing technical support and I’d already told him my opinion on ‘outside’ technical support many times. For another, it quickly became obvious to everyone that I didn’t pull any punches. The worry of my supervisor wasn’t my doing tech support per se, as I was very good at dealing with the customers, but was his worry about one customer in particular.
There was one guy that was a royal pain in the ass. He was calling to complain every few days about something. He was taking disability, no doubt for some problem that didn’t entirely exists, from a union job with the local auto plant. His union perks included unlimited access to legal assistance, so anytime anything wasn’t to his liking he threatened to sue. Many times my supervisor had suggested to him that if he was dissatisfied with his service that he could recommend other ISPs that serviced the area. Apparently he’d tried them and didn’t like the large national services like AOL, Genie or Compuserve because his threats of lawsuit didn’t phase them as much as the small, local mom-and-pop where I worked.
Alas, one day one of the young guy’s called in sick and the supervisor had to run across the street. As he came in the front door upon returning, the secretary out front shrugged at him and gave him a look like “I didn’t know what else to do!” and then informed him that the ‘problem guy’ was on the phone and I was the only one available to take the call. My super came dashing back into the back only to find me talking quite pleasantly and being very helpful to the guy on the phone. Almost too pleasantly. I was having him walk me through the trouble he was having. Trouble that inevitably was unrepeatable whenever someone got him on the phone. I suspected why long before I ever had an opportunity such as this to talk to him. I suspected whatever he was doing at the time was something he didn’t want us to know he was doing.
So I had him walk me through how he turned his computer on. I had him walk me through what he did before connecting, how he started a connection, what he heard, what came next. I knew all the answers already, I had heard others try to go through it with him many times but found nothing going wrong. But he was somehow delighted that someone was seeming to be paying such close attention to him and not simply seeming to want to try to find some way to get him off the phone again as soon as possible. No, I was making it quite clear that I really did want to find the source of his problem. Because I really did, having a good idea what it was.
Eventually he walked me through the process of starting up his mIRC client and which point his level of detail suddenly dropped to a generic ‘then I log into my chat room, but in no time I get disconnected’.
I asked “which chatroom is that?”
“Well any chat rooms I’m in.” he replied ambiguously.
“Well can you name one specifically? You are speaking to the head system administrator now, none of the regular technicians are here at the moment. Please be re-assured, there is probably no one working here that might have a better opportunity to solve your problem once and for all. But to do so, I first need to be able to detect it. And the only way I can do that is by repeating exactly what you do when it happens. You did say it happens just about every time. So what chatrooms? I will join the chat room with you and see if I can detect the problem. May I have permission to also put a sniffer on your traffic stream?”
He reluctantly agreed and my supervisor was now giving me a look like ‘What the hell are you up to? And why are you being so nice when you’ve told me many many times how you would deal with this guy if you ever got stuck on the phone with him?’ He started calling other people in and I could here him around the corner telling them “I think Scott is up to something and he’s finally gonna put that pain in the ass in his place somehow” Soon a crowd was forming around me.
I joined the chat room with the guy and quickly hesistated midsentence with my next inquiry of “so what do you do nex….. oh……”
I had a feeling what type of chat room it would be before I even suggested it, I also had a feeling his problem was related to an mIRC issue of a chat room of this type. The chatroom was labeled something like #countrymp3warez and it was an irc chat room specifically dedicated to the illegal trading of copyrighted music.
I tried to act like I was suddenly trying to be extremely discreet when I said, “Sir, we have a slight problem here. How can I explain this delicately. Are you familiar with legal terminologies?” I knew he would pretend he was even if he wasn’t, he threatened legal action enough times. “Have you ever heard of a concept known as ‘contigient liability’? No? Well you see the issue of internet ISPs and customer privacy as well as corporate liability has come up a number of times – long before the existence of the public internet even. The principle of contigient liability applies to anyone involved with a communications medium where they are required to respect the privacy of their customers but may unknowingly be allowing by way of their service an criminal, illegal or otherwise damaging action being perpetrated upon it. Mainly that the company is not liable for any damages from such an activity until such time as the existence of the activity is brought to their attention by any means.”
The guy tried to play dumb so I went on. “Well, this channel on which you are running into the problem is involved with the trading of copyrighted material subject to intellectual property laws. You mentioned you were having trouble with getting disconnected during file transfers, but at this point in time I am neither aware or unaware if your transfers are in any way related to the copyrighted material being traded on this or similar channels. If I inquire further, I will be required by law to report the activity and anyone participating in it to the local police department as a violation of international copyright laws or we can be held both criminally and civilly liable. Do you understand? If your activities are in any way related to downloading copyrighted music, and you admit that to me in the process of this call, I HAVE to report it. So I need to ask, do you wish to continue this technical support call at this time?”
No one at the company ever had to deal with the pain in the ass again.)
Before too long, they got contacted by their SGI representative who made them an irresistible offer on an upgrade for the box that was running the websites for their 3 big clients. They jumped at it and moved in this new and rather fancy Irix server. Unlike the old server, however, this one was a ‘headless’ design. Meaning it did not have a graphics card or monitor – it was expected that this type of server was to be administered remotely and all of it’s administrative functions were web or gui based. Very little could be done to configure or expand upon the web services without running a gui program on a remote ‘head’ system or some other Xwindows system that supported the Irix OpenGL graphics capabilities.
OpenGL was proprietary. It cost money. It wasn’t on ‘free’ linux boxes. And the company didn’t buy an Irix head. They asked me to start working on the irix box to expand the big-3’s services with some of the new functions of this box but I couldn’t because I couldn’t run the guis on anything. I pointed this out to them and told them they would have to get a head for it. They called the rep and he brought a loaner, but said up front we couldn’t keep it for more than a week. And when they learned what one would cost, they didn’t want to do it.
So I told them that there was a commercially branded Xwindows server that included the commercially licensed openGL graphics capabilities and pointed out they wouldn’t need more than one license because they could put it on any one of the linux boxes then use that box as the ‘head’ for the Irix box as well. I suggested they could get it with the Back-up GUI based software they had promised to buy me a few weeks prior but for which the credit card had never shown up despite my emailing a couple of times asking if it was still coming.
Something by this point in time made me start not only sending emails giving updates on my progress on things, but re-requesting the card to purchase these software packages as well as any other things that I was awaiting their action upon, but then to also print out a copy of those emails and keep them sorted in a folder on my desk.
Before too much more time passed, the british ninny apparently started seeing the flaw in his master plan to get his dreamjob. He still couldn’t weed me entirely out of the mix because he didn’t know Linux and didn’t want to know linux. We were now up to 8 seperate boxes running linux including my own box which now housed almost all my proprietary ‘tools’ to help me do the day-to-day functions in what I called the ‘bucket brigade’ because most of my work involved constantly putting out small fires by way of tiny recurring problems because they would never allow me segregated, uninterrupted time to program more comprehensive tools on the regular systems to alleviate them once and for all.
The Brit came up with a new plan though. He couldn’t talk the company into buying his preferred Macintosh systems for doing ISP services because none of the rest of them knew Mac. But the guy they had recently hired to do some community training classes claimed to know NT, so Gary bought four brand new NT boxes and installed them initially in the computer lab upstairs. Gary then also started moving devices that were previously working just fine on the linux network and systems to be exclusively connected to his new NT boxes.
One such device was a printer that I liked to use after hours to print out some of my graphics files that I dinkered with in my free time. When he moved it up stairs, he handed off to the training guy. I heard him cussing and moaning and groaning for hours. Apparently there was some kind of a fix needed to be put in place to support not only networking capability on the printer to get it working on the new NT servers, but to get it working at all. And Mr. “I know NT and will train the rest of you” couldn’t get it to work.
The brit, out of a sheer since of duty and trying to be polite had asked me if I wanted to also sit in on the NT training classes. I thought of the 8 linux boxes around me constantly complaining about one thing or another and asked myself if I really wanted to demonstrate a proficiency with NT as well? I politely declined. But that night I had wanted to print some more graphics files, so when the training guy came in the next morning, I heard him through the ceiling exclaiming something akin to “eureka, I go it to work!” – yeah right! I just kept my mouth shut. I didn’t need the administration of 4 new NT boxes added on my pile as well!
(Aside: It eventually did get out that I knew NT better than the instructor. I felt like Tom Selleck at the end of Quigly Down Under, where after going through the whole movie with a sharps long gun and being asked multiple times why he didn’t use a hand gun and answering “I don’t like them”, he is confronted by the aussie bad guy who fancies himself the Oz version of Wyatt Erp when it comes to the fast draw. He forces ‘Quigley’ to face him in a duel with colt revolvers taking his rifle away. As the bad guy lay bleeding in the dust after being gunned down by Selleck he stutters out “But, I thought…..” to which Selleck replies simply “I said I didn’t like them. I didn’t say I couldn’t use one”)
So on went the games, six months had come and went and I saw no significant raise and definitely not the amount I originally ‘required’. Their excuse was implied in the fact I was no longer the ‘chief’ administrator. The British sod backdoored me out of that distinction. I consoled myself with one year. I figured that was a reasonable time to consider as having given them a chance and working to earn my signing bonus. Fortunately enough, just about one year in they all three, my supervisor, the manager and the brit, showed up to ambush me.
They asked “why isn’t the backup system done?”. I hadn’t expected them to show up, but I leaned over to my folder and pulled out a stack of emails about 1/4″ thick. “oh… well… yeah. But what about that Irix box? why hasn’t anything been…” I layed another stack about 3/16th of an inch thick next to the ones about the backup software and lack of credit card to buy it detailing about an Xserver and the same problem. This process repeated through about 5 or 6 items, each was met with a small stack of printed out emails detailing how I was informing them that I could not proceed because I had already reminded them multiple times that I was waiting on them for X or Y.
They were ready to put me in my place and I completely and totally pulled the rug out from under them. I’ve never felt more empowered in any situation working for someone else in my entire life. When it was obvious they had completed their list of ‘what about’s, I asked simply, “is that all?” The manager just stood staring and muttered something like “this isn’t working for you is it?” apparently again trying to somehow lay the blame at my feet.
I wanted to give them a lecture on how it is absolutely ridiculous that a company that purports to be selling ‘internet services’ can’t even answer a f’king email that’s sent once or twice a week over and over for months at at time, but instead I just took a deep breath, sat up straight and took the power I was feeling over them at the moment and slammed them over the head with it.
“OK, this is how it’s going to work. You can call it whatever you want to call it. Say I quit, say it was mutual, but you will in no way say that I left you hanging out to dry or that I was incompetent in any way shape or form at doing my job. You will also write me a gleaming recommendation letter that I can take out of here as I seek other jobs. I’ll need the letter because there is no way in hell I am going to have anyone call you for a recommendation. I’ll expect the letter within the hour as my PC will be packed in the back by then and I will be wanting to go home at that time with no intention of coming back. Am I understood?”
They agreed, I got my letter, I told them where to send my last check and I never saw that hole again!