Target, get on the ball with this data breach

If you shopped at Target between 11/27/2013 and 12/15/2013, congratulations.  Your credit or debit card info is one of around 40 million up for sale in a thriving underground marketplace complete with wholesalers, distributors, retailers, and easy to use websites.  Replace your card right now before bad things happen.

Brian Krebs broke the story in his blog, Krebs on Security, and the public owes Krebs a debt of gratitude.   Here is the original story.   Here is a follow-up post.  Target blew it.  Target should have notified customers and broken the story itself.  But  instead of proactively notifying its customers, Target apparently responded to the Krebs blog, as did the rest of the popular press.

The more onion layers peeled back, the scarier this gets.  Where did that date range between Nov. 27 and Dec. 15 come from?  Apparently, banks buy samples of stolen credit card info from those same underground markets and look for patterns.  The big thing all these cards have in common is – you guessed it – transactions at Target during that time window.   That’s why the press is reporting the date range of 11/27 through 12/15/2013, not because of anything Target found and reported about its systems.

Let this sink in for a minute.   That date range came from looking at samples of cards already stolen and not from any analysis of whatever was penetrated to get the card numbers.  As of Christmas eve, 2013, we still don’t know what specifically was penetrated, which means we don’t know what else is at risk or what steps the public can take to protect itself.  Here is an article with some expert speculation, but it’s only speculation from the outside.

Target claims the vulnerability is now closed and offers reassuring press releases to soothe the general public.  But with no guidance on what was penetrated and what specific steps Target took to close the vulnerability, the press releases so far offer nothing of value.  The public trusted Target before the theft and now 40 million credit card numbers are up for grabs.  Why should the public trust Target now?  What’s different?

If anyone from Target reads this blog post, Crisis Management 101 suggests transparency and disclosure.  The worst thing you can do is hide.  Instead of reacting to events and putting out vague press releases that offer no useful information, get in front of this story and tell the public specifically what happened and what you’re doing about it.  Introduce us to the people working around the clock to plug the gaps.  Show us how hard you’re working to fix the problem.  Convince us that shopping at your stores won’t expose us to identity theft.  Treat this like a crisis, because it really is a crisis.

Are we all in this together, as your press releases promise, or are those just empty PR words?  Smart people who know how transaction systems are supposed to work are watching.

(First published on my Infrasupport website on Dec. 24, 2013.  I back-dated it here to match the original date.)

A long couple of days in the life of an IT professional

This story is one example of many for what the best IT professionals do to keep our skills current.

The story started in April, 2013 when an opportunity to deliver a project based on a software product called RHN Satellite from a company named Red Hat came along. Large companies use RHN Satellite for activities such as automated builds, patch management, auditing, configuration management, and other administration for Red Hat Enterprise Linux servers. Think of an IT shop that needs to roll out dozens or even hundreds of servers and set them all up the same way. Those folks need RHN Satellite.

I knew nothing about RHN Satellite, but I’ve earned a reputation as a quick learner and was confident I could master it.  This would help a customer bring in an important project and make some money for me.  A win for everyone.  So I said yes.  This is what the best IT consultants do; we say yes and we learn quickly.  The job is not for the feint of heart.

There was a training class in Dallas with open seats coming in one week and I quickly signed up. Dallas is more than 900 miles away and I’m too cheap to buy expensive plane tickets, so I drove the 14 hours to the training site, attended the class and learned enough about advanced Red Hat Enterprise Linux system administration to deliver the project.  I also invested in the certification test.  If I’m going to learn the product, I may as well also get some certification credit for it.

Red Hat certifications are unique in the IT industry. While most IT product vendors offer certification tests based on cleverly worded multiple choice questions, Red Hat tests are all lab based.   This means anyone who wants a Red Hat certification must demonstrate knowledge of that system by setting one up in a lab according to test specifications.  The tests are challenging and very few candidates pass on their first attempts, even instructors who teach the courses.

Equipment problems in the Dallas classroom forced Red Hat to reschedule my certification test.  I scheduled mine for July in Chicago and failed miserably.  I improved in Chicago in October, but not enough to pass.

I knew what I did wrong and how to fix it, but the next scheduled test in Chicago was 6 long months away.   Two other test sites had openings on Nov. 22.  One was in Atlanta, the other in Ottawa, Ontario, Canada.  The Atlanta site is 16+ hours away by car, the Ottawa site around 18 hours away.

I decided on the Atlanta site and planned to buy my seat sometime after Nov. 9 to maximize my credit card float.  But the Atlanta seats filled on Nov. 8, leaving Ottawa as the only available Nov. 22 choice.

Ottawa presented a logistical challenge.  Trips from the US to Canada require a passport and mine was expired.  Minneapolis has a passport office and I could renew my expired passport by bringing it in with an updated picture and $220.   But unable to find my expired passport, I had to start from scratch with a birth certificate from Idaho.  To get a copy of my birth certificate, I needed an official copy of another piece of paper documenting my legal name change back in 1978.  That piece of paper was buried in a vault and the only way to get a copy was a trip to the basement of the Hennepin County Government Center in Minneapolis, where the lady behind the counter said she would order it for delivery the next week.  In a miraculous sequence of events, the process accelerated and by Monday, Nov. 18, I stood on the steps of the US Passport office in Minneapolis with a new passport in hand.

Only one logistical challenge remained – sign up for the test in Ottawa.  But now, after spending hundreds of dollars and watching a logistical miracle unfold around my passport, all the seats in Ottawa were full.   I called Red Hat and talked to Lauren with the training group.  Lauren orchestrated another logistical miracle to add an extra seat, and I reserved my seat in Ottawa a few hours later.

Now the real work – prepare for the test, travel to the site, pass the test, and go home.  Air travel cost between Minneapolis and Ottawa started at roughly $1050, confirming my decision to drive.  The test was Friday morning, so I was on the road from Minneapolis by 4:45 AM Central time Thursday.   I arrived at the Stardust Motel on Carling Road in Ottawa at 11:15 PM Eastern time, 17 1/2 hours later.

Canadian customs officers have ultimate power at the border and can deny entry to anyone they want for any reason they want.  My trouble at the Sault St. Marie border crossing started almost immediately as I rolled down my car window and handed a lady in a little booth my passport.

“And what brings you to Canada today?”

“I’m taking a test.”

“How long do you plan to be in the country?”

“Overnight”

“And where is this test?”

“It’s in Ottawa.”  (I think alarm bells started going off her head.)

“Let me get this straight.  You’re driving all the way to Ottawa to take a test, then you’re turning around and going home?”

“Yes, that’s right.”

“You are aware that Canada is a sovereign country, right?”

“Uhm, Ok.” (Not sure where she was going with this.)

“And you know Ottawa is 11 hours from here, right?”

“Well, Google maps tells me it’s about 9 hours, but OK.”

“So why are you taking this test in Ottawa?”

“Because the site in Atlanta was full.”

“What is this test anyway?”

“It’s for advanced Linux system administration.”

“Advanced what?  Why are you traveling to a different country just to take a test?”

“C’mon, What foreign country?  This is Canada. We’re friends. ” (Note to self – Canadian customs agents apparently don’t like appeals to friendship.)

“When was the last time you visited Canada?”

“Uhm, well, I guess it’s been a while.  Why?”

“OK, you need to park right over there and go inside for more questioning.”  (Uh-oh. This can’t be good.)

She directed me to a parking spot where several people in uniforms waited to escort me inside.

One of my escorts asked me what documentation I had to prove I really was going where I said I was going.  Thinking about it, I only had some emails.  I could open them on my tablet so I brought it in with me.  Maybe the emails would satisfy them.  I’ve been to former communist countries with less hassle.

I watched car after car after car easily cross into Canada while I slowly walked inside the building, flanked by uniformed guards as the clock and my upcoming night’s sleep ticked away.  Walking in the building, I saw 5 more people in uniform behind a counter on my right laughing about a video on a computer screen.  They sent me to the farthest counter, where a fat, gruff, balding grey haired man in uniform talked to a group of three people. As I approached the counter, he ordered me to step back and sit down at a table and wait.

“They told me to go to this counter.”

“And now I’m telling you to sit over there and wait.”

“If it’s all the same to you, I’d rather stand and stretch if I have to wait.”

“Suit yourself, I’ll be back after a while.  Don’t come up here until I tell you to.”

And then he left with the three people trailing.  I stretched my legs and back, stiff after 9 hours of driving so far that day, thinking about the 9 additional hours still to come and the clock ticking while I waited on the Canadian government.

The officers watching the comedy video barely looked up.  I asked one if he could take care of whatever it was I needed to take care of and he said no, that guy was the only one who could do it.  My only option – cool my heals and wait.

I needed to pee.

After a few minutes, Mr. Authority returned, motioned me up to his counter and asked me what was going on.  I gave him my passport and then made my next mistake.

“The lady outside hassled me and told me I need to come in here and take care of it.”

Major mistake. And then I made it worse.

“She didn’t hassle you.  You said you’re here for work so she sent you in here. That’s what she’s supposed to do! Do you think we just let anyone in our country who wants to come in?”

Dumbfounded, I said, “I see at least 10 cars out there and they’re all flowing right through.”

“THEY’RE ALL CANADIAN CITIZENS AND THIS IS A SOVEREIGN COUNTRY!  SO HOW ABOUT I DON’T LET YOU IN THEN? YOU JUST CAN TURN AROUND AND GO HOME, HOW DO YOU LIKE THAT?”

I wanted to walk outside and survey the license plates on those cars passing through the border, but a little voice in my head told me to shut up before I got myself into more trouble.  This guy controlled what I needed and I had nothing he wanted.  He had all the power and I had none.  He was the master and I was a dog.  Time to become meek and beg for mercy.

“I wouldn’t like that at all. I’ve been up since 3:30 this morning.  I just want to get where I’m going, take my test and go home.”

“WAIT RIGHT HERE.”

He took my passport and disappeared into a little back room. He came back out a few minutes later.

“Ever been convicted of a crime?”

 I was about to say “speeding tickets”, but my literal answers had already gotten me into trouble.  Speeding isn’t a crime anyway.  So I answered, “No.”

“Ever been denied entry into any country?”

“Not until right now, no.”

“Show me this proof you’re taking some sort of test in Ottawa.”

I showed him one of the Red Hat test confirmation emails and he said, “That doesn’t say it’s in Ottawa. What else ya got?”

“Uhm – well, here, take a look at this email.” I brought up an email thanking Lauren for opening the additional the seat for me.

“I never heard of any company named Red Hat.  What do they do?”

“They’re a software company.”

“Where are they?”

“They’re in North Carolina and they have offices all over the world.”

He looked me over one last time.

“OK, I’m going to let you in. BUT YOU NEED TO WORK ON YOUR ATTITUDE!”

“Thank you.”  (resisting the urge to further express my feelings.)

“Go give this paper to those two standing over there and be on your way.”

“Thank you.”

I still had to pee, but not bad enough to ask anyone here to use a bathroom.  As the uniformed guards escorted me to my car, I asked if any of them wanted to search it.  They said no and I drove away.  Welcome to Canada, eh.

I passed the test.  I’m now a proud holder of a Red Hat Advanced Linux System Administration Certificate of Expertise.

Total cost – $480 for the November test, $480 for the October test, $2800 for the original training and test, about $400 for the passport and required paperwork, 4 trips to the Minneapolis passport office and Hennepin County Government Center, around $1000 for travel costs for the training and test trips, 36 hours behind the wheel for the round trip to Ottawa, $40 per month for upgraded cell phone service in Canada and Mexico, sore legs and back, and a lesson in humility from some Canadian border agents.

Hopefully it was worth it.

(Originally posted on my Infrasupport website in Nov. 2013.  I backdated it here to match the original posting.  At the time of the original posting, my one person company was a Red Hat partner.  I am also now a Red Hat employee.  Everything in this post is my own opinion and may not reflect what anyone at Red Hat thinks.  Life goes on.  That test I worked so hard to pass back in 2013 is now obsolete.)

www.healthcare.gov – A Classic IT Disaster

My family is OK with health insurance coverage for now and I live in Minnesota, which has its own website for navigating the local healthcare marketplace, so I don’t need to deal with the US healthcare.gov website to find health insurance. But I was curious after looking at the news the past several days. I find it amazing that an Internet website generates front page newspaper and breathless TV coverage and I wanted to see what the fuss is all about.

My wife’s sister and her husband live in Indiana, so I wondered how health care coverage would look in their state and county. Navigating the website was straightforward and I found a few sample plans and pricing. The prices looked high, but each page boldly displayed caveats that the prices displayed were probably high and lower prices were most likely available by filling out a real application.

The website also mentioned a worksheet called the Kaiser Family Foundation health insurance cost and savings calculator and provided a helpful link. Answer a few quick questions using this worksheet and the website promised to generate more accurate price estimates.

So I followed the link and came to a page with some helpful explanations. So far so good.

When I clicked Next, presumably to start answering those few quick questions, a popup window with red text and yellow background popped up that said,

Please review your answers on this page,
there was a problem with one or more of
them.

This question is mandatory.

I would love to fix my answers, but this page had no questions.

In less than 5 minutes, I found an obvious programming bug by taking a path through this website that any typical user would follow. Here is a screenshot, [This link is no good – see my Infrasupport blog post about how I recovered from my own IT disaster, here.] reduced in size to 50 percent to capture most of the page. Notice the popup message at the bottom about answering questions, but the page only has explanatory text. There are no questions to answer, but the website will not allow me to navigate away from this page without answering its nonexistent questions.

Now I see what the fuss is all about. If I stumbled across such an obvious bug in less than 5 minutes without even trying, what other bugs are lurking beneath the surface? Current news reports suggest this website is full of bugs, performance issues, and inaccurate information. After my experience, I believe the reports.

The consequences are predictable. Politicians with no IT experience beyond using Facebook and smartphones are screaming, and VIPs across the government with no IT experience are trying to hold other VIPs with no IT experience responsible for this mess.

Where have I seen this before? Perhaps with every big IT project ever conceived since the dawn of big IT projects? Why would any IT veteran be surprised this one has serious problems? This project has all the classic elements: A lofty goal at the beginning, political maneuvering among potential contractors for development money, squabbling constituencies during development, major changes in design and function throughout the process, no time for serious testing to meet a looming deadline, nobody in overall charge of the effort, and a search for scapegoats at the end when the project fails to meet expectations. And in a classic and insulting knee-jerk response, our most senior and clueless Executive Branch officials now promise us they’re bringing in the A-team of “experts” to fix it all.

The stakes are indeed high. By United States federal law, and affirmed by the United States Supreme Court, we Americans are forced alpha testers for a new piece of technology over which we have no control, and if the technology fails, we pay a fine. I see attorneys building entire careers from the lawsuits this debacle will generate.

The quick fix President Obama promises will not happen for many reasons. First, as a few grizzled IT veterans constantly remind me by metaphor, the calendar time for a 9 month pregnancy cannot be reduced by finding 9 people to each work one month. In IT terms, a pregnancy project only achieves its goal if one person does all the required tasks during 9 consecutive calendar months because the output from earlier months provides the input to later months.

Software development is not war and the Obamacare surge metaphor comparing this project with the wars in Iraq and Afghanistan is a load of BS. Adding more programmers does not necessarily improve development time because at some point in any software development project, adding more labor becomes counterproductive. Even if they are “experts”. When the United States Federal government sends in their newly found “experts”, these new people will first climb a learning curve before becoming productive, and will then face the same project dependencies and constituencies as the earlier, presumably less capable team that came before them.

President Obama’s proposal to fix healthcare.gov is like proposing new tires for the car to fix a broken engine. It’s nonsense.

Beyond rookie programming errors such as the one I found, the problems are fundamental. The real fixes will depend on all players cooperating and agreeing to a reasonable set of specifications. No matter how many “experts” our Government brings in, these experts have no power to persuade or coerce all the unrelated parties to work together. Consider:

  • insurance companies with no incentive to cooperate with government contracted software developers,
  • a federal government filled with too many high officials with too many big egos and too many new and brilliant half-baked ideas,
  • millions of users trying to use the system on the front end while developers scramble to fix bugs on the back end,
  • and nobody in overall charge,

and the result is a recipe for continued failure.

Future historians may well declare the new healthcare.gov effort as the biggest, most spectacular failure in the history of IT projects.

(Originally published on my Infrasupport website.  I back-dated here to match the original publication date.)

A new level of malware sophistication

I woke up today around 4 AM when one of our cats jumped on my stomach.  By now, it’s a pounce and jump operation because he knows if I catch him, I’ll throw him across the room and bounce him off a wall.  So he pops up from the floor to the window pane, pounces on my stomach from above my head, then flies off the bed, all in less than one second.  It’s an effective technique to wake me up, not so good to persuade me to feed him.

Some days, I really don’t like cats.

Since I was awake, I staggered out to the other room to do some computer work.  As I sat down in front of this Windows 7 system, I saw a window on the screen that made my blood run cold.  There, right in front of me, was a window telling me this computer had several virus infections and a “click here” button to clean them all up.  I lost count of the number of fake AV systems I’ve cleaned over the years, but this one was right here at home.

Fake AV, or fake antivirus, is a great scam.  Here is how it works:  Vicki the virus author decides she wants to make some illicit money.  So Vicki writes an evil program that pops up a window on a computer screen with an alarming and official looking message about dozens or hundreds of viruses found.   But the whole thing is a lie, designed to entice naïve users into giving up sensitive information.

Vicki will probably craft her program to display a reassuring ”Click here” button with a promise to make it all better.  When the user “clicks here”, her program will probably prompt for a credit card number, with the promise of a $24.95 download to take care of all the problems.  When the user enters the credit card number, the program will send the data back to Vicki, and Vicki can either have a great shopping spree at the credit card holder’s expense, or sell the credit card number in an underground market.

Vicki’s program may also leave behind a key logger or other malicious software designed to track user mouse clicks and keystrokes and send the results back to Vicki.  Vicki can mine this data at her leisure, stealing anything of value she wants.

It’s serious business.  If you “clicked here” for one of these programs, and especially if you gave a credit card number, stop reading this right now, call your credit card company, and cancel the credit card.   Also call your other credit card companies and your bank and sign up for a credit watch service.   Pull your computer off the Internet and have a trusted professional thoroughly clean it.  Don’t mess around with this – identity theft can make your life miserable for the next several years.

But Vicki has a distribution challenge – how does she distribute her evil program to millions of potential victims?  Enter many of today’s popular news, weather, sports, and gaming websites.   These sites make money by selling ads.  When a user visits, say, www.espn.com, that website will also download several banner ads from all over the Internet.   Those ads do not come from ESPN, they come from advertisers who pay ESPN.

So when you visit the ESPN website, you also visit several other advertiser websites who display their ads in areas ESPN assigns.  Any one of these can download programs to your computer for, say, displaying animation, touring the advertiser client products, or displaying a popup window claiming you have dozens of virus infections.   ESPN cannot possibly vet all its potential advertisers and must depend on the advertisers to keep their own websites clean.

I purposely picked on ESPN because ESPN had a well documented incident a few years ago, but all websites that sell third party ads have the same potential issue.  If any one of these ad websites are compromised, and that ad happens to display on your computer, your computer is in danger.  From the users’ point of view, it’s like a game of Russian Roulette.

From Vicki’s point of view, there are hundreds, maybe thousands of these ad websites.  Vicki only needs to find one with poor security so she can inject her program and make it act as an unwitting distribution point.

Vicki no doubt subscribes to an automated underground service that constantly probes these websites, looking for vulnerabilities.  When she finds an eligible candidate, she injects her payload, compromises the website, and sits back and waits for the credit card numbers to flow in.

Vicki’s evil payload eventually found its way to one of my computers when my wife visited an Internet game site the other day.

If you find such a program running on your computer, do not immediately shut down and reboot – this will generally trigger these programs to deliver their evil payload and they will own you after the reboot.  Instead, launch your Task Manager, find the offending process, kill it, find the offending program on your hard drive, delete it, then do a thorough virus scan using a reputable tool.  Or call me and I’ll guide you over the phone.

Curiously, this fake AV program was a little different than most.  This one claimed it was a Norton virus scanner and a quick trip into my Windows Task Manager found a process named nss.exe.  NSS.exe is, in fact, the name of the free Norton Standalone Scanner.  But in this case, it was an evil program pretending to be the real nss.exe.  I killed it and the fake AV window went away.

Next, I searched my hard drive for any file named nss.exe.  I found two occurrences, both in folders with Norton in the name, under C:\Program Files (x86).  Looking at these folders, I found dozens of .DLL, .exe, and other files.  This was definitely not a typical virus scenario.  Most directories containing malware programs have a single, hidden program with a random name.  This one was different – this one went to a lot of trouble to look like a real Norton installation.  The creation date for nss.exe was May 7, 2013.  The creation date for the directories I found were from July 28, 2013 at 5:39 PM.  The time as I write this is around 4:30 AM on July 30, 2013.

This gets even more interesting.   Looking at Control Panel…Programs and Features, I found references to two packages claiming to be Norton virus scan tools.  When I right-clicked and selected the “Remove” option, a new window popped up asking me if I wanted to install the Norton virus removal tool.  I also heard my hard drive rattle a little bit, suggesting to my paranoid mind this virus may have delivered its payload.

Nice try Vicki, but you failed to solve two problems:

  1. I never installed anything from Symantec or Norton on this computer, although it’s possible my wife may have done so.
  2. Even if somebody else installed this stuff without my knowledge, why would a removal option instead prompt me to install something new?

I think, with help from my wife,  I stumbled across a new variation on an old theme.  I’ll bet a zillion dollars, this is a particularly sophisticated fraud.  I think my hypothetical Vicki ripped off the Norton Standalone Virus Scan installation and replaced the main program, nss.exe, with an evil program of the same name.  It probably found its way to a compromised ad website, where my wife inadvertently downloaded it from the free Internet game site she likes to visit.

To be safe, I did a System Restore and restored that system to its state 4 days ago before the last Windows Update, or 2 days before the fake Norton installation.  After a reboot, Control Panel…Programs and Features no longer shows anything claiming to be a Norton product.  I deleted the offending directories and started up a full virus scan in another window using a popular tool named Malwarebytes.  The scan took one hour and 14 minutes to finish and I am pleased to report this desktop is clean.

With hindsight, I probably should have taken some screen shots and quarantined those directories so I could present it all in this blog post.   But, just like most users, this desktop is a tool and I need it up and running.  I didn’t think about documenting it all until after I removed it.

For my non-technical friends, the moral of all this?  Just like with your car, be on the lookout for unusual behavior.   If your car flashes the “Check Engine” light or exhibits unusual behavior, you look into it, right?  If your computer starts to act differently, you should also look into it.  Call me if the problem looks complicated.  You have important data inside that computer and believe me, you do not want criminals across the Internet messing around with your identity.

For my more technical friends, I think the malware arms race just ratcheted up a notch.  This looks like a new level of sophistication.  Watch out for variations on this theme as Vicki’s friends craft other pieces of malware to imitate other free virus scan products.

Vicki found the wrong user to mess with this morning.  Hopefully, you can also stop Vicki and her friends cold when their programs try to invade your computer.  Be vigilant.

(Originally published on my old Infrasupport website and backdated to match the original date here.)

Why “normal” people should care about IT

I did a presales call with a small dentist office a few months ago.  I have some dentist customers, so I’ve come to understand a little bit about how dentist offices operate.  But this office was, well, different.  The “server” was really an old, failing Windows XP PC tucked away in a dusty corner of an unused office.  Other workstations were in similar states of disrepair.  This office had a challenge – the receptionist’s brother maintained all the IT equipment, but he recently moved from Minnesota to Colorado and was no longer available to come onsite and resolve the latest emergency.

I promise – I am not making this up.

Apparently, nobody knew how to boot their “server” and they had to call the brother every morning to get the office up and running.  The process was generally to turn that central PC off and back on again and hope it booted. Once up and running, nobody was to touch it for the rest of the day.  Although PCs in the exam rooms had the ability to store a few patient updates locally, they all depended on this failing central repository to access historical patient data, including X-Ray images.  If that PC died, all the X-Rays and all patient data died with it.

The dentist/business owner said he knew he needed a server and we talked at length about setting one up.  Then I asked my key question:  “What happens if this PC you’re using as a server dies and you can’t access any patient X-Rays?”  His answer left me speechless.  “I don’t need computers to practice dentistry.”  The meeting went downhill after that, and this dentist office never returned another phone call or email.

I use that dentist as an example because, unfortunately, his attitude seems typical of so many business owners and otherwise intelligent executives.  Here is another quick story to drive the point home.  Several years ago, I was in a presales meeting at a bank to talk about IT security.  The banker proudly showed me the shiny new security audit report he undoubtedly paid a small fortune for and asked me to look it over.   I sat across the table from him, looked over the report, and commented it covered the bank’s website pretty well, but where was the section about the bank internal IT operations?  His reply – “Thanks for coming over” and he quickly hustled me out the door.  To this this very day, I don’t know what nerve I touched.  But I have some theories.

I think IT is boring for most “normal” people.  Most people don’t care about what DHCP servers do or the difference between 1 gb and 10 mb.  Some bankers probably never stop to think about the difference between their internal operations and public facing website.  At least one dentist never took the time to think through what would happen if all his patient records disappeared.   And because IT is boring and “technical” and costs money, it must be at best a necessary evil.  For most business decision makers I’ve met, IT is not an asset to be maintained and enhanced, IT is an expense to be minimized.

This is a shame.  Consider:

  • medical and dental clinics, who keep patient data inside a server instead of a large room full of paper files and film X-Rays.
  • transportation companies who can keep images of millions of invoices and other paperwork inside a computer network instead of whole buildings filled with file cabinets
  • email and the world wide web
  • automated airline check-in systems
  • online banking
  • and thousands or maybe millions of other applications we take for granted today.

What would happen if we turned all those off?  Think about a bank branch without access to the central databases.  Think about an airline without the automated ticketing and check-in systems we’ve become used to using.  Think about cutting off access to email and the world wide web.  Think about a dentist trying to run a modern office without access to computers and historical patient data.

If you are a small business owner, here is a challenge.   Turn off your servers and computers for one day and try to run without them.  Observe the chaos that will surely follow.  Try to calculate the lost revenue from all the customer service disasters that will happen.  Try to calculate the increased cost when everyone has to operate manually, with no access to any data.

I dare you to take up these challenges and send me some comments about your experience.  And then, let’s have a conversation about how to protect your critical assets and how you can use IT to at least gain competitive parity and maybe a competitive edge versus your competition.

(I originally published this in my old Infrasupport blog on May 30, 2013.  I back dated it here.)

Computer Whodunit Detective Story – the Conclusion

In part one of our computer detective saga, the story opened with a few users unable to access their emails. Similar to a Hollywood detective story, we followed the clues through several unexpected twists and turns, with each clue answering questions and generating new questions.  Continuing in the style of great whodunit detective mysteries, we eventually uncovered the culprit, a rogue DHCP server.  This changed everything.

And now the conclusion.

DHCP – Dynamic Host Control Protocol – is the reason we can connect our laptops and tablets and smartphones to the Internet.  DHCP servers assign all the attributes our devices need to enable communications.  Think of the Internet as similar to the telephone network, but with one important difference.  In the telephone network, your phone number stays the same no matter where your phone travels. On the Internet, an IP Address defines your device.  But unlike phone numbers, IP Addresses change, depending on where your device is located.  That’s why we need DHCP servers, to assign IP Addresses and other attributes to devices when they attach to an office network or the Internet.

Here is how DHCP works.  When you connect your device to a network, your device sends a broadcast to anyone on the local network who will listen.  It’s essentially a cry for help.  (Help!  Load me with what I need so I can talk to the world.)  The DHCP Server listens to the broadcast and downloads an IP Address and other attributes to the requesting device.  This is called an IP Address lease, and the lease expires after a settable amount of time, called a TTL (Time to Live).  Once the device acquires its IP Address lease, it can interact with the world.

DHCP is a thing of beauty when set up properly and works so well, only a few hard-core IT people think about it anymore.  Except when things go wrong.  And one of the worst things that can go wrong is a rogue DHCP Server wreaking havoc on the network.  When this happens, random devices get the wrong attributes and lose all ability to communicate.  Depending on how long the lease TTLs are set, sometimes the passage of a few hours can cure the problem, or sometimes make it worse.  The problem can “hop” from device to device as leases expire and new leases come online.  Sometimes devices can end up with duplicate IP Addresses that come and go and interfere with communications.  This can be maddening to troubleshoot.

The usual culprit in an office network is a wireless router somebody brought in from home.  This happens all the time as end users decide they want to build their own private wireless networks, but don’t think about the consequences to everyone else as their wireless router hands out home IP Addresses to random devices across the company network.

Obviously, the cure for a rogue DHCP server is to find it and get rid of it.   The challenge is how to find it?

Enter structured cabling.  Essentially, a structured cable plant runs network cables from stations all over the building to a central patch panel in the server room.  Each cable is labeled, preferably with the labels on both ends of the same cable matching.   All buildings should have a structured cabling.  Unfortunately, many don’t.  Fortunately, this one did.  And that proved to be a tremendous aid finding my rogue DHCP server.

Instead of walking the entire building and looking for a device that looked out of place, I set up a laptop near the patch panel and assigned the laptop a hard IP Address to fit the rogue DHCP server scheme.  After warning everyone their network connections may be disrupted briefly, I set up the laptop to continuously ping the rogue DHCP server IP address while I disconnected and reconnected each network cable.

The idea – one of those cables had to lead to the rogue DHCP server.  I would find the cable leading to my rogue DHCP server by watching for pings to stop responding when I disconnected that cable.  Once I found the correct cable, I could walk to the other end of that cable with a hammer and put the rogue DHCP Server on the other end out of its misery.

I eventually found it, chased it to the other end of the cable, and disconnected it.  It turned out, my friend James brought in a wireless router over a weekend to help with some work he needed to do.  He forgot to disconnect it and that was why my users started complaining on Monday morning.

The moral of the story?  These things happen and that’s why good troubleshooting techniques are invaluable.

Microsoft Office 2013 Retail – There’s a Sucker Born Every Minute

(Originally posted April 27, 2013 on my Infrasupport website when I was an independent IT consultant. I copied here and back-dated to match the original posting date.)

I spent an awful day yesterday with Microsoft Office 2013 Home and Business Edition.  Full disclosure – my company is a Microsoft Registered Partner and this blog entry won’t make me any friends in Redmond.  But right now, I am frustrated beyond belief and I will have trouble sleeping until I put electronic pen to virtual paper.

After more than 20 years of Microsoft producing a product named Office, by now everyone knows what it includes – a spreadsheet named Excel, a word processing program named Word, an email client named Outlook, a presentation package named Powerpoint, a personal database product named Access, and a desktop publishing program named Publisher.  Different editions of Office include different combinations of packages and licensing and Microsoft mixes them up with each new version.  By now, Office is the de-facto standard for electronic document formats.

With Office 2013, Microsoft combined the audacity that comes with monopoly power with technological incompetence.  What possible rational reason could anyone give to force customers to create a unique login on the Microsoft website for every single retail copy of Office Home and Business?  If you own, say, 50 computers and you have 50 copies of Office Home and Business, you need 50 Microsoft logins to make it work.

Sheer insanity.  Or is it?  Microsoft is filled with competent engineers and savvy marketers.  Microsoft did this for a reason, and this is really a story about a 21st century shakedown scheme.  But it’s buried underneath a pile of technical jargon so very few will notice.

With Office 2013, Microsoft offers three licensing choices, called Volume licensing, retail licensing, and a subscription service named Office 365.  Office 365 is new, the rest have been around a long time.

Volume licenses come with lots of flexibility businesses care about.  Companies can deploy volume licenses any way they see fit.  A volume license for Microsoft Office Standard edition includes only Word, Excel, and Outlook and lists for roughly $370.  Microsoft Office Professional Plus includes all the Office packages and lists for roughly $500 per seat.

Retail licenses cost less, but are less flexible.  For example, Office Home and Business includes Excel, Outlook, Powerpoint, and Word – more packages than Office Standard, but with a lower price of around $220.  The Home and Business license is only good for one computer.  Once installed on any computer, that license is married to that computer forever.  If your PC dies and you need to reinstall Office Home and Business, you need permission from Microsoft.

So far, so good.  Here comes the audacious part.

Starting with Office 2013, Microsoft purposely made Office Home and Business a nightmare to install by adding an artificial impediment.  Microsoft now requires a unique login on its website for every single individual copy of Office 2013 Home and Business.  For each individual login, you must specify the name, phone number, address, email address, and other identifying information.  After setting up this login, you can download and install your individually tailored copy of Office 2013 Home and Business.  The download is roughly 2.2 gigabytes. Customers who use T1 Internet connections will need almost 4 hours per download and each installation now requires its own download. 50 installations means 50 downloads.

If anything goes wrong – a network hiccup during the download, a wrong answer to a question, anything – you’ll spend hours fiddling with registry entries and deleting files by hand because it won’t remove cleanly. I had 4 identical brand new computers and spent most of a day cleaning the remnants of a botched installation on one, with lots of telephone advice from Microsoft Customer Support about undocumented registry entries.

And finally comes the new offering, Office 365.  It’s a Microsoft hosted solution, meaning you connect to a website and work on your documents from there.  The cost is $99 per year or around $10 per month.   No installation hassles, quick and easy to set up, no up-front financial pain for end users.  Your documents live inside a Microsoft cloud, so they are accessible globally and you don’t need a server anymore. Naïve CFOs and Purchasing Departments will love it.

P. T. Barnum reportedly once said ”there’s a sucker born every minute” and he may be laughing in his grave at this modern massive con job. Why would Microsoft price its hosted offering so low relative to a locally installed copy of Office?  Why would Microsoft take such apparently boneheaded steps to artifically complicate installations of Office Home and Business?  And why would Microsoft spend $millions for the cloud capacity to store and manage millions and millions of new user accounts?

Only one answer makes sense – increased revenue.   How does spending $millions to host all this stuff generate revenue?

I can think of only one answer – and I promise, you won’t like it.  Microsoft wants to be the repository for all your personal and business content.  Office 365 will capture your documents, Outlook.com will capture your email, Lync will capture your video meetings.  If Microsoft can make your installation experience expensive and miserable when installing on your own computer, and make it hassle free and low cost when hosting in its cloud, many people will opt for the path of least resistance and put their documents in the Microsoft cloud.  Millions of Office 365 users will blindly trust Microsoft with their most private data because getting started is cheap and easy.

Once Microsoft captures all your content, marketers will pay Microsoft a holy fortune to slice, dice, and analyze your content.  You will provide raw material for marketers and you will pay Microsoft for the privilege.   But marketers will pay much more.  Marketing will be the real Microsoft revenue source – your $99 per year subscription is just a few giblets on the real gravy train.

What to do about it?  If you don’t care if an army of marketers digs deep into your content, trust Microsoft.  If you do care about privacy, maybe now is the time to start looking at alternatives.  Several are available, including Libre Office and other free and minimal cost offerings.  If enough people start adopting some of today’s great alternatives, maybe Microsoft’s monopoly power can be tamed.  But if history is a good predictor, this probably won’t happen.

How a gross IT security lapse hurt a US Senate campaign

(Originally published on my Infrasupport blog on April 7, 2013.  I back-dated the posting here.)

This story is personal.   It is one of the best examples I’ve seen where poor IT security practices and the physical world collide and leave a trail of destruction.

Way back in 2006, I registered my name with the Norm Coleman for Senate campaign. Although the US Senate election was two years away, I felt kind of like an insider when the Coleman campaign sent me email updates.  Fortunately for me, I never gave the campaign a credit card number.

The 2008 Minnesota Senate election between Norm Coleman and Al Franken was too close to call.  There were recounts, court challenges, and recounts of recounted recounts.  Franken eventually won by a few dozen votes.

This is where it gets personal.

On March 10, 2009, I received this email, reproduced below with original spelling errors:

From: Wikileaks Press Office [mailto:press-office@wikileaks.org] Sent: Tuesday, March 10, 2009 9:29 PM To: undisclosed-recipients Subject: Norm Coleman leak

Senator Norm Coleman supporter / contributor list leaked.

Your name, address and other details appear on a membership list leaked to us from the Norm Coleman Senate campaign.

If you have contributed financially to the Coleman campaign there are additional details.

We understand that Norm Coleman became aware of the leak in January.

The information has been passed around out of public view.

We have sent you this note as a curtesy in case Norm Coleman has not contacted you previously.

We have not released the material yet, but may do so within the next few days.

In line with our policy of completely neturality for whistleblowers and political sources, the material will be treated impartially.  We support all those who engage in the struggle for political reform and wish you well.

For additional details, see: [Web links in the remainder of the email are no longer any good]

Apparently, my name and email address were now in the public domain because I filled out a web form on the Coleman for Senate website.  Not a big deal for me – I’m already on several spam lists anyway.  But information about all of Coleman’s online donors was also in the public domain, including credit card numbers and security codes.  This was a big deal.

Apparently, after the election and during one of the many recount challenges in January, 2009, the Coleman campaign decided to move its website.

Unfortunately, the campaign left a copy of its website content at the old hosting site, wide open for the whole world to see.  One of the files was an unencrypted spreadsheet listing donor contact information, credit card numbers, and security codes.  This is a wildly reckless violation of security best practices and PCI (Payment Card Industry) rules.  Credit card information should never be stored on the same system as a public facing website.  If the website is breached, the credit card information is also at risk.  This data should reside in a back end database server with carefully crafted access controls, putting another line of defense between this sensitive information and potential thieves.  And as a final line of defense, credit card information should always be encrypted, which at least makes it difficult for data thieves to exploit.

Organizations storing donor or customer sensitive information have an almost sacred duty to protect that information.  After all, these are the people  who fund and trust the organization.  With its amateur approach to security, the Coleman campaign demonstrated a reckless disrespect for its own donors’ trust and paid dearly for it.

Adria Richards, an IT consultant specializing in website security, found the old website content, took a screenshot of what she found, and posted the screenshot on her blog.  Here is the only remaining evidence I can find of Richards’ blog, and here is a PDF copy in case the web link goes bad.  The Minnesota Independent published an article on January 28 2009 about the incident.  Here is the article and here is a PDF copy.

While Richards’ detective work is admirable, she should have notified the Coleman campaign first, before publicizing the problem. Her failure to contact the campaign before publicizing her findings violated an ethical best practice.

Sometime between January 28 and early March, 2009, Wikileaks obtained a copy of the spreadsheet, and that led to the email I found in my inbox when I woke up the morning of March 10.  The public reaction came fast and furious.  Here is another Minnesota Independent article and PDF copy.  Here is a Computerworld article and PDF copy.  And here is a Minnesota Independent article and PDF copy with donor reactions. Predictably, donors were upset and at least one donor reported being victimized by credit card fraud.

For the next few days, the story saturated Minnesota TV and print media.  Although the Coleman campaign tried to defend itself in the press, it ended up with a major public relations black eye as the campaign alienated its own donors and supporters.

Coleman eventually lost the recount battle in one of the closest US Senate elections in United States history.  In early 2013, Coleman floated the idea of trying a rematch against Franken in the 2014 election. A few days later, Coleman announced he would not run in 2014.

I wonder how much Coleman’s poor IT security practices hurt his political career?  After studying this incident and Coleman’s bungled reaction, I know I don’t want Norm Coleman representing me in the US Senate or anywhere else.  I have a hunch many others feel the same way.

Computer Whodunit – a Computer Troubleshooting Detective Story

This story is a great example of characterizing a problem, getting closer and closer to a solution with each step, and why the process is so important.  The story flows like a detective novel, with Greg the gumshoe uncovering new clues with each new step, all leading to a surprising conclusion that generates more unexpected questions for subsequent episodes.

Opening scene

Like most detective stories, the day started innocently enough.

My friend and customer, Lynn, called with a common problem.  Her email was broken.   Many of my problem calls start with broken email because pretty much everyone uses email.  But sometimes problems are not what they seem and the path to a solution can take many twists and turns.  This was one of those times.

I built the IT network in Lynn’s office and I know its characteristics the same way Scotty knew the original Starship Enterprise.   I knew Lynn used Microsoft Outlook on her desktop, the server was named ehcserver1, and the server ran Microsoft Exchange.  The server is in the basement of the building and everyone connects over a series of Ethernet switches.   Time for a good problem description.

Greg: “What happens when you launch your Outlook program”

Lynn: “It just sits there for a while and then gives me an error message, something about the server.”

Greg: “When did it break?”

Lynn:  “It worked fine when I shut down yesterday, but when I came in this morning and turned on my computer, now it doesn’t work.  I promise, I didn’t change anything.”

I could push Lynn harder for more details, but this told me enough.  Her Outlook program was not able to find the Exchange Server.   And I know Lynn well enough to believe her when she tells me she did not change anything.  This suggested something out of her control must have changed.

The next logical step in characterizing the problem was to find out if the problem was specific to Lynn or more widespread.  Quickly polling a few people near Lynn, we discovered Bruce had the problem, but not Ayrica, Joe, or Mike.  Since at least one other user had the problem, this suggested the problem was not specific to any workstation setting.  The problem was something common to Bruce and Lynn, but nobody else.

Start Unraveling the Mystery

Experience suggests most email problems are really symptoms of a more general network or server issue.  Everyone reports email problems because email is the application they use most often.  But email depends on the overall network.  If the overall network is broken, email will also be broken.

To find out if the problem is specific to email or something deeper, try a different application and see how it behaves.

One rule about working with end users.  Always start with an easy test and then dig deeper as necessary.  People seem to appreciate it more that way.

Greg:  Let’s see if you can see other stuff on the network.  Click Start…Computer, try to open one of your network drive mappings and let’s see what happens.  What happens when you open, say, the V drive?

A network drive mapping is really a directory on the server.  The idea is, the desktop computer “thinks” it’s another hard drive, thus the drive letter, but really it’s a directory on the server.   This is far and away the most common use for servers in an office.

All IT support companies have their own style and I set up many of my customers with a “V” drive, accessible to everyone.  It’s a convenient place to test.   Why V?  Because V stands for eVeryone.   Why not use “E”?  Because some computers use “E” for a locally connected CD or DVD or USB card reader.  It’s generally easier to use high letters in the alphabet for network drive letter mappings and leave low letters for locally attached devices.

Here is a picture similar to what Lynn saw.  (The picture will open in a different tab on your browser.)  The red X on the network drive mappings does not necessarily mean they are offline.  The only test that generates anything meaningful – just double-click on the drive letter and observe what happens.  Either the contents or an error message will show up in a window.

When Lynn double-clicked on the V drive, she saw an error message.  So did Bruce.  Since another application depending on the server and network was broken, the problem was not specific to email, but instead something common to both email and viewing drive letter mappings on the server.  But only common to Lynn and Bruce.  Mike, Joe, and Ayrica were fine.

Whodunit?

Computer troubleshooting is often compared to a good mystery movie.  Uncover clues and follow them where they lead.  This one was starting to feel like a Hollywood whodunit.  Time for some more in depth tests.

I asked Lynn to launch an old-fashioned DOS command window and try a few commands.  In Windows 7, Click Start…All Programs…Accessories…Command Prompt.  In Windows 8, click the upper right corner of the display to launch the Start screen, click the Start icon, right-click anywhere, click apps in the lower right corner of the system tray on the bottom of the screen, find the Command Prompt, and double-click on it.  (How much money did Microsoft spend on this new, “improved” interface?)

I knew the server was named ehcserver1.  So in that Command Prompt window, I asked Lynn to type “ping ehcserver1″, press the enter key, and tell me what it said.  Here is a picture similar to what Lynn found.  Here is a picture similar to what Lynn should have found.

How was it possible that Lynn could not translate the name of her server?  Clearly, something was fundamentally wrong with the network.  But it only effected a few users.  The next step is a deeper diagnostic.  In that DOS command window, type

ipconfig/all

Here is a PDF file with a sample report and some annotations taken from a Windows 7 computer in the Infrasupport network.

The computers in Lynn’s network should all have IPv4 addresses that look like 192.168.10.nnn, where nnn is a number between 1 and 254.  The gateway should be 192.168.10.1, DNS Server 192.168.10.20.  I built this network; I know what these values should be.

Surprise plot twist

But in a surprise plot twist worthy of the best Hollywood has to offer, both Lynn and Bruce’s computers showed IPv4 Address, Gateway, DHCP Server, and DNS Server Addresses of 192.168 2.nnn.  Note the 2.nnn instead of 10.nnn.

No wonder Lynn and Bruce’s computers were broken.   They both had bogus IP Addresses that did not belong to this network.  This was stunning!

The only possible explanation:  Somebody introduced a rogue DHCP server into this network and it was competing with my real DHCP Server.

DHCP servers lease IP Addresses and other network parameters to computers in an office.  Although there are carefully crafted special cases, typically an office should have exactly one and only one DHCP Server.  If an office has multiple DHCP servers, it is not possible to predict which DHCP server will lease a computer its network parameters.  This means computers may appear to suddenly fail at random times, and for random lengths of time, as their old leases expire and a rogue DHCP server assigns them bogus new network parameters.

This was exactly the case here.  The rogue DHCP Server serviced both Lynn and Bruce’s computers, while the correct DHCP Server took care of Ayrica, Joe, and Mike.

The suspicious character with the shifty eyes did it – or did he?

Wonderful.  Problem identified.  Now, what to do about it?  See  part 2 for the exciting conclusion to the story.

(Originally published on my old Infrasupport website on April 6, 2013.  I backdated the posting here.)

Computer Troubleshooting 101 – Characterize the Problem

Just like most IT professionals, I get computer troubleshooting questions all the time from customers, friends, and family. A few are, um, well, memorable. For example, the one about email a while ago.  The conversation started out something like this:

Friend:  My email doesn’t work.

Greg:  (Trying to be helpful)  OK, what email program do you use?

Friend:  Huh?

Greg:  Well, you run a program on your computer to get to your email, right?

Friend:  No, I just click on “email”.  But now it doesn’t work. What’s wrong with it?

I don’t think we ever solved that problem.  And most IT people reading this, after they finish laughing at an all too familiar story, know why.  I didn’t have enough information to begin solving the problem, and my friend was unable or unwilling to provide it.

All IT people read articles with advice about communicating with “normal” people.   The articles usually scold us for speaking a language most people don’t understand.  Fair enough and guilty as charged.  But we have our “IT words” for a good reason, as do all other professions.  I’m not sure why we get picked on so mercilessly.  For you finance people – why is it OK to say “EBIT-DA”, but not OK for IT people to say, “DHCP server”?

This blog entry is a little different.  I’m an IT guy and I’m asking so-called  ”normal” people who do not speak IT as a natural language to stretch just a little bit.  If you can say non IT words like “EBIT-DA”, you can say some IT words too.  It won’t hurt, I promise.

Meet us in the middle for your own benefit.  We IT people are pretty good at solving problems – that’s why we’re IT people – but we need more than “it doesn’t work”.  If you want your problem solved,  we need more from you.  I’ve learned at the feet of some of the best in the business, and what follows are some great troubleshooting tips.

First, before solving the problem, we have to identify it.  We call this characterizing the problem.  The process is part science, part art form.

Here are some things you can give me to help you get back up and running again:

What exactly happens when it breaks?  What do you do and how does the computer respond?  Give me a sequence of events leading up to the problem.  Give me exact error messages, codes, and pictures of screen shots if possible.  Details are important because at least one of those details may be a significant clue.

Has the system ever worked as expected or has it always been broken?  If it worked earlier and is broken now, when did it break?  What changed between when it worked earlier and now when it’s broken?

“Nothing changed” is always the wrong answer.  If nothing changed, then the system would still behave the same as it did earlier.  My friend, Bruce had a cell phone email problem a while ago.  He insisted nothing chanaged and his email just stopped working for no reason.  We talked about it and ended up removing and adding the email account to his smartphone.  Email behaved properly after that, and then Bruce said, “Oh yeah – a big update for my phone came out a few days ago and my email broke right after that!”  My other friend, Bob was also in the room, and Bob said, “wow – that’s probably why my cell phone email stopped working too!”

That’s the power of characterizing the problem – sometimes it helps solve multiple problems.

If the system worked before and is broken now, something broke it.  That something may be subtle and difficult to find, and that’s why details are important.  So think back to everything that happened with your broken system around the time the problem started.  Put together a detailed sequence of events.  Write it all down if this helps.  If I had known about that cell phone software update with Bruce and Bob, we could have saved time and jumped immediately to the solution.

Is the problem reproducible at will, or does it only happen sometimes?  If reproducible at will, what are the steps to reproduce it?  And if only sometimes, what is different about when it works versus when it breaks?  One time, I had a Dell laptop that sometimes refused to connect to the office wireless network.  After hours of trial and error, we finally found a pattern – the problem happened when the laptop was running on battery power, but not on AC power.  This turned out to be a (questionable) feature and not a bug – somebody at Dell thought it was a good idea to conserve power by turning off the wireless adapter by default when running on battery power.  The cure – press a function key to turn it on.

The solutions to many problems seem obvious, but generally only after going through the exercise to find them.

Perhaps most important – compare and contrast how the system should behave versus how it actually behaves.  It’s your job to explain this clearly and in detail to an expert who cannot be as familiar with the history of the problem as you.

Answer these and similar questions and now we have a well defined problem.

Next comes finding a solution.  The process is also part science, part art form.  For the science part, we form a possible solution based on the problem definition, come up with a way to test it, then evaluate the results.  The process is usually iterative, sometimes tedious, and always slower than anyone wants.  For the art part, sometimes inspiration strikes and sometimes it’s right.  Check out this article for a great example of a troubleshooting scenario.  And watch this space for more articles about interesting troubleshooting scenarios as they come up.

(Originally posted April 4, 2013 on my old Infrasupport website.)