Mostly bad telemarketing

I’ve taken thousands of telemarketing cold calls over the years.  Many are deceptive, most are awful.  And a few are good.

Here is a common deception.  Many calls originate in call centers in India or the Philippines.  Callers use IP phones and connect via the Internet to a system in the US, which generates a caller-ID in my local area code.   When my phone rings, I see a caller ID that appears local, which makes me want to answer the phone because I think it might be a potential customer who wants to buy my goods and services.

The conversations generally start something like this:

Hello, this is Greg Scott.  (I always answer my phone this way.)

(long pause, sometimes with a series of clicks)

Yes, hello, I am trying to contact Agreg a Scote, uhm, with Infrasupport-a-tech company?

Well, yes, this is Greg Scott

Ah, good morning, sir Greg.  I am calling  because… (and we’re into the script flowchart).

Why do I dread these calls?  After all, the caller is courteous.  And the company she represents is only trying to find customers.  What’s not to like?

Well, plenty.  First, the call is an interruption.  I have to stop what I’m doing, switch gears, and make a decision whether to answer the phone.  After I answer the phone, I have to focus on the caller’s message instead of what I was working on before the phone rang.   I understand callers are trying to find customers and this is part of business.  I’ve done cold calls myself.  But since callers know they interrupted me, they should respect me and my time.

That leads to the next problem.  From the very first ring, overseas callers with automated dialers and IP phones have already disrespected me and my time.  Why is this company trying to fool me into believing it’s a local call?  How am I supposed to trust a company that tries to deceive me with the very first contact?  Why would I ever consider buying anything from such a company?

Focusing on the caller’s message is also challenging.  I speak English as a native language, my hearing is not what it used to be, and I have a terrible time understanding the thick accent on the other end of the phone.   And I am willing to bet, nobody in Bangalore, India is named Gary.  Or Bob, John, Ted, Mary, or any other common English name.

Here are two questions I want to ask these telemarketing firms – not the callers trying to do their jobs, but the boneheads who manage the callers.  If I tried to speak your native language and you heard my American accent, how much time would you need to figure out your language is not my first language?  If I adopt a telephone name native to your language, would it make a difference?   The obvious answers to those questions are about one second and no.  So why do you think I will believe your caller speaks English as a native language simply because you gave him an American telephone name?

The problem is compounded by poor sound quality.  After the packets containing the sound from these calls bounce around dozens of IP routers before flying across the public telephone network to my phone, the sound is often garbled, muffled, and distorted.  Combined with a thick accent, it is always difficult and often impossible to figure out what callers are saying.

For companies using these services – if you have such a low regard for me as a potential customer, what kind of service can I expect if I buy your product or service?

And it gets worse.

Lately, I’ve taken dozens of calls from machines pretending to be people.  Here is a typical call, synthesized from many:

The phone rings, showing a local caller ID.

Hello, this is Greg Scott

(Long pause – this is always the dead-giveaway.)

Why hello!  This is Nancy and I have an exciting offer for you!

Really – wow, thanks Nancy.  Are you a real person?

(Pause)

(Laughing)  Well of course I’m real, why do you ask?

Well, Nancy, you sounded like a machine.

Oh no, I can assure you, I’m a real person.  I’d like to talk to you about a great line of credit we offer.  If you’re interested, I’ll connect you to my manager and he can cover details with you.

Ah – thanks Nancy.  By the way, who won the baseball World Series last year?

I’m sorry, could you repeat that?

Yes – who won the World Series last year?

I’m sorry, but we’re not allowed to give out personal information.

What’s personal about that?

Thank you.  Goodbye.

Let’s see, what was wrong with this call?  After all, it solved the sound quality problem and the caller’s native language matches mine.  I can visualize a team of misguided engineers,  proud of their creative masterpiece, presenting the slideshow bullet items to a bunch of boneheaded executives in a boardroom and congratulating themselves on solving their telemarketing problem.

Here is my question for the clowns who dream up this stuff.  If your time and money is too valuable to use a real person fluent with my language to make real phone calls, why do you think my time and money is any less valuable?  Do you really think I will buy anything from you when you use a machine to waste my time and lie to me?

So after griping about bad calls, what about somebody who did it right?  Well, it happened one day last year when a nice lady from a training company called me.  Let’s call her Dee and her company, Training Inc.  These are both fictional names.  Dee did everything right.  It was obvious she looked at my website before she made contact because she tailored her pitch to meet my unique circumstances.  She asked me a bunch of questions about how I run my business.  She asked me about my training goals.   She was personable.  She spoke the same native language as me.  Instead of trying to fool me into thinking she was from this area, the caller-ID was from a different state.

I liked Dee.  We connected.  I don’t have any business for her right now, but when the time comes and I am able to send business her way, I will do so.  In fact, I liked Dee so much, I spent most of a Saturday updating and fixing my broken Exchange Server indexing so I could find her contact information.

If you are a telemarketer and happen to read this blog entry, first, thanks for reading.  If you’re spending money for overseas call centers with cheap IP phones, bad connections, and fake caller IDs, or if you’re trying to use machines pretending to be people, save your money.  Nobody in their right mind will buy anything from you when you approach them this way.  Instead, find somebody like Dee who will represent you properly.

Even in today’s high-tech, 24 hour, over stressed environment, the old-fashioned rules still apply.

(First published on my Infrasupport website, June 7, 2014.  I backdated here to match the original posting date.)

How to do bad customer service and destroy your reputation

I survived one of the worst customer service experiences ever this week.  We can all draw some lessons from this story.

The end user customer operates branch sites across the Midwest USA and uses Infrasupport firewalls to connect to the Internet and main office in the St. Paul, MN. area.  This branch site is in southern Illinois and uses a nifty new twist on my network failover system.

The firewall has a wireless LAN (WiLAN) card and several wired network interface cards (NICs). The new twist – connect a wired NIC to a low cost DSL connection. When the DSL modem drops, failover to the WiLAN connection and route through a cell phone carrier. And when the DSL connection comes back, fail back to the DSL wired connection. It works – this is the lowest cost redundancy anyone can buy.

This site opened about 6 weeks ago and the DSL connection wasn’t ready.  No problem.  We routed via WiLAN over the cell phone carrier and waited more than a month for Ma Bell Internet to provision the DSL service.

Ma Bell Internet is a fictitious name. As are the names of everyone else in this story other than me. Other than the names, every detail is true. As one person involved said, “you can’t make this stuff up.”

The story starts last Wednesday after I did the appropriate adjustments on the firewall to accommodate the new DSL connection.  Tom, the end user customer at the site, connected the patch cable from the DSL modem to the firewall and nothing worked.  Our troubleshooting pointed to a misconfigured or bad DSL modem.

We logged a service call with Ma Bell Internet, which triggered a week long comedy of errors. We called on Thursday and talked to – I’ll call her Misty – from tech support.   Here is a piece of our conversation:

Greg: You have a gateway at this IP Address. We can’t ping it from the site.

Misty: I don’t see why you can’t ping it. I can ping it from here.

Greg: Right – I don’t know why we can’t ping it. That’s why we’re calling. Nobody can route through that DSL modem and we can’t ping it.

Misty: I don’t understand why you can’t ping that modem. I can ping it from here, you should be able to ping it from there.

Greg: Right – we should.  And if it answered it would be even better.

Misty: So how come you can’t ping it?

Greg: Routers have two sides. We’ll call them the inside and outside. You’re on the outside and you can ping it. The site is on the inside and can’t ping it. Seems to me, something is wrong with the inside of that router.

Misty: Well if you can’t ping it, there’s not much I can do.

Greg: Yes there is!  It’s your router.  You need to fix it.

Misty: I tried connecting to it from here remotely to check its configuration but I’m not able to.

Greg: So – doesn’t that suggest to you something is wrong?

Misty: No. We connect those the way we’re ordered to connect them. If the order said no remote diagnostics then we wouldn’t have turned that on.  And I can ping it.

Greg:  So did the order for this one say no remote diagnostics?

Misty:  I don’t know what the order said – I don’t have it.

Greg:  So how do you know the order didn’t call for remote diagnostics?

Misty:  Because I’m not able to connect to it.

Greg: (Exercising patience)  OK, so how do we fix this?

Misty: You can run remote diagnostics from your site. Just hook a real computer up to it and connect to its website.

Greg: The problem with that is, we can’t access the modem. If we could access the modem we wouldn’t need to call you. And the only computer I have onsite is my firewall. Everything else is thin clients.

Misty: If you’re unwilling to run remote diagnostics, I can’t do much to help you.

Greg: Yes you can. Send somebody out there to fix the modem!

Misty: Maybe you can borrow a laptop from somebody and try the diagnostics.

Greg: There are no laptops to borrow at this site. We have my firewall – with no graphics so connecting to a GUI website won’t work anyway – and some thin clients. That’s it.

Misty: I can dispatch a technician, but it will be billable.

Greg: What??? Why is it billable?

Misty: Because you’re unwilling to run remote diagnostics.

Greg: No, not unwilling.  Unable.  We are unable to connect to this modem.  If we can’t connect to it, how do we run diagnostics on it?

Misty: If I dispatch a technician and he finds a building wiring problem or something else not from us, we’ll have to charge you.

Greg: Well of course. The wiring is a 20 foot cat5e patch cable.  That’s it.  That’s the building wiring.  One patch cable.  Send somebody out.

That was on Thursday.  I asked for somebody onsite Friday.  But with nobody available Friday, I had to settle for Monday.  My phone rang on Friday and I confirmed it – send somebody Monday, preferably Monday morning.

Monday morning came and I learned Ma Bell sent somebody to the site on Saturday.  Of course nobody was at the site on Saturday.  After some Monday phone calls, we scheduled another visit for Wednesday – nobody from Ma Bell was available Tuesday.  So now were were a week into this issue.  After opening the support ticket on Thursday, the soonest possible resolution would not happen until next Wednesday.

Wednesday morning about 9:45 my phone rang.  It was the onsite Ma Bell support technician.  Let’s call him BA.  You’ll see why in a minute.

BA told me the customer became angry and sent him upstairs to call me.  I apologized and said there was a systemic problem at Ma Bell Internet and he was the guy onsite who had to hear it all.   BA told me he also got mad at the customer and apparently started throwing boxes around the site.  And that was when emails started pouring into my inbox warning me about this technician and his attitude. Apparently, BA was angry the second he walked in the door and took out his frustrations on Tom, the end user customer.  Tom got tired of the abuse and sent BA upstairs to call me.

BA asked me a bunch of questions I didn’t know how to answer.  He wanted to know how to configure the router password and who to call for onsite tech support.  I asked BA – since he worked for Ma Bell Internet – if he shouldn’t know who to call for support?  BA launched into a diatribe about his employer and his frustrations about training and scheduling and his management.  Here is the portion I remember most vividly:

Greg: So how do you know these connections are good?

BA: We set them up dynamic and then connect them to the Internet. If they work, then we give them their static address and we’re done.

Greg: So how do you know they work when they’re static?

BA: Because they worked when they were dynamic.

Greg: Don’t you think you should test them when they’re static?

BA: That’s not what we do.

Greg: Maybe you should give the Ma Ball guys some feedback about…

BA: (Interrupting) They won’t listen.

Greg: How badly do they want to keep customers?

BA: They won’t listen. They’re managers and they don’t care!

Greg: OK. Well for now, we need to fix this modem. I need it to to have the static IP Address assigned to it, no NAT, no DHCP, and no firewall rules…

BA: (Interrupting again) Hold on there – Maybe those were English words but I have no idea what any of that stuff means. I’m not an IT guy.  I don’t know why customers keep trying to get me to solve their IT problems. I’m not an IT guy!

Greg: Well, that’s OK.  I’m an IT guy so I can cover that part.  Hang on a second – (recalling my notes from last week – fortunately I still had my scrap of paper handy and near the top of my pile of scraps of paper with notes from countless other customer engagements) I have the Ma Bell Internet toll free number right here.

I called the number and tried to conference us all together.  The conference call didn’t work.  I told BA I needed to hang up to clear my messed up conference call and I would call him right back and try again.

We hung up and I called the toll free number again. This time it worked and I pressed the phone buttons to talk to somebody in provisioning.  Provisioning sent me to Tech Support and I talked to a helpful (finally!) lady I’ll call Ingrid.

I told Ingrid that BA was standing by onsite and we needed to configure this modem.  Ingrid said they can’t disturb technicians who are onsite working.  I said he’s waiting for us to call right now.  I put Ingrid on hold, called BA onsite and his phone immediately went to voicemail.  And an email came into my inbox from Tom – BA was on the phone with somebody else.

I came back to Ingrid and explained BA was on the phone with somebody else and his phone went to voicemail.  And then my phone beeped.  It was BA.  I put Ingrid on hold again, told BA to hang up and stand by and we would conference him in.  Back to Ingrid – but my cell phone was now tied up because BA’s inbound call tied up my conferencing capability even after BA hung up.  Ingrid would have to do it.  So Ingrid tried to call BA.  Ingrid reported BA’s phone answered and disconnected.  She tried again, same problem.

My cell phone beeped again.  It was BA, reporting that somebody tried to call him twice.  He answered but could not hear anyone so he hung up.  I could hear frustration rising in BA’s voice.  So I explained to BA that Ingrid was trying to conference us all together and to stand by.  ”Hang in there, we’ll make this happen.  I promise.”

Back to Ingrid.  I told Ingrid what BA told me and Ingrid said she would hang up with me and try to call BA and conference me in.  I told Ingrid what we needed with that modem.  Give it the block of static addresses it’s supposed to have, turn off DHCP, turn off NAT, turn off all firewall rules.  Bless her heart, Ingrid knew what I was talking about.  She repeated it and we hung up.

Whew!

My phone rang a minute later.  It was Pat from Ma Bell Internet.  Pat had BA on the line and wanted to conference me in.  I don’t know what happened to Ingrid.  I said, “yes, absolutely!”

Finally – we had all the right people together on the same call at the same time. Now we could get to work.

Meantime, my email inbox chirped with requests from the customer main office for status updates.  Like most IT people, I can type and talk at the same time.  It’s a skill we all learn sooner or later.  So I updated the customer via email while I talked to Ma Bell Internet on the phone.

Pat asked if I wanted the modem to be a bridge or router. I told her I didn’t care, as long as this site could get to the Internet.  So Pat decided to try setting it up as a bridge.  Pat talked BA through the steps and suddenly, nobody from inside or outside could ping that gateway address anymore.  Woops.

So we had to configure it as a router.  BA groaned – “so that means I have to type in that long password string again?”

“Yes”, I said. “Sorry. I wish there was another way to do it, but you’re there onsite and I need your hands and eyes.”

Pat talked Mike through the steps to reset the modem and configure it again.  And after all that – after trying to teach a telephone support technician what routers do, after a week of bungled appointments waiting for Ma Bell Internet to send somebody, after reassuring an onsite technician with a bad attitude, after juggling conference calls that refused to conference,  after all that, here was the problem. This was why the site could not route to the Internet over that modem.

BA: And I’m setting the subnet mask to 255.255.255.0, right?

Greg: (Listening passively until now) NO!

Pat: Uhm,  no, change that last octet to 248. So the subnet mask should read 255.255.255.248.

BA: Oh – I always set it to 255.255.255.0. I don’t even know what it means.

Greg: Well, it’s important to get that one right.

Pat: (Didn’t say anything.)

And viola – it all worked. I started up an SSH session into my nifty onsite firewall and was ready to thank everyone for getting this up and running when everything dropped again.

After some more troubleshooting, Pat said, “I see some upstream errors here.” I said, “I’m pinging in another window and I notice the ping times zoom up from around 60 ms to more than 200 ms when my SSH session drops.”

Pat said, “Hold on a minute, I’ll bring somebody else in.” Pat put us on hold, leaving BA and I together.

After a few seconds, BA said, “Where did she go? Didn’t she say she was getting somebody?  What’s going on here?”

“Just hang in there.  Give her a minute to come back.”

A few seconds later, a man I’ll call Galen came on the line. Galen and Pat confirmed something unusual was going on.

And then BA asked us all to wait a minute. “I’m taking the modem offline for a minute.”

Pat said, “Oh – now I don’t see the modem anymore!” I said, “yes, BA said he was taking it offline for a minute.”

A few seconds later, the modem came back online.  BA explained he pulled the telecom cabling out of the punchdown block and punched the wires down again.

And after BA punched down the wires again, the connection stayed solid.  After watching for about 2 minutes, BA said, “I’m outta here!” and left.  Galen, Pat, and I stayed together for another few minutes and then I agreed Ma Bell Internet could close this case.

Wow!

What lessons can we extract from this experience? I named the onsite technician BA because he really did have a bad attitude.  He had a bad attitude because his company is dysfunctional and he handled the stress poorly, making a touchy situation even worse.  How do I know Ma Bell Internet is dysfunctional?  Look at the evidence.  Poorly trained telephone support technicians (Misty), broken scheduling, poor communications, overburdened and under trained onsite technicians.  This is fixable if the managers at Ma Bell Internet want to fix it.  If not, plenty of other Internet providers will eat their lunch.

(First published on my Infrasupport website, May 24, 2014.  I backdated here to match the original posting date.)

Spying – The pot calling the kettle black

Sometimes when high tech meets international politics, reality really is stranger than fiction.

First, a few enlightened members of our US Congress accused Chinese telecom equipment giant, Hauwei, of spying for the Chinese government. Here is one of many press articles, this one from October, 2012.  Here is another article from 2011.  Apparently, much of the fear on this side of the Pacific about Hauwei is because Hauwei founder and CEO, Ren Zhengfei was once a telecom technician in the Chinese People’s Liberation Army.  The company CEO served in his own country’s military years ago.  Therefore, today’s Chinese government will use equipment from his company to spy on the United States.

I wonder how many American CEOs once served in the US military?  Does it follow that their companies therefore spy on China?

This article from July, 2013 might be one of the best.   Quoting the first sentence in the article:

Former Central Intelligence Agency chief Michael Hayden said that at a minimum, Huawei had provided Chinese officials with “intimate and extensive knowledge of the foreign telecommunications systems.

Farther down, we see this nugget:

Hayden currently serves on the board at Motorola Solutions, and is a principal at security consultancy Chertoff Group.

Yup, that’s the same former Homeland Security Director, Michael Chertoff, who oversaw the US Government’s not-so-brilliant response to hurricane Katrina back in 2005.  Now he runs a consulting company, advising governments and big business how to keep their infrastructure safe.  And Michael Hayden works for him.

As for Motorola Solutions, here is how that company describes itself, from its own website at http://www.motorolasolutions.com:

Motorola Solutions provides business- and mission-critical communication products and services to enterprises and governments.

I should disclose a few things before going any further with this.  First, I am an American and proud of it.  By an accident of birth, I am blessed to live in the best country in the world.  I want the United States to compete fiercely and win all the competitive battles.  I don’t like Chinese counterfeiting, I don’t like spam relayed from Chinese email relay services, and I don’t want anyone spying on me.

I like to think I’m one of the good guys.  I want my country to also be one of the good guys.

I also like level playing fields.  I regularly go up against entrenched companies – American and foreign – and it frustrates me beyond belief when I offer superior solutions but lose because the entrenched competition successfully introduces FUD with the potential customer.  Introducing FUD – Fear, Uncertainty, and Doubt – is a time honored tradition in the high tech marketplace.  The conversations start something like this:

Mr. Customer, are you sure you want to look at this new solution?  You have a lot riding on this project, and even though this new upstart might offer some advantages and they’re less expensive than we are, is it really worth the risk?  After all, we’ll be adding that capability sometime in the next 20 years so they don’t really have any advantage anyway.  Doesn’t it make more sense to stick with us and what you already know?

And bla bla bla…

FUD is often no more than a line of BS, but fear is a powerful motivator.  FUD works – that’s why entrenched incumbents use it.

So now, along comes Hauwei, a Chinese company, and the guy who sits on the board of a direct US competitor accuses Hauwei of spying for the Chinese.  And he made his accusations nearly a year after a US Presidential Commission spent 18 months investigating Hauwei and found no evidence to support the accusations.  Read the details right here.

What’s really going on here?  Hayden and his boss are spreading FUD, wrapped up in the US flag and national security.   But it’s not really about national security.  It’s about keeping a competitor out of the US marketplace.  It’s good old fashioned protectionism mixed with a 21st century high tech twist.  It was never about national security, it’s about money.

And now it gets better.

Because the NSA – the organization Hayden used to run – could not keep its own secrets, we find out the NSA hacked into the Hauwei internal network and spied on Hauwei.  That’s the pot calling the kettle black.

Instead of Hauwei spying on us, we spied on Hauwei.  And got caught.

In what universe is it possible the Chinese are the good guys in this episode?

(First published on my Infrasupport website, March 26, 2014.  I backdated here to match the original posting date.)

Why we all should care about net neutrality

Many people will see the words, “Net neutrality” and groan about yet more tech gobbledygook and geeks who spend too much time pretending to be Mr. Spock and watching Star Trek re-runs.  Nobody on Main Street cares about net neutrality, right?  Isn’t this all just an arcane concept that never intersects with real people on Main Street?

Well, not so fast.

The real story – behind all the tech jargon – is as old as the first antitrust issue ever to come before the US Government more than 100 years ago.  And it will effect everyone who connects to the Internet, which is pretty much everyone these days.  For people who think tech is only for weenies, think money.  $Billions in money.  And all of it comes from your pocket.

Net neutrality means Internet Service Providers (ISPs) are supposed to treat all Internet traffic equally, end to end.  Every data packet should be treated equally to all other data packets, regardless of source or destination.  ISPs should be neutral carriers and not make judgments about favorable or unfavorable traffic.

Here is the issue.  Without net neutrality, large ISPs will have the legal right to mess with your traffic.  Large players will have monopoly power and will control your access to services you care about.

And what happens when any monopoly player offers its own, competing services?  Forget high tech for a minute.

Let’s say Alice runs a restaurant.  But Bob controls all the streets in town.  If Charlie wants to eat at Alice’s restaurant, Charlie has to travel over Bob’s streets to get there. What happens if Bob’s sister, Doris, opens a restaurant that competes with Alice?  Bob wants to make sure money stays in the family, so Bob sets up toll booths for all travelers on his streets. But people who eat at Doris’s new restaurant get their tolls refunded, courtesy Bob.  Of course, this puts Alice at a competitive disadvantage, so Alice eventually closes.  Before long, Bob controls all the restaurants in town.

Now back to high tech.  Today’s large cable companies offer bundles that include phone service, Internet service, cable TV, and premium services such as movies on demand.  These companies control both distribution and content.  They control many of the streets and some of the restaurants. They want to control all the streets so they can encourage you to eat at their restaurants.

If any single ISP becomes your only choice to connect to the Internet, that ISP controls your access to the services you care about.  ISPs can exercise that control with pricing and surcharge gimmicks, much like the antitrust monopolies of old.  But today’s ISPs also have even more powerful tools.   They can prioritize traffic or play other quality of service games, to treat traffic badly they don’t want to carry.

Today’s familiar services such as Amazon, Netflix, Hulu, Facebook, Google, LinkedIn, and others, at their core, are elaborate websites.  The path from your house or business to those services runs through the Internet.  Without net neutrality, ISPs can grant or deny or regulate or tax access to these services as they see fit.  If an ISP decides it wants to offer, say, retail services, what access policies will it set up for Amazon?  Let’s say you put your business in the cloud, but your ISP offers a competing cloud service.  What quality of service will your ISP give you?

This is not hypothetical.  Comcast, for example, blocks traffic coming from email servers located in home networks.   More ominous, thousands of Netflix users are complaining about bad Netflix movie quality when connected to Comcast.  Comcast counters that it has a right to prioritize traffic as it sees fit because it wants to protect occasional Internet users from heavy downloaders.  Following that line of reasoning, I wonder if Comcast prioritizes its own Movies on Demand service similarly to Netflix, which competes with its own service?

Net neutrality is under constant attack.  If open access to Internet services is important to you – and it should be – then familiarize yourself with the details around net neutrality and make your voice heard.  Your livelihood may depend on it.

(Originally published on my Infrasupport website on Feb. 17, 2014.  I backdated here to match the original posting.)

What is the right way to deal with IT security vulnerability disclosures?

With all the IT security issues in the news lately, suddenly IT security is everyone’s problem.  One natural question behind the headlines is, what is the right way to handle IT security vulnerability disclosures?

Here are some thoughts.

To keep things simple, let’s limit this discussion to three major players.  The real world is more complicated, but this is enough to illustrate the concepts. The first player is Bob, leader of an organization.  Next is Ingrid who discovers a security vulnerability.   And, of course, Trudy, the evil intruder we all love to hate.  Trudy spends most of her waking hours probing the Internet, looking for weaknesses she can exploit and secrets she can steal.

Let’s say Bob’s business operates a website and Ingrid finds a security vulnerability that exposes sensitive information about Bob’s customers.  How should Ingrid proceed?

Here is a blog post I put together a few months ago with an example of what happens when players proceed the wrong way.

This is what should happen.  When Ingrid finds the vulnerability, she realizes Trudy is already trying to exploit the weakness to steal personal information from Bob’s customers.  The race is on to fix the problem before Trudy exploits it for her own evil purposes.  And Trudy has a head start.

Ingrid has an ethical duty to immediately inform Bob about the problem and make Bob aware of the potential consequences.  Bob, always skeptical about gloom and doom warnings, listens to Ingrid because Ingrid makes a coherent and credible presentation about the problem.  Bob heeds the warning, fixes the problem, and quickly informs his customers and takes remedial action.  A newspaper or popular blog eventually publishes the story, giving credit to Ingrid for her dedication.  Evan, an executive from an influential software company, reads the story and offers Ingrid a job as Director of IT Security.   Everyone lives happily ever after, except Trudy, who was denied the opportunity to steal from somebody.

That’s how things should work.  But it doesn’t always happen that way.

Let’s say Ingrid presents the problem to Bob, but Bob ignores the warnings.  Now what?  Trudy is out there.  When Trudy finds Bob’s vulnerability, she will exploit it and steal from Bob’s customers.  Trudy might even drive Bob out of business.  How does Ingrid respond if Bob fails to respond?

Let’s say Bob uses software from a company named, say, Orange Computer, and Ingrid finds a security problem with that software.  Ingrid contacts the right people at Orange, but Orange sits on the problem and does nothing.  Trudy is out there.  If Orange fails to address the problem, Trudy will exploit it.  What does Ingrid do?

Ingrid’s only course of action in this case is to follow a best practice called responsible disclosure.  After trying to warn Bob.  After contacting Orange.   After taking all reasonable steps to inform the right people, and after waiting a reasonable amount of time for a response, and as a last resort, Ingrid has a duty to disclose the problem publicly.  Ingrid must assume Trudy and her friends are already quietly exploiting the problem, and Trudy will hurt too many people if Ingrid fails in her duty.

Ingrid also has a duty to protect herself.  She should document her attempts to contact Bob and the people at Orange Computer as appropriate because when the problem becomes public, it will ignite a firestorm of controversy with Ingrid in the middle.   This will create an opportunity for Ingrid to educate the public and a threat from people who blame the messenger for creating the problem.

Politicians will weigh in with uninformed opinions and instant experts hungry for publicity will offer canned analysis for gullible press outlets hungry for sensational stories.  The noise will be deafening; real information will be scarce.

Amid all the noise, what about customers, the people who use software from Orange Computer and the people who use Bob’s website.  How do they respond?

Customers should do independent homework and look for the real story.  Security vulnerabilities happen all the time.  Is this one just another sensational story or is it real?  What are the prudent steps to protect against it?  What are the plans from Bob and/or Orange Computer to address the problem?  What are the consequences of not addressing the problem?  Customers need to find credible answers to these questions and make informed choices on how to respond.

After the initial disclosure shock wears off, some other questions are appropriate. Who is Ingrid?  What were her motives?  How did she find the problem?  Before the problem went public, what steps did Ingrid take to contact the right people?

That scenario assumes Ingrid discloses the vulnerability responsibly.   What if Ingrid wants to make a name for herself and she discloses the vulnerability without first informing Bob?  In this case, Ingrid is really a bad guy disguised as a good guy and trying to gain notoriety at the expense of Bob’s company.

Bob learns about the problem on the TV news along with the rest of the world and his company phones start ringing a few seconds later as press outlets everywhere look for comments and controversy.   What does Bob do?

Bob faces multiple threats.  He faces a public relations threat from sensational press stories spawned by Ingrid’s improper disclosure.  Bob and his customers also face a material threat from Trudy, quietly exploiting the vulnerability at the expense of  Bob and his customers.

To meet the PR threat, Bob needs to get in front of a runaway public relations train and slow it down.  This is the time for visible leadership and Bob must get in front of the cameras and take charge.  Provide explanations and frequent progress updates, and answer questions honestly and directly to repair credibility with a skeptical public.

Simultaneously and behind the scenes, Bob must also immediately address the actual vulnerability because Trudy wants to steal from Bob’s customers.  This might mean bringing in outside experts, it may even mean temporarily suspending business.   It will cost money.  Probably lots of money.  But if Bob handles this crisis properly, it can also be an opportunity for Bob’s company to come out of it with more trust and more credibility than before.

What if  Bob himself is a bad guy?

In 2005, Mark Russinovich was Ingrid and multibillion dollar Sony Corporation was both Bob and Trudy when Sony compromised thousands of computers around the world by surreptitiously introducing a rootkit when anyone played a Sony BMG music CD on a Windows PC.   A rootkit is illicit software that modifies core system components and is designed to conceal itself from malware countermeasures such as antivirus products.  Bruce Schneir summarized the story here.  Mark Russinovich’s original blog post with details on his great detective work uncovering the problem here.

Russinovich found the problem and reported it publicly in his blog.   This was the right thing to do and Sony eventually paid millions of dollars to settle fines and class action lawsuits.

What if Bob is a government agency and Ingrid discovers a vulnerability or abuse of power?  Now the consequences might be global.  Scenarios like this have spawned long discussions over the generations about ethics and whistle-blowing.  Sometimes, Ingrid is a lonely crusader pursuing justice against powerful forces.  Other times, Ingrid is an egomaniac, pursuing her own interests at the expense of everyone else.  And Trudy is always out there, ready to strike at every opportunity.  Ingrid has a duty to proceed with caution and carefully weigh the consequences of any action.

If you find yourself in a position similar to my hypothetical Ingrid, how do you decide what to do?  Who is harmed, who is helped if you disclose the vulnerability?  And who is harmed, who is helped if you do not disclose it?  If you take action, are you serving justice or your own ego?  Confide in a few people you trust and make your choice based on honest answers to those questions.  Do it responsibly.   Careers and lives may depend on the choices you make.

(First published on my Infrasupport website Feb. 14, 2014.  I backdated here to match the original posting date.)

What should a small business IT security system look like?

Given the recent security breaches all over the news, what would a good Main Street business security solution look like and how much would it cost?  After all, if organizations such as the NSA and large retailers such as Target can’t keep their secrets safe, what chance does Main Street business have?

A pretty good one actually. Keep reading.

First, an assumption. No piece of equipment is hacker proof.  You must assume bad guys want to get inside your devices and use your equipment and your network for their own evil purposes.  They have specs for everything you own and probably know more about the internal workings of your equipment than you’ll ever hope to learn. They’re smart, they’re greedy, they collaborate, and they want what you have.

That’s the nature of the threat.  Here are the pieces to deal with it.

It starts at the firewall.  You need a real firewall with provision for multiple LANs.  A real firewall is a router with multiple segments and some rules to regulate how each segment interacts with the other segments.  Most credible DSL and cable modems can accommodate firewalls behind them if configured properly.  Here is a PDF file you can download with some firewall frequently asked questions.

Your firewall will have at least one public, Internet facing segment.  It might have more public segments if you want multiple Internet feeds from multiple providers so you always have a path out if one feed drops.  Multiple Internet feeds is probably overkill for a business like a Chinese takeout restaurant, unless that restaurant depends on, say, a website to operate hour by hour.

You may choose to have an HA (highly available) firewall system with redundancy at your boundary that can juggle multiple Internet feeds and do automated failover routing in case an Internet feed goes offline.  This may also be overkill for that Chinese food takeout restaurant.  It may not be overkill for a multiple site retail operation that depends on the HQ site always being available.  Start small and scale as the business grows.

It will have a “people” segment where you put your employee computers.  This is where you put in the typical rules you see in most business networks. You’ll want a credible antivirus solution on all your workstations in this segment.  It can also become elaborate. You can put in web filtering appliances to regulate which websites your users visit, for example. If you choose to host your own email or web server(s), you can put in rules to accommodate those, and rules to accommodate spam filtering. This is overkill for small operations and a logical growth path for larger businesses.

If you’re a retailer, your firewall will also need a POS segment for your Point of Sale systems.  A simple POS terminal might interact only with your credit card processors.  Credit card processors all have IP Addresses, so your firewall will have rules to allow anything in the POS network to interact only with those IP Addresses.  The firewall will also have a rule blocking anything between your “people” segment and POS segment.

If your POS network is more sophisticated, those POS systems might need to interact with, say, a database server.  That database server, in turn, may need to access servers in your “people” network.  In this case, carefully construct firewall rules to accommodate this traffic and log attempts at any other traffic.  This is overkill for that Chinese restaurant, but might be essential for a franchise of Chinese restaurants or a sophisticated retailer with, say, a loyalty program.

Maybe you want to offer wifi as a convenience for your customers. This is tricky to do properly because of the nature of wireless and because you don’t want your customer wifi to mingle with your employee wifi in your stores.  Isolate the customer wifi from your employee wifi and all your other segments.  The wifi segment is only a convenience for your customers to get to the Internet.  Nothing crosses the border between the customer wifi into the “people” segment or the POS segment.

And there you have it in a few short paragraphs.  A topology that does a wonderful job of enabling your business, serving your customers, and keeping bad guys out.  Total investment includes a properly built firewall and either a few physical network switches or a smarter switch with VLAN capability.  Budget a cost of about $4k to start. The actual cost might be a little less for small operations, probably more for larger operations.  The antivirus subscriptions and other support subscriptions will also cost some op-ex each year.

(First published on my Infrasupport website on Feb. 8, 2014.  I back-dated here to match the original posting.)

A Left-Handed IT Security Opportunity to Earn Public Trust

The sensational IT security stories just seem to keep coming.  Consider:

  • Researchers at antivirus companies decoded a mysterious computer virus named Stuxnet, apparently authored by our own NSA and the Israelli government, designed to attack Iran’s nuclear equipment.
  • Army Private Bradley Manning (now Chelsea Manning) stole hundreds of thousands of secret communications and videos and sent them to Wikileaks, which published them.
  • Edward Snowden, working as a contractor, stole thousands, maybe millions of documents detailing how the United States gathers intelligence information and fingering pretty much every American IT equipment vendor and large service provider.
  • 40 million credit card and PIN numbers are now up for grabs thanks to malware implanted in Target’s POS systems.  And personal information stolen from other Target databases on 70 million more people are also now up for grabs.
  • Apparently, Target is not the only retailer with a data breach.  News reports of another breach at Neiman Marcus now fill the headlines.  Others are sure to follow.
  • And because of the Snowden revelations, the United States government stands accused of paying and/or coercing a Who’s Who list of American IT equipment vendors and service providers to aid in spying on foreign and American citizens.  One breathtaking claim says the US Government paid $10 million to RSA, a leading IT security company and standard setter, to purposely weaken at least one of its encryption standards.
  • That same United States government effectively forced a Chinese company out of the US market by accusing it of spying for the Chinese government, while at the same time it coerced and enticed American companies to help the US Government in its spying.

TV news reports paint a picture of the NSA as a group of trustworthy professionals gathering all this data to protect an unsuspecting public.  I’m sure top professionals work for the NSA, but if the NSA is so institutionally smart, how did one rogue system administrator steal millions of documents and put the entire United States intelligence gathering capability at risk?  What happened to concepts such as least privilege and levels of accountability?  And why is the Stuxnet virus now in the public domain?  Did the authors really believe it would remain secret as it wormed its way around Iranian computers, looking for targets?

Sensational security stories are not limited to the US Federal Government.  The initial reports on the Target breach came on December 15, 2013.  See this blog post.  On Friday, January 10, 2014, Target disclosed another theft from the same breach involving personal information for 70 million additional people.  Let this sink in for a minute – Target and an army of forensic investigators examined Target’s infrastructure in detail for nearly a month before finding evidence of the additional theft.  How many other similar thefts have gone undetected?

The predictable result of all these revelations?  Erosion of trust, finger pointing, shock, outrage, and hyperbole everywhere.

While government, the courts, and an alphabet soup of secret security agencies and large companies sort all this out, how much of this matters on Main Street and what should businesses and individuals do about it?

The core of all security products and practices depend on trust.  That trust has been violated and that makes this critical on Main Street. Main Street companies can no longer trust their infrastructures are safe from government and criminal eavesdropping because the very products put in place to protect against it are tainted.

Great – we can no longer trust our IT infrastructure products.  What do we do about it?

Consider replacing critical IT infrastructure components with components built using the open source model.  Although this reads like arcane tech jargon, the concept is vital in today’s interconnected and insecure world.  Two general methods exist for building the software we use every day for browsing the Internet, processing transactions, connecting phone calls, and everything else.  These are:

  1. Proprietary
  2. Open

With the proprietary model, one company controls everything about a product.   Microsoft Windows, Microsoft Office, Apple IOS, Cisco routers, and many others use the proprietary model.  The good about the proprietary model is, companies (hopefully) stand behind their products and offer support and accountability.  The bad is, customers are left at the mercy of these companies and nobody knows what’s inside, which provides an opportunity for meddling by government or other bad guys.

With the open source model, one person or organization acts as a maintainer or lead developer of an ongoing project, and members of a world wide community contribute new features, bug fixes, and peer review.  The development process happens in full public view, which means no government agency from any country has an opportunity to introduce secret “back doors.”  Why would armies of thousands of unpaid volunteers do this?  For the same reason I write articles for this blog – for the recognition, which hopefully leads to service revenue.

The major challenge behind open source is, community developed means community supported, which means nobody is accountable when things go wrong.  To meet this challenge, companies such as Red Hat provide commercial support subscriptions for open source products.  My company, Infrasupport, is a Red Hat partner.  This provides the best of both approaches; accountability from the proprietary model and professional peer review from the open source model.

Enlightened IT departments will seize the opportunity from today’s supercharged security climate to secure their organizations’ IT assets using untainted, open source tools.  These organizations will earn back lost trust and the rewards that come with it.  The rest will bury their heads in the sand and hope the problem goes away.  But the problem will not go away.  Sensational stories will keep coming and market power will shift to those organizations with enough guts to take control of their own environments.

(First posted on my Infrasupport website on Jan. 13, 2014.  I backdated here to match the original publication date.)

Target, get on the ball with this data breach

If you shopped at Target between 11/27/2013 and 12/15/2013, congratulations.  Your credit or debit card info is one of around 40 million up for sale in a thriving underground marketplace complete with wholesalers, distributors, retailers, and easy to use websites.  Replace your card right now before bad things happen.

Brian Krebs broke the story in his blog, Krebs on Security, and the public owes Krebs a debt of gratitude.   Here is the original story.   Here is a follow-up post.  Target blew it.  Target should have notified customers and broken the story itself.  But  instead of proactively notifying its customers, Target apparently responded to the Krebs blog, as did the rest of the popular press.

The more onion layers peeled back, the scarier this gets.  Where did that date range between Nov. 27 and Dec. 15 come from?  Apparently, banks buy samples of stolen credit card info from those same underground markets and look for patterns.  The big thing all these cards have in common is – you guessed it – transactions at Target during that time window.   That’s why the press is reporting the date range of 11/27 through 12/15/2013, not because of anything Target found and reported about its systems.

Let this sink in for a minute.   That date range came from looking at samples of cards already stolen and not from any analysis of whatever was penetrated to get the card numbers.  As of Christmas eve, 2013, we still don’t know what specifically was penetrated, which means we don’t know what else is at risk or what steps the public can take to protect itself.  Here is an article with some expert speculation, but it’s only speculation from the outside.

Target claims the vulnerability is now closed and offers reassuring press releases to soothe the general public.  But with no guidance on what was penetrated and what specific steps Target took to close the vulnerability, the press releases so far offer nothing of value.  The public trusted Target before the theft and now 40 million credit card numbers are up for grabs.  Why should the public trust Target now?  What’s different?

If anyone from Target reads this blog post, Crisis Management 101 suggests transparency and disclosure.  The worst thing you can do is hide.  Instead of reacting to events and putting out vague press releases that offer no useful information, get in front of this story and tell the public specifically what happened and what you’re doing about it.  Introduce us to the people working around the clock to plug the gaps.  Show us how hard you’re working to fix the problem.  Convince us that shopping at your stores won’t expose us to identity theft.  Treat this like a crisis, because it really is a crisis.

Are we all in this together, as your press releases promise, or are those just empty PR words?  Smart people who know how transaction systems are supposed to work are watching.

(First published on my Infrasupport website on Dec. 24, 2013.  I back-dated it here to match the original date.)

A long couple of days in the life of an IT professional

This story is one example of many for what the best IT professionals do to keep our skills current.

The story started in April, 2013 when an opportunity to deliver a project based on a software product called RHN Satellite from a company named Red Hat came along. Large companies use RHN Satellite for activities such as automated builds, patch management, auditing, configuration management, and other administration for Red Hat Enterprise Linux servers. Think of an IT shop that needs to roll out dozens or even hundreds of servers and set them all up the same way. Those folks need RHN Satellite.

I knew nothing about RHN Satellite, but I’ve earned a reputation as a quick learner and was confident I could master it.  This would help a customer bring in an important project and make some money for me.  A win for everyone.  So I said yes.  This is what the best IT consultants do; we say yes and we learn quickly.  The job is not for the feint of heart.

There was a training class in Dallas with open seats coming in one week and I quickly signed up. Dallas is more than 900 miles away and I’m too cheap to buy expensive plane tickets, so I drove the 14 hours to the training site, attended the class and learned enough about advanced Red Hat Enterprise Linux system administration to deliver the project.  I also invested in the certification test.  If I’m going to learn the product, I may as well also get some certification credit for it.

Red Hat certifications are unique in the IT industry. While most IT product vendors offer certification tests based on cleverly worded multiple choice questions, Red Hat tests are all lab based.   This means anyone who wants a Red Hat certification must demonstrate knowledge of that system by setting one up in a lab according to test specifications.  The tests are challenging and very few candidates pass on their first attempts, even instructors who teach the courses.

Equipment problems in the Dallas classroom forced Red Hat to reschedule my certification test.  I scheduled mine for July in Chicago and failed miserably.  I improved in Chicago in October, but not enough to pass.

I knew what I did wrong and how to fix it, but the next scheduled test in Chicago was 6 long months away.   Two other test sites had openings on Nov. 22.  One was in Atlanta, the other in Ottawa, Ontario, Canada.  The Atlanta site is 16+ hours away by car, the Ottawa site around 18 hours away.

I decided on the Atlanta site and planned to buy my seat sometime after Nov. 9 to maximize my credit card float.  But the Atlanta seats filled on Nov. 8, leaving Ottawa as the only available Nov. 22 choice.

Ottawa presented a logistical challenge.  Trips from the US to Canada require a passport and mine was expired.  Minneapolis has a passport office and I could renew my expired passport by bringing it in with an updated picture and $220.   But unable to find my expired passport, I had to start from scratch with a birth certificate from Idaho.  To get a copy of my birth certificate, I needed an official copy of another piece of paper documenting my legal name change back in 1978.  That piece of paper was buried in a vault and the only way to get a copy was a trip to the basement of the Hennepin County Government Center in Minneapolis, where the lady behind the counter said she would order it for delivery the next week.  In a miraculous sequence of events, the process accelerated and by Monday, Nov. 18, I stood on the steps of the US Passport office in Minneapolis with a new passport in hand.

Only one logistical challenge remained – sign up for the test in Ottawa.  But now, after spending hundreds of dollars and watching a logistical miracle unfold around my passport, all the seats in Ottawa were full.   I called Red Hat and talked to Lauren with the training group.  Lauren orchestrated another logistical miracle to add an extra seat, and I reserved my seat in Ottawa a few hours later.

Now the real work – prepare for the test, travel to the site, pass the test, and go home.  Air travel cost between Minneapolis and Ottawa started at roughly $1050, confirming my decision to drive.  The test was Friday morning, so I was on the road from Minneapolis by 4:45 AM Central time Thursday.   I arrived at the Stardust Motel on Carling Road in Ottawa at 11:15 PM Eastern time, 17 1/2 hours later.

Canadian customs officers have ultimate power at the border and can deny entry to anyone they want for any reason they want.  My trouble at the Sault St. Marie border crossing started almost immediately as I rolled down my car window and handed a lady in a little booth my passport.

“And what brings you to Canada today?”

“I’m taking a test.”

“How long do you plan to be in the country?”

“Overnight”

“And where is this test?”

“It’s in Ottawa.”  (I think alarm bells started going off her head.)

“Let me get this straight.  You’re driving all the way to Ottawa to take a test, then you’re turning around and going home?”

“Yes, that’s right.”

“You are aware that Canada is a sovereign country, right?”

“Uhm, Ok.” (Not sure where she was going with this.)

“And you know Ottawa is 11 hours from here, right?”

“Well, Google maps tells me it’s about 9 hours, but OK.”

“So why are you taking this test in Ottawa?”

“Because the site in Atlanta was full.”

“What is this test anyway?”

“It’s for advanced Linux system administration.”

“Advanced what?  Why are you traveling to a different country just to take a test?”

“C’mon, What foreign country?  This is Canada. We’re friends. ” (Note to self – Canadian customs agents apparently don’t like appeals to friendship.)

“When was the last time you visited Canada?”

“Uhm, well, I guess it’s been a while.  Why?”

“OK, you need to park right over there and go inside for more questioning.”  (Uh-oh. This can’t be good.)

She directed me to a parking spot where several people in uniforms waited to escort me inside.

One of my escorts asked me what documentation I had to prove I really was going where I said I was going.  Thinking about it, I only had some emails.  I could open them on my tablet so I brought it in with me.  Maybe the emails would satisfy them.  I’ve been to former communist countries with less hassle.

I watched car after car after car easily cross into Canada while I slowly walked inside the building, flanked by uniformed guards as the clock and my upcoming night’s sleep ticked away.  Walking in the building, I saw 5 more people in uniform behind a counter on my right laughing about a video on a computer screen.  They sent me to the farthest counter, where a fat, gruff, balding grey haired man in uniform talked to a group of three people. As I approached the counter, he ordered me to step back and sit down at a table and wait.

“They told me to go to this counter.”

“And now I’m telling you to sit over there and wait.”

“If it’s all the same to you, I’d rather stand and stretch if I have to wait.”

“Suit yourself, I’ll be back after a while.  Don’t come up here until I tell you to.”

And then he left with the three people trailing.  I stretched my legs and back, stiff after 9 hours of driving so far that day, thinking about the 9 additional hours still to come and the clock ticking while I waited on the Canadian government.

The officers watching the comedy video barely looked up.  I asked one if he could take care of whatever it was I needed to take care of and he said no, that guy was the only one who could do it.  My only option – cool my heals and wait.

I needed to pee.

After a few minutes, Mr. Authority returned, motioned me up to his counter and asked me what was going on.  I gave him my passport and then made my next mistake.

“The lady outside hassled me and told me I need to come in here and take care of it.”

Major mistake. And then I made it worse.

“She didn’t hassle you.  You said you’re here for work so she sent you in here. That’s what she’s supposed to do! Do you think we just let anyone in our country who wants to come in?”

Dumbfounded, I said, “I see at least 10 cars out there and they’re all flowing right through.”

“THEY’RE ALL CANADIAN CITIZENS AND THIS IS A SOVEREIGN COUNTRY!  SO HOW ABOUT I DON’T LET YOU IN THEN? YOU JUST CAN TURN AROUND AND GO HOME, HOW DO YOU LIKE THAT?”

I wanted to walk outside and survey the license plates on those cars passing through the border, but a little voice in my head told me to shut up before I got myself into more trouble.  This guy controlled what I needed and I had nothing he wanted.  He had all the power and I had none.  He was the master and I was a dog.  Time to become meek and beg for mercy.

“I wouldn’t like that at all. I’ve been up since 3:30 this morning.  I just want to get where I’m going, take my test and go home.”

“WAIT RIGHT HERE.”

He took my passport and disappeared into a little back room. He came back out a few minutes later.

“Ever been convicted of a crime?”

 I was about to say “speeding tickets”, but my literal answers had already gotten me into trouble.  Speeding isn’t a crime anyway.  So I answered, “No.”

“Ever been denied entry into any country?”

“Not until right now, no.”

“Show me this proof you’re taking some sort of test in Ottawa.”

I showed him one of the Red Hat test confirmation emails and he said, “That doesn’t say it’s in Ottawa. What else ya got?”

“Uhm – well, here, take a look at this email.” I brought up an email thanking Lauren for opening the additional the seat for me.

“I never heard of any company named Red Hat.  What do they do?”

“They’re a software company.”

“Where are they?”

“They’re in North Carolina and they have offices all over the world.”

He looked me over one last time.

“OK, I’m going to let you in. BUT YOU NEED TO WORK ON YOUR ATTITUDE!”

“Thank you.”  (resisting the urge to further express my feelings.)

“Go give this paper to those two standing over there and be on your way.”

“Thank you.”

I still had to pee, but not bad enough to ask anyone here to use a bathroom.  As the uniformed guards escorted me to my car, I asked if any of them wanted to search it.  They said no and I drove away.  Welcome to Canada, eh.

I passed the test.  I’m now a proud holder of a Red Hat Advanced Linux System Administration Certificate of Expertise.

Total cost – $480 for the November test, $480 for the October test, $2800 for the original training and test, about $400 for the passport and required paperwork, 4 trips to the Minneapolis passport office and Hennepin County Government Center, around $1000 for travel costs for the training and test trips, 36 hours behind the wheel for the round trip to Ottawa, $40 per month for upgraded cell phone service in Canada and Mexico, sore legs and back, and a lesson in humility from some Canadian border agents.

Hopefully it was worth it.

(Originally posted on my Infrasupport website in Nov. 2013.  I backdated it here to match the original posting.  At the time of the original posting, my one person company was a Red Hat partner.  I am also now a Red Hat employee.  Everything in this post is my own opinion and may not reflect what anyone at Red Hat thinks.  Life goes on.  That test I worked so hard to pass back in 2013 is now obsolete.)

www.healthcare.gov – A Classic IT Disaster

My family is OK with health insurance coverage for now and I live in Minnesota, which has its own website for navigating the local healthcare marketplace, so I don’t need to deal with the US healthcare.gov website to find health insurance. But I was curious after looking at the news the past several days. I find it amazing that an Internet website generates front page newspaper and breathless TV coverage and I wanted to see what the fuss is all about.

My wife’s sister and her husband live in Indiana, so I wondered how health care coverage would look in their state and county. Navigating the website was straightforward and I found a few sample plans and pricing. The prices looked high, but each page boldly displayed caveats that the prices displayed were probably high and lower prices were most likely available by filling out a real application.

The website also mentioned a worksheet called the Kaiser Family Foundation health insurance cost and savings calculator and provided a helpful link. Answer a few quick questions using this worksheet and the website promised to generate more accurate price estimates.

So I followed the link and came to a page with some helpful explanations. So far so good.

When I clicked Next, presumably to start answering those few quick questions, a popup window with red text and yellow background popped up that said,

Please review your answers on this page,
there was a problem with one or more of
them.

This question is mandatory.

I would love to fix my answers, but this page had no questions.

In less than 5 minutes, I found an obvious programming bug by taking a path through this website that any typical user would follow. Here is a screenshot, [This link is no good – see my Infrasupport blog post about how I recovered from my own IT disaster, here.] reduced in size to 50 percent to capture most of the page. Notice the popup message at the bottom about answering questions, but the page only has explanatory text. There are no questions to answer, but the website will not allow me to navigate away from this page without answering its nonexistent questions.

Now I see what the fuss is all about. If I stumbled across such an obvious bug in less than 5 minutes without even trying, what other bugs are lurking beneath the surface? Current news reports suggest this website is full of bugs, performance issues, and inaccurate information. After my experience, I believe the reports.

The consequences are predictable. Politicians with no IT experience beyond using Facebook and smartphones are screaming, and VIPs across the government with no IT experience are trying to hold other VIPs with no IT experience responsible for this mess.

Where have I seen this before? Perhaps with every big IT project ever conceived since the dawn of big IT projects? Why would any IT veteran be surprised this one has serious problems? This project has all the classic elements: A lofty goal at the beginning, political maneuvering among potential contractors for development money, squabbling constituencies during development, major changes in design and function throughout the process, no time for serious testing to meet a looming deadline, nobody in overall charge of the effort, and a search for scapegoats at the end when the project fails to meet expectations. And in a classic and insulting knee-jerk response, our most senior and clueless Executive Branch officials now promise us they’re bringing in the A-team of “experts” to fix it all.

The stakes are indeed high. By United States federal law, and affirmed by the United States Supreme Court, we Americans are forced alpha testers for a new piece of technology over which we have no control, and if the technology fails, we pay a fine. I see attorneys building entire careers from the lawsuits this debacle will generate.

The quick fix President Obama promises will not happen for many reasons. First, as a few grizzled IT veterans constantly remind me by metaphor, the calendar time for a 9 month pregnancy cannot be reduced by finding 9 people to each work one month. In IT terms, a pregnancy project only achieves its goal if one person does all the required tasks during 9 consecutive calendar months because the output from earlier months provides the input to later months.

Software development is not war and the Obamacare surge metaphor comparing this project with the wars in Iraq and Afghanistan is a load of BS. Adding more programmers does not necessarily improve development time because at some point in any software development project, adding more labor becomes counterproductive. Even if they are “experts”. When the United States Federal government sends in their newly found “experts”, these new people will first climb a learning curve before becoming productive, and will then face the same project dependencies and constituencies as the earlier, presumably less capable team that came before them.

President Obama’s proposal to fix healthcare.gov is like proposing new tires for the car to fix a broken engine. It’s nonsense.

Beyond rookie programming errors such as the one I found, the problems are fundamental. The real fixes will depend on all players cooperating and agreeing to a reasonable set of specifications. No matter how many “experts” our Government brings in, these experts have no power to persuade or coerce all the unrelated parties to work together. Consider:

  • insurance companies with no incentive to cooperate with government contracted software developers,
  • a federal government filled with too many high officials with too many big egos and too many new and brilliant half-baked ideas,
  • millions of users trying to use the system on the front end while developers scramble to fix bugs on the back end,
  • and nobody in overall charge,

and the result is a recipe for continued failure.

Future historians may well declare the new healthcare.gov effort as the biggest, most spectacular failure in the history of IT projects.

(Originally published on my Infrasupport website.  I back-dated here to match the original publication date.)