Sometimes it’s not a cyber-attack

It’s good to know that at least my family listens to my constant Internet safety lectures.  I wish more business leaders would do more than talk about taking security seriously.  I am under constant cyber-attack.  Every single day, more than 100 phishing emails hit my inbox.  Some are clever.  One cussed me out for sending a bogus invoice, conveniently attached to the message.  Another cussed me out for not paying an invoice, also attached.  Many claim to come from UPS or USPS or Amazon with news that the package I was expecting had a delivery problem.  Open the attachment for details.  My “Bullseye Breach” book website regularly comes under attack, most recently from a Russian IP Address.  Since “Bullseye Breach” is a book about how Russians steal forty million customer credit cards from a large retailer named Bullseye Stores, I guess the only surprise is that it took the Russians so long to attack it.

So when my daughter came to me with strange cell phone behavior, I knew it had to be another attack.

She was trading text messages with another mom to set up a play-date for my grandson.  The other mom offered to have my grandson over to her house to play with her son, and my daughter offered to stay and help.  Boys can be rambunctious when they get together.  This was one of my daughter’s messages, quoting with permission:

“Sounds good.  I am cool with staying and hanging out if you want.  I just don’t want you to feel like overwhelmed or anything.”

The other mom responded and they continued their text conversation.  I still don’t see the appeal of text messages as a primary form of communication.  Those teeny tiny keys and auto-correct drive me nuts.  If we’re both tapping little buttons on a phone, why not just talk to each other?  Maybe it’s a generational thing.

In the middle of her conversation with the other mom, two identical text messages from two different unknown local phone numbers came in.  The messages were, “who dis?” followed by forwards of my daughter’s messages to the other mom.

Shocked and afraid, my daughter asked me to help figure out how somebody invaded her phone.  Why was somebody stalking her from two different phone numbers and taunting her with her own text messages?  How did some lowlife intercept her text messages and play them back for her?  What did they want?

I was curious myself.

Looking over the conversations, the texters knew my daughter’s name and the date and time she planned to meet the other mom.  But we knew nothing about the texters.  Time to put on my tough guy dad hat.  I texted one of the numbers with “who are you and what do you want?” and was about to try to identify the other number and call the Police, when her phone rang from the first number. My daughter looked at me and handed me the phone.

“Hello,” I said in the strongest dad voice I could muster.  (It’s not the weapons you bring to the fight that count, it’s what the other guy thinks you bring to the fight that counts.)

To my surprise, the caller was a woman and she was just as mystified as my daughter.  She said she received a text message about staying and hanging out from this number, but had no idea what that meant or what was going on.  She knew my daughter’s name because my daughter used it in another message in the conversation thread.  The mystery was, how did this unrelated third party end up with a copy of part of my daughter’s half of a conversation with the other mom?

Curious, we called the other number.  That was also a woman, but she thought my daughter was a guy sending inappropriate advances.  What does “hang out” really mean anyway?  We had a long talk and cleared it up.

Apparently, my daughter’s cell carrier, T-Mobile, had a text message routing problem and sent copies of text messages to unintended phone numbers that night.  Imagine receiving a text message about hanging out and don’t feel overwhelmed with no other context from a strange phone number.  But this time, there was no cyber-attack, no stalkers, no perverts.  Just a T-Mobile tech glitch with suspicion layered on top.

Lessons?  Yup, a few.

  • It’s not always a cyber-attack or an evil cyber-stalker.  Sometimes it’s a tech glitch.
  • If you want to share intimate messages with somebody, best to do it by voice or face to face.  Text messages can be mis-routed.  I saw it first-hand.
  • And “who dis” must be a common text greeting.  I need to learn a new language.

(Originally published on my Infrasupport website, March 17, 2016.  I backdated here to match the original posting date.)

We all need Apple to win the FBI encryption dispute

In December, 2015, two terrorists in San Bernardino, California, committed a horrific and gruesome crime when they murdered 14 coworkers and seriously injured 22 others.  Law enforcement caught up with these murderers a few hours later and they died in a shootout.  Good riddance.  If you commit an act of terrorism, you deserve the harshest consequences society can offer.

But this blog post isn’t about terrorism.  It’s about the aftermath these terrorists left behind in an encrypted Apple iPhone 5c.  Three months later, the FBI is unable to break into that phone to examine its contents.

The phone belongs to the San Bernardino County Department of Public Health, where one of the murderers was an employee, and is now in FBI custody.  The phone’s contents are encrypted and the phone may be set to brick itself after a small number of penetration attempts. Apple itself has no way to access it.  Here is a blog post with details.

The FBI wants Apple to engineer a special firmware update for this specific phone to allow the FBI to bypass the phone’s security and look at its contents.   The FBI secured a court order compelling Apple to cooperate.  Here is a PDF with the FBI motion.  Here is a PDF with the court order.  Apple CEO, Tim Cook, expressed Apple’s opposition to the order in an open letter, published on the Apple website.  Here is a PDF copy in case the website link goes bad.

And now the fight is on.  It’s the long awaited clash of privacy rights versus counter terrorism.  And although I question the value of anything stored inside the specific phone at the center of this fight, the big picture stakes could literally be life and death.

Naturally, politicians are weighing in.  In this article Donald Trump called Apple “disgraceful.”  Trump also said, “We should force them to do it. We should do whatever we have to do.”

And in a USA Today Opinion piece, Senator Richard Burr, R-NC, Chairman of the Senate Select Committee on Intelligence, said, “The newest Apple operating systems allow device access only to users — even Apple itself can’t get in. Murderers, pedophiles, drug dealers and the others are already using this technology to cover their tracks.”  Here is a PDF of Senator Burr’s article in case the link goes stale.

On a personal level, if my family or friends were victimized by a terrorist attack, I would do everything in my power to gather information to bring the attackers to justice, and if encryption got in my way, I would bust whatever heads I needed to break it, legal or not.  I would not care about bigger policy issues.

But when I look dispassionately at the bigger picture, I am forced to conclude Apple is right and the FBI is wrong.  And the longer I look at the issue, the stronger my convictions become.

Tim Cook framed the Apple arguments around privacy and trust and a slippery slope to tyranny.  And his arguments are persuasive and right on.  But the arguments so far on both sides miss a larger point – the perceived trade-off pitting privacy against law enforcement is not the most important issue.  More important than any trade-off, weakening encryption, even to break into this one phone, hurts the fight against terrorism more than it helps and the government has no business trying to compel companies to break the security of their products.

Apple could break into this one phone and maybe the government might uncover a few names.  Maybe.  But at what long term cost?  Senator Burr, if Apple loses this fight, then murderers, pedophiles, drug dealers, and others will simply find another encryption tactic to cover their tracks.  If the government wins this skirmish with Apple, we will all pay a long term price in the more important war against crime and terrorism.

In a Feb. 19, 2016 interview on CBS This Morning, Assistant New York City Police Commissioner, John Miller, took Apple to task by asking Apple how many victims in Paris and San Bernardino were Apple customers.  Miller is right to frame the debate in life and death terms.  So my question for Miller, if I ever get a chance to ask is, how many more people will die if law enforcement forces tech companies to weaken encryption?  How does it make sense to cripple the good guys when the bad guys won’t follow the rules?

Like it or not,  strong encryption is here to stay.  It’s a fundamental part of 21st century society.  We can no more roll back encryption than we can replace cars with horses and buggies.

Don’t believe me.  Just use recent history as a guide.

For many years, the open source IPSEC community refused to accept contributions from US citizens because of the threat of US Government regulation.  So encryption technology continued to progress, just without input by the United States.   If we return to that broken way of thinking, we will blind the United States when dealing with our enemies.  Not only will we not be able to decipher encrypted communications, we may not even know they’re going on.

I spent 16 months writing and publishing “Bullseye Breach,” an educational book disguised as an international fiction story about how Russian criminals steal 40 million credit card numbers from a large US retailer named Bullseye Stores.   No amount of government regulation inhibiting or regulating encryption would have helped in any real-world breach scenarios, and the arguments suggesting the government act as a safe storage location for encryption keys has more holes than Swiss Cheese.  Just ask any victim of the recent OPM breach about the safety of US Government servers.  If people who apply for security clearances can’t trust the United States Government with private information, why should the general public trust the Government with millions, perhaps billions of encryption keys?

That’s why Apple must win this fight. To stop a first step down a slippery slope and keep the playing field level between the good guys and bad guys, so the good guys have a chance to fight back.  Crippling encryption cripples the good guys.  It delivers exactly the opposite result the government and all of us want.

One final note:  I am now a Red Hat employee.  For people unfamiliar with the tech industry, Red Hat is the preeminent open source software company and is rocking the IT industry.  The opinion expressed here is mine, and may not reflect what the leaders at Red Hat think.  But I’m right.

(Originally published on my Infrasupport website Feb. 19, 2016.  I backdated here to match the original posting date.)

How a printer inkjet cartridge clobbered one of the world’s largest retailers

This is a story about how a depleted printer inkjet cartridge stopped Sams Club from selling cell phones on the day of a big Apple announcement.  Not a shining moment for enterprise IT.

My wife, Tina’s smartphone gave us three good years of life.  But with crashes and hangs and a battery that would not stay charged, it was time for a new phone.  I know I can buy phones and supplies directly from my carrier, but  every time I compare equipment prices, Sams Club is less expensive.  And that’s what led us to our local Sams Club store a few days ago.

Charles in the cell phone department helped us select a phone and set it up.  All went smoothly.  Just one more thing to do – print and sign the paperwork.  Charles clicked the “Print” button on his POS system and that started a chain of events that destroyed this store’s ability to sell phones and tablets.   But it would take us more than two hours to come to that conclusion.

When the paperwork would not print, Charles checked the Lexmark CS510de printer at this station and found it needed a new black inkjet cartridge.  Charles found one, but it didn’t fit.  He looked all over the store, but there were none.  Why not just send the paperwork print job to another printer in the store?  Because this POS system was locked down and nobody had permissions to set up different printers on it.  The POS system and this network connected printer were mated.

Time for plan B – just swap printers with another station.  Bring in a different printer and assign it the same IP Address as the original printer.  Two roadblocks with this.  First, this was the only Lexmark CS510de printer in the store.  But there were other Lexmark printers.  Maybe one of them was close enough that the driver would be compatible.  So there was a way around the first roadblock.

Navigating through the menus on the printer touch panel, we ran into the next roadblock.  We needed to look up the IP Address on the original printer so we could configure the replacement.  Unfortunately, we quickly found that this printer prohibits looking up any network parameters when it’s not ready.  And it can only be made ready when it has good inkjet cartridges.  No inkjet cartridge, no IP Address.

Insert my editorial question here.  If anyone from Lexmark reads this post, what were you guys thinking and who made the boneheaded decision to prohibit accessing any network parameters when the printer is offline?

With no way to look up the IP Address for the original printer from either the printer touch panel or the locked down POS system, it was time for plan C – find an inkjet cartridge for this printer.  Somewhere.  Surely it was in stock at a store in the area. Right?  Charles jumped in his car and tried the nearby stores and came up empty.  With no replacement inkjet cartridges and no ability to look up the IP Address of the original printer, there was no way to fix the original printer or set up any replacement to mimic it.

Why not just call the corporate IT Department?  Surely they had tables of which IP Addresses belonged to what devices.  But by now it was evening Central time on Friday and nobody was available in Corporate IT to take a call.

Somewhere during all this, the department manager was also involved.  And two other department members.  And me, tinkering with printer settings.  The problem consumed three employees and a customer for two hours.

A brand new Apple phone was available for sale starting Saturday, but this department could not sell anything with a broken printer.  So now the problem was bigger than just my paperwork.  We tried to fix it.  But sometimes the obstacles are insurmountable.  The manager gave me a $50 gift card for my trouble and apologized.

What could Sams Club have done differently to stay out of this predicament?  Having a few spare inkjet cartridges is the obvious answer.  But more important, who designed a business system that depended on finicky inkjet cartridges that can fail at any time?

The system should have had redundancy built in with every component.  A spare printer, a spare POS system with the ability to print to a default printer and alternates, and a standard printer model at all stations throughout the store.  Why use an inkjet printer in a high volume printing station?  Why not a real laser printer?  And why use a printer that prohibits looking up its IP Address?  Where was the corporate IT Department?  Sams Club stores open at 7 AM on the US East coast and close at 9 PM on the US West Coast.   Why was nobody in IT available during prime time Central timezone shopping hours who could address the situation?

I wonder how much in sales this store lost, especially on the day of a major Apple announcement, because of a depleted printer inkjet cartridge?

(Originally published on my Infrasupport websiter Oct. 6, 2015.  I backdated the post here to reflect the original posting date.)

The fallout continues on the OPM data breach

It seems the Chinese plundered the United States Office of Personnel Management (OPM) at will for at least a year.  Here is my original blog post about the OPM data breach nightmare.

If you’re a CEO of a major organization and you still think Internet security is abstract and doesn’t apply to you, I hope you have a nice retirement package set aside.  Don’t believe me?  Just watch the ongoing revelations about the OPM data breach.  The news just keeps getting worse.  The latest tally is 22 million people.  It’s the biggest and maybe the worst data breach in US Government history and it cost Director Katherine Archuleta her job.  I imagine a few more heads will roll over the next few days.  Here is a link to a NY Times article with details.

Want to see one of the best examples of government CYA in action?  Take a look at this press release from a company named Cytech.  PDF here in case the original link goes bad.  Apparently, a Cytech April 2015 demo uncovered a set of unknown processes on some Windows systems.  I’m guessing they were Windows systems – none of the reports overtly mention it.  Cytech worked with OPM to chase down those processes and the rest is history.

But wait – a sales demo uncovering the worst data breach in US Government history makes high government officials look bad.  Spokespeople to the rescue.  Here is a Fortune article with the response from OPM spokesman Sam Schumach.  PDF here in case the link goes bad.  I’ll quote Sam’s first sentence:

“The cyber intrusion announced last week affecting personnel records for approximately 4 million current and former federal employees was discovered through enhanced monitoring and detection systems that OPM implemented as part of an aggressive effort in recent months to strengthen our cybersecurity capabilities. …”

You can read the rest in the Fortune article.

Pause for a minute.  Beyond CYA posturing, what are the real-world consequences of this debacle?  Well, for one thing, personal information for everyone who applied for a US Government security clearance since 2000 is now compromised.  If you applied for a US Government clearance and you contacted somebody in a hostile country who helped the United States, it’s likely the Chinese learned about it back in 2014.  Do I need to connect any more dots?  Still think IT security is abstract and doesn’t apply to you?  Real, flesh and blood people who wanted to help the good guys may have died because the United States Federal Government only paid lip service to taking your security seriously.

Now back to CYA posturing.

I’m not sure I would want to be in Cytech CEO Ben Cotton’s shoes right now.   Imagine this scenario.  A large government agency invites your company to do a sales demo for your flagship product.  You spend days, weeks, maybe months and a fortune in investor private sector money preparing.  You put it all at risk because that’s what we do in the private sector.

And it seems to pay off when you unexpectedly uncover a huge mess.  And then you help remediate the problem because it’s the right thing to do.   Word leaks out, speculation runs rampant, and you feel forced to do a press release in response because everyone is naming your company anyway.  But now the people running the agency that invited you in look bad and they put out their own statements contradicting you.  What are the odds you’ll earn a sale from your hard work?  No good deed goes unpunished.

And there’s more.

After the news about the breach came out, OPM offered free credit monitoring for victims.  The questionable value of this free credit monitoring is well documented, and once the monitoring period ends, then what?  But forget about that – how did OPM notify victims?  By sending an email with a “click here” link.  To millions of Federal employees.

Why is that significant?  Because that’s how phishing schemes operate.  “Dear customer.  We at your bank found an irregularity.  Please click here to make it all better.”  Bla bla bla.  Except the email didn’t come from your bank, it came from a con artist on the other side of the planet who wants to plunder any information in your computer.  It’s one of the oldest and most well known con-jobs on the Internet.  And people still fall for it.  See my blog post, “How to spot a phishy email,” for more.

So guess what?  Almost immediately after OPM sent its “click here” email, scammers and spammers duplicated it and sent identical emails with their own “click here” links pointing to their own shady websites.  Take a look at these articles, here and here.  Talk about rubbing salt in the wound.

Now take a look at this link.  It’s the National Institute of Standards’ cybersecurity framework.   That’s right.  The United States Federal Government literally wrote the book on cybersecurity.  And keeps it updated.  It’s a shame the leaders at the United States Federal Government HR office apparently didn’t read it.

Finally, if you’re mystified and curious how these breaches happen at the grass roots, and if you’re not, you should be, take a look at my new book, Bullseye Breach.  Here is a link.  It’s a story about how a fictional large Minneapolis retailer named Bullseye Stores loses 40 million credit card numbers to some Russian crooks.  I used fiction as a vehicle because the world already has enough how-to books that nobody reads.  So I used fiction and a compelling story to hopefully keep your attention.  Every CEO should read this book – it might save you from putting out a press release explaining how you take security seriously after a major breach.

(First published on my Infrasupport website on July 11, 2015.  I backdated here to match the original posting date.)

Here we go again – data breach at Trump Hotel properties

Here is yet another data breach headline, this time from the Donald himself, Trump Hotel properties, published yesterday (July 1, 2015) by Brian Krebs.  Here is the link to the article.

It seems the banks uncovered a trail of credit card fraud leading right back to Trump Hotel properties.  This one has apparently been going on since Feb. 2015.

We’re early in the cycle of this latest sensational data breach, but they all follow the same pattern.  Watch for it with this one.  Here’s how they unfold.

  1. Lax or dysfunctional management ignores all the warnings about potential IT security problems.  Those techies – all they want to do is spend money on tech toys.  We sell hammers or hotel rooms or clothes.  Or we’re a Government HR department.  Or we make movies.  We’re not a tech company.
  2. A sensational news story hits the wires.  Millions of credit card numbers stolen!  Personal information stolen by the Chinese!  Fortune 500 company brought to its knees!
  3. The CEO or other leader of the breached organization puts out a press release.  “We take our customers’ privacy seriously.”  The press release includes a generous offer of worthless free credit monitoring for potential victims for a year.
  4. PR teams gear up as leaders in the breached organization fill the airwaves with excuses and all the important steps they’re taking to mitigate this breach.  They use words like “sophisticated” and “criminal syndicate” or “nation state” to describe the attackers.
  5. Columnists and bloggers express outrage.  (That’s what I’m doing right now.)
  6. Lots of people share commentary about how awful this all is and the poor state of our security.  But nobody shares any specifics about conditions leading up to the breach, how the bad guys penetrated the victim organization, or the get-well steps.   (I saw one exception to this in a KrebsOnSecurity.com post about the Sally Beauty breach.)
  7. Embarrassed Boards of Directors and other VIPs outdo themselves with knee-jerk reactions as they pour a fortune into closing the barn door after the horses have already escaped.
  8. Sometimes, a major news magazine does an in-depth story about the personalities involved at the victim company a few months later.
  9. The story eventually fades away and the public is left to believe that breached companies are helpless victims of sophisticated criminal syndicates or nation-state sponsored terrorists.  There’s nothing anyone could have done about it.

Don’t believe this crap for even one second.  Every single sensational data breach we’ve read about was preventable.  Every single one.

Want to fix the problem instead of putting out CYA press releases?  Here’s what needs to happen – and it doesn’t cost a fortune.

First, a tactical step:  Improve the topology.  Put the most valuable systems behind an internal firewall with a white list and log access to it.  Notify the right people if the systems holding that critical data try to communicate outside the white list.

Second is vigilance.  When we peel back the onion layers on these breaches, we find too many people asleep at the switch.  Or nobody minding the store.  Pick your metaphor.  The Chinese run rampant through the US Office of Personnel Management network and nobody notices traffic flying to China?  What’s up with that?  The North Koreans run rampant through Sony Pictures and nobody notices?  Let’s call this what it is – carelessness from the people who should know better.

And that leads to the third step:  Openness.  This is counter-intuitive, but organizations should publish what they do for security.  This doesn’t mean give away passwords and encryption keys.  But publish their standards and methods.  In detail.  Present at conferences, do media interviews, and open up to community scrutiny.  This is a departure from traditional large organization operating procedure and I can already hear the screams of agony:  “If we tell the world how we do security, then everyone will know and it will be worthless!”

I answer that with a question: “Given recent sensational data breach headlines, how’s the current operating procedure working out?”  Right now, only the bad guys know the relevant details and they’re plundering us.  So level the playing field.  Open it up.  The surviving encryption methods are all open and well-known.  And hardened because they’ve passed a gauntlet of public scrutiny.  Business and government should take a lesson.

Do those three things and IT security will naturally gain the attention it needs at the top levels of business and government and appropriate investments will follow.

Finally – want to read a novel with a realistic story about how a sensational data breach unfolds?  Check out my new book, Bullseye Breach.

(Originally posted on my Infrasupport website on July 2, 2015.  I backdated here to match the original posting.)

The Chinese may now have personal information on 4 Million US Government employees

Yet another sensational data breach headline – not even shocking anymore.  Yawn.  But listening to the story on the radio on the way home last night after being slaughtered in softball again, I started thinking.  And I dug a little deeper into the story when I got home.  I was shocked.

The systems penetrated belong to the US Government Office of Personnel Management.  Yep, that’s the United States Federal Government Human Resources Department.  It holds personal information for everyone who works for the US Federal government.  It’s the agency that hands out security clearances.  Think about this.  Let it sink in.

The Chinese broke into the system that US Government investigators use to store information about background checks for people who want security clearances.  That’s right.  If you applied to the US Government for a security clearance, it’s a good bet the Chinese know a lot about you now.  Which means you’ll probably be the target of some finely crafted spear phishing campaigns for the next several years.

And that’s only one system out of 47 operated by the Office of Personnel Management (OPM).  It’s not the only one the Chinese penetrated.

Update:  According to this Washington Post article, (PDF here in case the link breaks) the Chinese breached the system managing sensitive information about Federal employees applying for security clearances in March 2014.  The latest OPM breach targeted a different data center housed at the Interior Department.

Update June 12, 2015:  The original reports were bad.  Now it’s even worse.  It now seems the Chinese have detailed information on every US Federal employee.  14 million, not 4 million.  And people may die directly because of this breach. But even now, we don’t know the extent of the damage.  This article from Wired Magazine sums it up nicely.

Reactions from high government officials were typical. They all take the problem seriously.  Bla bla bla.  According to the Wall Street Journal:

“We take very seriously our responsibility to secure the information stored in our systems, and in coordination with our agency partners, our experienced team is constantly identifying opportunities to further protect the data with which we are entrusted,” said Katherine Archuleta, director of the Office of Personnel Management.

Here’s another one, from the New York Times:

“The threat that we face is ever-evolving,” said Josh Earnest, the White House press secretary. “We understand that there is this persistent risk out there. We take this very seriously.”

This one from the same Washington Post article is my favorite:

“Protecting our federal employee data from malicious cyber incidents is of the highest priority at OPM,” Director Katherine Archuleta said in a statement.

Do I really need to ask the question?  Katherine, if it’s such a high priority then why didn’t you address the problem?

As I mentioned in a blog post way back in Feb. 2014, about dealing with disclosures, we’ve heard lots of noise about this breach but very little useful information.  Here’s what we do know.  I want to thank David E. Sanger, lead author of the New York Times article, “U.S. Was Warned of System Open to Cyberattacks,” for sending me a link the 2014 Federal Information Security Management Act Audit report.  In case that link breaks, here is a PDF.

We know the Chinese penetrated the OPM in fall 2014 and stole at least 4 million records over the next six months.  That’s it. As usual, nobody I can find is forthcoming with details.

The report from the Office of Inspector General (OIG) gives us some clues.  Apparently, the various program offices that owned major computer systems each had their own designated security officers (DSO) until FY 2011.  The DSOs were not security professionals and they had other jobs, which means security was a bolted on afterthought.  In FY2012, OPM started centralizing the security function.  But by 2014, only 17 of the agency’s 47 major systems operated under this tighter structure.

All 47 major systems are supposed to undergo a comprehensive assessment every three years that attests that a system’s security controls meet the security requirements of that system.  It’s a rigorous certification process called Authorization.  Here’s what the report said:

“However, of the 21 OPM systems due for Authorization in FY 2014, 11 were not completed on time and are currently operating without a valid Authorization (re-Authorization is required every three years for major information systems). The drastic increase in the number of systems operating without a valid Authorization is alarming, and represents a systemic issue of inadequate planning by OPM programming offices to authorize the information systems that they own.”

Remote access also had problems.  Apparently the VPN vendor OPM uses claims the ability to terminate VPN sessions after an idle timeout.  But the idle timeout doesn’t work and the vendor won’t supply a patch to fix it.

Identity management was also weak.  Although OPM requires multi-factor authentication to enter the network, none of the application systems do.  So if a Chinese bad guy penetrates the network, he apparently has free reign to everything in it once inside.  And since OPM had no inventory of what systems it owned or where they were or their use, OPM had no way to know the Chinese were plundering their data.

It adds up to a gigantic mess.  And an embarrassment, which probably explains why nobody wants to talk about details.

Wonderful.  So what can a small IT contractor from Minnesota offer the multi trillion dollar United States Federal Government to address this problem?  Here are some suggestions from an outsider who wrote a book about data breaches.

Three attributes will keep our systems safe.  Sharing, diligence, and topology.

Sharing drives it all.  So first and foremost – move from a culture of hierarchy, secrecy, and “need to know” to a culture of openness, especially around security.  What does that even mean?  For an answer, check out the new book by Red Hat CEO Jim Whitehurst, “The Open Organization,” published by the Harvard Business Review.

The Chinese, and probably others, penetrate our systems because a government culture of secrecy and “need to know” keeps our teams isolated and inhibits collaboration and incentives for excellence.  It’s a traditional approach to a new problem the defies tradition.  I’ll bet the Chinese collaborate with each other, and probably also with the North Koreans.

Instead of a closed approach, adopt an open approach.  Publish source code, build communities around each of those 47 systems, and share them with the world.  To protect it better, share how it all works with the world.

And when breaches happen, don’t tell us how you take security seriously.  You’re supposed to take security seriously.  It’s your job.  Tell us what happened and what steps you’re taking to fix the problem.  Instead of hiding behind press releases, engage with your community.

And use open source tools for all your security.  All of it.  Firewalls, VPN systems, IDS/IPS (Intrusion detection/Intrusion prevention systems), traffic analyzers, everything.  Breaches occur with open source software, just like proprietary software, but when they happen, the open source community fixes them quickly. Why? Because the developers’ names are on the headers and they care about their reputations.  You won’t need to wait years for a VPN patch in the open source world.

Openness doesn’t mean granting access to everyone.  Openness means building communities around the software systems OPM uses and accepting patches and development from the community.  Community members are compensated with recognition and opportunities for paid engagements.  OPM is rewarded with hardened, peer reviewed software driven by some of the smartest people on the planet.

When teams move away from hierarchy to an open culture, diligence and topology will follow.  There is no substitute for diligence and no technology to provide it. Teach everyone to be diligent and practice it often with drills.  Reward the cleverest phishing scheme or simulated attack and reward the cleverest defense.

And topology – put layers of security in front of key databases.  Put in appropriate access and authorization controls for key databases to ensure personal information stays personal.  Consider physically segregating these database systems from the general network and setting up a whitelist for their interactions with the world.

None of this proposed culture shift needs to cost a fortune.  And in fact, in this era of doing more with less, might save taxpayer money by igniting passion at the grass roots of the OPM IT staff.

Am I proposing radical change to a government that resists change?   Yup.  So why do it?  I’ll answer that question with my own question – given the recent headlines and your own Inspector General audit reports from the past several years, how’s the current method working out?

(I first published this on my Infrasupport website on June 6, 2015.  I backdated here to match the original posting date.)

What is redundancy anyway?

I’ve been in the IT industry my entire adult life, so sometimes I use words and just assume everyone thinks they mean the same thing I think they mean.  I was recently challenged with the word, “redundancy.”

“What does that even mean?” asked my friend.

“It means you have more than one.”

“So what?”

“So if one breaks, you can use the other one.”

“Yeah, everyone knows that, but what does it mean with IT stuff?”

Seems simple enough to me, but as I think about it, maybe it’s not so simple.  And analyzing how things can fail and how to mitigate it is downright complex.

Redundancy is almost everywhere in the IT world.  Almost, because it’s not generally found in user computers or cell phones, which explains why most people don’t think about it and why these systems break so often.  In the back room, nearly all modern servers have at least some redundant components, especially around storage.  IT people are all too familiar with the acronym, RAID, which stands for Redundant Array of Independent Disks.  Depending on the configuration, RAID sets can tolerate one and sometimes two disk failures and still continue operating.  But not always.  I lived through one such failure and documented it in a blog post here.

Some people use RAID as a substitute for good backups.  The reasoning goes like this:  “Since we have redundant hard drives, we’re still covered if a hard drive dies, so we should be OK.”  It’s a shame people don’t think this through.  Forget about the risk of a second disk failure for a minute.  What happens if somebody accidentally deletes or messes up a critical data file?  What happens if a Cryptolocker type virus sweeps through and scrambles everyone’s files?  What happens if the disk controller in front of that RAID set fails?

Redundancy is only one component in keeping the overall system available.  It’s not a universal cure-all. There will never be a substitute for good backups.

Virtual environments have redundancy all over the place.  A virtual machine is software pretending to be hardware, so it’s not married to any particular piece of hardware.  So if the physical host dies, the virtual machine can run on another host.  I have a whole discussion about highly available clusters and virtual environments here.

With the advent of the cloud, doesn’t the whole discussion about server redundancy become obsolete?  Well, yeah, sort of.  But not really.  It just moves somewhere else.  Presumably all good cloud service providers have a well thought out redundancy plan, even including redundant data centers and replicated virtual machines, so no failure or natural disaster can cripple their customers.

With the advent of the cloud, another area where redundancy will become vital is the boundary between the customer premise and the Internet.  I have a short video illustrating the concept here.

I used to build systems I like to call SDP appliances.  SDP – Software Defined Perimeter, meaning with the advent of cloud services, company network perimeters won’t really be perimeters any more.  Instead, they’ll be sets of software directing traffic to/from various cloud services to/from the internal network.

Redundancy takes two forms here.  First is the ability to juggle multiple Internet feeds, so when the primary feed goes offline, the company can route via the backup feed. Think of two on-ramps to the Interstate highway system, so when one ramp has problems, cars can still get on with the other ramp.

The other area is redundant SDP appliances. The freeway metaphor doesn’t work here. Instead, think of a gateway, or a door though which all traffic passes to/from the Internet.  All gateways, including Infrasupport SDP appliances, use hardware, and all hardware will eventually fail.  So the Infrasupport SDP appliances can be configured in pairs, such that a backup system watches the primary. If the primary fails, the backup assumes the primary role. Once back online, the old primary assumes a backup role.

Deciding when to assume the primary role is also complicated.  Too timid and the customer has no connection to the cloud.  Too aggressive and a disastrous condition where both appliances “think” they’re primary can come up.  After months of tinkering, here is how my SDP appliances do it.  The logic is, well, you’ll see…

If the backup appliance cannot see the primary appliance in the private heartbeat network, and cannot see the primary in the  internal network, and cannot see the primary in the external Internet network, but can see the Internet, then and only then assume the primary role.

It took months to test and battle-harden that logic and by now I have several in production.  It works and it’s really cool to watch.  That’s redundancy done right.  If you want to find out more, just contact me right here.

(Originally posted on my Infrasupport website, June 4, 2015.  I backdated here to match the original posting.)

The real-life story of an identity theft victim and what she did about it

I have a friend, let’s call her Mandy.  Mandy is an identity theft victim.  Mandy is not her real name because this is a private story and she wants to maintain her privacy.  She’s willing to share it, anonymously, because she read “Bullseye Breach” and she knows what I do for a living.  She’s hopeful that her story might help others in a similar situation.

For anyone who still thinks the law enforcement bureaucracy will help you when you’ve been violated in this manner, Mandy’s story will change your mind.  And hopefully this deeply personal story will help persuade you that IT security is important and you need to take it seriously.

I am privileged to post Mandy’s story, in her own words.

#####

Living in a nice neighborhood can give you a false sense of security. Maybe you know most of your neighbors and don’t think twice about leaving your windows open all day to let in cool air.  Maybe you don’t even lock your doors at night.

I’ve never been that trusting. I grew up in a South Florida neighborhood where it seemed like we were receiving flyers on a weekly basis about break-ins.

They left an impression on me. Once out on my own, I always made sure my doors and windows were locked, but turns out that didn’t matter.

On the morning of Nov. 7, 2005, someone pried open a locked window and got into my home anyway. My husband and I returned from work around the same time that evening to find our home ransacked.

The thief or thieves must have spent a long time inside because everything, and I do mean everything, that was both portable and valuable was gone. Every room in the house had been gone through.

Missing were thousands of dollars worth of electronics, including a laptop computer that contained personal information and a video camera with precious video of my son inside; all of our checkbooks and bills that had been written out but not yet sent; a set of extra keys to our house and one of the cars; and the coin collection I had been building since I was a kid.

May sound hard to believe, but it wouldn’t have been so bad if that was all that had disappeared. What’s ten times more devastating is the fact that my family also fell victim that day to what has become the number one crime in America — identity theft.

Like so many people I know, we had our social security cards and birth certificates in a fire box under the bed. The thief found the key to the box in my underwear drawer and cleaned it out.

I feel stupid for having left the key in such an obvious place, but my husband has convinced me that if they hadn’t found the key, the thieves would have just taken the whole box anyway. I should have hidden it better.

We spent all of November and December worrying about how our information was going to be used, but nothing bad happened. Then the other shoe dropped the night of January 11th.

Because of the fraud alert we put up on our credit reports after the break-in, someone from Dell Computer called our house around 10 o’clock that night. He said he had J. on the other line and was calling to confirm his identity.

My husband was not the man on the line with Dell. We were being violated again.

After hanging up with Dell, we ran our credit report and found out that a few days earlier, someone had tried to secure a home mortgage in our name.

When I got to work the next morning, I looked up our client contact at one of the credit bureaus, called her up and started asking a lot of questions. She couldn’t answer all of them, so she put me in touch with Kevin Barrows, the former FBI agent who is credited with busting up one of the country’s largest identity theft operations in 2002.

He told me, “Because you put the fraud alert up and filed a police report, you will not be liable for anything the identity thief does; but at the same time, you do need to get his inquiries and the false addresses he gave off your credit report as quickly as possible.”

That night, I embarked on another round of letter writing. The next morning it was off to the post office again.

Early on in the process, I had read an article that recommended all communications with the credit bureaus be sent certified with return-receipt. I’ve spent close to $100 sending letters that way so far.

That’s in addition to the thousands of dollars spent installing an alarm system, fixing our broken window, replacing a damaged sliding glass door; rekeying our house and car; replacing stolen documents; etc. Some, but not all of our losses, were covered by insurance.

Just when we thought we had the situation under control, my husband and I started getting calls from credit card companies calling to confirm our identity because of the fraud alert on our accounts. One after another… I lost count around 30… We would tell the people on the other line that no we did not authorize the opening of an account.

Right away after the calls started coming in, I pulled our credit reports again and found mention of multiple inquiries made by creditors we had never heard of, plus a mysterious address in Illinois added to both mine and my husband’s accounts. I called the police department in that city to report that someone at that address was fraudulently using my address to try and establish credit.

Believe it or not, the detective I spoke with actually told me they had received similar reports from others about that exact address, but there was nothing they could do because it was a federal crime. I was referred to the Post Master General, I presume because the thieves wanted to get credit cards fraudulently sent to them through the mail.

The person I spoke with took down my information and referred me to the FBI. The agent I spoke with at the FBI told me there are too many cases like mine for them to pursue all of them. They referred me back to the local police dept in the jurisdiction where the theft happened. My hometown police department basically said, “Sorry, there is nothing we can do about a crime being committed across state lines.”

I am sharing my story in hopes that I can help make the recovery process easier for someone else.

Here are the steps I’ve taken since the day of the break-in:

  1. Called the police to file a report. (This is a critical step. You will need that report in order to get extended fraud alerts issued).
  2. Called the credit bureaus. (Work your way through the automated menus until you find the option to get a fraud alert issued. Experian, Equifax and TransUnion are required to share information with each other, but to give yourself peace of mind, contact all three anyway. I did.)Equifax: 1-800-525-6285; www.equifax.com; P.O. Box 740241, Atlanta, GA 30374-0241Experian: 1-888-EXPERIAN (397-3742); www.experian.com; P.O. Box 9532, Allen, TX 75013

    TransUnion: 1-800-680-7289; www.transunion.com; Fraud Victim Assistance Division, P.O. Box 6790, Fullerton, CA 92834-6790

  3.  Called the banks to get all of my accounts frozen immediately after discovering the theft. Went into the branches I do business with the morning after the break-in to get new account numbers issued; and also secured a safe deposit box to store personal information in from now on.
  4. Cancelled all of my credit cards. The thieves only made off with the two they found in the fire box, but I have no way of knowing if they went through my files to get other numbers too.
  5. Called all my creditors to see which ones had received payment on my accounts. Sent new checks with a letter of explanation for the lack of a stub to the others.
  6. Had my mail stopped so the thief couldn’t return to the house and steal our mail. Went to the post office daily for over a month until I was able to find, purchase and install a secure mailbox.
  7. Went to the Department of Motor vehicles to get new driver’s licenses issued with new numbers. We have no way of knowing if the thieves came across our old numbers when they went through our file cabinet.
  8. Went to the Social Security office to request new copies of our cards.
  9. Filed a complaint with the Federal Trade Commission (FTC), which shares information about identity theft with law enforcement agencies across the country.You can file a complaint with the FTC using the online complaint form at www.ftc.gov; or call the FTC’s Identity Theft Hotline, toll-free: (877) ID-THEFT (438-4338); or write Identity Theft Clearinghouse, Federal Trade Commission, 600 Pennsylvania Avenue, NW, Washington, DC 20580.
  10. Sent letters to the Department of Vital Statistics in the three states in which our family members were born to get new certified birth certificates. Also had to get a new copy of our marriage certificate.
  11. Once things settled down, called a few alarm companies, took bids, then hired one to install a home burglar alarm for us.
  12. After receiving confirmation of the initial fraud alerts from the three credit bureaus in the mail, sent in letters requesting a 7-year extended alert along with a copy of my police report.
  13. Signed up for 3-in-1 credit monitoring so I’ll know instantly the next time someone fraudulently applies for credit in our name.

#####

If anyone reading this wants to contact Mandy, just contact me and I’ll work on setting it up.

(First published on my Infrasupport website.  I backdated here to match the original posting date.)

How can organizations avoid sensational data breach headlines?

I was in a Barnes and Noble bookstore a few days go, pitching my new book, “Bullseye Breach,” to one of the folks working behind the counter.  I know all the big decisions are always made at corporate headquarters, but nobody invited me to corporate headquarters and I have to start somewhere.  So I started at this store.

While pitching for all I was worth, a lady who said she works at the Target Corporation Credit Department here in the Twin Cities walked up to the counter.  Many have suggested I patterned my fiction story in “Bullseye Breach” after the real world Target breach – I’ll leave that for readers to judge.  I had a copy of my book with me and she seemed interested.  Which helped my ego tremendously.  Those million book sales start with the first one.

We talked for a while and she said, “It’s a shame we’re all so vulnerable.  No matter how big you are, no matter how much you’re loved in the community, no matter how much good you do, a group of crooks can still break in over the Internet and do this to you.”

That triggered a diatribe from me about believing press releases and people who should have known better not doing their jobs.  I said lots of other things, most of it politically incorrect.  To my surprise, she thanked me for being passionate about this topic and even insisted on buying the copy of my book I had with me on the spot.  I walked away dumbfounded and grateful.

That encounter put a whole series of thoughts in motion.  Since I insisted that organizations can protect themselves, that being a victim to cybercrime is not inevitable, what would I do if somebody actually invited me to corporate headquarters to provide advice and counsel to the CIO?

So here is the advice I would offer.

First is topology.  Retailers, isolate your Point of Sale systems from the rest of your network and keep a whitelist for where they can interact.

Set up automation to notify the right people if those POS systems try to interact with anything outside that whitelist.  Other industries may have similar issues, but retail POS systems are special because untrained store clerks interact with them and they interact with payment processors across the Internet.  Their interactions with the internal network and the rest of the world need to be strictly regulated and monitored.  If the topology had been right, and the right people heeded the warnings, none of the sensational data breach headlines we’ve read about recently would have happened.

That leads to diligence.  No matter what technology is in place, there is no substitute for human diligence.  People are and always will be the last and best line of defense against attack.  Train end users to stay away from the wrong websites and not to fall prey to phishing schemes.  Run drills.  Do probes.  Test often and discuss results.

But even with the best diligence and awareness training and drills, a company with 1000 employees means 1000 potential attack vectors.  Inbound spam filtering and outbound web filtering can help, but sooner or later, somebody will visit the wrong website or click on the wrong email attachment.  That’s why the right people need to pay attention to the inevitable warning signs and take action when warranted.

Which leads to sharing.  This is counter-intuitive, but the best way to defend against attack is to share how all the defenses work.  In detail.

This comment to a Brian Krebs blog post deconstructing the 2014 Sally Beauty breach is a great example.  It was a gutsy call for Blake Curlovic to publicly share the detailed information about this breach, both in the Krebs article and in subsequent comments, and the information he shared will be invaluable to future IT Departments fighting bad guys.

In cryptography, the algorithms are public.  Everyone knows them.  That’s why we have strong cryptography today – the surviving algorithms have all been peer and public reviewed, attacked, and strengthened.  CIOs should operate similarly.  Openly discuss security measures, expose them to public and peer review, conduct public post mortem incident reviews, publish the results, and adjust the methods where necessary.

Bad guys are already reviewing, discussing, and probing security in the shadows.  Bad guys have a whole supply chain dedicated to improving their ability to plunder, complete with discussion forums and specialists in all sorts of dark endeavors.  The bad guys have unlimited time and creativity and the good guys are out gunned and out manned.

Against such an adversary, what CIO in their right mind would want to stand alone?

This doesn’t mean CIOs should call press conferences to brag about the latest security tool.  But CIOs should be visible at conferences and should contribute keynotes and other presentations in a running dialog to help continuously improve the state of the art.  They should also be engaged in online forums discussing and refining the latest ideas.  And when it makes sense to appear in front of the written and TV press, they should take the lead and use the forum to educate the public.

Smart good guys should join forces out in the open for the common good.  Contribute to and profit from a thriving marketplace of good ideas and everyone wins.

(Originally published on my Infrasupport website, May 13, 2015.  I backdated here to match the original posting.)

Should government have the the power to access encrypted communications?

The short answer is, no.

The pro argument says law enforcement needs this tool to fight crime and terrorism, and we can build appropriate safeguards into any law to prevent abuse.  The con arguments point out the danger in granting more power to the government, suggesting that safeguards have limited value.

I’ve read through the pros and cons and concluded it’s a bad idea to grant the government power to access encrypted communications.   Nobody wants to give terrorists and other bad guys a free ride – but as many have pointed out elsewhere, bad guys will find their own ways to do encryption regardless of any US law.  So if we pass a law essentially crippling encryption technology in the United States, we hurt the good guys and help the bad guys.  Tell me how this makes any sense.  We’re all better off with a level playing field.

With a law granting the government this power, even loaded with safeguards, what’s to stop corrupt individuals from abusing it? Attempted abuses of power are already easy to find. There was a case in Minnesota a few years ago when male law enforcement professionals looked up driver’s license records for a few female troopers, politicians, and news media celebrities.  In another case, the IRS as an institution put up roadblocks to make it unnecessarily difficult for some nonprofit groups to gain tax exempt status because individuals in positions of authority apparently disapproved of these groups.  So if we grant the government even more power, imagine the possibilities for abuse and tyranny on a massive scale. It would be 1984 in the 21st century.

Some have advocated an approach combining new technologies with court approval as a safeguard against such tyranny.  The ideas essentially come down to inventing an electronic lock-box to hold everyone’s decryption keys.  Law enforcement can access the lock-box only with appropriate court orders.  The idea sounds nice, but it’s short-sighted and foolish.  Does anyone seriously believe a determined group of bad guys would have any trouble coming up with an attack against such a lock box?  Does anyone seriously want to trust our cryptographic keys with the same government that brought us healthcare.gov and sensational headlines around NSA break-ins?

But my opinion is not worth the disk space to store it. Don’t believe me? Just look at what happened to US cloud providers shortly after the Snowden revelations. Look at what happened to RSA’s credibility after the stories about RSA and the government being in cahoots started circulating.  Now imagine what would happen to confidence in the entire United States data grid if such a law were to pass.

Why would anyone trust any service provider with anything important if the government can access all of it? My private information is mine, and I choose who sees it. Not the government. And I promise you, if I have information I care enough about to keep private, I’ll find a way to safeguard it regardless of any law.

Carrie Cordero and Marc Zwillinger recently wrote a point/counterpoint article on this topic in the Wall Street Journal, here.  In case that link breaks in the future, I saved a PDF here.

There are other ways to fight back against the bad guys besides granting tyrannical power to the government.  I wrote an education book about IT security, disguised as an international fiction thriller titled, “Bullseye Breach.” Take a look at the website, right here.

(Originally published on my Infrasupport website, April 21, 2015.  I backdated here to match the original posting date.)