How a printer inkjet cartridge clobbered one of the world’s largest retailers

This is a story about how a depleted printer inkjet cartridge stopped Sams Club from selling cell phones on the day of a big Apple announcement.  Not a shining moment for enterprise IT.

My wife, Tina’s smartphone gave us three good years of life.  But with crashes and hangs and a battery that would not stay charged, it was time for a new phone.  I know I can buy phones and supplies directly from my carrier, but  every time I compare equipment prices, Sams Club is less expensive.  And that’s what led us to our local Sams Club store a few days ago.

Charles in the cell phone department helped us select a phone and set it up.  All went smoothly.  Just one more thing to do – print and sign the paperwork.  Charles clicked the “Print” button on his POS system and that started a chain of events that destroyed this store’s ability to sell phones and tablets.   But it would take us more than two hours to come to that conclusion.

When the paperwork would not print, Charles checked the Lexmark CS510de printer at this station and found it needed a new black inkjet cartridge.  Charles found one, but it didn’t fit.  He looked all over the store, but there were none.  Why not just send the paperwork print job to another printer in the store?  Because this POS system was locked down and nobody had permissions to set up different printers on it.  The POS system and this network connected printer were mated.

Time for plan B – just swap printers with another station.  Bring in a different printer and assign it the same IP Address as the original printer.  Two roadblocks with this.  First, this was the only Lexmark CS510de printer in the store.  But there were other Lexmark printers.  Maybe one of them was close enough that the driver would be compatible.  So there was a way around the first roadblock.

Navigating through the menus on the printer touch panel, we ran into the next roadblock.  We needed to look up the IP Address on the original printer so we could configure the replacement.  Unfortunately, we quickly found that this printer prohibits looking up any network parameters when it’s not ready.  And it can only be made ready when it has good inkjet cartridges.  No inkjet cartridge, no IP Address.

Insert my editorial question here.  If anyone from Lexmark reads this post, what were you guys thinking and who made the boneheaded decision to prohibit accessing any network parameters when the printer is offline?

With no way to look up the IP Address for the original printer from either the printer touch panel or the locked down POS system, it was time for plan C – find an inkjet cartridge for this printer.  Somewhere.  Surely it was in stock at a store in the area. Right?  Charles jumped in his car and tried the nearby stores and came up empty.  With no replacement inkjet cartridges and no ability to look up the IP Address of the original printer, there was no way to fix the original printer or set up any replacement to mimic it.

Why not just call the corporate IT Department?  Surely they had tables of which IP Addresses belonged to what devices.  But by now it was evening Central time on Friday and nobody was available in Corporate IT to take a call.

Somewhere during all this, the department manager was also involved.  And two other department members.  And me, tinkering with printer settings.  The problem consumed three employees and a customer for two hours.

A brand new Apple phone was available for sale starting Saturday, but this department could not sell anything with a broken printer.  So now the problem was bigger than just my paperwork.  We tried to fix it.  But sometimes the obstacles are insurmountable.  The manager gave me a $50 gift card for my trouble and apologized.

What could Sams Club have done differently to stay out of this predicament?  Having a few spare inkjet cartridges is the obvious answer.  But more important, who designed a business system that depended on finicky inkjet cartridges that can fail at any time?

The system should have had redundancy built in with every component.  A spare printer, a spare POS system with the ability to print to a default printer and alternates, and a standard printer model at all stations throughout the store.  Why use an inkjet printer in a high volume printing station?  Why not a real laser printer?  And why use a printer that prohibits looking up its IP Address?  Where was the corporate IT Department?  Sams Club stores open at 7 AM on the US East coast and close at 9 PM on the US West Coast.   Why was nobody in IT available during prime time Central timezone shopping hours who could address the situation?

I wonder how much in sales this store lost, especially on the day of a major Apple announcement, because of a depleted printer inkjet cartridge?

(Originally published on my Infrasupport websiter Oct. 6, 2015.  I backdated the post here to reflect the original posting date.)