HAL: Robot Redux

I had an on-line chat with a robot named Betsy last night.

It all started when I broke my digital camera. I dropped it while it was powered up, and the lens and shutter jammed in the extended position.

So, I did what all geeks do. My camera was an HP product, I logged onto the HP web site.

After a quick read of the FAQ’s, ( finding nothing helpful) I pointed my mouse to “more help…” and clicked.

I was immediately offered the opportunity to ‘chat’ on-line with an HP technician.

I gladly accepted the invitation to chat. This seemed to be a technically savvy and creative way for a consumer to solve a problem. Having someone with problem solving experience on a specific product would be the most efficient way I could think of to diagnose and fix a problem.

The chat connection was established, and a message followed that said Betsy would be the HP technician joining me in chat.

Now, let me break off this saga about my broken camera for a minute to tell you another story that happened 25 years ago.

In the 1980’s a flurry of articles appeared in the computer trade rags concerning the coming of Artificial Intelligence software (AI). AI was being touted in many computer magazines as software that your computer could use to make decisions based on human knowledge by performing analysis that could eliminate the need for human intervention in the decision process.

In those days when articles on new products gained a certain ‘critical mass’ in the trade magazines, you knew that this might be the ‘next big thing’ in the software industry. It was my practice to obtain demo or beta versions of these products in an effort to keep up with the rapidly evolving software support industry.

My position, at that time, was manager of an IS/IT Department that supported both the Engineering Department that designed and modified a number of custom-built products, and the Manufacturing Department that built and shipped products based on specifications from Engineering.

These were products with a wide range of configurations, but with various design and safety l requirements constraining the combinations of choices. The products had severe legal liabilities and potential safety issues if an incorrectly engineered product failed at a customers site.

You could select any size unit, for instance, but once the size was determined, there would be a load factor. The thickness and type of steel selected depended on the size, the load, and the process for which the customer wanted the unit. These interrelated permutations and combinations of the selections quickly pyramided with regard to length, motor size and a host of other product specifications.

So out of myriad possible combinations- only a hand-full could actually be manufactured. Of the (almost) unlimited types of units that could be configured, there were safety and structural requirements that reduced the usable products to less than 100 styles.

Our first attempt was to write a traditional table driven software system that would aid our Engineers in providing specifications. The program design quickly devolved into complex decisions tables. The decision table paradigm was too slow, complex and cumbersome to build , and was prone to errors both as program 'bugs' and in the products as well.

The product errors were not the usual programming ‘bugs’ for which you could test. The traditional decision table approach would allow the specification of a product that either could not be manufactured or (worse) was dangerous and should never be built for a customer to use in their facility.

The traditional table approach was meant for decisions that were linear. That is, each outcome was the result of following a single path through the tables. For this specification generator program to work, decision tables would have to be constructed for every conceivable style of product that we had ever built or might build in the future. There were no ‘linear decision paths’ for this product.

The decision tables we had to work with at that time were all single choice, one direction, and limited scope when multiple options were included. The state of 'programming decisions' would now be considered a ‘typical telephone answering menu” style of programming. In other words, pick “X” from a list- and leads you to the next list. Pick “Y” from that list, and so on. Only a very few of our products could be designed in this flat table model. We needed a multi-dimension table to provide for variations in constraints.

Since we rarely anticipated some of the configurations requested by customers, this meant that any order for a new style would involve rebuilding our complex tables if we continued with the traditional flat decision table model.

[Even worse, the initial program requested by the Engineering Department was for all product lines. Our initial programming tests were based on a single product. The complexity of the design for one product meant that each product was going to require a completely new version of the programs with additional cost and development time – and little reusable code from previous program versions.

Our attempt at homegrown software was further constrained by the difficulty of any future changes to the specifications of existing products. The initial programs and decision tables were so intricate that a disturbance in one section of code or one table could make pieces of code elsewhere fail- and often the failure could not be uncovered in testing. The program might work even though the product it specified did not meet safety or liability requirements.

We told our Engineering management that the ‘traditional approach’ to making a decision tree program was of no use on this problem. The selections were not linear and there was no foolproof way to insure the integrity of the interconnected selections because of the sheer complexity of the decision making process.

That is when I had the Department investigate the possibility of using AI software that promised to create rational and correct decisions from complex and, and often, unanticipated relationships in data.

The time and effort to develop AI software on the tools then coming into the computer marketplace was not so much programmers time as it was of verifying that the Engineering design specs were accurate and complete. This project bogged down when the Engineering Department could not allocate engineers to validate specs. This was assigned to their most experienced personnel- they were also the most valuable in supporting sales efforts.

For a change, the programmers were not the ‘bad guys’ in the missed project deadlines.

Got it?

OK. Fast forward to today’s world, and let’s pick up my broken camera problem.

Shortly after I accepted the HP web page offer to chat with Betsy, the technician, her chat window became active:

Betsy: “Hello, my name is Betsy. What is your Name?”

A brief exchange followed while I provided my name and the serial number and model of my camera. And this is where I think the AI ‘robot came into the picture:

Betsy: “Please explain in detail the problem you are having”

Me: “My camera is jammed”

Betsy: “Let me see if I understand your problem correctly”

Betsy: “Your camera lens is jammed in the opened or closed position?”

And so the conversation continued.

Each time I provided information on the problem, Betsy gave me a test to perform.
As I reported the outcome of the testing, I was led to more complex steps.

As the stilted questions continued- blank and emotionless- and I threaded through the ‘technicians’ assistance – I got the feeling that I was chatting with an AI program generated by a computer (robot if you will) not a real person named Betsy.

This became more evident as Betsy and I worked our way through the trouble-shooting list. I could see how an AI program could narrow down a problem in the way this was being done. And it was obvious, as a business decision, that paying a person to do this over and over was not cost effective.

This had all the requirements for a classic AI program.

When Betsy asked me to push the reset button on the camera I informed ‘her’ that I could not find the reset button, She presented me with a product selection list and accompanying diagrams in a separate window. I was led through a diagram of my camera to find the reset button (cleverly hidden on the camera body).

Her detailed instructions sounded like this:

Betsy: “Did you properly locate the reset button”?

Me: “Yes”

Betsy: “Did you press the reset button as instructed?’

Me: “Yes”

Betsy: “ Is your problem fixed?”

Me: “No”

And when none of her further ‘expert advice and suggestions’ resulted in resetting my lens and shutter to the closed position, she went through another process for having the entire camera replaced. No one repairs electronics equipment any more (Betsy told me that).

How do I know there was not a person on the other end actually chatting from a script?

Clue number one is that I am sure in the past 20 years AI programs have not only become more sophisticated in their processing, but they have now been given standard ‘chat response’ interfaces and can connect to web chat software so that a human is no longer needed on the manufacturers end of the chat ‘conversation’.

Just as ATM’s replaced bank tellers, a machine (robot?) could deliver the same technical support that was once delivered by humans. It just makes good business sense to me.

But more to the point is the artificial look and feel of the chat conversation itself. If you saw the movie “2001” you know there was a supremely intelligent computer named HAL that ran the spaceship while the astronauts were in a deep sleep for their long space trip.

This conversation with Betsy had a HAL-like quality to it.

Verb tenses were sometimes out of sync with the tense that a human responder would have used. The syntax was often very stilted and structured. Not like people normally speak (or chat).

In addition, several years ago I used a similar HP support web site for a PC problem. The technician was a real person and the phraseology and interaction of the ‘chat’ had an entirely different feel and rhythm to it.

I guess the interesting aspect of this is that in the manufacturing world in which I worked most of my career we always considered robots as huge machines that knew how to properly position and weld a truck frame with amazing accuracy, or monitor precision tooling operations well beyond the ability of human operators. There was a lot of talk around early manufacturing robotics about ‘lights-out’ factories: entirely operated by robots.

But in the interactive, web delivery world of today, ‘robot’ has come to have a wider meaning. With on-line chat (voice response in the future) we will not only see the precision results of robot help in manufacturing, but also find that we deal with robots in the most unexpected places every day when on the web.

My feelings about Betsy the robot were confirmed when Betsy ‘asked me’ when a telephone call would be appropriate to set up the logistics for the return of my camera for a replacement. That pretty much meant, to me, that Betsy was a machine. I gave ‘her’ a time on the following day for someone to call and make the arrangements.

In a very awkward way, Betsy asked about changing my requested time:

Betsy: “Is it possible that this telephone call could be taken within one to one and one half hours from this time?”

That question ‘felt’ like a computer program had checked the queue of waiting calls for the human operators on duty and calculated a maximum 1.5-hour wait time. (That is akin to the message when you use a telephone menu and are told, “The wait time for the next representative is 90 minutes”. The syntax of Betsy’s response sounded like the wait time was loaded into a script as “one to one and one half hours”.

If Betsy had been a person instead of a machine I would have expected a less structured and stiff presentation of the question and the response. Actually, had Betsy been a human it would have been more efficient to process the details for the return of the camera at that time and not defer to a phone call at a later time.

The odd part is that after Betsy’s help I felt that - although I would have preferred a more immediate solution- the problem had been resolved to my satisfaction. I was pleased to have a ‘fix’ for my broken camera. And I was doubly pleased that I got immediate response when I logged on the web site- instead of waiting on hold for a human being to become available to chat.

So, my robot friend, Betsy, was quick, responsive, efficient and knowledgeable. Not a bad customer support package, I’d say. Since I was a very satisfied HP customer my last chat comments with Betsy-the-robot went like this:

Betsy: “I am glad I could be of service. Thank you for choosing HP. Have a good evening”.

Me: “You have a good evening too”

Betsy: “Thank you”

I still can’t believe it. I wished a machine ‘good evening’.

Disasters: Real and Imagined

If you have spent any time around a computer operation you know what a “Disaster Recovery Plan” (DRP) is all about. So, skip a couple of paragraphs down or go to the war stories in Parts 2 and 3 of this blog.

Otherwise, I’ll give you a quick description of what a DRP does. How it is supposed to work. And, why it usually doesn’t work like planned when you need it most.

First of all, it is a useful analogy to think of any ‘computer disaster’ as a row of dominoes standing on end. If you tip the first one, you start a chain reaction tumble from one to another until they all are down (the classic domino effect). That mental picture fits the playing out of any disaster that impacts computers and computer operations in companies that I have seen.

Did you ever see one of those ‘Guinness Book of Records’ TV shows where some idiot has stood 10,000 dominoes on end and tumbles the first one, and the falling dominoes run off in every direction tumbling dozens of others. As a matter of fact, computer disasters are like dominoes in another way: Those TV extravaganzas always end up revealing the unexpected. A logo or slogan or something spelled out by the fallen dominoes. That is exactly the same result as a computer ‘disaster’. The unexpected final results are always a surprise. Everyone feels that their DRP had anticipated every scenario and that is rarely the case.

When a disaster strikes (sometimes caused by Mother nature- but often caused by us humans), one by one the normal processes in your computer operation spirals out of control. Sometimes the initial trigger event and the total collapse of your ability to process are spread over hours or even days (as in rising flood water). Sometimes it is like a bolt of lightening that is over before you know it with the same widespread damage.

In every company where I have been involved in the computer operations there has either been an existing DRP or a major project launched to create a DRP to anticipate and prepare for specific emergency procedures.

One blind spot for many senior managers is that they often see the DRP as a problem for the IS/IT department. The strategic importance of information processes by company’s today means that disaster is a problem for all departments- not just the ‘computer geeks’.

The challenge to ‘it’s just a geek thing’ hurdle for a company writing a new DRP is to get every department in the company to realize it is everybody's problem and for them to commit resources and get involved in the development of the DRP. In today’s information world, losing computer systems in an organization is not like a patient with a broken arm- it is more like a patient with a potentially fatal heart attack.

So, if you are assigned to a company team that is creating a DRP- what do you do?

First, you have to figure out the more obvious disasters you might face. Near a river: think flood. In an area with tornadoes: think total wipe out. In an area of frequent, severe thunderstorms: think lightening and destruction of your networks by electrical surges. You get the idea. Then evaluate a lot of 'what ifs' relating to potential threats. Then make a ranked list of all the things a company must do to survive during recovery from a disaster.

DP's should fill two needs: how to restore the company to pre-disaster processes, and how to keep operating while disaster recovery is underway.

Next, you think of the ‘key’ people who will be the most useful in assessing and recovering from a disaster. As you might guess, this is usually not top management. It is usually the mangers of various operating areas with tons of company experience- and hopefully a network of acquaintances and business relationships within your company (and outside companies and suppliers) who can be relied upon to respond quickly if/when disaster strikes.

Most important- you assign someone to be the “Disaster Manager”. This is the ultimate go-to guy during a disaster. Like the conductor of a symphony, this Manager wields the baton and keeps everyone involved in the recovery on the same page of the music. This manager should be fairly senior in the organization. He must have the authority to sign off on funding for emergencies supplies and support.

Nothing is worse than needing instant response to a problem that is spreading out of control by the minute, and finding that you cannot find anyone to sign a purchase order to get vendors or equipment to solve the problem and stop the hemorrhage.

So, that is the framework of starting to create a DRP. Rather than give a shopping list of items to include, you can probably see more easily what a DRP actually should contain if I first take you into an imaginary disaster in Part 2. Then in Part 3 we will contrast that disaster plan exercise with an actual disaster to learn more lessons.

Finally, before we dive into some real-world examples in Part 2 and 3, never forget that a DRP has to be tailored to your specific situation. There are many books, guides, and even sample plans to use as a starting point. Michelangelo was asked how he made the statue "David". His reply was that he started out with a large block of granite. Then he chipped away everything that didn’t look like David.

You can apply that same advice to any prototypes, textbooks or sample templates of a DRP that you use as a guide in writing your own.

Disasters: Real and Imagined. Part 2

Background

All major corporations have financial auditing firms and frequently those firms are asked to apply their auditing and IS/IT skills to audit operations in addition to the traditional accounting processes. Particularly in this day when information systems are the backbone of a company- the auditors will ‘audit’ the company data and computer systems with regard to security, data integrity, etc.

One thing you can be sure they will do in a systems audit is look at the Disaster Recovery Plan (DRP).

I worked for a company that had a very large computer Data Center that housed about 50 computers for the entire North American sector of the company. It was located in the middle of a Dallas industrial park in a multi-thousand square foot building with offices for a support staff of about 200

The auditors for the Dallas Data Center not only provided oversight in the creation and maintenance of the DRP, but also had the authority to perform unannounced evaluations of the effectiveness of the plan. The Auditors were required by their contract to perform a ‘live test’ of the DRP at least once every 2 years.

Bang, bang, you’re dead.


It is routine part of good practices for a computer center to take regular backups of all of their databases and files. This Data Center backed-up every file on all 50 computers every Saturday night and Sunday morning. The full backup of all the systems took about 16 hours. The backup media consisted of printed reports, tapes, and disc copy of all the data on the computers. All of the backups were sent to a ‘hardened’ commercial site miles away from the Data Center. The storage site was physically secured by fences and electronic surveillance and was underground to protect it from Mother Nature.

Imagine the surprise in the Data Center computer room on a quiet Sunday morning when, just as the last backups were winding down, the Company Treasurer (actually the #2 man in North American corporate management) walked into the computer room with the Chief Auditor. The Chief Auditor informed the senior computer operator that, “a massive tornado just demolished the Data Center building and there is only one survivor."

Let the games begin.

One computer operator was determined to be the ‘sole survivor’. He was given the opportunity to make one phone call. The Auditors purposely picked an individual that was not a manager. This low-man-on-the-totem-pole survivor did not know about nor had ever seen the Data Center DRP. In a panic-the computer operator just called his bosses boss. Even though his real boss was sitting in his office next to the Computer room at that moment, he was one of those already officially pronounced as a 'tornado' casualty.

Remember the tumbling domino analogy in the introduction? Well now we will see it kick in as the formal DRP comes off the tracks and crashes:

Let’s see: First tipping domino: Devastating tornado. 'THUNK!' Second tipping domino: no one knows what to do. 'THUNK!' Third tipping domino- no one in authority is in the building.'THUNK!' The domino effect is in full play before the DRP gets off the ground.

There were a number of other dominoes going ‘THUNK’, but no one noticed them until later in the exercise. Four key technical staff members had been testing new software on Saturday night while the computers they needed were not busy. The software gurus were still running tests when the disaster was declared. Like the on-site Computer Manager and the rest of the operations staff, the 4 who would have normally been at home would not be available to help with technical problems for the remainder of the evaluation: faux casualties of the faux disaster. More on that 'tipping domino' later. ‘THUNK’,

The poor ‘sole survivor’ in the computer room listened as his boss’s boss (aroused from a deep Sunday morning sleep) picked up the phone. Not only was his brain fog-bound by sleep, but also he was not even sure who this person several layers down on the organization chart was that was calling at that early hour. But, he was wide awake and out of bed in a heartbeat when he heard the message. He had been a part of the DRP Team when the plan was written. He knew what had to be done.

He quickly called the Vice President of the Dallas Data Center to inform him -since he was the 'link' to Corporate Headquarters. Next, he called the secure storage facility where the backup tapes are stored. Someone in the ‘underground bunker’ at the secure location answered immediately (ahh- the plan is working).

When informed of the situation, the storage facility clerk checked the Data Center backup storage logs. The logs were maintained by the off-site facility to identify and locate backup material for any date so that it could be brought from the vaults when needed (as in this situation). Some backups were stored for up to 7 years for tax purposes, and each weeks backups were carefully catalogued. But if you listened carefully as the clerk pulled out the logs for the Dallas Center you could hear another domino tipping.'THUNK!'

The logs showed that that the latest backup tapes for the week just finished were still at the Data Center. Regular pickup is scheduled on Mondays. 'THUNK'! The backups were sitting on the loading dock waiting for the scheduled pickup. Since the auditors declared the entire building a no-man’s land, the current backups could not be used for this DRP test. 'THUNK'!

All is not lost. Each day there are incremental backups that are also stored in the off-site facility. But that also means means that instead of 24 hours to restore the single condensed weekly backup (declared D.O.A.), there will be an additional 18-24 hours added to the total disaster recovery time to restore the previous Sunday backup plus the 7 incremental daily backups for each day of the previous week.

The DRP called for restoring data on a schedule that would have everything back up and running Monday afternoon. Now the lost time will stretch recovery into Tuesday – and that is if nothing else goes awry. 'THUNK'

Not a pretty picture to report to International headquarters. Had this been a real disaster instead of a simulation. If this was the real thing then on Monday everyone in the rest of the country would have opened for business as usual only to find that they had no data in their databases and no access to their computers in the Data Center until (at least) Tuesday. That could have toppled a lot of dominoes.

All large-scale Data Center disaster plans in those days included an alternative data recovery facility at a different geographic location. In the case of the Dallas Data Center, the off-site data recovery center was in New Jersey.

The ‘cold computer’ contract made with data recovery companies meant that with a phone call you could request that they warm up the equipment and when you walked into their front door you would start operating immediately as though it was your home Data Center. The recovery centers maintained computers that were technical copies of their clients hardware. These centers also had the ability to attach to your networks and also provided office and work space. The recovery facilities were designed to clone complex infrastructures such as the Dallas Center.

No ‘Thunk’ of a domino there. The call was made and the New Jersey disaster recovery center prepared to get the Dallas operation back on-line.

With the backup computer system now coming up, the quick-thinking manager knew that he had to round up the Off Site Recovery Team that was described in the DRP. ‘BIG THUNK!’ The only copy of the DRP that he had was in his office in the ‘demolished' Data Center. He would have to direct this recovery from his recall and memory of a document written a year ago that he had not read since. He could get the ‘official copy’ from their bank vault on Monday morning, but he had to act now.

Kuh-Thunk! Kuh-Thunk! Two of his Team had recently moved. The phone numbers in his little black book were no longer correct. There are many small satellite communities around Dallas, and he did not know where they had bought homes. More time lost tracking them down to notify them (which begs the issue of whether they had changed and updated their addresses even if he his copy of the DRP).

Now the clock seems to be running in ultra-fast mode. The Team is notified. One of the Team members performs the task of getting airline tickets (the agency that would normally schedule them is closed on weekends ‘THUNK!’). Another knows that his assigned task is to arrange a pickup of the tapes from the ‘underground’ storage facility and get them to the airport.

Since the ‘underground storage bunker’ had already been alerted by the call from the boss- all they needed was the security password to authorize preparation of the tapes for shipment. At least that went according to the DRP.

Things appeared to be more under control by the time the Team was on the plane on their way to New Jersey (with the backup tapes in the seat next to them). One of the practical plan elements was that you could not depend on a shipping company. Packages could be lost be in an accident delaying the backup tape arrival at the remote disaster center in New Jersey. The DRP required that they have a Dallas Data Center 'baby sitter' from the secure site to the recovery site at all times.

As a gruesome aside, one point that was missed because they did not have the DRP checklist to follow was that the tapes and the Team were supposed to be on different planes. The Team was also supposed to split into two flights.

Sunday night is now turning into early Monday morning. Team and tapes are finally in the data center in New Jersey. The cloned Data Center computers are humming and the data recovery phase can finally begin. It has taken a full business day to get to this point in the recovery.

But even though this part of the plan went well, there are more dominoes to tip over.
The dominoes started tipping again as soon as the backup tapes had finished loading on the computers, and it was time to start production runs.

Many of the production jobs that run on a routine basis in the Dallas Data Center had process upgrades and enhancements installed over the past year. Pieces of the software on the Dallas computers had been upgraded to newer versions as well. The job control statements and run instructions used in the Dallas Center to set up and run the production jobs was no longer the same as the copies of the run books sent to the New Jersey recovery facility the previous year when the contracts were signed. 'THUNK!'

Some of the systems issues created by the by the mismatch in instructions, software, etc. required the experience and expertise of the 4 software engineers that were ‘causalities’ in the faux tornado.‘THUNK’

As the test of the DRP continued there were more operating issues to be resolved. More domino ‘THUNKS’ occurred before the Auditors finished their evaluation.

Tuesday evening the Auditors declared the simulation of the disaster completed- even though many tasks remained unfinished or incomplete. 'MANY THUNKS’! The final assessment of this exercise by the Auditors was that most of the files and systems were fully restored. Data was loaded properly. But that the system fell short of being 100% ready for running the business.

In truth, the Auditors allowed the off site test to skip some computer runs and ignore some problems and errors because the reason for the evaluation was to determine the effectiveness of the DRP - not obtain 100% recovery accuracy as they would in a real disaster.

Including the number of days that would have been required to play catch up for lost work on Monday and Tuesday, the auditors estimated that the disaster would have costs the entire international company a full week of business and income in North America.

The 'success' of this DRP in this simulation could probably be summed up in the old computer saying that "Any plan is better than no plan at all".

Thank goodness it was just a drill.

Disasters: Real and Imagined. Part 3

Background

As we saw in Part 2 of this series, Even with a plan, disasters have this bad habit of being the personification of Murphy’s Law.

After the lessons learned in Part 2, we can stand our dominoes back up. For this next war story the ‘Thunk!” of the tumbling dominoes will take a quite different direction - and the lessons will be different. This is a story of a real disaster with real consequences. Not a hypothetical test of a DRP.

The disaster is about a flood: Worse, a man-made flood that could have been prevented.

Here is what happened.

It was Thanksgiving weekend. The 2nd shift on Wednesday night went home at midnight. Because of the four-day break, they were the last workers for the week. The plant was dark and deserted while everyone was home eating turkey and enjoying the holiday.

It started to rain Thanksgiving Day and rained Friday, Saturday, and Sunday. Not a gentle refreshing rain, but a ‘gully washer’. ‘THUNK’!

I have to digress a minute and tell you about the physical dimension of this disaster. The plant was about 20 years old. It originally consisted of a two story, brick Administrative building, which was about 100 feet from the plant. The plant was a multi-thousand square foot, steel structure with a flat roof.

At some point in the history of the company a one story office building was constructed between the Admin building and the plant. It was to house the Manufacturing Engineers Department. They constructed a front and back wall and used the brick wall of the Admin building on one side and the plant wall for the 4th wall. They built a flat roof over the four walls and they were ready to go.

Since I have mentioned the flat roof of both buildings, you may already see the next ‘THUNK’ coming.

The weather had been miserable for the entire month of November, and as a result the pine trees on the plant property were dropping pine needles in large quantities and they were blowing onto the flat roof of the Engineering offices: “THUNK!”

The miserable weather had prevented the normally routine removal of the accumulating pine needles. By the week of Thanksgiving the storm water drains for the roof of the Engineering offices were almost totally clogged with pine needles. ‘THUNK’!

Then Murphy’s Law kicked into high gear. The flat roof of the manufacturing plant – because of its size- required 24-inch drainpipes to take rainwater from the roof to a storm drainage system behind the plant. The size of the roof and the size of the drains meant that the pipes transported a tremendous flow of water in a heavy rainstorm

One of the downspouts had developed a minor leak- and (again) because of the weather-had not been patched by maintenance crews.‘THUNK’ In the torrential rain, the water running off of the multi-thousand square foot factory roof started to enlarge the hole. Finally the drain pipe simply cracked wide open. ‘THUNK’!!

A deluge of rainwater started running out of the ever-expanding hole instead of flowing to the ground and into the drainage pond. The flat roof of the Engineering offices – already a swimming pool from the pine needle blockage was now adding most of the runoff from the factory roof as well. The depth of the accumulating water on the Engineering office roof rapidly increased. ‘THUNK!’

Under the weight of the deepening water, the roof finally collapsed. Water poured into the Engineering offices and then down the halls and into the main Administration building. From watermarks on the walls the water depth was about 2 feet in both buildings.

The Flood

I was at home. It was on Sunday of the four-day break, the phone rang. Caller said we had ‘water’ on the floor of the offices. Oh yeah. That was like calling the Titanic sinking a ‘boating accident’. From the tone of the phone call, I expected to find something akin to a bad window leak when I got to the plant.

The driveway to the parking lot was blocked. The plant was surrounded by police cars, fire trucks, emergency vehicles, and most of the senior management of the plant. They were huddling in clusters and trying to stay dry in the (now) lightly falling rain while they decided what to do next. This was definitely not going to be a 'window leak'.

This plant had a 440-volt electrical service. It is not turned off like a switch for a light bulb. So with knee-deep water in the office area, the Fire Marshal would not allow anyone inside any buildings until the power company came and disconnected the electrical service.

We were not sure who to call: ’KLUNK’ No names. No phone numbers. No DRP remember? Fortunately the Fire Marshal knew what to do and called in the electric company crews to shut off all power on that part of the grid.

The power company shut off the power. The Fire Marshal made a quick inspection of the offices then called for a full structural check of the darkened buildings to determine potential security or safety hazards before anyone would be allowed to enter. The clock is running on this disaster and we don’t even know the magnitude of the problem yet. ‘THUNK!’.

As you can see, we are now several hours into a full-blown disaster and have been able to do absolutely nothing to start any recovery efforts. Our only assessment of the damage was gleaned from peering through the front door into the darkened offices.

Finally, by first light, the building was deemed safe and a few key managers entered the building to assess the extent of the damage.

We had no Disaster Recovery Plan. So this recovery was going to be improvised as we went along and as needs dictated what was to be done. The success and speed of reaction would rely entirely on the knowledge and experience of the managers (none of whom had experienced a disaster like this). It was a true 'flying by the seat of our pants' episode.

The computer facility had been relegated to the second floor some year’s prior in an ‘office space political struggle’ in favor of more first floor space for another department. In this case, it was a battle I was glad we had lost. All of the computers and related equipment were dry and undamaged on the 2nd floor.

But, the phone switch – the guts of all internal and external phones as well as the Corporate computer network connection was right next to the doorway of the flooded Engineering offices. The bottom half of the phone switch was useless. Worse, much of the wiring had become a spider web as more and more components had been added to our communications systems in the last few years. Much of that wiring chaos would have to be sorted out and replaced before we replaced all of the soaking equipment.

Back in those days there was only one portable phone in the entire company (and it was not your Razar- it weighed about 10 pounds and looked like a WW II walkie-talkie)- At times the impromptu disaster team was handing the portable phone around the room so that each team member could place a call to a vendor or make other critical calls. Some members eventually went home and used their residence phones because one phone shared by 10-12 managers was not allowing us to move very quickly to bring in additional resources.‘THUNK!’.


The Plant Manager had a car phone – another unusual feature in the early 90’s. He conducted business from his car most of the first day so that he could use his phone without interruption. Since it was important for him to make on-site decisions frequently in those first hours, he did not have the luxury of using his home as an office.

Needless to say, we lost the entire first floor computer network and all phone connections from either direct water damage or secondary damage. All paper work in the lower two feet of any cabinet or desk was practically unusable. We hired temporary workers using hair dryers on the important and irreplaceable drawings and other product material to salvage as much as we could. There were no off-site copies of these business critical documents.‘THUNK!’

As far as computers were concerned, we ended up replacing all of the terminals and PC’s and completely rewiring the entire first floor office network.

This was- needless to say- a disaster recovery with no plan of attack for getting business back up and running. With 12 to 18-hour days, a lot of ingenuity, and a lot of grit, things returned to some sense of normalcy in about 30 days.

The entire recovery- including rebuilding the Engineering Annex and major rewiring of networks and phones took almost 6 months.

These two examples of a Disaster Recovery Plan are real life stories. This ‘Flood’ story points out the chaos of no DRP in terms of assessing priorities and managing the restoration of the business.

But, in all fairness, I do not think anyone in his/her wildest imagination would have ever anticipated a flooding roof collapse and had a specific DRP response ready for this disaster.

In a similar vein, the Dallas DRP did not anticipate that key people would not be available, and that no matter how well you plan- if you don’t revisit and update the plan on a regular basis its effectiveness deteriorates with regard to systems, people, and operations. Simply put: as the business changes, the plan must change with it.

And the changes must be made available to those who will be touched by the plan, and especially organizations outside the four walls of the company on whom you will depend.

So what is the moral of both Part 2 and Part 3 of this tale of woe?

Simply this: Don’t make a disaster recovery plan to anticipate how you will act when a certain type of disaster occurs (i.e. tornado, flood, etc.). Mold the plan in what the military calls an “Order of Battle”. The plan should indicate who will be responsible for what areas of the business. Establish the priorities for the business to keep the first efforts after a disaster focused on business continuity.

Even a generic plan will serve the company well. As some smart person once said, “any plan is better than no plan at all”. In the case of an information system DRP it is crucial that the plan of attack be coherent and rational not that it second-guess the potential causes of disasters.

Hey! Where's the alt-ctl-delete?

My daughter is fascinated by my old slide rule that I used in my Engineering School days in the 1950's. She looks on it the way one might look on the 'rain sticks' from the Belgian Congo that were used to 'bring rain' to the grasslands by primitive people.

So, let's talk slide rules.

Back in the real caveman days (i.e. just before I was born, she thinks) you could rub two sticks together and make fire.

But, in my 'caveman' days you rubbed two sticks of wood together and got products of two numbers, sines, cosines, all sorts of useful engineering calculations: Neanderthals called their two sticks kindling. I called mine my slide rule.

This miracle of its day was the engineers mathematical right hand. The model I used had the cumbersome engineering description of log-log-deci-trig- duplex . That is kind of like saying you have a 512K /40 GB PC with a 19" monitor. In both cases it is jargon, and in both cases it fits the same purpose. It describes the capabilities of the tool you are using.

More on that mouth-full name later.

Many pre-1960 engineering accomplishments in the world were built to one degree or another relying on slide rules for computational support. They were the engineers laptop portable and hand calculators all in one.

So what was the 'trick' in rubbing two sticks of wood together and creating mathematical solutions. It certainly wasn't an Ouijai Board (if you're too young for that comparison visit: http://www.museumoftalkingboards.com/WebOuija.html).

Maybe if we build a primitive slide rule in this blog we can get an idea of how it works.

Take two plain old yardsticks. Put a red mark at the 'zero' inch mark on one of the yardsticks. yardstick. Now Let's say, without taxing our minds too much, that we want to add 3 and 2. Well if we slide the 'zero mark' on the yardstick to the 3 inch mark of the other yardstick and then on the original stick with the zero mark we move our finger up 2 inches- Viola: 5.

Get it? Three inches plus two inches takes you to 5 inches.

Dumb and arduous way to add two numbers, isn't it. Not only that, but with these yardsticks the biggest number you can add must not exceed 36. Very useless, I'd say.

So there are two major problems with using a yardstick for a slide rule. One is that is is practically useless to just be able to add two numbers. It still doesn't multiply, divide, do sines and cosines, etc. The other limitation is that the precision is limited by its physical length.

What do you remember about logarithms?

Maybe you remember that as you go from 1 to 10 the numbers are not the same distance apart. As a matter of fact going from 1 to 2 in a book full of logarithmic tables is about twice the distance as from 2 to 3. Another feature of log tables is that they only go to 9.9999.(or so).

But. the one thing you might remember is that if you want to multiply 3.22 by 5.783 you only have to add the logarithms of the two numbers together and look up the logarithm resulting sum to find the answer. So, you add them and look up the sum in a log table you will get your answer of 18.6126

That property of logs made the slide rule possible: adding them is multiplying and subtracting them is dividing.

So, back to our real slide rule. If we place the 'zero mark' on the slide rule 3.22. Then proceed down the rule to 5.783. Bingo! It is right across from our answer of 18.6126.
So we really did use the property of adding two numbers just like our simple minded yardstick. Instead of adding 2+3 (the yard stick equivilent of addition) the slide rule added the logarithms of two numbers and produced the results for 3.22 X 5.783 for the product of two numbers.

Numeric computations can be provided by using their log values and converting the calculations to simple problems of addition. That means that if you do the problem we just solved in reverse you would end up doing division.

There is another advantage to logarithms. Precision. If I have 12.256 or 1225.6 or .12256- the logs are all the same one. Only the number of zeros and/or significant digits are changed. There is even a rule for the decimal place depending on whether you moved the slide to the left or right. Pretty damned nifty, I'd say!

There are also classes of solutions where you either chain together a series of movements of the 'slide' or flip the slid-rule on its back and read an answer from the other side. The design of these was very complex and many types and models were available. Mine happened to be the 'pick up truck' model. Not real pretty and fancy, but capable of doing a lot of work.

The deal on naming slide rules was to describe their capabilites and processes. That is where the ungainly name of log-log-deci-trig-duplex came from for mine. It was logarithmic and the logs were base 10 ("deci) and it did trigonometric functions and it had scales on both sides ("duplex").

They came in many shapes. There was a circular one. I once worked at an aeronautical wind tunnel facility that had 30 inch slide rules because they increased the precision.

In the end though, it was nothing more than only a ‘technological abacus’.

It had a lot of limitations. The ability of the user was one. The precision was another.

Precision depended on the size of the ruler, the number of significant digits in the numbers in the problem you were solving, and keen eyesight. If you had a number like 3.14165 you might find that you get get it to "3.14" by reading the scales. then, depending on the size of the slide rule and your eyesight you had to apply some windage and get the accuracy of the "3.1416". And, quite honestly, on a small slide rule like mine- the rest of the number disappeared into uselessness.

Having said all of that, you might wonder what was the big deal.

Well in a time of no PC's, no HP hand calculators, and no computers smaller than an 18 wheeler it had virtues no longer valued: Portable. No batteries. No ALT-CYL-Delete restarts, and no BSOD.

What a concept.

My Heroes Have Always Been Cowboys

Truth of the matter is that except for Albert Einstein, I don’t really have any heroes in the traditional definition.

I guess I am such a geek that what I have are ‘professional heroes’. Individuals that have written advice and philosophy's that have served me well in tough job situations and- I hope- made me a better IT professional as a result of reading and applying their ideas.

These are probably going to be people you have never heard of. But you can Goggle them and get lots of background if they pique your interest.

First and foremost- in the project management arena is a guy named Fred Brooks.

There once was an IT magazine called "Datamation". It was one of those professional publications that lived on its advertisers and was provided free to people in the IT business. One month there was an article written by Fred Books titled: “Why is the Software Always Late?”

At that point I had been in the business about 20 years, and had never been on a project that finished either on time or under budget (usually neither). Personally, I had tried many approaches to make a project finish in the time and costs constraints of the original plan.

One of the crude approaches I tried at one time was to thoroughly and carefully estimate the time and cost of the project and then submit a plan that was double those calculations.

Even with a factor of two, that project was late and over budget.

Probably the most common problem on projects plan failures is "scope creep".

A project is defined. The objective is approved by all the participants. Then as the project is underway, someone convinces management that since we are doing "X" in this project and "Y" is almost the same thing- why don't we include that in the scope of the project.

You can add the additional time to do "Y". But seldom is the time to integrate it into an already underway project either clear or even considered.

I tried to get around that one by having the scope on a written and signed document by senior management. Any change had to be over their signatures. Well, a fair number of managers still had the influence to go to senior managers and still get the 'scope' expanded.

So, it was late and over budget once again.

The "DATAMATION" article by Fred Brooks came along when I had pretty well exhausted my bag of tricks and was frustrated that my best attempts to create realistic project plans never seemed to work. So I read the Brooks' article with interest (expecting the usual blather and blah, blah, blah on an academic level- or ill-advised and idealistic suggestions).

It contained an absolute gem of inspiration for my future project planning.

The article revolved around ‘the mythical man-month’ and deconstructed the way all projects were managed in those days (the 80’s). It identified a powerful, but not obvious, fly in the planning soup.

Fred Brooks’s writings on ‘the mythical man month’ are well worth digging out and reading before you try to estimate the time and cost of your first project.

Then in the mid 90’s I ‘discovered’ the works of Dr. W. Edwards Deming. In a nutshell, he espoused the idea that no system is ever finished. You must practice ‘continuous improvement’. That, by his measure, was how you achieved a quality product: and how you had happy customers. His philosophy was that when you design a system you view the completion of the first project as just the beginning of a long and continuous process improvement.

Since he was one of the principal architects of the recovery of the Japanese economy after World War II- his writings were not only thoughtful, but also based on a solid foundation of results. In fact, the entire Asian industrial revolution in the last 50 years- and their leadership today in so many industrial areas (not just Japan, these concepts have been adapted by many Asian industries) is based on the basic principles that he used back in the 50's.

Finally, I always respect the work of Northcote Parkinson. You may be familiar with the famous “Parkinson’s Law” (‘work will expand to fill the time available to it'). But he has many things to say about individual productivity.

In a large computer system project productivity is a major issue and the misapplication or misuse of productivity is usually one of the hidden factors in a failed or late project.

These 3 are not philosophical or religious gurus that will require long and arduous study to apply their principles. They are three down-to-earth writer/thinkers who provided me with some simple guidance and a few more tricks to add to my project planning so that my recommendations to my management would be more rational in making projects successful.

Guerrilla Systems Implementation

Did you know that two of the most technologically advanced airplanes of their time - the U2 and the RS-70, were built by a small band of engineers working in an isolated hanger at the edge of a large aircraft companies air field?

True. Lockheed Aircraft has thousands of designers and engineers. But, when they needed a new and radical airplane that departed from 'traditional design' they were smart enough to let a small band of designers go off by themselves, outside the bounds of the usual development bureaucracy, and get the job done. The design team called the small building away from the big headquarters "The Skunk Works". These 'gurrilla designers' were aware that they were pulling themselves away from the usual promotion and job opportunities and would not necessarily be well thought of by the corporate culture. But they also knew that it was the only way to get the job done.

How effective was it? These were the only two 'spy planes' we needed from the start of the cold war in the 50's to the present. Not a bad record, I'd say.

It is rare that the opportunity to break new ground comes to anyone in a large corporate structure. Corporations are not, in general, bold and daring. When that opportunity comes to you - be a risk taker and go for it. Whether it succeeds or fails, it will be the most fun you can have in a career.

Example:

My company was one division of a very, very large international corporation. As a result, we could not afford to operate a large warehouse, or own the large mainframe computers needed to control inventory of replacement parts and ship them to our dealers from the warehouse . One of our large -actually the largest division- cousins provided that service for us.

This was definitely a two edged sword. On the one hand, we did not have to worry about warehouse or computer systems and software. We paid them a fee and they did it all.

The other edge of the sword was that since we were only about 10% of their warehouse operation we found it almost impossible to get modifications to their programs and processes that better fit our customers needs.

One of our request to them was to take the cumbersome and difficult phone-line connected terminals in our dealers operations and put them on the web.

We discovered that they had already started that project and that - even after the software was installed and tested- it would take several year to get it in their own distributors locations. Under the best of conditions our division might be waiting 5 years for something we needed immediately to remain competitive.

I listened to their well-reasoned Corporate Strategy. I returned to our headquarters and immediately got permission to launch an off the radar, low key investigation of a less complex solution that we could move out to our dealer locations on our own in a year.

Internet technology was just taking hold in corporate America at that time. So, credit goes to my management because they were willing to risk a little 'cutting edge' technology to get a leg up on our competitors.

First, since we were free from our big cousins technical rules, we decided to build an internal Internet. Back in those days that was a private Internet on a closed network that was not on the World Wide Web: an Intranet.

Then we found a company that would go around to our dealers and quickly set up the hardware, software and training for each dealer. Our big cousin had an entire department to do that, but they were already stretched too thin to be of any help so we went outside the bureaucracy.

Next we looked at software. We were faced with a multi-thousand line code system , an IBM mainframe based system on green screen terminals. This system had hundreds of man-years of development behind it. There was no way my guerrilla team could replace that. And because it was the backbone of our corporate cousins operations there was no way they would give us access to modify it (and I don't blame them for that decision- very wise one).

Since we couldn't get our hands on the guts of the system, we decided to 'clone' it and piggyback on their well tested and developed programs.

We found a piece of software that could take those ugly green screen terminal pages that our dealers now faced and present them as windows/mouse applications. Basically, we had a web page of an ugly main frame system- with the advantage that no one had to touch the code of the mainframe.

Quick and dirty? Yes indeed.

Do the job? You bet.

In one year we had all of the North American dealerships and all their branches on a web site to order parts directly from our bigger cousins warehouse.

At a presentation to the management of the divison that owned the mainframe and warehouse after we finished all of our implementaton, they were astonished to find that we had the entire North American dealer base on this intranet. Sometimes, bureaucracies are not even aware that this kind of results is possible.

This system worked well and continued to function while the corporate people decided on "Web strategies", proper design, fancy screens, etc. All the things that make a really top notch Corporate Web Site.

There is a message for systems designers in that story.

Sometimes, when the business imperatives are strong enough, you can provide a smashing solution which will carry the day by simply reducing the problem to a few small and attainable goals.

Why did this one work? First, and foremost- the team was empowered to do what it took as long as it did not jeopardise our business. Second, it was under the radar screen of the people who delight in enforcing constrictive rules that are more bureaucratic than business focused. And, finally, we had management that recognized how the boundaries of the big sister division were crippling our business.

The epilogue to this tale is that when the '5 year plan' corporate web page came to life, they were forced to use most of our solution as the central part of it for several more years because they had still not solved all of the issues of moving off of the mainframe computer.

Horse to water II

Well, here's a different horse but let’s lead him to the trough and see what happens.

Part II is the good news story about technology:

A major international company had bought an extensive, but specialized warehouse distribution company in Chicago. I was invited to look at it and determine the state of their automation and systems and provide a systems 'vision' for where to go with their systems.

The distribution warehouse had been a division of an Asian company, and overall was pretty technically savvy, but they did not have the automated tools to work well with their customers in the U.S.

The company that had purchased this distribution warehouse wanted the systems to at least parallel their own so that at some future time they could be converted to the existing software.

The first step would be fairly rapid and provide some enhancements as well as building a base for a more ambitious second step to follow. My task was to design a cost effective stop-gap that could easily be converted to the purchasers internal system.

It was a pretty straight foreword assignment. And I gave them a presentation on a Vision Statement" pretty quickly. Everyone was satisfied with the estimated cost, timing and technology of the project and by the break for lunch that day it was decided to proceed.

After lunch, we returned to the conference table for a wrap-up. Just as I thought we were done, the European manager of the company asked me if the same approach could be used with their internal products in Europe. They had many problems that paralleled this newly acquired warehouse- multi-lingual, multi currency (before the Euro), equipment engineered to different countries requirements etc..

I asked for a break. Everyone went for coffee. I did my old ‘staring out the window’ trick. One tool in my bag was the ability to develop a vision of where I would like a system to be, and then deconstruct the vision into practical project steps.

When everyone returned from the break, I got up and did an off-the-cuff ‘chalk talk at the board. There were questions and comments- then the European manager asked, “How much would it cost”.

Twenty five years of experience (at that time) had demonstrated time and again that systems had three- almost equal- cost components. Software. Hardware. Programming implementation and personnel costs.

I had recently seen a piece of software that – while probably not the one they would buy – gave me an approximate cost. I figured (on the back of an envelope- as they say) the hardware costs. And! Viola! My ‘rule of 1/3rds applied was holding- so I added in the final third for people and programming and I gave them a number.

The European manager raised an eyebrow. My estimate was several million dollars higher than anything he had budgeted for systems over the next few years. But, he was profuse in thanking me for developing a realistic vision and being frank in its potential time and costs.

Whew. Didn’t fall into the tough and drown.

Two years later, the same European Manager called and invited me to a meeting in Paris. They needed to get serious about my ‘vision plan’ and wanted to get down to details. He had, obviously, forgotten how we came to that vision. Because he asked me to send him all my ‘data’ with time, costs, etc. so they could prepare for the Paris meeting. I had to remind him that this was an off-the-cuff, but experienced guesstimate. I pointed out that he already knew everything about the plan that I did. There was no paper trail.

So we agreed that would be the first item on the Paris agenda.

As the Paris meeting progressed, it became evident that the headquarters people of this major international company were about to enter into a contract with a large software supplier that- to protect the innocent- I will call the TBQ software package. This package was notorious throughout the industry for being way over ambitious, way, way expensive, and usually requiring 2 to 4 times the initial estimate of project cost and time.

When I heard this might be forced on them by the international management, I took a piece of paper from my notebook and wrote the figure $10 million dollars on it. Someone asked what I was doing and I told them, “I have just written down the minimum this will cost you with that company”, I didn’t tell them it was over twice what I had originally proposed.

I told them that I would reveal the number to them once they had a bid from the TBQ company. I was trying to give them some ammunition to go back to their management and object to being force fed a software company they neither wanted nor needed.

At a presentation of time and costs by the TBQ company some time later at the Corporate Headquarters in Brussels turned out to be a shock to the managers and no-news to me.
The initial costs was between $9 and $10 million. But, it ‘could’ go up- TBQ was always careful to hedge their bets.

After the software marketing team left the room, I pulled out my piece of notebook paper and passed it around.

So, the good news on this is that I led that horse to water (technologically) but made them aware of the potential of drinking from the trough. They knew exactly what had to be done and the cost and time to achieve it.

From that point forward, they were diligent and persistent in getting the system they wanted. They had fully absorbed and understood the potential dangers.

Moral to this “horse trough” story is that if you can prepare your users understand the technology and its costs they will be able to prevail even when they don’t have total control of all the pieces.

This Project Team was eventually mandated to use the corporate -purchased software package. Software the Team did not want. But, at least they knew from the start of the project that they would have to force the software vendor to make modifications to meet their needs instead of just accepting what the supplier offered as standard.

Applying 'off the shelf' software that 'almosts' fits your needs will guarnatee that the cost, time, schedules, and patience of management will be fractured by a factor of 2X to 5X.

But, allegorically, I led them to this trough of not too clean water- and they quickly perceived that to 'quench their thirst' they would have to drink carefully.

Horse to Water I

The full title of this piece should have been, "You Can Lead A Horse to Water,
Just Don't Let Him Drown in the Water Trough".

This is a good news/bad news piece about expectations and delivery of systems.

So we can end on a happy note, I'll start with the bad news. I can (with tongue in cheek) title this episode “How I got screwed with a screw”: In part duex, I'll tell a good news version.

Our company was having financial problems. Mostly losing customers to lower priced competitors. I was not surprised when I was called in one day to meet “The New Division Manager”. The corporate folks had brought in a ‘turnaround expert’.

Literally translated in business jargon that means, whack and cut costs with a chainsaw (mostly people) and get the company profitable so that we can sell it.

One of our products was a heavily engineered, custom built device called an Archimedes Screw, This was a shaped metal device that turned like a barber pole, but which moved ‘stuff’ from one place to another. A very important piece of machinery if you processed little things like pet food all the way up to big things like rocks and gravel.

The orders for these screws were time and labor intensive. Read that ‘costly’. Not only that, but if order input got heavy – and that would have produced more revenue- the backlog of engineering work to be done would mean that we would be promising delivery in months rather than weeks. Read that as ‘lost revenue’. The traditional response from buyers was to go to a smaller competitor who could delivery quickly. Read that ‘lost market share’. No company executive wants to hear ‘lost revenue and lost market share’ in the same sentence.

Back to our company version of General Sherman slashing and burning as he went.

He threw this problem in my lap and suggested that solving it was a good way for IT to ‘pay its way’. Read that as “maybe we’ll only cut half your staff instead of 2/3Rd's”.

This was in the very earliest days of Windows and PC’s. The first version (Release 1.0) of Visual Basic had just been introduced. So I rounded up a couple of the people in the IT Department who had lots of product experience and long company history and we went off to a conference room to solve the problem.

Sounds simple enough. And if you look a the problem from ten thousand feet it probably looks that way: Figure out how they bid the product. Make that into a flow chart. Program it in Visual Basic and take advantage of the new Windows environment. But when you get down on the ground and start doing the real-world stuff, it gets messy.

First, I said Windows technology was new. So if you take staff whose only systems experience is on traditional main frame computers with dumb terminals and ask them to design an interactive Windows system- guess what?

It will run on Windows and look like windows, but it will behave like an obsolete ‘dumb terminal’ system. First order of business was to get everyone in tune with interactive design instead of sequential steps from point A to point X. (This totally begged the issue that we had the first release of a new Microsoft product which had its own challenges).

With all that as a background, we grabbed a copy of the order request form from Engineering. Looked simple enough. Step 1, step 2,….etc, Except that we found that the engineers did not do the steps in sequence.

There were some items calculated at the end of this order form that were done first. For instance, you had to calculate the ‘tons of product moved per hour’ before you started the design to make sure you could both build the custom piece and that it would not fold and crumble when used in real life.

We immediately realized that the Windows program was not going to be so simple as putting the filling out of the form in a PC environment. We started grilling engineers and came up with the ‘real process flow’. Not the one on the order form. It turned out that the order form was nothing more than a convenient way to keep all the data on a customer order in one place. It was not a road map on how to build a screw.

In actuality, and I have seen this often since then, there is little relationship between data on a form and the sequence and process by which it is gathered and entered. Just as a 'for instance': Some things are so common and well known that no one ever writes them down on the form. Also, in a manual system, there are often notes written in the margin that are essential, but not a formal part of the processing.

We were under severe time constraints as ‘General Sherman’ was at the city limits and taking no prisoners. So we made (In hindsight a terrible idea) a decision to build a mock up to show him. He was a person of no patience and little time for any individual project, so we knew we had a very brief opportunity to lead this 'horse to water'.

Getting his immediate attention (and approval) in the quick interview he allotted us meant that we would build a system he already '"bought in to" and not show up all of a sudden in a month with something he hated and lost the months design time as well.

Sounds like a plan.

When we finished, we had a completed Windows interface that could be exercised using fixed data we built into the program to demonstrate 3 or 4 examples of our most difficult products.

That enabled him to see the flow of the design. The visuals we would use. The convenience of point-and-click, etc.

For a first time out of the starting gate for new developers- it had a pretty good ‘Windows look and feel’. We felt it could be given to our field sales guys (no PC experience at that time) with laptops and they could actually place finished order straight to production t instead of long lead time RFQ (request for quotes) that could take weeks to get out of engineering.

Just what the doctor ordered. Cut the lead time, up the sales successes, and squash the competitors. What could be better?

Full of boyish enthusiasm, I pitched it to'General Sherman'. He immediately saw that we had an Aladens Lamp to make all his increased sales dreams come true.

We had truly led that old war horse to water.

And that is when he threw us in the trough and we had the ugly sensation that we were about to drown.

His next statement was-“ I want you to get these out to the field at once”. And then he left the room. On his busy schedule, the meeting was over and the die was cast.

Oh, that’s just swell. I show him a conceptual prototype and he thinks it is ready for the big leagues.

This prototype model was based on fixed data to demonstrate the design. There was no computerized Engineering database anywhere in the company. The data used on the forms was in books and extrapolated from charts.

Designing, verifying and loading the database- another biggie could take 6 months or More (depending on how many engineers could be put on the task).

So my marching orders went from an ‘atta boy’ to “ready Shoot Aim”. I could feel the dirty horse trough water filling my lungs.

Moral of this story: Be very careful in handling technology. It is kind of like a nuclear weapon. It can be a strong and powerful tool in getting things accomplished with those in power, but if it ever runs amok, it can start a chain reaction that will consume you.

Next blog we’ll look at how to lead the horse to water and make damn sure the water is ready to drink before the horse takes a sip.
.

If it ain't broke- don't fix it: And, how to know it ain't broke.

I was doing a consulting job for a company that wanted to install bar code on all of their inventory and products. We live on bar code today. Every time we self-check in a supermarket it is the bar code that makes the whole idea work.

But there was a time when bar code was new- and like all new technology- very shaky.
But, the leading edge companies (or those who wanted to be) saw that the way to get a competitive leg up was to get a jump on the new technology.

That is a recurring theme in this blog. Companies wanting to be on the edge of the new stuff, and how it more often than not gets them in 'the stuff'.

So, I made a presentation to the Board of Directors. Now, as a "Consultant" I should have been looking for a way to get a juicy contract out of these guys. The Chairman of the Board liked me- and the rest of the Board didn't know or care as long as the Chairman said it was OK.

Lesson #1, Be careful of any group that "doesn't care what system you install". In this case, the Board was more than willing to participate in a great success and also cut-throat enough to make it "The Chairman's stupid idea" if it failed.

I had done several bar code systems before this. That was part of the reason I got to do the 'Feasibility Study' part of this project for the Chairman.

Much to his chagrin, I spent my allotted hour with the Board of Directors reviewing the 2 most unsuccessful bar code projects I had done. I spelled out in bloody detail the cost, the carnage, the failed deadlines, and any other bits and pieces of the projects that reeked of disaster.

I put my little laser pointer in my pocket at the conclusion and sat down.

The room was silent.

Some members of the Board with a vendetta for the Chairman, saw a golden opportunity slipping away.

The Chairman stood up, shook my hand, and asked if i would like to come to work for their company. They had been hosting a series of other consultants, vendors, hardware suppliers and other leeches who had drawn a picture of paradise with bar code.

The Chairman was smart enough to know that the vendors and suppliers had whipped his staff into a frenzy that could not have possibly been as easy as the sales and marketing pitches had made it sound.

Lesson #2. I have worked with salesman who knew what I was trying to do and worked for the long-term good of the products they sold. They are a blessing. If you cultivate them they can get things done for you sometimes when the deadlines get tight. But I have worked with vendors and their marketing staffs who are interested in maximizing their income. They will 'deep six' you in a heartbeat. figure out early in the initial discussions of a project what motivates your vendors.

So, was this a case of a company that had a system that was broken- and was I just holding them off from diving into the deep end with what was (then) new-fangled technology (i.e. bar coding) and reaping the success?

No, what I was responding to was their own environment and what I had found in my Feasibility Study.

I spent 6 weeks walking around and talking to people in their operation. Folks on the production floor. The finance guys. Managers. One thing became clear in all these chats:

This was an old-line company that had very little data discipline. Huh? How do you spank data?

When you bring in a computer you introduce black and white exactness into everything you do. And that exactness becomes an unquestioned part of the business culture in a company.

More on the bearing of cultural environment on success/failure in another blog.

I was having a chat in one of my interviews and the person did nothing but complain about how much work the computer had created for everyone since it was installed. They pointed out a "for instance". they still had to cut and pasted mailing lists from computer printouts- instead of having the computer print them.

Boy, I hear you saying. that is really el stupido.

Seems that when people sent them coupons, request for info, etc. the names were entered in a computer file. Unfortunately, no one ever setup a routine discipline to be followed. So some were in the computer first name then last. Some last name then first. AND just to make it perfect- some put first and last on the same line in the computer.

The funny (NOT) incident that made me realize that this company was far from ready was a request for information from a church. Their name of the church was something like "Christ the King"...blah blah blah.

Murphs Law kicked in. Needless to say, it got entered incorrectly. The computer generated form letter to them- instead of saying "Dear Pastor Jones"- stated with "Dear Mr Christ". Neither the church nor the Chairman of the Board was amused.

My message to the Board was summarized like this: There are ten steps to get a successful bar code implementation. When you get to step one- call me and we can talk about what you have to do for the other nine.

Sometimes a company can go along doing business with systems that have served them well. They may not be pretty, but they work. The idea that a company with data chaos is going to get catapulted ahead of the competition by state-of-the-art technology doesn't sound like such a good idea to me.

Many vendors and suppliers will tell you otherwise. That's how they get wealthy. That's why it is up to the systems and project guys to get this idea to the top people early in the game.

You can't take a company that is flying hang gliders and send them to 'Top Gun' school and expect them to get in a supersonic jet and do anything but crash and burn.

The key to giving this company good advice was to recognize that it wasn't their system that was broken. So why bother to fix it.

What needed to really be fixed was the overall ability of the company to implement ANY system. Before they coould even start on the path to bar code, there was a lot of basic systems work to be implemented and inculcated into the companies culture.