About Software Requirements…
Software projects may be started any day of any year. There is no special day that could possibly influence the outcome of the project. What is important is to understand and follow some basic software engineering principles. That said, the ability, know how and experience of the team members always tends to play a significant role in the outcome of the project.
In order to describe some basic software engineering concepts I will provide with a set of scenarios and recommendations which if followed should make most (never say all) software projects a success. So why most software products and projects are a failure? Read along and you should find the answers for yourself.
Software development is both and science and an art. As a matter of fact, most (as was previously stated, never say all) engineering disciplines are both a science and an art. For example, in civil engineering, an architect might design a structure (i.e., building, bridge, etc) which should be strong enough to support the stress and fatigue of it’s intended use and last for the expected / specified lifetime. Structures are designed and built differently based on location. A four-lane bridge for general traffic (automobiles, trucks) is designed and built differently when the intended location is Southern California than when its location is somewhere in Maine. One of the sites the bridge is prone to experience rather warm temperatures, lots of traffic, earthquakes, strong winds, etc, etc, etc. The counterpart in Maine might expect lower temperatures and perhaps salt corrosion if the structure is near or on the water. Both bridges will share some basic requirements but they also have unique ones. Being a layman in civil engineering, I can understand how requirements for structures will differ. That does not make me a civil engineer nor gives me the ability to design one.
Algorithms are at the core of any software development project. You use your favorite web browser and look up the definition. Most people would just type the word “algorithm” and follow some of the links returned by the search engine of their choice. I tried Google and at the top of the search I found a link to Wikipedia and a dictionary. A dictionary tends to provide a basic definition that is good enough to understand the term (e.g., a set of rules for solving a problem in a finite number of steps, as for finding the greatest common divisor). If you look at Wikipedia you may find something like “An algorithm is an effective procedure for producing a well-defined function”. You need to realize that Wikipedia contents are generated by contributions of many people who in most cases do not have the experience or background or have some agenda to contribute. Personally I seldom look at the contents of Wikipedia. If I do, I check references and then narrow my search to reliable sources (typically papers and books from accredited universities). I have taken several courses on computer algorithms and have also read several books on the subject. It just happens that this quarter I have registered to attend a computer science course on advanced algorithms at the University of Minnesota. The book for the course is “Algorithms” third edition by Thomas Cormen, Charles Leiserson, Ronald Rivest and Clifford Stein. I just happen to have contacted Dr. Rivest years ago when I developed the first content addressable storage regarding possible drawbacks in the use of MD5 digests to be used as names / handles for files. Dr. Rivest is an expert in software encryption and hashes. The book defines an algorithm as follows: “Informally, an algorithm is any well-defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values, as output”. Hopefully you are able to see the nuances in the definition and be motivated to delve further into the subject matter with the invitation by the initial definition presented in the book. Please note that by reading a good textbook you will not become an expert in the subject. You also need the experience and hopefully you will be able to come up with better than average elegant approaches (algorithms) to solve software issues that you may be confronted with in a project.
MapReduce is allegedly the mechanism / technology / algorithm used by Google to manage huge amounts of data. Allegedly Google has developed a proprietary mechanism (Google’s patent #7,650,331) to manage large sets of data using a distributed architecture. Google does not make use of a SQL database (i.e., SQL Server, MySQL, etc). Apparently Google uses no-sql. By reading books that describe to some extent the technology, MapReduce is nothing else that the implementation of the divide-and conquer paradigm that has been used for decades in multiple algorithms such as merge sort. Being able to file a patent does not require an elegant or unique algorithm. As the Google patent demonstrates, one is able to patent an old idea just applied to a different scenario (in this case a distributed system). Given that America is becoming a very litigious society, it is the best interest of companies to be able to fill their shelves with patents. The ability to do so tends to be a function of the quality of the methods / processes developed by software engineering teams.
Computer Science (CSci) as its name states is a science backed up by mathematics. CSci is also considered art. Once again, going back to a dictionary, art is defined as: Works produced by such skill and imagination. When viewed under this definition, CSci requires both education and experience. We all have reached some level of formal education. When studying any science or mathematics, we first read the textbook and attend lectures on a subject. In order to solidify our knowledge we perform some exercises and tend to solve some problems. Sooner or later in our professional careers (assuming you exercise in some technical, scientific or mathematical field) we have encountered problems that due to our education and experience have been able to recall one or more techniques that when applied to the situation at hand, tend to solve the issue. Based on education and experience some solutions are more efficient (consume less resources) and elegant (simple to describe, implement and maintain) than others. I have made on multiple occasions reference to the popular idiom “There is more than one way to skin a cat”. As I always say, “yes there is, but of the entire possible set, there is one that is worst and one and only one that is best”. The ability to most of the times produce solutions that lean towards the best side is the result of education, experience and ability which can easily be described as art.
“The Art of Computer Programming” is what one may say another example that computer programming is not only a science backed up by mathematics, but also an art in itself. Donald Knuth has written (and as far as I know he is still doing so) a collection of books that deal with the description and analysis of may computer-based algorithms. Professor Knuth started work on the book set in the 1960’s. As I recall in the 1980’s most respected software developers and computer scientists frequently referred to the works of professor Knuth searching for algorithms that could be used when confronted with a new design. I have owned a couple copies of “The Art of Computer Programming”. The main reason for this is that the set has been growing through the decades. I currently have this set of book at home on a desk I use to study, read and think. Perhaps this is an indication that I am not a young pup.
Most of us at several points in time (some more than others) get ill and have the need to seek medical attention. Depending on the severity of the ailment, we make an appointment over the phone with a clerk. The day of the appointment we show up and a receptionist collects information and typically the co-payment for the visit. After a few minutes a nurse leads us to a consulting room making a quick stop by the scale. After taking our temperature and blood pressure we are asked a few questions as to why we are interested in seeing a general / family medicine practitioner. Finally we see a physician. If the ailment is obvious the physician prescribes some medicine and out we go to the pharmacy. In most cases after a few days we are feeling well. If the issues affecting us are not than obvious, we get sent to the lab for a battery of tests. Typically some tests are of chemical origin (sugar level, white cells count, etc) while others may be imaging (i.e., x-ray, CT, MR, etc) using some form of x-ray source. The results might lead to a diagnosis or the issue might be elevated to a specialist. If the issue requires surgery, one would be referred to a surgeon. It is clear that just being associated with a healthcare organization does not qualify an individual to perform brain surgery. The same holds true for software development. I know of a receptionist that used to make minimal wage at a software development company. One day I was her reading some “For Dummies” book. A month later she got a raise that increased her hourly rate by a factor of five (5). She was now a software developer.
If you watch some television you must have seen a commercial for a prescription drug used to treat / prevent acid reflux. I watch (typically sleep in front of the television screen) television about an hour a day. Around 08:00 PM, based on workload, I tend to start shutting down my computers. I typically run three (3) different computers. Given the resources available on the different computers and their intended uses, it is not technically or economically feasible to virtualize the dozen machines that I have at home. That said I do run VirtualBox on a couple machines mostly to access a few VPNs (Virtual Private Networks) that have side effects on the network and to run different operating systems (Linux and different flavors of Windows) on the same hardware. Getting back to watching television, it was not until a few months ago that for the first time my wife and I, at insistence from one of my sons, that we got DVR from the cable company. Now we record a few programs during the day that we might watch when we are available to spend an hour or so in from of the flat screen. No matter what my wife and I have going on during the day, on workdays around 08:30 PM, we watch World News with Diane Sawyer on ABC News. If you watch television around 05:30 PM, it becomes obvious the demographics watching based on the type and amount of commercials. In particular there is one that start with an actor wearing a white physicians coat attempting to use a jack hammer on what appears to be a road construct ruction zone in a downtown area. The person does not fit in the scene. In the next scene a husky male, wearing blue jeans, boots and a tool belt is wondering in what appears to be a pharmacy aisle looking for medication to cure the symptoms that he appears to portray (heartburn / acid reflux). In a following scene the physician is discussing with the patient the symptoms, which lead to a prescription for the pharmaceutical in question. The moral of the story / commercial “you do your job and let the physician do his” or something among those lines. Perhaps I should get a lawyer and demand some type of royalty from the AtraZeneca Nexium commercial.
One can easily draw an analogy between technicians, engineers, system managers, software developers, computer scientists, QA, chief technical officers, chief information officers, IT managers, and many other people whose job is associated with computers and / or software at different levels (users, super users, system administrators, maintenance, developers, analysts, scientists, etc) whose jobs at different organizations are associated with providing computer and / or software based services / products. Such organizations tend to have receptionists, clerks, supervisors, managers, etc, etc, etc. An individual just by belonging to such organizations do not qualify to design software. There is of course the exception when the person reads a book “For Dummies” and in a couple weeks is developing software.
Requirements are extremely important. They should unambiguously document the use of some software to be / being developed. The idea is quite simple. If the intended use of a software product cannot be unambiguously documented then no one knows what to do. If no one knows what to do there is no way to generate a design or a set of test procedures to test the software because no one knows what the software is supposed to do. This simple logic is seldom put to work. Most software projects are started and in many cases completed or canceled before a useful requirements document are generated. Different methodologies have different ways of generating and maintaining requirements. Most methodologies are flawed. Some make sense but are not well known or their steps are not easy to follow. One may be led to believe that there is no good software development methodology. This is far from being true. As an analogy one can take diets. Not sure how many diets are available. Some seem to work for some for some periods of time. Most fail miserably. The fact is quite simple. Our bodies are engines. Engines need fuel. If our bodies burn 1,500 calories per day and we intake 2,000 we will just be gaining weight. On the contrary, if we change our life style by incorporating exercise and reducing our caloric intake, then if we consume 3,000 calories per day but due to exercise and other healthy habits burn 3,000 calories in the same time period, we might not live one hundred years, but on average we will have a longer and healthier life. Like I said, there are many ways to skin a cat but there is only one way which is best. Burn the same amount of calories as one consumes on a daily base and forget any other type of diet. In reality software development is several orders of magnitude more complex so things are not so simple as a diet. What is extremely important is to have unambiguously defines requirements written in a succinct way. Writing a Victorian novel is not the proper way to write software requirements. The detail and number tends to be a function on the level of expertise of the lead system architect and the development team as a whole. Being involved in many software development projects and having knowledge of different methodologies attain this level of expertise. As we all know, no matter how the metaphor is applied, there is never such a thing as a free lunch.
Design by committee is a term that is typically used for both design and requirements even though these are two separate aspects (requirements and design) of any software development project. A good designer needs education and experience. A person without a software background is able to design basic software. Today wizards allow individuals with little or no computer science education to create software (specially when it is a GUI web based) not understanding or knowing what they are doing. Such approaches tend to work now and then form small companies. As the demands for the software grow, there are no wizards that can help. One needs to understand what is going on and be able to make the necessary changes or come up with an alternate solution. While developing software one runs into situations when the desired effect is not achieved with the initial approach. One then needs to look for alternate ways to achieve the goal. This process is based on know how and experience. In cases when one needs assistance, one should be able to look for it consulting with other team members or vendors. In many cases, the higher the position in a software development project, the least the person is willing to accept that help is needed. When too much involvement by the entire team on all aspects of the project is reached, no one is responsible for failures. Humans tend to blame others. When the software is properly partitioned and individuals are assigned to specific modules, developers tend to take responsibility for their part. This is especially true when a separate group does testing. That way the team as a whole responds to issue encountered by QA. If one looks at standards that have been developed by many individuals from different companies (e.g., DICOM), the results tend not to be elegant, bloated and the software implementations are poor (slow and consume lots of resources). When single individuals come up with something (e.g., Linux) the result tends to be elegant and performs well. This does not mean that a single individual developed Linux or that all members that participated in the committees that developed DICOM contributed. The rule of thumb is to assign parts or a project to a single individual and develop the system in such a way that all interactions are based on clients making requests and a server responding to them. This can actually be done with client / server interactions or via the use of APIs. The key is to make each module responsible for their operation based on the requirements for the software. Once again, being able to partition a project of any size requires know how and experience.
Lead by example is something that is not easy to do and even harder to follow, but when achieved, results are typically success. The idea is for the senior / lead system architect to show how things are done. It all starts with requirements and their associated documentation, through design with associated documentation, implementation and associated unit testing to product delivery and if needed initial support. Less experienced engineers, when properly motivated and interested in learning and advancing tend to imitate success. As usual this is easier said than done. If one has a single rather simple project under the belt and the software is supported by an army of technical service personnel, then that might not be the person to put in charge. The same holds true for people that work as system administrators and are asked to design software with little education and experience. Being able to issue database queries and use a report writer to generate reports with colorful graphics does not qualify the person to become a lead system architect. Once again, politics get in the way of reason and after a few years when the software is a failure people stops and thinks on what was done. At that time it is too late for the software and the project. It is better to address most (all is not possible) situations before they turn into issues and then into problems.
A couple decades ago I was involved in the development of the first document management system ever. The project started by processing documents for insurance companies and then shifted to address medical images. The general manager had a PhD degree in nuclear physics. He used to work for a well-known company (Xerox), which at the time had a quite known research center in Palo Alto, California. On paper this individual fit the bill to manage the project. This individual had quite an amenable personality. He might have been top notch in nuclear physics but had little understanding (to say the most) on hardware and software development. At the time, there was no off-the-shelf standard hardware or software to put together the product. Top executives at the fortune 500 company backed up the project. The first and probably largest mistake was assigning the lead system architect role to an individual from IT. He might have had experience with databases and report writing, but that was it. The lead architect had to experience or knowledge on hardware, networks, imaging, etc, etc, etc. As the project blew out its schedule, the perceived solution was to throw at it more money. With that extra funding additional resources were brought in. The project lasted several years. Towards the end, software QA was a 24 x 7 operation. Each day new issues were discovered. At some point in time after over 100M dollars (in 1980’s value) the original project was cancelled. Somehow after a few months a second pass for the project, under the same guidance was authorized. The system architect blamed the computer language used to program. The first pass used C so the second was done in Pascal. I left when the first project was cancelled. After a few more years (do not know the actual cost) the project failed and it was cancelled for a second time. The lead system architect was able to get authorization with very few resources to run a final and third pass. This time the software was written in Ada (do you know what Ada is?). Of course, the language has nothing to do with the success or failure of a project. After a year or so the project was finally put to rest permanently. What is interesting to note is that the lead system architect returned to IT to deal with databases and report writers. I ran into him at a seminar in a hotel in downtown Minneapolis. He was doing just fine back were he started a decade or so before.
Testing is important in any software development project. The same individuals developing the software cannot do testing. Developers must create the methods to perform unit testing. System testing must be performed by a different organization. Such tests must be developed following the software requirements. QA should never see the design documentation nor be exposed to the source code of the project. The issue is that testing then becomes a function of the actual code and not of the requirements.
For software to be successful it must be flexible. If the software works perfect for only the specified set of test input and it fails under different situations, the quality is not there. The software must be designed in a way that it can evolve not breaking previous uses and allowing for foreseeable changes in the business. Very few software modules are able to adapt to changes and survive a decade or two. When the software is properly designed and implemented with simple yet robust technology, the software product tends to be useful for long periods of time. When little though is put into a project, the resulting software tends to become obsolete in as little as a year or two. Some companies thrive in such models because they imply recurrent business. If you do not understand this, take a look at the different technologies provided by Microsoft. Most of them do not last more that a couple years. Every time they are replaced with something quite different with the promise to solve all problems. This is type of behavior / approach is known as the “silver bullet”.
Team organization is quite important for any software development effort. This is true from projects with a single person to ones with hundreds of people in multiple disciplines, which may interface with products / modules form other companies. Putting together a team is important in order to have a successful project and quality software result. Tem members should be selected based on expertise and track record. Getting back to the healthcare analogy, you would not want to have surgery performed on you by a receptionist just because you like him / her. You can always have some simple surgical procedure done by a general surgeon, but if a specialized procedure were required, only a specialist would be considered. Software is no different, yet people tend to go against basic logic and allow people with limited qualifications to take charge.
Politics in projects, no mater the size of the organization or the team (at least two people are needed to disagree but a single person with his / her own agenda qualifies) needs to be addressed and managed for a project to succeed. This subject appears to be foreign to most people, yet many books have been written about the topic. I have personally read a few. Software development is a human activity. We as humans tend to resist change and have preconceived ideas as to what needs to be done. This is just one reason for having a simple, short and complete set of requirements for a software project. By no means this statement patronizes the waterfall model. There are different methodologies better suited to collect and maintain requirements during a software development project. A properly managed project assigns team members to different tasks / modules based on their education level, experience and track record. I have run across several individuals with advanced degrees (PhD) that could not develop software even if their life depended on it. The issue is that humans tend to be polite and not tell individuals that they should be doing something else or they just tell other members their thoughts but no action is taken. This builds friction which when paired with lack of expertise, produces an environment in which success is typically not achieved. The best way to prevent political problems is to put together the proper team and assign tasks that are palatable to the individuals. By having quality requirements and proper architectures, team members can see how their contributions to the software make a difference. If a problem is encountered, developers are more inclined to look at their pieces and correct them when needed, than just blaming other modules (pointing fingers). Some times developers go the additional mile by making the necessary changes to their modules to compensate for the shortcomings of others while still keeping the integrity of the software.
After reading most of the contents of this blog entry you might come to the conclusion that the best software development team is a completely homogeneous group. This is not the case. Diversity is important. The issue is to make sure that diversity comes from backgrounds that may contribute to the project and that such individuals are not charged to lead. Different ideas should always we considered but when all is said and done, someone needs to make a decision as to how the group should proceed. During my professional career I have had the opportunity to work for small, medium and large size companies. I have also done contracting / consulting work. My background is a mix of electrical engineering and mostly computer science. I have worked in quality assurance groups and have supported systems at customer sites. That diverse background has helped me come up with innovative approaches to solve design problems. Some time ago I was working for a company in the east coast. I was in charge of developing a color console for a process control system. The company was developing the entire system. The system could control the environment in buildings or an oil refinery. The product was software and hardware based. My team had a PhD in psychology. Obviously the individual was intelligent. He was also very dedicated. Married without children, just a cat at home. Because of this, we were able to work long days and enjoy them. In a few occasions this team member would get stuck in a task. Without technical assistance he would spend several days in a rut. I recall one occasion when we were dealing with a driver for a UART. Given that his background was not in computer science or electrical engineering, the concept of how to control the chip to process RS232 input and output in queues at the interrupt level was overwhelming. After I notice that we were not moving forward, we got together and in a day, with a different design, the issue was addressed and we were able to continue to make very good overall progress. A year or so ago I was optimizing a file transfer for an application on Windows. The specific transfer operation was not intended for what it was designed. I spent a few days optimizing the command line interface using a similar approach as to the RS232 driver I worked with my team member a couple decades ago. Of course the issue was over the TCP/IP network and had multiple data buffers and queues and was working with multi core processors. The point is that knowledge and experience were brought up to solve two separate but similar problems decades apart. Diversity in backgrounds is most of the time beneficial specially when used the proper way.
There is a cryptography expert that has made his home the Twin Cities of Minneapolis and St. Paul. I have read a few of his books and periodically skim over articles in his newsletters. He has a friend / co-worker that has repeatedly made the comment (perhaps is more than a comment but a believe) that the software development industry is in shambles. This person claims that all software is full of bugs, insecure and not stable (and he is not just referring to the Windows platform). I have read some of his articles and based on my experience have to somewhat agree with his opinions. The matter of fact is that software, no matter which company is producing it, has many issues that can be traced down to all of the development phases. In general properly trained individuals with proper experience do not develop software or the team members and management have different goals. In most cases the goal is just to get a promotion and raise and move or to a higher position in the same or different company. Company managers in most cases are not interested in the quality of the product but on making money. This behavior is so widespread in our country that we have become oblivious to it. A few months ago the biography of Steve Jobs was published. When it became available, given the current set of books sitting on my nightstand, study desk and other places I do not wish to mention but most people should be able to infer, I decide to skip. My wife got the book for her Kindle. She already finished reading it. Several times a day I would ask about what she was reading or she would just volunteer comments. I feel like I read the actual book. Steve Jobs was not an easygoing person. He had several traits that I just dislike. On the other side of the token, Steve was a very passionate person as far as the Apple products and services. He always set his goals to achieve perfection. Although that was not possible, Apple products are far superior to Microsoft’s. This is quite obvious at the Mall of America in Bloomington, MN. On the first floor the Apple store is just across the Microsoft counterpart. I do not visit the Mall every day, but when I do it calls my attention that the Apple store typically has at least double the amount of customers. This is how typically taste tests are conducted. The subject is presented two (2) options and based on statistics, the best tasting food is the one more often selected. Without a doubt the winner is Apple. Good job Steve!!!
How to collect and validate requirements for a software product is just one more important part in the software development process. The same holds true with design, implementation, testing, documentation, etc, etc, etc. The design aspects, not covered in this blog entry, describe how the software product is designed to cover all the requirements. Basic design concepts and some techniques used in cloud computing will be described in a future blog.
If you have a comment / question please feel free to leave a message. I will respond and post it on this website.
The Naïve American
