“Agile Development” – The Software Industry’s New Meme

“Agile” as A Meme

Since the introduction of Agile Development in the early 2000s, it has increasingly spread throughout the software industry in the same manner a popular cult is formed; based on supposition, mythology, and rumor for which little real statistical proof has ever been provided.

A more accurate description of the growing popularity of this development paradigm is that of a sociological “meme”. Wikipedia defines the word “meme” using basically the same definition as found in the online version of the Merriam-Webster Dictionary, a standard in English studies, as the following…

“A meme is “an idea, behavior, or style that spreads from person to person within a culture”. A meme acts as a unit for carrying cultural ideas, symbols, or practices that can be transmitted from one mind to another through writing, speech, gestures, rituals, or other imitable phenomena with a mimicked theme. Supporters of the concept regard memes as cultural analogues to genes in that they self-replicate, mutate, and respond to selective pressures.”

The Wikipedia description goes on to elaborate on the transfer of memes in the same way that is found in biological evolution.

“Proponents theorize that memes are a viral phenomenon that may evolve by natural selection in a manner analogous to that of biological evolution. Memes do this through the processes of variation, mutation, competition, and inheritance, each of which influences a meme’s reproductive success. Memes spread through the behavior that they generate in their hosts. Memes that propagate less prolifically may become extinct, while others may survive, spread, and (for better or for worse) mutate. Memes that replicate most effectively enjoy more success, and some may replicate effectively even when they prove to be detrimental to the welfare of their hosts.^”

In other words, memes are transferred in a culture the same way that natural selection works in biology; except for the fact that with memes, which are transfers of information, they become successful not as a result of intelligent research and understanding by the Human transfer points but in somewhat the same manner that fads are often found to grow in popularity, through simple popularity of an accepted strain of thought.

Birth Pains

When “Extreme Programming (XP)” was developed prior to the Agile movement, it was defined in the fourth year of a five-year projected development timeline for the Chrysler C3 Payroll system, a system that was to be developed to replace the existing payroll processes. The people that developed the XP development paradigm thought they had found a “silver bullet” to software development with their dilution of planning and analytical design coupled with the new idea of “paired programming” mixed with a “Code First” strategy.  One would think that after working with such development concepts over the four years of their project, they would have been actually on to something. Not waiting for the final results in the form of a completed and delivered project into production, the “Extreme Programming” creators went out on tour around the US to popularize what they believed they had developed as the answer for speeding up software development in a time when technical management was coming under increasing pressures to get their deliverables in to production faster. These promoters elicited a lot of interest and the basis for a new software development paradigm was born.

Unfortunately, in the 5^th and final year of the project, the entire thing collapsed under the weight of these newly derived concepts proving that they were erroneous from the beginning.

Nonetheless, a new software industry “meme” had been born as a result of the popularization efforts by the creators of XP. This new meme would have found its way into extinction had it not been for the efforts of a new set of developers (with probably the same people who created XP) who came up with the “Agile Manifesto” promoting a more refined methodology where the best tenets (as they believed them to be) of XP development could be incorporated, while adding alternative and\or refined concepts to the “Agile” paradigm, which were seen as corrections to the original XP model. So with “Agile”, paired programming was an option and not a requirement but nonetheless still promoted. The “Code First” strategy was still followed while planning and design were diluted with the idea of collaborative development with the user community; another concept that has proven to have mixed results since many users really don’t care how their systems get built.

In fairness to the developers of both the XP and “Agile” techniques, there were two major, external forces that lent their influence to these developments.

First was the excited frenzy that surrounded the rise of the Internet as a commercial medium for transactional processing, which in turn gave rise to the “DotCom Bubble, where clueless venture capitalists and to some extent organized crime invested millions if not billions in an outrageous number of new start-up companies all organized around bringing new forms of consumerism to the Internet.

Such start-ups were under heavy pressures to develop new applications that would allow everyone to buy literally anything on the Internet from credit to physical goods. The “meme” born from XP made its way into these small firms who were all under the impression that they had found something completely new and unique with this new form of rapid application development paradigm. Combined with the inexperience of the new youth oriented culture of these new start-ups, the development techniques they employed against frivolous plans and goals that in many cases made little sense, all came together in the first software development business “bubble”, which eventually blew up in their faces pushing the newly minted “DotCom Era” into a sudden free-fall it never recovered from.

However, the idea of using the Internet for commercial processes was not as farfetched as the failure of so many companies made it appear to be. The Internet was certainly a viable medium to work with so everyone went back to the drawing boards to begin again.

Saved From Extinction

At the same time, the outsourcing of US technical jobs was becoming as much a frenzy in the early to mid-2000s as corporations saw the Internet as a communication medium that could be used for connecting technical departments to foreign counterparts in countries such as India and China that could provide similar technical resources for far lower costs than what could be found in the United States; or so it was thought as no one gave any thought to the costs in administration, management, quality control, and project control to name a few ignored areas in this rush to cut the bottom line.

This rising inequity within the US corporate sphere gave new impetus to US technical personal to adopt techniques that rushed out deliverables at greater speeds trying to prove that US developers were as every bit as good as their lowly-paid foreign counterparts. The result was that instead of maintaining the original quality of the US software community it instead slowly adopted the “Agile” paradigm allowing corporations in turn (though they were doing it anyway) to reduce or eliminate vital components to the development of quality software such as business analysts, system analysts, software designers, architects, quality control centers and the like pushing increased responsibilities and pressures onto software developers and engineers all with the ridiculous idea that software development teams could do it all.

Under such circumstances the “meme” that was born with XP found a new nesting place in the developing Agile Community of proponents. The result was the growing contention that software developers could do everything while also adopting in-depth knowledge of the businesses they were working in. Article after article appeared in the field’s technical journals proposing as much and technical managers looking for ways to reduce costs and speed up the delivery of their products “drank the Cool-Aid” giving this nascent meme a new lease on life instead of the extinction it should have experienced.

Today, hardly a technical site ignores this as one article after the other is published proclaiming the benefits of Agile, Team-Agile for enterprises and the new paradigm of DevOps, which appears to increase responsibility by including operations personnel as part of the software development team as well as their responsibilities. The industry’s propaganda engines have been on full throttle for quite a while and the deterioration within the software development industry has demonstrated this. Even Microsoft has fallen prey to this infectious disease of high-speed development without proper structural planning and analysts have recently made note of the deterioration in their product quality after switching to the “Agile” paradigm.

Nonetheless, in all the years that this meme has infested the software profession, very little hard-core analysis has been performed in the same fashion that long-term analysis of earlier software development efforts and the many impediments to quality product delivery had been analyzed. There are some articles pointing to the successes and failures of the Agile methodology, but for the most part they are few and far in-between with little to no actual in-depth discussion has to how the projects were actually implemented. People who have written such articles that accurately describe Agile as a failure in their own experiences are merely told by proponents that it wasn’t implemented correctly or some facets of the paradigm were not in included.

What Did We All Do Before “Agile”?

However, prior to even XP, the “bible” of software development (“Rapid Development”), was published in 1996 by Stephen McConnell, which outlined every major aspect on how to develop good, durable software products without all of the hype of the paradigms today. This book is still so popular it is still available on Amazon.com in its original 1^st edition publication. And prior to its publications, there were many examples documented of how good software projects were structured. In one such case, an entire book that included all of the framework source-code for a highly successful project for the state of South Dakota demonstrated how Visual Basic 6.0 was used for both the project’s desktop and web-based applications. The concurrent usage of this system was over 1500 users! “Agile” was not included just standard, well-defined N-tiered development processes.

Instead of looking deeply into the past for what actually worked, creators of XP and Agile only took a short glance and found its only target being that of the “Waterfall Approach” to software development demonizing it as the worst possible way to create new software while promoting the new concepts as the saviors of the software industry. And if one were to apply the “Waterfall Approach” to every type of software development project, that accusation would be quite correct. However, the way Agile is being promoted is in fact the same way that the “Waterfall Approach” is being demonized since Agile promoters see it as the panacea for all types of projects, though they have never offered any substantial proof in order to make such a justification.

The reality is that the “Waterfall Approach” is merely one type of development paradigm, which has been quite successful for the large, complex projects that it was originally designed to be used against. However, there are approximately 10 different types of software development paradigms that have been in use for years that the current crop of software professionals appear to have either ignored or never bothered to research. If they had, their theories on “Agile” would fall apart. To demonstrate this, here are the standard development models that have all been used to date in terms of software engineering analysis…

• Pure Waterfall
• Code & Fix (what many organizations still use to their detriment for all projects)
• Spiral
• Modified Waterfall
• Evolutionary prototyping
• Staged Delivery
• Evolutionary Delivery
• Design to Schedule
• Design to Tools
• Off The Shelf Software Implementation

Each of the development models above have their strengths and weaknesses and each has to be researched for its appropriate use for a new project under development.

Surprisingly however, the “Code & Fix” model is what Agile today mostly mirrors, which is fine for simple maintenance tasks but useless for complex maintenance and complex project development. In this former vein, Agile has proven somewhat successful but has found its limitations when applied to these other larger projects.

Nonetheless, like “Code & Fix”, Agile is being used for many different types of projects as a result of the effectiveness of the belief that “Agile” can be applied just about anywhere in the software development spectrum (an underlying foundation to the meme contention). And being as embedded in the software industry’s culture as it is, “Agile Development” has become the model for all such development making it not only a “cultural meme” in the profession but a dangerous one similar to what happens with cults; unquestioned belief.

Questioning the Dogma

There are those who have questioned the promotion of the “Agile” model and have done so quite well, though such people are ignored in today’s conformist society since for most it is now too difficult to do any real critical thinking on matters within a culture that promotes sound-bite thinking.

Capers Jones, a highly respected software engineering analyst, has an excellent paper that demolishes the concept of paired-programming, which can be found here.

In his paper, Jones demonstrates the faulty economics that paired-programming forces on technical organizations as well as demonstrates statistical analysis showing the failures in such a development technique.

In another well written paper by Lajos Moczar and published in CIO magazine, he demonstrates that “Agile” simply lacks any common sense in its approach to software development. The paper is short but concise in its presentation and can be found here.

Finally, in a third paper, written by Alex Papadimoulis, “Agile development” is compared to building a pyramid sideways. The comments are as interesting to read as the article as you will find both supporters of Alex’s contention as well as detractors. Admittedly, the concept is a little far-fetched, but the author makes some very good points.

And not surprisingly, such papers and subsequent analysis today is rather difficult to come by as it took quite a while to dig these up in research. Instead, there should be a lot more out there with similar research and points of view but there isn’t as “Agile” thinking has literally taken over the psychology in the business application development corner of the software profession. And anyone who tends to disagree with this is basically told they don’t know what they are talking about.

This process has also shown the same deterioration in critical thinking in software development that has been shown in many aspects of US national sociology, especially education, as demonstrated in well documented, sociological studies. With mobile-computing taking center stage in most popularized software development today, one needs no longer the critical skills in complex system development that were once the hallmark of professional US software developers and engineers. Instead, mobile computing requires a rather limited of skills in order to build non-complex applications considering that only so much can fit on a smart-phone with only a proportional increase in complexity for tablet-based software. And yet, what is actually being developed for these devices?

In fact, if one were to take a look at the Windows App Store and its breakdown of stored applications (which has probably the same amount as the Android Store has), practically 90% of such applications are devoted to games and entertainment leaving one to wonder exactly how necessary such “junk consumerism” is to our national, psychological health.

“Agile”, which has done a rather good job in reducing software development processes to nothing more than sound-bites in terms of any real discussion that surrounds it has in fact seen a number of what appear to be well designed documents discussing various aspects of “Agile”. However, the graphics used in these documents all of which have shown nothing more than photographed scratches on paper for diagrams of the intended topic appears to demonstrate the rather casual approach to the serious subject of software development that this paradigm supports. One would think that the authors, if they were serious about their subject matter, would have created better graphics that supported their contentions. It is these images that tended to stand out as if testifying to the idea that “Agile” is as casual as the images appear to be.

The “Agile” meme, which can now be found in just about every aspect of modern, business, software development may in fact be quite supportive of the fluff that passes today for real software projects as is shown by its success for simple maintenance tasks. However, until we see real, critical analysis on a far, wider scale regarding the “Agile” development methodology we can assume that its realities are being ignored (or hidden) while the entire business development aspect of the profession is seduced into believing that the “magic silver bullet” for software development has finally been found.

It is a telling tale of why wisdom is wasted on the old and youth is wasted on the young…