Agile software development has been around for over a decade. While many now consider it to be a mainstream approach, there is a need for deeper understanding about how best to make it work for organizations.
To gain a deeper understanding of the key drivers of success, challenges and relationships that various factors play in the successful adoption of Agile, Intelliware designed a survey on the state of Agile adoption in organizations today.We surveyed 117 individuals representing a broad range of industries, job roles, company sizes, and experience with Agile. Given our local presence in Toronto, Canada, the majority of our respondents – approximately 85% – were in Canada. There are several key findings from this survey:
- “The ability to adapt to changing priorities” is the most common benefit of Agile.Close collaboration leads to more adaptive organizations. There is a significant positive correlation between the Agile benefits of “greater collaboration” and “ability to adapt to changing priorities” with an organization’s success with Agile.
- Organizations want to adopt Agile, but often aren’t having a great deal of success. There’s a significant gap between the importance organizations place on Agile and their capability with it.
- Augmenting your existing team by hiring full-time staff with Agile is only part of the equation. Agile experience alone cannot be relied on, as we found that experience drives Agile capability but only to a limited extent.
- Lip-service doesn’t work. “Approach – using Agile elements in a non-Agile framework” was the most frequently selected barrier in our survey; 24% of respondents stated this as the lone barrier that is inhibiting further adoption of Agile.
- Lean Startup is driving Agile adoption, but it’s mostly with organizations that have been practicing Agile for some time.
Agile software development practices are becoming increasingly prevalent these days. Articles and blogs abound discussing the pros and cons of this approach to software development. In fact, now the term “Agile” is being attached to all sorts of other business processes, ranging from business strategy to marketing*. To gain a deeper understanding of the key drivers of success, challenges, and relationships that various factors play in the successful adoption of Agile, Intelliware designed a survey on the state of Agile adoption in organizations today.
This survey and corresponding analysis is intended to provide a general benchmark of the state of Agile adoption. We asked 6 broad questions:
- How long has your organization been using Agile software development methodologies?
- How important are Agile practices to your overall business strategy?
- How would you rate your organization’s Agile capabilities?
- What benefits of Agile practices are important to your organization?
- What is inhibiting further adoption of Agile practices in your organization?
- How much influence have “Lean Startup” concepts influenced your organization’s desire to adopt Agile development practices?
The survey was short enough to help generate a sufficient number of survey responses but long enough to capture enough data that would allow for a meaningful analysis. These questions provide the opportunity for us to get a better understanding of the business drivers, capabilities and challenges companies are facing in their adoption and use of Agile practices. We also included a question on Lean Startup, since this method is gaining traction as a viable alternative to traditional product development and employs several Agile practices. To add to the analysis, we wanted to learn more about the companies represented by the survey respondents. We researched the respondents’ companies to identify company size and industry. Company size was categorized by number of staff: Small-to-medium sized businesses (SMBs) = <100, Midmarket = 100-999 and Enterprise = 1000+. While we saw some variance by industry and company size, our sample size was not large enough to report conclusive findings based on these variables. Throughout the report, we refer to several statistical techniques. For definitions of these techniques, please see the Appendix.
Over the course of 7 weeks, we surveyed 117 individuals representing a broad range of industries, job roles and company sizes. Given out local presence in Toronto, Ontario, Canada, the majority of our respondents – approximately 85% – were in Canada. At a high level, respondents represent:
- A variety of job roles and companies of all sizes
- Multiple industries, such as banking, insurance, ICT(information, communication and technology), retail
- Organizations with varied Agile experience. On average, respondents represent companies with just over two years of experience with Agile
Based on the results, there is a gap between organizational capability with Agile and the importance of Agile to those companies. In general, people do not believe that their Agile capabilities are on par with Agile’s importance to their organization.
We believe that importance can be seen as a proxy for expected results from Agile. Therefore, this could also have been interpreted as the gap between the desired outcome from Agile practices and the ability to achieve this outcome.
The following pages provide an overview of this gap as well as an exploration of the influences of organizational capability based on these 5 variables:
- Benefits of Agile
- Barriers to Agile
- Practice Makes Perfect? The Role of Experience
- The Influence of Job Role on Agile
- Agile Segments – Two Groups
Respondents were asked to rate how important Agile practices are to their organization’s overall business strategy as well as how capable their organization is with Agile. The data shows a gap between the importance an organization places on Agile and its competence with it.
The average score for Agile’s importance to business strategy was 6.82 out of 10 yet the average capability rating was only 5.66 out of 10. Figure 1 illustrates how respondents stated organizational capabilities are not in line with the importance of Agile for their organization.
While the significance of this gap can be debated in subjective terms, there is a significant gap between them in statistical terms.* This gap could be caused by a number of things. Perhaps the barriers to Agile are getting in our way? Maybe many of the benefits of Agile are recognized as important but remain unrealized? Or, could it be that we are simply our own worst critic?
Figure 1: The gap between organizational capability with Agile versus the importance of Agile
*At the 95% confidence interval, importance scores would range from 6.35 to 7.29 and capability scores would range from 5.20 to 6.11.
*Percentage of respondents who agreed that these Agile benefits are of importance to their firm
One of the most interesting findings is that there is a relatively strong relationship between the two benefits of “greater collaboration” and the “ability to adapt to changing priorities” and the importance of Agile practices to overall business strategy.* In other words, organizations that believe Agile is important place a higher value on collaboration and adaptability. The same is true for capability with Agile. Further, there are relationships among some of the benefits of Agile (see Figure 2). While none of the benefits have a strong correlation (i.e. nothing above 0.5) with one another, there are several that are positively correlated with statistical significance (between .20 and .35), ), such as “better fit of features to product needs” and the ability to adapt to changing priorities”.** Other Agile benefits that have virtually no correlation to one another such as “lower costs” and “better fit of features to product needs” are not listed.
|Benefits by Correlation||Correlation|
|Fit to product needs & the ability to adapt||0.320|
|Fit to product needs & time to market||0.311|
|Reduced risk & ability to adapt||0.279|
|Lower costs & time to market||0.268|
|Reduced risk & greater collaboration||0.259|
|Reduced risk & fit to product needs||0.245|
Figure 2: Benefits with at least a moderate level of correlation, i.e. > 0.2. Note that correlation ranges from +1 between (perfect correlation) to –1 (perfect negative correlation). 0 indicates no relationship.
Understanding the relationship between two benefits of Agile is helpful to get a sense of which benefits go well with one another. But alone, they leave an unclear picture of how these various benefits of Agile relate to broader business benefits. Consequently, we grouped these benefits together to create related “factors”, or broader categories of benefits. Using factor analysis, we found that the benefits listed in our survey could be loosely grouped into the following three factors:
- Software quality
- Speed and agility
- Resources: saving time and money
“Software quality” includes four variables (or benefits): better fit of features of product needs, reduced risk, greater collaboration, and ability to adapt to changing priorities. “Speed and agility” includes time to market and ability to adapt. “Resources” include lower costs and faster time to market, which could be interpreted as saving staff time as well as satisfying the market with releasing products faster. These three factors show the true breadth of Agile benefits: effectiveness (software quality), efficiency (speed and agility) and money (resources).*** “Greater Collaboration” has a 0.30 correlation and 0.001 significance with Agile importance and has a 0.35 correlation and 0.000 significance with Agile capability. The “ability to adapt to changing priorities” has a 0.35 correlation and 0.000 significance with Agile importance and has a 0.31 correlation and 0.001 significance with Agile capability.
**Statistically significant at the 0.05 level, i.e. p<.05.
***To see the factor scores, see the Appendix.
* Percentage of respondents who agreed that these barriers are inhibiting further Agile adoption in their organization
One of the most interesting findings is that the barrier of “management – not supportive of Agile” is negatively correlated with both the perceived importance of, and capability with, Agile.* In other words, those who see management as a barrier to further Agile adoption in their organization are more likely to say that their organization places lower importance on Agile and are less capable with Agile relative to their peers. This is an unsurprising yet interesting finding because it shows how critical it is to have management support for Agile. A grassroots Agile effort may be the easiest way to implement Agile in an organization but it is by no means guarantees success if management isn’t involved. The barrier of “approach – using Agile elements in a non-Agile framework” is noteworthy. Interestingly, while it was the most frequently selected barrier in our survey (46% of respondents), it is the only barrier that does not have any significant correlation (p<0.05) with another barrier. Sixty three percent of those who selected “approach” as a barrier selected two or fewer barriers that are inhibiting further adoption of Agile practices at their organization, with 24% selecting approach alone. In other words, there are many people who find that their organization’s approach to Agile is the only or just one of a few barriers they face to improve Agile adoption. Using factor analysis again, we found that the barriers listed in this survey could be loosely grouped into the following two factors: 1.Practice barriers 2.Environmental barriers “Practice barriers” include only two important variables (or barriers): approach and culture. “Environmental Barriers” include four important variables: staff, office layout, management, and culture. In essence, what’s getting in the way of Agile adoption appears to be either the environment in which one is trying to be Agile or the approach that one is taking with Agile.**
“Management – not supportive of Agile” has a -0.39 correlation and 0.000 significance with Agile importance and has a -0.37 correlation and 0.000 significance with Agile capability.
**To see the factor scores, see the Appendix.
Figure 3: Capability and importance ratings by Agile experience
While greater experience with Agile generally leads to better Agile capabilities, the rate of improvement is underwhelming. One might expect capability to increase more steadily with experience. It seems as though organizations cannot rely on Agile experience alone to improve their capabilities. One theory is this is due to the common barriers to Agile, such as culture, approach, and staff (as described in “Barriers to Agile” section). Regarding the perceived importance of Agile, the good news is that experience and average perceived importance are positively correlated. That is, the greater one’s experience with Agile, the more important it becomes. The bad news is that the rate of growth is rather slow – only about 7 percent per year after the first year with Agile. This may be due to the results or challenges from gaining experience with Agile. It’s worth noting that this finding is based on the respondents’ perceptions. Consequently, it is possible that organizations are improving their capability at a steady and greater rate but are overestimating their organization’s Agile capabilities when new to Agile and are underestimating it once they are experienced. It is also possible that there is more than meets the eye with Agile and that those who are new to it underestimate what it entails.
People at all job levels share common views about Agile (see Figure 4). While it’s encouraging that the results show that Agile is seen as at least moderately important across job levels, this may be the result of response bias. After all, those who respond to an Agile survey likely think that Agile is somewhat important. The findings are similar for organizational capabilities with Agile; moderately important across job levels but likely influenced by response bias (see Appendix for details). While we saw some variance in these ratings by job role, our sample size was not large enough to report any conclusive findings, as the margin of error is too large.
Figure 4: Capability and Importance of Agile to Organizations Strategy – Perceptions By Job Role
For those who are unfamiliar with Lean Startup, it is a method for developing businesses and products and is primarily adopted for software development. Its central tenet is “validated learning”, which is a process to get end-user customer feedback and then iterate accordingly. The Lean Startup movement and Agile are viewed as complementary.
The Lean Startup concept of validated learning goes well with the Agile principles of regularly reflecting on how to become more effective and of welcoming change. Both Agile and Lean Startup use iterative processes, from development to product releases. Given these similarities, we were curious to understand how much influence Lean Startup concepts are having on the organizational desire to adopt Agile development practices.
We found that Lean Startup is influencing Agile adoptions, but to a limited and specific extent. When we asked survey respondents how much influence, on a scale of 1 to 10 with 10 being the highest, Lean Startup concepts have had on their organization’s desire to adopt Agile development practices, the median response was 4 and the most common response was a 1. However, despite this minimal impact, when we considered experience with Agile, the effect of Lean Startup on Agile became more evident.
Experience with Agile is positively correlated with the perceived influence Lean Startup is having on Agile adoption. This implies that those who have been Agile for longer are more likely to belong to an organization whose Agile adoption rate is influenced by the Lean Startup concept. Figure 4 illustrates that for every experience category, the longer one has practiced Agile, the higher the level of influence that Lean Startup tends to have on their organization’s Agile adoption.
Because the sample size was not large enough to report any conclusive findings, it seems as though industry sector may be an influence of Lean Startup on Agile adoptions. On average, ICT respondents provided a 4.22 rating whereas Financial Services respondents provided a 2.70.*
Figure 5: Lean Startup’s Influence on Agile
*For the ICT rating, there is a 95% confidence level of a rating between 3.5 to 5.0. For the Financial Services industry, it is only 1.8 to 3.6 at the same confidence level.
We explored whether there are any natural groups of people with similarities related to Agile. Using cluster analysis, we determined that while there were no tightly-knit clusters in our sample, there were two loosely-connected groups that can be described as follows:
- The Veterans: Those who consider Agile to be among the top priorities for their organization, are very good with Agile, and are influenced by Lean Startup
- The Rookies: Those who consider Agile as only having moderate importance to their organization, are subpar at it, and are not influenced by Lean Startup
- Experience: about 3 years
- Importance of Agile: 9 of 10
- Capability with Agile: 7 of 10
- Influence of Lean Startup on Agile: 6 of 10
- Believe Agile helps their organization have greater collaboration
- Do not believe staff or company culture are inhibiting Agile adoption
- Experience: about 1.5 years
- Importance of Agile: 5 of 10
- Capability with Agile: 4 of 10
- Influence of Lean Startup on Agile: 2 of 10
- Do not believe Agile helps their organization have greater collaboration
- Believe both staff and company culture are inhibiting Agile adoption
The response data from this survey reveals several insights into the adoption of Agile in today’s organization. There are two main areas that should be considered by anyone in an organization that is Agile or is considering this approach:
- Agile is enabling a more adaptive organization, and,
- Despite a will to be Agile, there is a gap between good intentions and actual capability.
Organizations are recognizing the imperative to be nimble and action-oriented in order to manage changing priorities. The results of this survey indicate that Agile is a key enabler for this: from “the ability to adapt to changing priorities” to “greater collaboration”, organizations are coming more adaptive because of Agile.
Although Agile is helping companies become more adaptive, the Agile capability gap needs to be addressed. Organizations are headed in the right direction in terms of placing strategic value on Agile. Now they need to address their actual ability to implement and maintain a high-performing Agile culture. Hiring experienced Agile practitioners is only part of the solution. Other aspects of the organization must be addressed, most importantly, meaningful support from the top – not only in embracing the idea, but also by removing the barriers that inhibit Agile and creating an environment that supports Agile.
Definitions of Statistical Terms
Correlation: A statistical measure that indicates the extent to which two or more variables fluctuate together. A positive correlation means that the variables tend to be either large or small together. For example, education and income are positively correlated. A negative correlation means that as the value of one variable increases, the other(s) decrease(s). For example, TV viewing and grades (for students) are negatively correlated. Correlation ranges from +1 between (perfect correlation) to –1 (perfect negative correlation).
Statistically significant: The probability that an effect is not likely due to just chance alone. P-values are often coupled to a significance or alpha (α) level, which is also set ahead of time, usually at 0.05 (5%), as they are in this report.
Factor Analysis: a statistical method used to describe variability among observed, correlated variables in terms of a potentially lower number of unobserved variables called factors. It helps identify groups of related items (called factors). For example, in a decathlon, each athlete competes in 10 events, but several of them can be thought of as “speed” events, while others can be thought of as “strength” events, etc. Thus, a competitor’s 10 event scores might be thought of as largely dependent on a smaller set of 3 or 4 types of athletic ability.
Cluster Analysis: the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters).
Factor Analysis Table – Benefits
|Benefits||Factor 1 (Software quality)||Factor 2 (Speed and agility)||Factor 3 (Resources: time & money)|
|Time To Market||0.182||0.541||0.533|
|Fit to Product Needs||0.606||0.329||-0.064|
|Ability to Adapt||0.563||0.482||-0.241|
The 3 Categories of Agile Benefits. Note that the figures listed above are “factor loadings”. The closer a factor loading is to 1, the more representative the variable is to the factor. E.g. Lower Costs is very representative of the factor “resources: time & money” with a factor loading of 0.884.
Factor Analysis Table – Barriers
|Barriers||Factor 1 – Environmental Barriers||Factor 2 – Practice Barriers|
About Intelliware Development Inc.
Intelliware is a custom software, mobile solutions and product development company headquartered in Toronto, Canada. Intelliware is a leader in Agile software development practices which ensure the delivery of timely, high quality solutions for clients. Intelliware is engaged as a technical partner by a wide range of national and global organizations in sectors that span Financial Services, Healthcare, ICT, Retail, Manufacturing, and Government.
Lawrence Ludlow is Director, Process at Intelliware Development. He has over 24 years experience managing technical projects, the last 16 in the software development field. Lawrence holds a Bachelors degree in Civil Engineering from the University of Waterloo and a Masters degree in Environmental Engineering from the University of Toronto. Lawrence is a Director and co-founder of the Toronto Agile Community group who organize the annual Agile Tour Toronto conference. For the past 10 years Lawrence has also been the administrator for the XP/Agile Toronto users group.
Joe Kevens is a B2B marketing Specialist at Intelliware Development. He has 6 years experience in a variety of marketing and business development roles at PricewaterhouseCoopers, the Greater Toronto Marketing Alliance, and Eloqua. Joe holds a Bachelors degree in Commerce with a Marketing Management major from Ryerson University and a Masters degree in International Business from the Queen’s School of Business.