A Framework for Assessing Frameworks
When combining multiple frameworks and methods into a single unitary approach, it helps to determine, as objectively as possible, which ones are most robust (and indeed, how robust the finished product is). I propose here a modest approach to judging the merits of any particular framework using inter-related dimensional criteria, abbreviated as CERTS for mnemonic purposes.
The essential dimensions and considerations/tradeoffs are as follows.
- Comprehensiveness: How much of the Area of Concern does the method or framework address?
- Effectiveness: How well-suited is the framework to the problems it does address?
- Repeatability: How easy or hard is it to follow the same steps from one situation to the next?
- Transferability: How easy or hard is it to share the method or framework with others?
- Sustainability: Does the method or framework actual produce relatively better results with less labor or does it just tend to add overhead?
Together, these five dimensions fall into two broad categories of attractiveness: Fitness (how well the framework or method is suited to problem solving) and Implementability (how easy or difficult it is to employ the framework or method). The diagram immediately below demonstrates the grouped dimensions of this assessment paradigm, with explanatory discussion to follow.
Comprehensiveness
As I mention in other sections of this work, any core method must be judged first and foremost by whether it addresses all identified aspects of the area(s) of concern. It is little help to come to a common aspect of the problem space, only to find that the methodology or framework currently employed has little to say about it (which, incidentally, created the need for a full-spectrum approach outlined in these pages).
One particular and striking strength of a full methodology, like CSC Catalyst, is its comprehensiveness in addressing a wide range of problems in the Business-IT landscape. TOGAF, being focused on architecture, is somewhat less comprehensive, although Enterprise Architecture does cover a lot of what one finds in Catalyst or the Accenture Development Method. Special-purpose frameworks and methods, of course, fill the gap or provide depth in critical areas. ITIL is one example of this, as a focused approach to IT services management.
In general, I find the best approach is to select a method that is most comprehensive (broad) and supplement it, where needed, with focused methods having more depth. The goal in integration, of course, is complementarity and synergy.
In terms of the inter-relationships between Comprehensiveness and other dimensions, an increase in Comprehensiveness tends to foster an increase in Effectiveness but a decrease in Transferability and Sustainability because the cost of the extra range gained is greater overhead and complexity, resulting in relatively more effort in communicating the method or framework to others.
It is important to note that this is a relative or qualified judgement as apposed to an absolute. I would not negatively judge the Comprehensiveness of a given approach against my full problem space if what I really need is a focused method in a particular slice of that space. So the key question is "Comprehensiveness relative to what?" I may judge a method as lacking Comprehensiveness across the full spectrum (not suitable as a core method) but exceptionally good at covering what it does. Indeed, I routinely incorporate several special-purpose or focused methods for the very reason of their Comprehensiveness (and Effectiveness) in what they do cover (and in that they make up for lack of depth in certain parts of my broad, comprehensive method). Thus, the best approach is to determining the usefulness of a method in this dimension is perform this judgement against various slices of the problem space, such that a profile is developed. In other words, if the method or framework has promise, measure twice (or ten times) and cut once.
Effectiveness
Effectiveness in this approach means a judgement of the consistent quality and power of the results when the framework is used or method applied. Whereas Comprehensiveness asks "what ground do you cover?", Effectiveness asks "How well do you cover that ground?"
The generally observed positive relationship between the two acknowledges a natural tendency of fully developed frameworks and methods to also be good at what they do. But while correlated, they do diverge in some important ways and at times, the peanut butter can be spread quite thin, resulting in opposite tendencies.
Needless to say, it is helpful to note that while there may be correlations, these dimensions necessarily measure two different aspects of method or framework attractiveness. As with Comprehensiveness, of course, it is also helpful to ask "Effective in precisely what slice of the problem space?" Any judgement of Effectiveness is relative to the ground being covered.
I always find it interesting to make the two judgements closely together (and with respect to the same problem topology subsets) when looking at an approach to fully form my opinions on Fitness and also see if I believe C and E were positively or negatively correlated across aspects of my problem space as the approach was developed and built out.