Valid Validation

Management Today (June 2011) recently reported that "The overwhelming majority of consultants do not deliver on client expectations, new research reveals. The Consulting Client Satisfaction Survey from the RFP Company, which examines the factors behind clients' overall satisfaction with results achieved when working with management consultants, shows that more than 70% of consultant engagements do not meet, let alone exceed, the results the client expected. Although respondents acknowledged the strengths of consulting firms in the areas of budget management and familiarity with project content, results show that these do not contribute strongly their probability for delivering results that meet or exceed client expectations. The quality of the individual consultants assigned and their focus on the assignment were identified as areas for improvement. But it was not all finger pointing, with clients acknowledging their responsibility to select an appropriate consulting firm for their needs and to proactively engage with them."

With this in mind  I ran a simple online quiz on a discussion thread and asked professional people who considered themselves strategic planning consultants to respond to this statement: "I am interested to learn what processes people in this group use in determining the validity of the advice and management tools/techniques that they promote."

After about three weeks, I collated and categorised the responses:

Category 1: "It's not my responsibility"

*none

*fairly relaxed about the 'validity' of techniques or advice

*results are the ultimate measure of a performance 

*what the client wants 

*it is the client's responsibility 

*consider [client] unique capabilities i.e. define what they do better than their competitor 

*It is the customer that must be able to validate the resource that it is being acquired

Category 2: "I know".

*use intuition

*infer or surmise 

*common-sense 

*evaluate own resources and achieve whatever goal must be achieved

*don't promote anything that I haven't used and tested, and been satisfied with the results.

* try to look behind others' conclusions about causes and effects to see if my learning offers alternative interpretations; which I tend to trust more.

Category 3: "What's the evidence?"

*evidenced-based methodologies
*testing propositions and references through checking into previous results achieved
*avoid offering services to people who seem captivated by things like the 7 habits (and most other generalisations which look only at the successes, i.e. ignore cases where exactly the same behaviour was followed by failure. ... exceptions for those who clearly have deep understanding of their business, especially if they are prepared to work together.

Category 4: "Mainstream fashions"

*marketing promotes the thinking 

Category 5: "Empirical evidence"

*grounding in peer reviewed empirical research (but has limitations)
*introduce me to someone who has done this, so we can discuss their circumstance and experience 

Validation goes beyond surmising (i.e., suppose that something is true without having evidence to confirm it.) and inference (i.e., the act or process of deriving logical conclusions from premises known or assumed to be true). Validation goes beyond personal experience, knowledge and skills, which can all be easily biased and incomplete. 

Validity generally refers to the extent to which a concept, conclusion or measurement is well-founded and corresponds accurately to the real world without just relying on an individual interpretation. There is nothing trivial or easy about methods of validation. There is always some form of theoretical construct (mental models and attributions) through which an individual’s observations are interpreted. 

Therefore, when considering, suggesting or promoting management tools and techniques, some courses of validation action might include: 

Looking beyond only personal experience - challenge  ingrained theoretical constructs. 

Consider the credentials of the author/s? (e.g., "management guru" rhetorical author, academic, practitioner or researcher?)

Seek out peer group assessment or review of the method/tool. 

Find the evidenced-base (e.g., academic and/or empirical evidence?) 

Evaluate the level of data that supports the claim (i.e., validated data, hearsay or evidence-based?)

Search for actual implementations of a method/tool and  outcomes evaluation. 

Determine if the method/tool is based on generalisations and platitudes (e.g., 7 habits, Excellence,) and overlooks anything contextual? (e.g., Not-for-Profit oganisation compared with For-profit Organisation) 

Recommended reading: Client-Consultant Collaboration: Coping with Complexity and Change. Eds. Anthony F. Buono and Flemming Poulfelt, 2009