Is Leaders' Talk Cheap?

Earlier this year, two sociologists provoked a probing debate about the integrity and value of interviewing as a research method.  Colin Jerolmack and Shamus Khan, respectively of New York University and Columbia University & Berlin School of Creative Leadership (Steinbeis University), launched their broadside in the May 2014 issue of Sociological Methods & Research.  While this methodological deep dive and accompanying critical discussion were made in the context of a specific discipline, the challenges mounted have relevance for research in all fields where interviews are widely used.  For management and business studies, in particular, in which interviews of leaders are often central to substantiating and defending accounts and insights of organizational and industry practice, such questioning has potentially far-reaching implications.

Though Jerolmack and Khan’s argument is dense and sophisticated, one of their major concerns is captured in what they call the ‘Attitudinal Fallacy,’ which claims that correlations of reported attitudes with situated behavior are never high enough to presume equivalence. ‘Attitudes are poor predictors of action’, they contend, and since interviews are shaped by and convey subjects’ attitudes, they constitute an inherently unreliable approach to researching the reality of these individuals’ actions. 

Respondents to the original piece closely engaged this issue of ‘ABC’, or Attitude-Behavior Consistency.  For instance, Karen Cerulo, a researcher of cognition and culture at Rutgers, acknowledged the longstanding discrepancy or word and action while nevertheless urging that critics don’t overstate the problem.  Sometimes what people think or say does inform what they do, she notes, and so we should still value, if with more circumspection, the content of interviews.  Moreover, even if words and actions are sometimes specifically inconsistent, individual level data can sometimes illustrate more general patterns of action worth highlighting.

To their credit, Jerolmack and Khan don’t reject interviews entirely but instead call for greater methodological pluralism in research.  Rather than relying entirely or disproportionately on interviews, in other words, they urge increased deployment of ethnographies, deep or participant observation, qualitative data gathering, little or big data, and holistic analysis.  In research on business and leadership, especially, questioning the integrity of interviews and calling for multiple research methods compounds the challenge of gathering data and insights about individuals and organizations. 

This is not only an issue for researchers of business and management research and analysis but for readers and practitioners.  Ask yourself: How many recent books or articles on leadership or effective business practice have you read that rely on interviews, typically of successful leaders about their beliefs and experiences, to advance a distinctive insight or approach?  Content drawn from interviews or other first-person sourcing of business leaders’ viewpoints is especially common, indeed is a major selling point, in popular business writing and journalism.  

Connecting Individual Attitudes with Organizational Behaviors

The words of leaders are typically taken by readers and listeners as reliable expressions of their behaviors – past, present, or future.  The credibility of those words, and those speaking or writing them, is crucial, of course, and turns back to the fundamental challenge of ABC.  While we may debate whether Marissa Mayer’s online strategy for Yahoo! is the right one, for example, we accept that her words capture what she and the company are actually doing.

Yet the matters of credibility and consistency are further tested by the break between individual claims or characterizations and the often widely disparate actions of organizations.  Even as leaders speak, the connection to collective behavior – as summary, explanation, cause, plan – remains opaque amidst so many possible causes.  As much as we want to ground our understanding of corporate and collective reality in personal attitudes, explaining corporate or organizational behavior demands wider attention to persons, situations and the actual interactions between them.

Research and Access in Business and Organizations

Jerolmack and Khan identify another shortcoming, the ‘Accounting Fallacy’, to emphasize the fallibility of individual accounts and self-reporting and how they cannot accurately stand in for analyses of actual behavior.  To generate wider explanations of the situations and interactions in organizational life requires multiple research approaches.  Yet in most corporate settings, these are impossible for outside researchers and analysts to pursue.  Confidentiality, distraction, and the priorities of time and other resource allocation typically limit or preclude more robust access to people and situations.   

The usual response to this lack of access is to judge leaders’ words by specific indicators of their organizations’ performance.  Rather than methodological pluralism in research about corporate actions, particular outcomes – like products, services, and financial, innovation or creative results – become the basis for assessing those words.  This makes certain sense.  After all, we invest in some firms, say Alibaba, because of the value they can deliver or promise to customers and in the marketplace.  Yet even in a world of inevitably incomplete information, attributing the successful or failed results of complex organizational behavior to individual words is a poor substitute for ascertaining more robust and varied understanding of the actual collective situations producing the results. 

Interviews as Public and Professional Performances

Another source of uncertainty in trying to connect the words of leaders to their organizations’ actions is the recognition that many interviews, whether public or ostensibly not, are performances.  Both the carefully crafted pronouncements of leaders and their closely managed professional personae affect whatever claims researchers and others may be able to make about them.  Even when the words may not correspond to actual behaviors of the organization, present or past, they may nevertheless be presented in the interests of the company, its brand management, and public relations.

Ultimately, an ethical aspect emerges here: do we trust what leaders say?  And even if we recognize that some words may be exaggerated (recall Steve Jobs’ ‘reality distortion field’), do we justify that hyperbole in terms of other ends, like increasing shareholder or market value or hyping and delivering new products?  While the trust issue exists in all interviews, where we can question the correlation of self-reports of behavior, the test of consistency between words and actions becomes all the more complicated when leaders may be trying to achieve multiple ends and speak to multiple audiences in the same interview.

Authorities, Elites, and Celebrities

If the interview responses of leaders are shaped by a range of organizational interests, the interactions with researchers and others that prompt these responses can be strongly impacted by the status or reputation of the leader himself or herself.  We choose to interview or otherwise solicit the perspectives of leaders in the first place because of their experience or position, expertise or accomplishment.  Yet that very appeal to authority can limit the thoroughness of the interview and consequently the integrity of the verbal data emerging from it.

At a practical and human level, interacting with very senior or elite leaders involves unequal power relations that can be difficult to navigate.  Consider the prospect of interviewing Richard Branson or Sheryl Sandberg or Martin Sorrell.  Establishing the kind of access, trust and rapport so important for substantive exchanges and then producing meaningful data is particularly challenging with the involvement of subjects like these who possess great seniority, reputation, and even celebrity.

Caveat Lector

Taken together, these considerations suggest that interviews with leaders, since they typically concern collective or corporate behaviors as well as those leaders’ reports of their own individual actions, would benefit from more multidimensional analysis and layered understanding of their accounts.  While this call resonates with Jerolmack and Khan’s general imperative to expand the range and integration of methods employed in researching social or corporate action, it takes on special significance in a field of research where leaders’ words are accorded such value and prominence.  For external researchers of organizations, gaining meaningful access and developing robust contextual understanding of the special situations (and constraints) of organizational life are ongoing challenges.  For readers of business and management research as well as more popular analyses reliant upon the words of leaders, the methodological debate should prompt us to question, constructively but more diligently, the accuracy of the claims leaders make about their own actions as well as those of their organizations.