The Varieties of Democracy Institute (V-Dem) incorrectly categorizes India as an "electoral autocracy." The problems with V-Dem do not derive from its statistical methodology, but from its idiosyncratic choice of indicators, inexplicable scaling decisions, and vulnerability to expert biases. In short, the problem isn't in the models; it's in the data. Or as the computer scientists say, "garbage in, garbage out."

The V-Dem Electoral Democracy Index is the primary democracy evaluation published by Sweden's Varieties of Democracy (V-Dem) Institute. In its 2022 report (based on 2021 data), V-Dem ranked India 100th in the world for Electoral Democracy, down from 65th in 2014. By 2023 (2022 data), India had dropped to number 108. If that's an accurate reflection of the state of Indian democracy, it is alarming. But is it accurate?

The V-Dem democracy rankings are highly respected and widely cited, but they suffer from serious flaws. They are calculated on the basis of five sub-indices. Two of these are objectively scored based on an analysis of constitutions: 'Elected Officials' and 'Universal Suffrage'. Among the 131 countries given perfect scores for 'Elected Officials' in the 2022 report (2021 data) were Belarus, Cuba, Libya, Palestine (West Bank), Russia, Syria, and Vietnam.

The V-Dem documentation notes that for this sub-index "a popular election is minimally defined and also includes sham elections with limited suffrage and no competition." Among the 174 countries given perfect scores for 'Universal Suffrage' were all of the above, plus China, Hong Kong, Iran, North Korea, Palestine (Gaza), Myanmar, and Venezuela. The V-Dem documentation notes that this sub-index takes into account only "legal de jure restrictions, not restrictions that may be operative in practice."

Since most countries receive identical perfect scores for the two objective components of the V-Dem Index, nearly all of the differentiation among countries in their rankings derives from the three subjective components—i.e., from expert evaluations.

Three of V-Dem's sub-indices are subjectively scored based on expert evaluations: 'Clean Elections', 'Freedom of Association', and 'Freedom of Expression'. India was ranked 91st in the world for 'Clean Elections' in the 2022 report, down from 59th in 2014. V-Dem rates the 2019 elections in India as less "free and fair" than the 2021 Legislative Council elections in Hong Kong and only marginally better than the 2021 National Assembly elections in Vietnam. In V-Dem's estimation, India's electoral system in 2019, 2020, and 2021 was substantially less "free and fair" than at the height of the Emergency in 1976, when elections were suspended and more than 100,000 of the government's political opponents were imprisoned without trial.

India was ranked 114th in the world for 'Freedom of Association' in the 2022 report, down from 92nd in 2014. The repression of civil society in India today was characterized as falling between "Mugabe's Zimbabwe" and "Franco's Spain." India was ranked 119th in the world for "Freedom of Expression", down from 61st in 2014. Government censorship of the press was characterized as being roughly on a par with that in Iran.

The Varieties of Democracy Institute has always ranked India's democracy relatively poorly because its objective sub-indices in effect give India no credit for its objectively strong representative democracy based on genuine universal suffrage. But the recent downgrading of Indian democracy by V-Dem has been driven entirely by subjective expert evaluations that rate India's electoral environment today as worse than at the height of the Emergency. A big part of the problem is that V-Dem's electoral indicators fail to differentiate between genuine and "sham" democracies, with the result that one-party dictatorships can and often do score higher than real democracies for the quality of their elections. But the methodological problems that plague the V-Dem rankings run must deeper than this.

Indian democracy - the view from abroad

This paper examines the methodology that underlies the democracy evaluations published by the Varieties of Democracy Institute (the "V-Dem" rankings), using India as a case study. It focuses on the 2022 V-Dem report, but there were no major methodological changes in 2023, and its findings still apply. All three major international democracy rating organizations (the Economist Intelligence Unit, Freedom House, and the Varieties of Democracy Institute) place India relatively low in their democracy rankings. But the best-trusted and most sophisticated of these rankings, the V-Dem Electoral Democracy Index, ranks India worst of all. India was ranked 100th in the world in V-Dem's 2022 report (based on 2021 data), making it one of the worst performers among countries that have any form of democracy at all. This represents a substantial drop from the 2014 V-Dem ranking of 65th in the world at the time the present prime minister of India, Narendra Modi, took office.

Op-eds decrying the deterioration of Indian democracy under Mr. Modi and the Bharatiya Janata Party (BJP) he leads have appeared in respected Western newspapers like the Wall Street Journal, the New York Times, and the Washington Post. The Washington Post went so far as to editorialize in 2021 that India "cannot be readily described as a democracy." The New York Times has recently reported as news the claim that "Mr. Modi has bent to his will the courts, the news media, the legislature and civil society." In less exalted corners of the internet, more extreme language equating India's current governance institutions with "fascism" have become commonplace. The Indian government, of course, strongly objects to these characterizations—as one would expect.

Ordinarily, disinterested observers with no particular connection to India would trust the opinions expressed in Western newspapers and academic evaluations like V-Dem's over the protestations of a petulant foreign government. But those who read more carefully might notice that all of the articles cited above were authored (where authorship was attributed) by people who were born in India or neighboring countries—as was the India focus section in V-Dem's 2021 democracy report. It may be deeply politically incorrect to question analyses on the basis of the national origin of the analyst, but it is nonetheless a fact of life that analysts routinely carry their personal politics into their professional opinions. The reality is that international narratives about Indian democracy are strongly conditioned on the potentially biased opinions of a very small number of analysts of South Asian origin.

That's why the V-Dem rankings are so important. The V-Dem rankings are compiled for 179 countries, based on a highly sophisticated statistical methodology. They are produced by academics at Sweden's University of Gothenburg, not by a (potentially politically-motivated) think tank or consulting firm. The V-Dem project is primarily funded by the European Research Council, and although its methodology might incorporate subtle biases favoring Scandinavian conceptions of democratic norms, these biases would have been a constant over time. Moreover, unlike the Economist Intelligence Unit, V-Dem publishes extensive methodological documentation for its rankings, and unlike Freedom House, V-Dem focuses specifically on the evaluation of democracy.

The world rightly prioritizes the V-Dem democracy rankings as the most authoritative evaluations available. As a result, the V-Dem rankings find their way into third-party governance metrics published by the likes of the World Bank, the US Agency for International Development, and the Cato Institute. These in turn can be integrated into country risk profiles, bond pricing models, and other private sector analyses. It is thus very important, not only for India, but for all less-developed countries to know that the V-Dem rankings are fair, reliable, and above all: accurate. When the United States is relegated to 29th place, well below Costa Rica (#3), Estonia (#5), and Chile (#25), and scoring less than a hundredth of a point higher than Argentina (#30) and Cyprus (#31), it has essentially no affect on global perceptions of American democracy. But for a developing country like India (#100), the potential effects of being ranked on a par with chronically-unstable Lebanon (#99) are much greater.

Few people actually believe that India's democracy is of only marginally higher quality than that of the Ivory Coast (#101), a country whose current president won office with a 94% majority after he defied a constitutional ban on running for a third term and his main opponent was sent into exile to avoid a 20 year prison sentence. This prompts the question: how do these kinds of anomalies rise in such a carefully-calibrated instrument as V-Dem's Electoral Democracy Index? And although many countries may have anomalous placements on the V-Dem rankings, the Indian anomaly is of particular interest. With roughly 1.4 billion people, India is by far the world's largest democracy. It is also by far the oldest democracy among developing countries, and a democratic model for the non-Western world. With these considerations in mind, it seems appropriate to conduct an analysis of the V-Dem methodology using India as a case study.

V-Dem's two objective components

The flagship Electoral Democracy Index published by V-Dem is the outcome of an extremely sophisticated set of statistical models. In a recent interview, Staffan Lindberg, the principal investigator of the V-Dem project, described V-Dem's approach as "the gold standard latest, Bayesian item response theory modelling" using "supercomputers" and "astronomer math" to "achieve cross-country comparability." The statistical models used by V-Dem are, indeed, world-class. Lindberg himself is a highly-respected and highly-cited professor of political scientist. His own specialty is African democracy, but in the V-Dem project he is supported by a world-spanning team of democracy experts. It must be stressed: there are no reasonable grounds on which to question Lindberg's professional credentials or the professionalism of the V-Dem project as a whole. V-Dem is an incredible exercise in academic number-crunching.

That said, it sometimes happens that the most sophisticated statistical analyses are built on poorly-conceived numerical and methodological foundations. At the highest level, this seems to be the case with the V-Dem models. Despite their highly advanced mathematical skills, V-Dem's programmers seem to have overlooked some very basic properties of the data that underly their calculations.

For all its underlying statistical sophistication, V-Dem's Electoral Democracy Index is conceptually a straightforward arithmetic function of five constituent sub-indices (although the actual calculation of the index is more complex). Two of these sub-indices are objectively scored based on an analysis of country constitutions (Elected Officials and Universal Suffrage) while three of the sub-indices are subjectively scored based on expert evaluations (Clean Elections, Freedom of Association, and Freedom of Expression). Each of these sub-indices is scored on a scale from 0 to 1, as is the final Electoral Democracy Index, which is calculated using a formula that includes both a multiplicative component (the five sub-indices are multiplied together) and an additive component (the five sub-indices are averaged, with the two objective sub-indices given half the weight of the three objective sub-indices). The exact formula necessarily involves assumptions about the relative weighting of the various sub-indices (and how to combine the terms), but the methodological documentation notes that "because most of the variables are strongly correlated, different aggregation formulas yield very similar index values." This is entirely in line with expectations.

Unfortunately, however, the two objective sub-indices (Elected Officials and Universal Suffrage) are very poorly distributed.

The Elected Officials sub-index is designed to measure "(a) whether the chief executive is elected ... and (b) whether the legislature ... is directly or indirectly elected." No fewer than 131 out of 179 countries evaluated receive a perfect score of 1 for Elected Officials. A further 20 countries receive scores of 0.9 or above. As a result, nearly all of the cross-national variability in Elected Officials derives from the differences between these 151 countries and the 38 remaining countries that score below 0.9.

A bizarre peculiarity of the Elected Officials sub-index is that for coding purposes "a popular election is minimally defined and also includes sham elections with limited suffrage and no competition." Thus countries that do not in fact hold elections can and do receive perfect scores on this sub-index. Countries with no elections that receive perfect scores for Elected Officials include Cuba, Libya, Palestine (West Bank), and Vietnam. Additional non-democratic countries with perfect scores include Belarus, Russia, and Syria.

Interestingly, India falls just short of a perfect score for Elected Officials, apparently because (until recently) two of its legislature's members were appointed (not elected) to represent the country's small Anglo-Indian community. This arrangement was repealed by constitutional amendment in 2020, but V-Dem's coders seem not to have picked up the change in time for the 2022 report.

The Universal Suffrage sub-index is designed to measure "what share of adult citizens as defined by statute has the legal right to vote in national elections." A remarkable 174 out of 179 countries evaluated receive a perfect score of 1 for Universal Suffrage, with one additional country (Thailand) receiving a near-perfect score of 0.994. The four remaining countries are the UAE (0.36), Qatar (0.035), Somalia (0.002), and Saudi Arabia (0). Among the countries receiving perfect scores for Universal Suffrage are the seven rogue countries cited above for their perfect scores on Elected Officials, plus China, Hong Kong, Iran, North Korea, Palestine (Gaza), Myanmar, and Venezuela.

Like the Elected Officials sub-index, the Universal Suffrage sub-index suffers from an extraordinarily counterintuitive coding choice. In ascertaining the proportion of a country's adult population that is allowed to vote, it "covers legal de jure restrictions, not restrictions that may be operative in practice de facto." Thus many communist dictatorships like China, North Korea, and Vietnam receive perfect scores. In the historical V-Dem data, the old Soviet Union received a perfect score of 1 even during Stalin's purges. The Elected Officials sub-index is a purely theoretical indicator of democracy.

The Universal Suffrage sub-index is an even more meaningless indicator. Literally all of the cross-national variability in Elected Officials derives from the differences between the 174 countries with perfect scores and the five that for one reason or another fall short. It is no coincidence that Saudi Arabia (#179) comes in at the bottom of V-Dem's democracy rankings; its score of 0 for Universal Suffrage ensures that it automatically receives a total score of 0 for the entire multiplicative component of the Electoral Democracy Index.

The lack of cross-national variability in V-Dem's two objective democracy indicators represents a serious methodological flaw. Since 131 of the 179 countries evaluated receive identical perfect scores for Elected Officials and Universal Suffrage, these two components contribute relatively little cross-national variability to the overall Electoral Democracy Index—and thus they have very little impact on the international democracy rankings. A country like India, which genuinely elects its officials under genuine universal suffrage, receives the same objective sub-index scores as Cuba and Vietnam. The result is that India's (objective) electoral democracy counts for little on the V-Dem Electoral Democracy Index. Instead, nearly all of the differentiation among countries in their V-Dem rankings derives from the three subjective components—i.e., from expert evaluations.

V-Dem's three subjective components

International rankings on V-Dem's Electoral Democracy Index depend almost entirely on the three subjective sub-indices measuring Clean Elections, Freedom of Association, and Freedom of Expression. Ignoring the two objective sub-indices entirely, a recalculation of democracy scores based only on the three subjective sub-indices is almost perfectly correlated (r>0.995) with the actual Electoral Democracy Index. India's ranking calculated on the basis of the subjective sub-indices only would be 101st in the world, one spot lower than its actual ranking. Despite decades of holding genuinely democratic elections, India gains nothing at all compared to countries that hold "sham" elections (or no elections at all). This constitutes a clear failure of V-Dem's underlying methodology.

Yet the legitimacy of the V-Dem project rests in part on its highly sophisticated methodology. In the interview quoted above, Prof. Lindberg seemed keen to emphasize his team's use of "supercomputers" to calculate "astronomer math," freely offering to share V-Dem's data and programming code "if somebody has access to supercomputers and wants to rerun these calculations" while underscoring that "it is difficult to understand the math for non-experts, naturally." Regarding the project's reliance on expert evaluations to generate the underlying data on which those calculations are based, he stressed that "When you do expert surveys, you are not looking for an opinion. You are looking for knowledge." He asserted that the experts consulted by V-Dem "have sound scientific knowledge about the area" each assesses. He used variations on the word "scientific" to describe V-Dem's expert analyses no less than five times.

But political science is not a science, and political scientists' expert evaluations of democracy indicators are not akin to chemists' measurements of the temperature of a fluid. The reality that expert evaluations are imprecise and deeply subjective is illustrated by the very fact that V-Dem specifically seeks out "the scientific knowledge of specialised country experts." In the natural sciences, measurements are not generally carried out by top experts; they are outsourced to students, laboratory technicians, or external contractors. V-Dem's commitment to consulting only the best-qualified experts on every subject in every country highlights the reality that each evaluation is in fact a subjective judgment call, not an exact measurement.

Take for example V-Dem's expert panel evaluation of the autonomy of the Election Commission of India. This is a metric on which the V-Dem expert panel has recorded an extreme deterioration since 2014: on a scale that runs from 0-4, India has declined from an average score of 3.6 to a score of 2.3. That represents a fall from Swiss levels to Zambian levels of autonomy in the term of a single government. Yet it is difficult for anyone who does not actually sit on the Election Commission to know whether or not such a sea change has actually occurred. What V-Dem calls "the scientific knowledge of specialised country experts" is in reality little more than a matter of subjective personal opinion. In the virtual absence of any empirical basis on which make judgments, there is wide scope for personal politics to influence experts' answers.

Prof. Lindberg asserted in his interview that "the use of the scientific knowledge of specialised country experts is accepted worldwide as the gold-standard methodology." Although expert opinion is sometimes used in social research, there certainly no such consensus about its dependability. The leading sociologists who study expert interview research methodologies (the experts on expert interviews, if you will) concluded in a 2009 overview of the field that the most important lesson to be drawn about expert interview methodologies is: "First and foremost the realization that the naïve image of the expert as source of objective information ... has long become problematic."

Expert surveys are much more likely to produce reliable results in areas that are a mater of verifiable fact (e.g., "in your country, are political candidates subject to asset disclosure rules?") than in areas that are a matter of unverifiable speculation (e.g., "in your country, do political candidates lie on their asset disclosures?"). Yet the subjective evaluations that feed into V-Dem's Electoral Democracy Index are, for the most part, based on questions that require unverifiable speculation on the part of the country experts. That makes them extremely vulnerable to biases introduced by the personal politics of the expert evaluators themselves. This is illustrated below with examples drawn from each of V-Dem's three subjective sub-indices.

Clean Elections

V-Dem's Clean Elections sub-index is designed to measure "to what extent are elections free and fair." In 2021, India scored 91st in the world on this sub-index, down from 59th in 2014. India ranked 20 places below Hong Kong (#71) and only two places above Vietnam (#93). These perverse results were enabled by the specific choices made by V-Dem in constructing the Clean Elections sub-index. The sub-index is based on a combination of eight separate expert evaluations: EMB autonomy, EMB capacity, the integrity of the voter registry, vote buying, other voting irregularities, government intimidation, other electoral violence, and the overall freedom and fairness of elections. Despite receiving low scores for the overall freedom and fairness of their elections—in Vietnam's case, an unlikely score of 2 ("ambiguous") and in Hong Kong's case, a score of 0—the two non-democracies were able to make up for this with high scores on secondary metrics.

For example, in Hong Kong and Vietnam there was much less reported vote buying, the voter registries were near-perfect, there were virtually no irregularities at the voting booth, and there was little or no violence at the polls. Even the electoral commissions in Hong Kong and Vietnam matched India's for capacity (though not for autonomy). All told, by running well-ordered sham elections in one-party police states, Hong Kong and Vietnam outscored India on many of the components of Clean Elections, resulting in overall scores that counterintuitively place them at (Vietnam) or above (Hong Kong) India's level.

An alternative check on the validity of V-Dem's Clean Elections sub-index is offered by an inter-temporal comparison with the India of the Emergency, the period in 1975-1977 when prime minister Indira Gandhi of the Indian National Congress (INC) suspended elections, ruled by decree, amended the Constitution without debate, and imprisoned more than 100,000 of her political opponents without trial. Since 2019, India has been rated worse for Clean Elections than at the height of the Emergency in 1976. Today's India scores worse on almost every metric. The 2019 election was rated less free and fair than the 1977 election; the Electoral Commission is rated as less autonomous than in 1976; electoral irregularities are reckoned to be greater today; and (bizarrely) government repression is rated as much worse than in 1977—when many opposition leaders were still in prison.

Freedom of Association

V-Dem's Freedom of Association sub-index is designed to measure "to what extent are parties, including opposition parties, allowed to form and to participate in elections, and to what extent are civil society organizations able to form and to operate freely." In 2021, India scored 114th in the world on this sub-index, down from an already-low 92nd in 2014. At first glance, this might seem an odd result for a country with eight recognized national parties, dozens of regional parties, and highly competitive elections. Yet India bans several terrorist-linked political parties, requires parliamentary candidates to place security deposits of ₹25,000 INR (roughly $300 USD), and has a tradition of coalition government. All of these factors weigh heavily on India's Freedom of Association ranking due to V-Dem's implicit assumption that all of these practices are anti-democratic.

While these relatively objective evaluations may be highly questionable, India scores even worse on the more subjective components of Freedom of Association. On Civil Society Organization Entry and Exit ("to what extent does the government achieve control over entry and exit by civil society organizations into public life") India ranks 146th in the world—once again, just below communist Vietnam. The text descriptor for this metric states that:

The government licenses all Civil Society Organizations and uses political criteria to bar organizations that are likely to oppose the government. There are at least some citizen-based organizations that play a limited role in politics independent of the government. The government actively represses those who attempt to flout its political criteria and bars them from any political activity.

For Civil Society Organization Repression, India ranks almost as low, in 139th place (this time just above Vietnam). India's civil society repression was rated on a par with that in Palestine (Gaza), Rwanda, and Cameroon, all one-party dictatorships. Interestingly, V-Dem gives historical examples to aid experts in evaluating countries on this metric; on these examples, India falls somewhere between "Mugabe's Zimbabwe" and "Franco's Spain." As with all subjective evaluations, the credibility of India's placements on V-Dem's two civil society metrics is in the eye of the beholder. It is worth noting, however, that India's performance on these two subjective indicators has fallen precipitously since 2014, despite relative stability in the regulations governing civil society organizations.

Freedom of Expression

V-Dem's Freedom of Expression sub-index is designed to measure "to what extent does government respect press and media freedom, the freedom of ordinary people to discuss political matters at home and in the public sphere, as well as the freedom of academic and cultural expression." In 2021, India scored 119th in the world on this sub-index, a dramatic decline from 61st in 2014. This represents the most serious deterioration in India's raking on any of the five V-Dem sub-indices. Government censorship of the media in India is now rated on a par with that in Iran, and all other indicators of media and academic freedom have declined since 2014. While Freedom of Expression indicators are not now as low as they were during the Emergency, they are lower than at any other time since independence. This is obviously a highly subjective area of analysis, with no unambiguously factual metrics on which to anchor expert evaluations.

Conclusion

V-Dem claims on its methodology page to have "developed innovative methods for aggregating expert judgments in a way that produces valid and reliable estimates of difficult-to-observe concepts." There is much truth in this. The Bayesian item response theory models used by V-Dem are indeed highly sophisticated. Some statisticians might favor other modeling strategies, but the models used by V-Dem are widely accepted and certainly fit for purpose.

The problems with V-Dem do not derive from its statistical methodology, but from its idiosyncratic choice of indicators, inexplicable scaling decisions, and vulnerability to expert biases. In short, the problem isn't in the models; it's in the data. Or as the computer scientists say, "garbage in, garbage out."

First, the V-Dem rankings suffer from serious errors introduced by the selection of indicators to include. V-Dem includes a large number of indicators that do not necessarily measure what they seem to measure. As a result, totalitarian regimes can be recorded as having "free and fair" elections by ensuring that their sham elections are conducted efficiently, with wide participation, and without violence. On V-Dem's primary democracy indicators related to elections, one-party dictatorships can and often do score higher than real democracies.

Second, the poor measurement properties of some of the V-Dem sub-indices introduce serious statistical problems. V-Dem's two objective democracy sub-indices are scaled in such a way that most countries receive perfect scores. As a result, these objective sub-indices contribute very little to V-Dem's ultimate democracy rankings, which depend almost entirely on the three subjective sub-indices. On V-Dem's supposedly "objective" measures of democracy, one-party dictatorships can and do receive perfect scores.

Finally, the V-Dem rankings are—by design—highly vulnerable to respondent bias. V-Dem's surveys generally ask experts questions about which they might have well-informed opinions, but cannot possibly have direct research-based knowledge. As a result, most of V-Dem's subjective indicators are based on unverifiable speculation. By asking for broad evaluations instead of factual determinations, these questions almost ensure that experts' answers will be conditioned on their own personal politics.

These errors don't seem to have been intentional, and there is certainly no reason to believe that they were motivated by ill-will toward India (or any other country). But they are errors, and they do seriously compromise the V-Dem rankings. Unfortunately, these errors can only be corrected through a complete overhaul of V-Dem's survey questionnaire and historical database. The errors identified in this paper are deeply baked into V-Dem's back-end data collection processes, and cannot be corrected by a front-end statistical patch.

It may seem unreasonable to demand a root-and-branch reformulation of the V-Dem democracy rankings simply because they return low scores for today's India. But India is not the issue here; India is only the canary in the coal mine. The problems identified in this paper are problems with V-Dem, not problems with V-Dem's India scores. Until they are fixed, the V-Dem rankings should not be taken seriously as indicators of the health of the world's democracies.

Salvatore Babones is the executive director of the Indian Century Roundtable and an associate professor at the University of Sydney. His book Methods for Quantitative Macro-Comparative Research is a standard source for the statistical analysis of international comparisons. He is currently researching a book on Indian democracy.