Beyond the Bell Curve: The possible limitations in IIT Madras’s NEET marks study

CHENNAI: The National Eligibility Cum Entrance Test (NEET) 2024 for admissions to undergraduate medical programmes has been mired in controversy. Issues include an unusually high number of perfect marks, emerging evidence of question paper leaks, subsequent arrests of individuals involved in the leak, and alleged incompetence of the National Testing Agency (NTA) in conducting the entrance test.

In response to these concerns, a few NEET UG 2024 candidates filed a writ petition at the Supreme Court, highlighting multiple alleged irregularities by the NTA. To gain more insights into the NEET marks, the Supreme Court asked the NTA if it was possible to use data analytics from an expert government agency.

Consequently, the NTA assigned the Indian Institute of Technology Madras to analyse the NEET marks data. IIT-M submitted a report examining various parameters, including candidates’ marks across different ranges, marks distribution, and city and centre-wise rank distribution. Through their quantitative analysis of the data provided by NTA, IIT-M concluded that no abnormality was observed in the NEET 2024 marks.

However, the study raised several pertinent questions regarding its limitations, interpretation, and potential extrapolation of results.

One significant constraint was the heavy reliance on a bell-shaped distribution curve. This approach assumes that cheating would disrupt the normal distribution of candidates’ marks.

However, due to the large number of candidates, widespread but subtle cheating might shift the entire curve without changing its form, thereby evading detection. Therefore, while statistically no abnormality was observed as per IIT-M’s analysis, we cannot interpret this as definitive proof that no malpractice occurred.

The analysis’s scope was another point of concern. By focusing on only the top 1.4 lakh ranks out of 23.3 lakh candidates, the study might have missed patterns or irregularities occurring beyond this range. Further, of the top 1.4 lakh, analysis appears to focus heavily on high-performing students, potentially neglecting similar scrutiny of middle and lower mark ranges where malpractice could also occur.

A truly comprehensive analysis should examine all mark ranges with equal rigour. Moreover, the study’s methodology could benefit from cross-verification with other performance indicators, such as students’ performance in classes 10, 11, and 12. Comparing NEET marks with these metrics could help identify inconsistencies that might indicate malpractice and increase the granularity of this quantitative data analysis.

Additionally, limiting the analysis to only two years (2023 and 2024) is insufficient to identify long-term trends or cyclic patterns. A longer timeframe could reveal subtler trends or changes in malpractice methods over time.

Over-emphasis on detecting ‘mass malpractice’ or benefits to a ‘localised set of candidates’ may overlook smaller-scale or more dispersed forms of cheating. Even isolated incidents of candidates accessing the question paper could avoid clustering in specific locations, making them harder to detect through this type of analysis.

Attributing the increase in marks (550-720 range) solely to a 25% syllabus reduction may oversimplify the factors influencing improved performance. Other variables like changes in exam difficulty or the inclusion of more similar or peripheral questions could also play significant roles. Similarly, the claim that more coaching centres in Sikar, Kota, and Kottayam have resulted in students from these places scoring top ranks is quite an extrapolation and cannot be substantiated with just two years of NEET marks.

As a researcher, while I find these limitations significant, they should not be seen as criticising or undermining IIT-M’s efforts. Rather, they serve as a reminder of the complexities involved in detecting malpractice in high-stakes exams and the precautions needed when interpreting analysis findings. It should be noted that no research and statistical analysis is beyond limitations.

However, IIT-M should have explicitly spelled out the limitations of this analysis. In doing so, their analysis would have been more objective rather than appearing to simply express the sentiments of the NTA.

(The writer is an independent social and behavioural science researcher)