July 8, 2026/2 min read

Inside an Autonomous Hackathon Screening Agent: Architecture, Scoring, and Failure Modes

by Akshat Kushwaha · updated August 18, 2026

India Innovates 2026 had roughly 3,000 submissions, each usually a PDF or PPTX link, and a shortlist deadline. the system needed to classify the file, render it, infer the relevant problem statement, assess the work, verify the claims it could verify, and leave a record of the decision.

the pipeline#

code

submission URL → dedupe → render slides → extract context
               → map problem statement → evaluate evidence
               → verify GitHub where present → score + audit trail

the score separated deck quality, alignment to the actual prompt, prototype credibility, and supporting media. the hard gate mattered most: no credible prototype evidence and no real repository activity meant an automatic rejection, even if the slides were polished. at hackathon scale, presentation and proof are not the same thing.

failure modes were product requirements#

duplicate rows and repeated links were skipped before work began. transient errors retried with backoff. a hard quota failure stopped the run rather than letting thousands of later requests write bad results. model output was validated before any score became permanent. workers coordinated so the same submission could not be evaluated twice.

for large batches, an asynchronous queue can lower cost, but it makes monitoring even more important. before a run, the diagnostics check keys, database access, expected tables, duplicates, and recent failures. catching a bad configuration before submission one is much cheaper than discovering it after submission 1,400.

the system sustained around 7.5 submissions a minute, so approximately 3,000 entries took about 400 minutes and about $20 in API credit.

i do not think the lesson is “use an LLM to judge everything.” the lesson is narrower: if you automate a decision that affects people, build the evidence checks, stopping conditions, and audit trail before you celebrate the throughput.

drafted with an LLM, then reviewed and edited by me before it went up.

the pipeline#

failure modes were product requirements#

leave something useful, or usefully sharp.