July 8, 2026/2 min read

inside an autonomous hackathon screening agent

by Akshat Kushwaha · updated November 5, 2026

India Innovates 2026 had roughly 3,000 submissions, each usually a PDF or PPTX link, and a shortlist deadline. the system needed to classify the file, render it, infer the relevant problem statement, assess the work, verify the claims it could verify, and leave a record of the decision.

the pipeline#

code

submission URL → dedupe → render slides → extract context
               → map problem statement → evaluate evidence
               → verify GitHub where present → score + audit trail

the score separated deck quality, alignment to the actual prompt, prototype credibility, and supporting media. the gate that mattered most was the hard one, no credible prototype evidence and no real repository activity meant an automatic rejection, even if the slides were polished, because at hackathon scale a presentation and proof aren't the same thing.

failure modes were product requirements#

duplicate rows and repeated links got skipped before any work began, and transient errors retried with backoff. a hard quota failure stopped the run instead of letting thousands of later requests write bad results. model output was validated before any score became permanent, and the workers coordinated so the same submission couldn't be evaluated twice.

for large batches an asynchronous queue can lower cost, but it makes monitoring matter even more. before a run the diagnostics check keys, database access, expected tables, duplicates and recent failures, because catching a bad configuration before submission one is much cheaper than discovering it after submission 1,400.

the system sustained around 7.5 submissions a minute, so approximately 3,000 entries took about 400 minutes and about $20 in API credit.

i wouldn't take "use an LLM to judge everything" from this, the lesson is a narrower one. if you automate a decision that affects people, build the evidence checks and stopping conditions and audit trail first, before you celebrate the throughput.

drafted with an LLM, then reviewed and edited by me before it went up.

the pipeline#

failure modes were product requirements#

leave something useful, or usefully sharp.