You turned on AI resume screening. The first batch of scores came back. The candidate your hiring manager loves got a 64. The one nobody recognizes got a 91. Now the team does not trust the tool.
The problem is almost never the AI model. It is the criteria and weights you gave it — or, more likely, the defaults you never changed.
Recent keyword data from DataForSEO Labs (United States, English) shows “ai resume screening” at roughly 390 monthly searches, with commercial intent. Teams are actively searching for this. Many of them will hit the same calibration wall.
Why default weights fail
Every AI scoring tool ships with defaults. Those defaults are reasonable guesses: weight experience heavily, give moderate credit for skills match, add some value for education and certifications. They are built to work across thousands of job types.
Your role is not “thousands of job types.” Your role has specific tradeoffs. Maybe you care more about industry experience than tool proficiency. Maybe certifications are irrelevant for your team but the AI gives them 15% weight.
Default weights produce default rankings. Default rankings produce teams that either override every score manually — defeating the purpose — or accept scores they should not trust.
The calibration problem is human, not technical
Salesso’s 2026 recruitment data report found that 83% of companies now use AI for resume screening. But a Harvard Business School and Accenture study found that U.S. employers eliminated over 27 million job seekers through automated screening — many fully capable of performing the work.
The issue is not that AI cannot screen. It is that the criteria driving the screen were set once (or never set at all) and never revisited. Calibration is the missing step between “we have AI scoring” and “our AI scoring helps us hire better.”
For a broader overview of how AI candidate scoring works and common pitfalls, see our AI candidate scoring explainer.
Step 1: Start with must-haves, not weights
Before you adjust any slider, get the hiring manager to separate requirements into two buckets:
Must-haves (pass/fail):
- Work authorization for your location
- Required certifications or licenses
- Minimum years in a specific domain (be honest about whether this is a real requirement or a preference)
- Language requirements
- Willingness to work on-site, hybrid, or travel
Scored criteria (weighted):
- Relevant technical skills
- Industry or domain experience
- Evidence of outcomes (not just responsibilities)
- Leadership or management experience
- Cultural or team-dynamic signals from application materials
Must-haves should be binary filters, not scored. If a candidate does not have the right work authorization, a high skills score should not push them past the threshold. In Canvider, CriteriaMatch handles this layer — setting hard criteria that AI checks independently from the scoring weights.
Step 2: Agree on what “strong” looks like
This is where most calibration efforts skip a step. The team agrees on five criteria. They assign weights. But nobody defines what a 4 out of 5 looks like versus a 3.
Run a 20-minute exercise with the hiring manager:
- Pick two to three resumes the team has already reviewed and agreed on
- Score each one against the criteria, independently
- Compare scores and discuss discrepancies
- Write one-sentence definitions for each score level on each criterion
This is not academic. It is the difference between “the AI gave them a 78” and “the AI scored their systems experience at 4/5 because they listed three production-scale deployments.”
Step 3: Review outputs after the first batch
The first batch of scored candidates is a calibration test, not a final ranking.
After the AI scores the first 10 to 15 applicants:
- Check the top five. Are these people the hiring manager would actually want to interview? If not, which criteria are over-weighted?
- Check the bottom five. Is anyone down there who should be higher? What did the scoring miss?
- Look for patterns. Are candidates with short resumes systematically penalized? Are career changers getting low scores despite relevant transferable experience?
Adjust weights based on what you find. This is not “fiddling with the AI.” It is the same thing you would do if you trained a junior recruiter to screen — review their work, give feedback, adjust.
Step 4: Revisit weights for each new role
Weights that work for a senior backend engineer will not work for a customer success manager. Different roles demand different criteria balances.
When a new role opens:
- Start from the previous role’s weights as a template, not as gospel
- Check whether the must-haves changed (they usually do)
- Ask the hiring manager: “If you could only look at two criteria, which two?”
- Adjust weights so those two criteria represent at least 40% of the total score
This takes ten minutes. It saves hours of misranked candidates and debriefs about why the AI “got it wrong.”
The bias calibration layer
AI screening bias is not hypothetical. A 2025 study published on arXiv found that general-purpose language models used for resume screening show demographic impact ratios as low as 0.809, well below the 0.957 near-parity benchmark that domain-specific hiring models achieve. Intersectional disparities were even wider.
Calibration does not eliminate bias, but it reduces the surface area:
- Define criteria based on job-related evidence. “Worked at a top company” is a proxy signal. “Shipped a production system serving 10K+ users” is evidence.
- Audit score distributions. After a scoring round, check whether any demographic group is systematically scoring lower. If so, examine which criteria are driving the gap.
- Use the AI as a first pass, not the final decision. The hiring manager reviews the short list. The AI prioritizes the inbox. Humans decide.
New regulations in New York City, Illinois, Washington State, and the EU now require some form of bias auditing for AI hiring tools (Articsledge, 2026). Even if your jurisdiction does not mandate it yet, calibrating against bias is good practice and good hiring.
How Canvider supports calibration
Canvider AI Score lets you set and adjust scoring weights per role. CriteriaMatch handles the pass/fail layer separately so hard requirements do not get mixed into weighted scores. And Collaborative Candidate Assessment gives the team a shared view of how scored candidates compare — so calibration feedback happens in context, not over email.
The workflow: set criteria, score the first batch, review together, adjust, and score again. Each round gets sharper.
Start calibrating this week
Pick your next open role. Before any candidates apply:
- Separate must-haves from scored criteria
- Assign initial weights with the hiring manager
- Define what “strong” means for each criterion in one sentence
- After the first 10 applicants, review the top and bottom five together
- Adjust and re-score
AI scoring is a lever. Calibration is what points it in the right direction.