Response evaluation
Assess responses against the task, context and stated success criteria.
LANGUAGE MODELS / HUMAN EVALUATION
LLM evaluation.
Reasoned judgments.
Structured human assessment of AI responses against agreed criteria for factuality, relevance, instruction following and safety. Clear rationales, with uncertainty made visible.
12+ years of relevant professional experience · Direct founder involvement
SYNTHETIC RUBRIC / HUMAN REVIEW
ResponseEvidence
before preference.Criteria → Judgment → Rationale
before preference.Criteria → Judgment → Rationale
LLM EVALUATION CAPABILITIES
Human judgments.
Explicit criteria.
Structured response review can reveal where outputs diverge from task expectations. Altema focuses on the agreed evaluation task, not model development or a promise of model-level certification.
Preference rating
Compare candidate responses and record a preference with a criterion-based rationale.
Factuality
Check claims against supplied evidence; flag unsupported assertions and uncertainty.
Relevance & instruction following
Review whether the response addresses the request and respects explicit constraints.
Safety evaluation
Review outputs against client-defined safety criteria within the agreed scope. Domain fit and sensitive-content requirements are assessed first.
Rubric-based assessment
Apply defined scales or categories, calibrate examples and document ambiguous decisions.
ILLUSTRATIVE RESPONSE COMPARISON
Ground the summary.
Explain the preference.
A fictional maintenance notice provides a controlled source. Compare the two synthetic responses against the same instruction; no real model or client result is represented.
FICTIONAL SOURCE
“For the Atlas demonstration, a maintenance window is scheduled for Tuesday from 02:00 to 03:00 UTC. During the window, dashboard access may be interrupted. API requests are expected to continue. The notice does not specify whether existing sessions will be signed out.”
Instruction: Summarize this notice in two sentences. Preserve uncertainty and use only facts stated in the source.
“The Atlas demonstration has a maintenance window scheduled for Tuesday, 02:00–03:00 UTC, when dashboard access may be interrupted. API requests are expected to continue; the notice does not say whether existing sessions will be signed out.”
Retains the source’s distinction between a possible interruption and an expected continuation.
“The Atlas demonstration will be unavailable on Tuesday, 02:00–03:00 UTC, because a security upgrade will sign everyone out. API requests are guaranteed to continue without errors.”
Introduces a security upgrade and forced sign-out, while turning expectations into guarantees.
Illustrative evaluation
- Factuality
- A stays within the source; B introduces unsupported claims.
- Instruction Following
- Both use two sentences; only A preserves uncertainty and source-only facts.
- Relevance
- Both address maintenance, but B adds an unsupported cause and outcome.
- Evidence-based rationale
- “May be interrupted” and “expected to continue” support A; neither guarantees nor forced sign-out are supported.
- Preferred Response
- Response A — the preference follows the stated evidence and constraints.
Reveal the illustrative assessment
| Criterion | Response A | Response B |
|---|---|---|
| Factuality | Matches the stated window, possible dashboard interruption and expected API continuation. | Claims a security upgrade and forced sign-out that the source never states. |
| Instruction Following | Two sentences, source-only facts, uncertainty preserved. | Two sentences, but violates the source-only and uncertainty constraints. |
| Relevance | Focuses on the maintenance impact and the unresolved session question. | Addresses the topic, but introduces an unsupported explanation and widens dashboard interruption to full unavailability. |
| Evidence-based rationale | Uses “may” and “expected”; explicitly retains what the source does not specify. | “Will be unavailable,” “security upgrade,” “sign everyone out” and “guaranteed ... without errors” go beyond the source. |
| Preferred Response | Response A, for factual support and adherence to the instruction. | Not preferred under this rubric; no general claim about a real model is made. |
Illustrative preference: Response A. This is a worked synthetic example, not a measured project result, benchmark, or assessment of a named model.
WHAT YOU RECEIVE
Outputs with context.
Final scope, tools and export requirements are confirmed before work starts. The items below describe proposed deliverable types, not a promise of every format or integration.
Structured evaluation records
Ratings, preference judgments and concise rationales in a structure agreed for the project.
Error and ambiguity notes
Flagged factuality, relevance, instruction or safety issues linked to the applicable criteria.
Pilot review summary
Observed rubric ambiguities and recommendations for clarifying the evaluation task. No statistical model-performance claim is implied.
QUALITY BY DESIGN
Agree on quality.
Then evaluate the work.
A focused pilot helps establish the task rules, review method and acceptance criteria. No generic accuracy promise replaces a task-specific agreement.
Define rubric
Agree on criteria, evidence sources, rating anchors and escalation rules.
Calibrate
Review sample responses to align interpretation before the main task.
Evaluate
Apply the rubric and document uncertainty or conflicting criteria.
Review findings
Check consistency and deliver the agreed records with contextual notes.
PROJECT QUESTIONS
Clarity before
commitment.
Scope and suitability are reviewed directly with the founder.
Does factuality review always include external research?
No. The scope must specify whether review is grounded in supplied evidence or requires additional verification. External research access, expertise and time must be agreed separately.
Can you assess a specialist domain or every language?
Domain knowledge and language requirements are assessed before accepting a task. Broad or specialist coverage is not assumed.
Is safety review a compliance certification?
No. A scoped review against defined criteria does not establish legal compliance, certify a model, or guarantee that all harmful outputs will be discovered.
Can results be used to compare model versions?
That depends on the evaluation design. Sampling, rubric stability and comparison methodology must be agreed before any general performance conclusion is drawn.
START WITH A CLEAR SCOPE
Bring the challenge.
Let’s define the pilot.
Independent, founder-led delivery. Capacity, timing and requirements are assessed for each project.