Applied AI interview guide

Unofficial prep for AI engineer interviews focused on GenAI systems: RAG, agents, evaluation, and production operations. This is not an Anthropic certification guide.

How to answer

Structure most answers as:

1. Clarify goals, constraints (latency, cost, risk), and success metrics
2. Propose an architecture with trade-offs
3. Evaluate how you would measure quality and catch regressions
4. Operate monitoring, failure modes, and rollout

Interviewers reward judgment under constraints more than buzzwords.

Domains

LLM APIs and integration

• Assemble streamed tool/content blocks; honor stop reasons before side effects.
• Bounded exponential backoff with jitter; layered timeouts; idempotent side effects.
• Schema-constrained outputs + server validation; prompts versioned like code.
• Adapter interfaces for multi-cloud with pinned model IDs and contract tests.

RAG and retrieval

• Chunk by structure with metadata (ACL, version/date); re-index on change.
• Hybrid lexical + dense when exact tokens matter; rerank shortlists.
• Ground answers; cite evidence; refuse when weak; treat retrieval as untrusted.
• Evaluate retrieval (recall@k) and answer faithfulness separately.

Agents and orchestration

• Prefer deterministic workflows when steps/validators are known.
• Prefer agents for open-ended exploration — name loops, tool hallucination, weak stops.
• Control flow: tool_use → execute → tool_result → continue; cap iterations.
• HITL before money moves, deletes, or external communications.

Framework selection

Do not choose a framework from a feature checklist alone. Start with the required control flow, state lifetime, failure recovery, side effects, and audit needs. Keep model, tool, state, and event contracts behind application-owned interfaces to reduce lock-in.

LangGraph details

• Reducers define how parallel node updates merge into shared graph state; avoid implicit last-write-wins behavior.
• Checkpointers persist thread-scoped execution snapshots. Stores hold durable data across threads.
• interrupt() plus a stable thread ID supports approval/edit flows. Resume must be safe if the interrupted node executes again from its start.
• Conditional edges and subgraphs make routing explicit; stream node or message events according to the UI’s observability needs.

CrewAI details

• Flows provide state, events, branching, and production control; Crews perform bounded collaborative work inside that structure.
• Sequential processes suit known ordering. Hierarchical processes add a manager that delegates and reviews, but also add cost, latency, and another failure point.
• Guardrails validate or transform task output. Checkpointing/persistence prevents completed work from being repeated after interruption.

AutoGen details

• AgentChat offers higher-level agents and team patterns; Core provides the event-driven runtime for custom distributed behavior.
• Use simple team patterns before GraphFlow. GraphFlow adds directed sequential, parallel, conditional, and looping execution when strict ordering is required.
• Execution flow and message visibility are separate concerns: filter messages so each agent receives only relevant context.

Evaluation and observability

• Versioned golden sets + rubrics; run on every prompt/model change.
• Calibrate LLM-as-judge against humans; keep held-out slices.
• Traces: prompt/version, retrieval, tools, outputs, latency, cost (redacted).
• Canaries/shadow evals before cutover; cluster failures for targeted fixes.

Evaluate outcomes and trajectories

• Score task outcome (was the goal achieved?) separately from trajectory quality (were the right tools selected, arguments valid, steps efficient, and side effects correct?).
• Use deterministic tool stubs and recorded fixtures to replay failure paths without touching production systems.
• Test long-horizon behavior across repeated runs. Report success rate and uncertainty instead of treating one nondeterministic run as pass/fail.
• Include malformed tool results, timeouts, partial failures, loop pressure, prompt injection, and denied approvals in agent eval suites.
• Calibrate model judges against human labels; version judge model, prompt, and rubric; defend the judge from untrusted evaluated content.
• Gate releases on a quality/latency/cost Pareto envelope, not quality alone.

Observe the whole run

• Create one root run span with child spans for model calls, retrieval, tools, guardrails, handoffs, and human approvals.
• Propagate trace context through queues and agent-to-agent messages so asynchronous work stays connected.
• Record model/prompt/tool/schema versions, routing decisions, token and cost counters, retries, termination reason, and outcome labels.
• Redact or hash sensitive prompt/tool fields before export. Control high-cardinality attributes and tenant-specific retention.
• Combine head sampling for baseline coverage with tail sampling for errors, slow runs, high cost, loops, and policy failures.
• Turn incident traces and user feedback into versioned regression cases; alert on task-success decline, loop rate, tool failures, guardrail bypass, and cost per successful task.

Official references: LangGraph overview, CrewAI documentation, AutoGen AgentChat, and OpenTelemetry. Framework APIs evolve; verify version-specific behavior in official documentation.

Production ops

• SLOs on TTFT and p95 by query class; stream for perceived speed.
• Cost: caching, cascades, batch for offline, $/successful task.
• Fallbacks, circuit breakers, load shedding, degraded keyword modes.
• Feature flags for fast prompt/model rollback.

Prompting and context

• Durable policy in system templates; live task in user turns.
• Manage long context with summary/RAG — not infinite concat.
• Few-shot: format-matched, including confusion-pair edges.

Safety

• Trust boundaries for user/retrieved content; validate tool args.
• Least-privilege tools; output redaction; abuse quotas.
• Safety evals: injection, leakage, high-risk tool scenarios.

Applied ML fundamentals

• Cosine similarity is not entailment; measure embedding retrieval on your data.
• Prefer classical code/rules for deterministic exact work; LLMs for messy language.
• Hybrid: structured source of truth + LLM explanation layer.

Practice loop

1. Take a Full Mock (40q) under light time pressure.
2. Review misses by domain — rewrite the correct option in your own words.
3. Rebuild one small system (RAG endpoint or tool loop) and add an eval slice.
4. Re-drill weak domains in Study mode.

Good luck.