"AI engineer" is on every CV now. Some of those engineers have put AI features in front of real users, measured them and kept them working. Others have built a weekend demo. Both describe themselves the same way, and a standard interview rarely tells them apart.
This is a practical guide for CTOs and founders: what "AI-native" should mean when you hire, the questions that separate real experience from demos, and how to run a 7-day trial that tells you more than any interview.
What "AI-native" should mean
We use the term for two skills. A good hire has both.
They use coding agents well. They hand agents the volume, such as boilerplate, tests, refactors, documentation and first drafts, and keep the judgement for themselves. They read every line an agent writes before anyone else sees it. Speed is the result. Review is the job.
They build AI into products. They know where models fail, and design for it: an eval set before any tuning, citations users can check, an approval step before anything irreversible, personal data kept out of prompts, and a budget for cost and response time.
What it doesn't mean: a collection of prompt tricks, a list of every new framework, or a portfolio of chat demos. Ask about production, not novelty.
Questions to ask, and what a good answer sounds like
Ask for specifics. Strong candidates answer with names, numbers and mistakes. Weaker ones answer with tools.
- "Walk me through an AI feature you put in front of real users. What broke first?" A good answer names a real failure, how they found it (a trace, a user report, an eval) and what they changed. "Nothing really broke" is a red flag.
- "How did you know it was good enough to ship?" You want to hear about an eval set: how many cases, where they came from, what the pass mark was, and who agreed it.
- "Where did a person approve what the AI did, and why there?" Good answers put the approval just before something irreversible, such as sending, posting or paying. If they have never thought about it, they haven't shipped agents that touch real data.
- "Show me a pull request where a coding agent wrote most of the code. What did you change?" Look for specific edits: a missed edge case, a test that tested nothing, a dependency they removed. If they changed nothing, they didn't review it.
- "What do you never let an agent do?" Expect firm lines: no production access, no secrets, no unreviewed database migrations, no merging its own work.
- "How did you keep personal data away from the model?" A good answer describes removing it before the call, and why. A setting on the provider's dashboard is not the whole answer.
- "What did it cost per request, and how did you bring that down?" People who have run AI in production know this number, at least roughly. People who have only built demos usually don't.
- "How would the next engineer pick this up?" Listen for docs, decisions and notes kept in the repo, and small pull requests that each say what changed and why.
Red flags
- Every answer is about a tool, and none is about a user.
- No numbers, for quality, cost or speed.
- They can't explain the code in their own portfolio, line by line.
- They want production access in the first week.
- They talk about the model as the solution, rather than one part of the system.
How to run a 7-day trial
An interview tests how someone talks about work. A trial shows you the work. Seven days is enough to see how an engineer reads a codebase, plans, communicates and ships, if you set the trial up properly.
Before day one
- Pick a real task from your backlog. Not a toy, and not a take-home test. It should be small enough to finish in a week, and useful enough that you would merge it.
- Write down the success criteria. Three to five, each one checkable. "AI answers cite their sources, pass a 50-question eval set, and ship behind a feature flag" is a good example. "Improve search" isn't.
- Set up access safely. NDA first. Accounts you control, only the access the task needs, and no production access. Plan to remove all of it the day the trial ends.
Day by day
- Day 1: read and agree. A kick-off with your tech lead. The engineer reads the relevant code and confirms the success criteria in writing. Good questions on day one are a good sign.
- Day 2: plan. A short written plan: the approach, the risks, what they will build first, and what they need from you. Your lead approves it before the code starts.
- Days 3 to 5: build in small pieces. Small pull requests, each with tests and a short note on what changed and why. Your lead reviews every one. Watch how they take feedback.
- Day 6: show it working. Tests and evals pass, and the feature runs behind a flag in a real environment. Anything that didn't work is written down, not hidden.
- Day 7: review and decide. Go through the criteria one by one, with the engineer and your lead.
What to score on day seven
- Outcome: were the success criteria met? Where they weren't, is the reason clear?
- Code: would you merge it as it is? Could someone else maintain it?
- Judgement: what did they push back on, simplify, or flag as a risk?
- Communication: did you know where things stood each day without asking?
- Handover: are the decisions written down in the repo?
Give judgement the most weight. Coding agents have made writing code cheap. Knowing what to write, what to leave out and when to stop is the skill you are paying for.
How we run ours
This is how trials work at Wanile, and you can hold any provider to the same standard.
- The first 7 days are free.
- The engineer works on a real task from your backlog, against success criteria agreed with you on day one.
- Code and IP are yours from the first commit.
- If it doesn't work, you walk away and owe nothing.
- If it does, you keep the engineer month to month, with a short written update every Friday and 2 weeks' notice to stop.
The full terms are on How we work, and the engineers you could start with are on their profiles.