An AI-powered feature can work perfectly in a demo and still cause problems once customers start using it. A support assistant might invent a refund policy. A shopping chatbot could recommend products that are out of stock. An AI agent with access to customer accounts might perform an action the user never authorized.
These aren’t necessarily problems with the underlying model. They often come down to how the application was designed, what information the model can access, and how its responses are checked.
If you’re planning to bring AI talent onto your team, previous experience with ChatGPT or other popular AI tools shouldn’t be your main hiring criterion. You need people who understand the engineering work around the model and can explain how their decisions will affect customers, maintenance costs, and the business itself.
Production Experience Matters More Than an Impressive AI Portfolio
A candidate shows you an AI assistant that answers questions about uploaded documents. The interface looks good, the answers are relevant, and when it comes to demonstration, it only takes around five minutes.
What you haven’t seen is how the assistant behaves when a document contains contradictory information, the user asks an unrelated question, or the model provider becomes unavailable.
That’s the gap between a prototype and a production application.
Previous projects are worth discussing in detail. Find out which parts of the application the candidate built and whether they stayed involved after launch. Developing a chatbot interface is one thing. However, implementing document retrieval, access controls, and production monitoring requires a different set of skills.
Problems encountered in production can tell you even more. An assistant might have started returning outdated answers because its document index wasn’t updated, or struggled with response times as the amount of processed data grew. Find out how the developer investigated these issues, what changes they made, and whether those changes worked.
It’s also worth distinguishing between experience with AI models and experience with the surrounding software. A developer familiar with Python, FastAPI, and model APIs may be well suited to building a conversational feature without having trained a model from scratch.
For many commercial products, that’s exactly the expertise required.
Find Out Whether They Can Challenge the Proposed Solution
Imagine an online retailer wants to introduce an AI shopping assistant. Customers should be able to describe what they’re looking for, compare products, and receive recommendations.
A developer could immediately suggest an LLM connected to the product catalog through a vector database. That might work. But there’s a more important question to answer first: what problem is the retailer trying to solve?
If customers struggle to find products because search filters are inadequate, improving the existing search functionality may be cheaper and more effective. If they want to describe complex preferences in natural language, a conversational interface becomes more interesting.
The distinction matters because language models introduce costs and behavior that conventional software doesn’t. Their responses can vary, and they may also produce inaccurate information even when given relevant source material.
An experienced developer should be comfortable discussing alternatives.
For example, retrieval-augmented generation (RAG) can help an assistant use current product descriptions or support documentation without retraining the underlying model. Fine-tuning may be appropriate when a model needs to learn a specialized task or output pattern, but it isn’t a reliable substitute for retrieving frequently changing facts.
There’s also no reason to let an LLM calculate a discount when the business already has explicit pricing rules.
Good technical judgment is particularly valuable in AI product development because choosing an unnecessarily complex architecture can leave a company paying for capabilities it doesn’t need.
Look Closely at How They Connect AI to Business Systems
Consider a customer asking an AI assistant to change the delivery address for an existing order.
The assistant needs to identify the request, retrieve the correct order, check whether the customer is authorized to modify it, and determine whether the order is still eligible for changes. Only then should the application submit an update.
The language model can help interpret the request. It shouldn’t be responsible for deciding whether the customer has permission to access the order.
This distinction is easy to overlook in demonstrations where developers use sample data and simplified workflows.
Production LLM integration requires conventional backend engineering alongside knowledge of model behavior. API calls can fail, generated responses may not follow the requested format, and regarding external services, they can impose rate limits. If an application expects structured data, it needs to validate the output before using it.
A candidate familiar with OpenAI’s function calling or comparable tool-use capabilities should be able to explain how these mechanisms interact with application-level controls.
Ask them to walk through a failed transaction. What happens if the model requests an address change but the order management API times out? Does the assistant tell the customer the change was successful? Can the system retry without accidentally performing the operation twice?
These details reveal whether the developer has thought beyond generating a plausible response.
Customer Data Requires More Than a Privacy Statement
Suppose a company operates a subscription platform where customers can ask an AI assistant about their invoices, account settings, and payment history.
The assistant needs access to relevant records, but that access must be limited to the authenticated customer. A poorly designed retrieval system could expose information belonging to another account, even if the language model itself is functioning as expected.
The developer should know where authorization checks belong and how to preserve them when AI-generated requests reach backend services.
Third-party model providers add another consideration. Before sending customer information to an API, the team needs to understand the provider’s data retention terms, processing arrangements, and available privacy controls. These conditions can differ between consumer products, business subscriptions, and API services from the same provider.
For UK businesses, data privacy decisions may also involve UK GDPR obligations, including lawful processing, data minimization, and appropriate security measures.
Security testing should cover prompt injection as well. A malicious customer might attempt to persuade an assistant to reveal restricted information. Instructions hidden inside retrieved documents can also influence model behavior.
OWASP’s Top 10 for LLM Applications identifies prompt injection, sensitive information disclosure, and excessive agency among the risks developers need to address.
No prompt can guarantee that a model will ignore every malicious instruction. The application needs independent controls over what information can be retrieved and which actions can be executed.
For developer vetting, a practical scenario is more revealing than asking whether someone knows about AI security. Describe an assistant connected to customer accounts and ask the candidate how they would prevent unauthorized access. Their proposed architecture should make those protections understandable.
Ask for Evidence That the Feature Can Be Tested
An AI assistant answers 90 out of 100 test questions correctly. Is it ready for customers?
That depends on the questions, the nature of the errors, and what happens when the assistant gets something wrong.
If the ten incorrect answers concern store opening hours, the consequences may be limited. If they involve refund eligibility or account balances, the same accuracy figure tells a different story.
Testing needs to reflect the product’s actual risks.
For a customer support assistant, developers can build an evaluation dataset using representative questions, approved answers, and examples of requests that should be transferred to a human agent. Tests should also cover ambiguous wording, missing information, and attempts to obtain restricted content.
Frameworks such as DeepEval and LangSmith can help automate parts of this work. They can compare outputs, run repeated evaluations, and help teams investigate changes in behavior.
But automated evaluation has limits. Using another language model to judge an answer can produce misleading scores, particularly when the expected response requires domain-specific knowledge. Human review remains necessary for important cases.
The testing process also needs to continue after release.
Imagine a team switches from one model version to another to reduce costs. The new model performs well on common questions but becomes less reliable when extracting order numbers from messages. Without a regression test covering that task, the problem may go unnoticed until customers report it.
Candidates who have maintained production AI features should be familiar with this kind of tradeoff. Ask how they would decide whether a model update is safe enough to deploy.
Check Whether Their Cost Estimates Hold Up
Running costs deserve attention before an AI feature goes live. A support assistant handling 20,000 conversations a day would generate approximately $12,000 in monthly model charges at two cents per conversation. Raise the average to five cents, and the bill reaches $30,000. These figures are illustrative, but they show how much a small difference in per-conversation costs can matter at scale.
A realistic estimate needs to account for the selected model, expected traffic, input and output tokens, and any additional processing. Developers should be able to explain their assumptions and identify which parts of the application are likely to consume the most resources.
Long conversation histories and large amounts of retrieved content can drive up token usage. Prompt caching, shorter context windows, and smaller models can help control expenses, although their usefulness varies. Trimming conversation history may remove important context, while a cheaper model could struggle with more demanding requests. Caching is worth considering when enough prompt content can be reused to offset its costs.
Performance deserves attention too. A model that produces excellent answers but takes 15 seconds to respond may be unsuitable for an interactive shopping experience.
An experienced engineer should be able to discuss acceptable response times, expected traffic, and the cost of meeting those requirements before the business commits to an architecture.
Use a Realistic Product Scenario During the Interview
A portfolio review and technical interview won’t tell you everything about how someone works. A short design exercise can fill some of the gaps without asking candidates to build an unpaid prototype.
Give them a scenario related to your product.
For instance, a travel booking platform wants an AI assistant that can answer questions about reservations and help customers request changes. The candidate should explain what information the assistant needs, which operations require confirmation, and how it would handle requests the system cannot complete.
The candidate’s approach should account for cancellation policies, supplier API limitations, payment adjustments, and changes in availability while a request is being processed. These details affect how the booking assistant should work and what it can realistically handle.
The discussion should also give you an idea of how well the developer communicates with nontechnical colleagues. Can they explain the implications of an architectural decision without getting lost in implementation details?
For external hires, responsibilities need to be agreed on before development starts. This includes architecture, code reviews, testing, deployment, and ongoing support, along with repository access, documentation, as well as knowledge transfer arrangements.
These responsibilities are easy to leave vague during recruitment and much harder to resolve halfway through development.
Make the Hiring Decision Around the Product’s Risks
Before choosing between candidates, identify the parts of your proposed feature where failure would have the greatest consequences.
For a shopping assistant, inaccurate recommendations may be the main concern. A financial services application will have stricter requirements around sensitive information as well as potentially harmful responses. An AI agent that can modify customer records needs carefully controlled permissions and reliable transaction handling.
Use those risks to guide the final technical discussion.
A candidate doesn’t need experience with every model or framework on the market. They do need relevant engineering skills, a credible approach to testing, and the judgment to recognize when a proposed AI feature is more complicated than the business requires.
The most useful hiring evidence comes from specific decisions candidates have made, problems they’ve solved, and limitations they’re willing to acknowledge.