Decision guide / Artificial intelligence

AI or automation: how to choose for each process.

Use rules for exact conditions, integration to move data between systems and AI when the task requires interpreting variable information. Then decide what can act automatically and which results need review.

By Kenea · Published and reviewed: 8 September 2026.

The difference is how the result is produced

Automation means a system performs process steps under defined conditions. It may use rules, calculations, data exchanges or AI models. “AI or automation” is therefore not always a choice between two products: an automation can include an AI task.

Here, AI mainly means models that interpret or generate text and documents. Other AI techniques, such as predictive models, need their own design and evaluation. Having a chatbot does not demonstrate that an operational process is integrated.

A rule can also be poorly defined, and an integration can fail without using AI. The technical choice changes what you need to check; it does not remove the need for testing.

A defined workflow and an agent have different needs

Anthropic distinguishes workflows with predefined steps from agents that dynamically choose their path and tools. Autonomy can help with open-ended tasks but introduces more decisions to evaluate. Before adding it, check whether a simpler workflow already solves the case. Source: Building effective agents.

Reading a request, proposing a category and placing it in an inbox is a bounded path. Allowing an agent to query applications, change records and write to third parties broadens the scope. Specify which actions it may take, with which data and when it must stop. The word “agent” does not replace those decisions.

How to combine rules and AI in one process

Hypothetical example: a company receives maintenance requests by email. A design could separate five steps; this does not represent a Kenea installation or result.

1. Receipt: record the message reference and avoid processing the same input twice.
2. Interpretation: suggest the affected equipment, location and issue type from the text.
3. Validation: check that the equipment exists and required fields are present.
4. Review: show ambiguous requests to the responsible person, with evidence for each suggestion.
5. Registration: create a task with accepted data and check that the destination returns the expected reference.

AI would assist with interpretation. Request identity, permissions and business checks would remain system responsibilities. If the model suggests a location absent from the inventory, the workflow should flag it instead of filling in the detail on its own.

This division also helps locate problems. Misclassification calls for reviewing examples and criteria; a duplicate record calls for investigating the integration. Changing the model does not necessarily fix both.

Evaluate a sample that resembles real work

Gather authorised inputs covering common formats, incomplete documents, language variations, ambiguous cases and out-of-scope situations. Define expected results with someone who knows the process. Reserve examples for checking changes instead of repeatedly using the same ones during tuning.

Evaluation must assess the task’s result, not just whether the response sounds convincing. Anthropic explains this distinction and recommends defining inputs, success criteria and repeatable checks. Outputs can vary between runs, so one successful example does not establish stability. Source: Demystifying evals for AI agents.

• Quality: which fields, categories or claims are correct, and which errors have greater consequences.
• Coverage: how many inputs it resolves, refers for review or cannot handle.
• Review: how long a person takes to check and correct each result.
• Time: how long users wait and how the process handles multiple requests.
• Cost: model usage, search, storage, integration and supervision.

Compare the proposal with the current process and a simple alternative. Fast drafts may add little if checking them means repeating the task. For document extraction, inspect identifiers, dates, amounts and free text separately: one average may hide important errors.

Do not treat a model’s stated “confidence” as a calibrated probability of correctness. If a threshold will route cases for review, test its relationship to observed errors in a representative sample. Keep review in place where evidence is insufficient.

Define what people review and what the system may do

“There will be human oversight” is insufficient if nobody knows what to check. Reviewers need the source data, proposal, checks already performed and available actions: accept, correct, request information or stop the case. They also need allocated time and a queue that does not hide pending work.

Separate reading from writing. A trial may start by querying data or preparing drafts. Authorising a later change requires an explicit decision on scope, validation and recovery. Avoid granting full application access for a task needing only a few fields.

Documents and emails may contain instructions intended to change model behaviour. OWASP describes this as prompt injection. The application must treat that content as data to examine, maintain permission boundaries and validate actions outside the generated text. Source: OWASP Gen AI Security Project, Prompt Injection.

Before using real documents, agree what information is needed, where it is processed, who can access it and how it is deleted. Provider and model selection should consider these conditions alongside quality, cost and operation.

When to proceed and when to change approach

Proceed when the sample explains the improvement, errors are visible and there is a workable resolution procedure. If exceptions require more work than the process saves, it needs a different scope. If source data is insufficient, improving collection may be more useful than asking a model to infer it.

Document the task, comparison approach, acceptance criteria, action limits and who will review future changes. Repeat relevant checks when models, instructions, documents or tools change. An initial demonstration supports investigation; acceptance requires evidence of the complete path.

Kenea’s applied AI work starts by evaluating that fit. You can explore your process in the free assessment, review what to automate first or explore our solutions before choosing an implementation. The tool gives guidance from your choices; a commissioned technical assessment requires agreement on scope, price and schedule.

Explore my process Review the integration

Sources and scope

Primary sources consulted on 8 September 2026: Anthropic, Building effective agents (19 December 2024; the page notes later tool changes), Anthropic, Demystifying evals for AI agents (9 January 2026), and OWASP, LLM01:2025 Prompt Injection. These are design, evaluation and security references, not endorsements of Kenea or a particular provider. The example and matrix are editorial proposals.