Generative AI did not repeal software engineering. It made mature engineering more valuable. The demos suggest magic; production systems still need pipelines, selection discipline, infrastructure, guardrails, prompt management, governance, IP clarity, and cost control.
This post expands on that thesis: GenAI projects succeed when they are treated as software products with probabilistic components—not as science fair exhibits bolted onto a website.
The problem: treating GenAI as an exception to engineering
Teams often split into two failure modes:
- Research theatre — notebooks, one-off prompts, no ownership, no SLOs, no rollback.
- Wrapper shipping — a thin UI on a public API with no evaluation, security model, or cost envelope.
Both can produce a splashy pilot. Both struggle in production when prompts drift, models change, users abuse inputs, or finance asks what the unit economics are.
GenAI introduces uncertainty into outputs, but it does not remove the need for requirements, testing, observability, and change management. It increases them.
Why it matters
Organisations are pouring attention into copilots, agents, and content systems. Without engineering maturity, they accumulate AI debt quickly: untested prompts, model lock-in, data leakage, unpredictable bills, and no way to know if quality improved.
There is also a market implication. ISVs and IT services firms that productise frameworks—evaluation harnesses, gateway patterns, guardrail libraries, delivery playbooks—can help customers move from pilots to operable systems. The opportunity is not only “build a chatbot.” It is “industrialise how chatbots and agents are built.”
What mature GenAI engineering includes
Pipelines
Data and document pipelines for RAG, training, or evaluation must be reproducible: ingestion, cleaning, chunking, permission propagation, refresh, deletion. If your corpus cannot respect access control and retention, your GenAI feature is a compliance incident waiting for a calendar invite.
Model selection
Choose models against task requirements: quality, latency, cost, context needs, tool-calling reliability, deployment constraints. Pin versions. Re-evaluate on change. Selection is an engineering decision with a product owner—not a one-time blog-post default.
Infrastructure
Serving, queues, caching, rate limits, secrets, GPU/CPU capacity where relevant, and multi-tenant isolation. Streaming UX needs back-pressure thinking. Tool-calling agents need sandboxes and permission boundaries.
Guardrails
Input/output filtering, policy checks, allow-listed tools, PII handling, jailbreak resistance, and human confirmation for high-impact actions. Guardrails belong in software and process, not only in hopeful system prompts.
Prompts as artefacts
Version prompts. Review them. Evaluate them. Own them. (Prompt debt is real; treat instructions like production configuration.)
Governance
Who may launch an AI feature? What data may leave the boundary? What audit logs are required? How are incidents handled when the model is wrong in a costly way? Governance should enable safe speed—not invent paperwork after a scare.
IP and data rights
Training data provenance, licence constraints, customer data isolation, output ownership, and retention. Legal and engineering need a shared checklist early—not after the sales deck ships.
Cost
Token spend, tool spend, cache hit rates, cost per successful task. Build budgets into design. A feature without an economic model is a science project on a cloud bill.
A practical delivery shape
Think in layers:
- Product outcomes — the user job and success metrics
- Workflow design — where generation helps vs where deterministic software should run
- Model + retrieval — chosen against evals
- Platform — gateway, auth, logging, caching, guardrails
- Operations — monitoring, incident response, cost review, retraining/refresh cadence
Most “GenAI projects” that hurt organisations skip layers 4–5.
Patterns that work
- Start with a narrow use case and a golden evaluation set
- Prefer retrieval + policy over fine-tuning when the need is freshness and control
- Keep side-effectful actions behind explicit tool permissions
- Log prompt/model versions with requests
- Separate experimental playgrounds from production tenants
- Design fallbacks: search, templates, human handoff
Patterns that fail
- No baseline: you cannot prove the model helps
- Unlimited context dumping as architecture
- One shared API key for everything
- “The model will refuse harmful stuff” as the security strategy
- Ignoring non-deterministic test strategy (eval suites, sampling, shadow traffic)
Opportunity for ISVs and IT services
There is room for opinionated frameworks:
- Reference architectures for RAG and agents
- Evaluation and red-team harnesses
- Cost and quality dashboards
- Industry-specific guardrail packs
- Delivery kits that connect architecture, security, and change management
The winners will sell operable systems thinking—not only model access.
Practical recommendations
- Appoint a product owner and an engineering owner for every GenAI feature.
- Require an eval pack and a cost model before production.
- Put a gateway/abstraction boundary in place early for core flows.
- Make guardrails and logging non-optional platform capabilities.
- Review IP/privacy constraints in the same gate as security review.
- Budget platform work; do not fund only demo UI.
Closing
Generative AI changes what software can do. It does not change what software must be: reliable, observable, securable, and economically sane.
If you want GenAI in production, hire and organise for software engineering—with probabilistic components under discipline. The magic is optional. The engineering is not.