AI-native delivery

AI-native data pipelines

Large language models are primary contributors to every Avizing engagement, in how we build your Apache Hop platform and in what the pipelines themselves can do. Here is what that means in practice, and the guardrails that come with it.

AI-native delivery

Built with AI, built for AI

Large language models are primary contributors to every Avizing engagement: in how we design and build your platform, and in what the pipelines themselves can do.

AI as a primary contributor to the build

Our engineers work alongside LLM-based assistants from the first design session to the last runbook. Architecture options get explored and compared quickly, tests and documentation are generated alongside the code and kept current, and designs stay flexible because changing course is cheap.

  • Architecture options explored and compared in hours, not weeks
  • Tests, runbooks, and documentation generated with the code and kept current
  • Every AI-assisted change goes through the same pull request review, CI, and promotion path, with a named engineer accountable for it

Pipelines and workflows that can call an LLM

The Apache Hop pipelines and workflows we build can hand records to a language model as a normal step: classify free text, extract fields from documents and emails, normalize messy values, summarize, or flag data quality problems that rules cannot express. The guardrails are part of the step, not bolted on afterwards.

  • Hosted models or private models running inside your cloud boundary, your choice
  • Sensitive fields redacted before any call leaves your environment
  • Structured, schema-validated output; malformed replies are retried or routed for review
  • Token budgets, response caching, and per-run cost reporting
  • An evaluation set runs in CI so prompt changes are tested like code
HOP PIPELINEREAD RECORDSupstream transformsREDACTPII masked per policyLLM CALLversioned prompt templatehosted or private modeltoken budget, cache, cost logVALIDATEJSON against schemaTARGETwrite rowsrowssafe textreplyvalid rowsfailed validationREVIEW QUEUEretry or human checkprompt changes tested in CIEVALUATION SETknown inputs, expected outputsOnly redacted text leaves the pipeline. With a private model,nothing leaves your cloud account at all.Purple marks the model call and the evaluation loop that keeps it honest.
An LLM step inside a Hop pipeline: records are redacted, sent to a hosted or private model with a versioned prompt, and the structured reply is validated against a schema before it continues downstream. Replies that fail validation go to a review queue instead of into the target.

Working with AI agents

What agents are good at, and what to guard against

AI coding agents are a normal part of how we deliver now, and they are only worth having when the risks are handled deliberately. This is what we have found they do well, where they bite, and the working practices that keep them safe.

Where agents earn their keep

  • Volume work at speed: converting dozens of pipelines, adding an audit column everywhere, or generating the tests and documentation that would otherwise be skipped
  • Exploration: three architecture options with working code in the time it used to take to write one
  • Tireless review: reading every line of a large pull request and flagging the inconsistencies people skim past
  • Onboarding: explaining an unfamiliar Hop project, a legacy Kettle job, or a cloud setup to a new team member on demand

Where they bite

  • Confident mistakes: an invented transform name, a plausible but wrong join, a step that passes lint and fails on real data
  • Collisions: several agents and people editing one shared checkout leave half-applied changes and merge conflicts nobody asked for
  • Scope creep: an agent asked for one fix that also tidies twenty other files, burying the change that mattered
  • Data and secrets exposure: credentials pasted into prompts, or production data sent to a hosted model without redaction
  • Cost and drift: unbounded token spend, and prompts or conventions that change silently over time

How we work with agents safely

  • One worktree per task

    Each agent and each engineer works in its own Git worktree on its own branch, never in a shared checkout that someone else can switch or dirty underneath it. Changes from many sources land as separate, small pull requests, so conflicts surface at merge time, where they are cheap to resolve, instead of mid-edit, where they corrupt work.

  • No production credentials

    Agents run with least-privilege identities against development or sandbox environments. Production access belongs to the deployment pipeline, not to anything that types.

  • Small, reviewed, tested

    Agents are asked for one change at a time. Every pull request runs lint, unit tests, and the evaluation set, and a named engineer reviews it, on the Hop Web canvas where that helps.

  • Redaction and budgets

    Sensitive fields are masked before any prompt leaves your boundary, and token budgets with per-run cost reporting keep spend visible and bounded.

  • Provenance

    Commit messages and pull requests record what was AI-assisted and which instruction produced it, so an odd change can be traced and understood later.

ONE SHARED CHECKOUTEngineerAgent AAgent Bcheckout feature-acheckout feature-bcheckout feature-cWORKING COPYfiles from whichever branchwas switched to last, pluseveryone's uncommitted editshalf-applied changes, surprise conflictsmainwhatever got committedONE WORKTREE PER TASKEngineerworktree a/branch feature-aAgent Aworktree b/branch feature-bAgent Bworktree c/branch feature-cPULLREQUESTSCI, reviewmainconflicts resolved here, visiblyEach directory is isolated; nobody switches or dirties another's files.Every branch switch changes the files under someone else's edit.
Left: several contributors switching branches inside one shared checkout, where someone's half-finished edit sits underneath everyone else's work. Right: one worktree per task, each on its own branch, meeting only at pull request time where conflicts are visible and cheap.

Questions

AI agents and data pipelines, briefly

Do AI agents replace data engineers?
No. Agents do volume work, exploration, and review at speed, but every change they make is reviewed by a named engineer and reaches production through the same pull request, CI, and promotion path as any other. The engineer stays accountable for the result.
Why Git worktrees instead of switching branches?
A worktree is a separate directory checked out to its own branch. When several agents and people are making changes at once, giving each task its own worktree means nobody switches or dirties the files someone else is editing. Conflicts still happen, but they show up at merge time as clean, reviewable diffs rather than as corrupted half-edits in a shared checkout.
Is our data sent to an AI vendor?
Only if you choose a hosted model, and then only after sensitive fields are redacted inside your environment. With a private model running in your cloud account, nothing leaves your boundary at all. Credentials are never placed in prompts or repositories.
How do you keep AI-generated pipelines from breaking production?
Agents work in development or sandbox environments with least-privilege identities. Every pull request runs lint, unit tests, and an evaluation set of known inputs and expected outputs before an engineer reviews it, and promotion to production is a separate, gated step.

Next step

Free Consultation

Tell us about your data platform, or the one you wish you had. A short call is usually enough to say whether we can help and what it would take.

Get in touch(877) 881-6238