Apache Hop consulting

Apache Hop, from first pipeline to production platform

We help teams adopt Apache Hop and run it the way an enterprise expects: projects in Git, Hop Server and Hop Web in containers, credentials from a secrets manager, and a runtime that fits your cloud. This page covers what Hop is, how we deploy it, and what a cloud-native Hop platform looks like.

Apache Hop

Apache Hop, deployed properly

Apache Hop is the open source data orchestration and integration platform from the Apache Software Foundation. Pipelines and workflows are designed visually and stored as plain text files, which makes Hop a natural fit for Git, containers, and cloud-native operations.

We help teams adopt Hop, migrate to it from Kettle and Pentaho Data Integration, and run it in production the way an enterprise expects: reproducible builds, no secrets in repositories, and nothing edited by hand on a server.

New to Hop? The project site and documentation live at hop.apache.org.

  • Pipeline and workflow development

    Hop projects structured for reuse and review: shared metadata, environment configuration, LLM steps for classification and extraction where they earn their place, and conventions your team can keep following after we leave.

  • Containerized Hop Server, Hop Web, and runners

    Immutable Docker images that bundle Hop with your project, built by CI from a tagged commit, so every environment runs exactly what was tested. Hop Web gives your team the designer in a browser, hosted in your own cloud.

  • Cloud-native runtime

    Hop on Kubernetes, Amazon ECS, or Cloud Run, with scheduled and long-running workloads, autoscaling where it pays off, and Apache Beam runners when a job outgrows a single node.

  • Migration from Kettle and PDI

    Import existing transformations and jobs, modernize the parts that need it, and move to Hop without a big-bang rewrite.

GIT REPOSITORYHop project, workflows,environment configCI PIPELINElint, unit tests,build imageIMAGE REGISTRYHop + project,tagged by commitRUNTIMEKubernetes, ECS, or Cloud RunHop Serverhop-run jobsBeam runner for big jobsDATAsources,targetspush, PRimagedeployreads,writesSECRETS MANAGERVault, AWS, Azure, GCPleast-privilege identitiesMONITORINGlogs, metrics, run statusalerts to on-callBACKUPSmetadata, config,run historycredentials injected at run time,never stored in Git or imageslogs, metrics, run statusscheduled snapshotsEVERY ENVIRONMENT, SAME IMAGEdev, test, and production differ onlyby configuration and credentialsPurple marks the runtime and the credential path that must never be shortcut.
How a Hop deployment fits together: pipelines live in Git, CI builds a tested container image, the runtime pulls credentials from a secrets manager at run time, and logs, metrics, and backups flow out to your operations tooling.

New to Apache Hop?

What you see on the canvas

Hop is a visual tool. You build by placing steps on a canvas and drawing the connections between them, called hops. There are two kinds of canvas, and once you can tell them apart you can read any Hop project.

PipelineA pipeline moves data

customers_load.hplPREVIEWRUNrowsrowstruefalserowsCustomers CSVCSV file inputSelect valuesrename, retypeFilter rowsemail is valid?Lookup regiondatabase lookupCustomers tabletable outputRejects filetext file outputEvery transform runs at once.Rows stream along the hops; no waiting on a whole file.Hops can branch.Valid rows go one way, rejects the other.
A pipeline canvas: each icon is a transform, each arrow is a hop that rows travel along. Rows stream through every transform at the same time, and a transform like Filter rows can send rows down different hops depending on a condition.
  • Transforms read, change, and write rows: file and table inputs, lookups, filters, joins, calculations, outputs.
  • All transforms run at once. A row can be leaving the output while later rows are still being read.
  • Click any transform to preview the rows flowing through it while you build.

WorkflowA workflow decides what runs, and when

nightly_load.hwfSCHEDULERUNalwayssuccesssuccesssuccessfailurefailureStart02:00 nightlyFile arrived?check file existscustomers_loadrun pipelinebuild_reportrun pipelineMail: donesend mailMail: alerton-call, with the logon successon failurealwaysThe failure path is part of the design.
A workflow canvas: each icon is an action, run one after another from Start. Hops carry a condition: green runs the next action on success, red on failure, black regardless. Workflows are where pipelines get scheduled, chained, checked, and reported on.
  • Actions do one thing each: run a pipeline, check a file exists, run SQL, send mail, evaluate a condition.
  • Colored hops make the failure path as visible as the happy path, so nobody has to guess what happens when a load breaks.
  • A workflow is what you schedule. Pipelines do the data work inside it.

  • Transforms vs actions

    If the icons pass rows to each other, you are looking at a pipeline. If they run in order and branch on success or failure, it is a workflow. Every Hop project is built from these two pieces.

  • The canvas is a plain file

    Each pipeline is a .hpl file and each workflow a .hwf file, stored in your Git repository. Changes show up as readable diffs, get reviewed like code, and can be promoted through environments by CI.

  • Metadata does the wiring

    Database connections, run configurations, and environment settings live in project metadata, not inside the canvas. The same pipeline runs against development or production by switching environment, with credentials supplied at run time.

Illustrations are simplified. The real Hop GUI uses the same shapes: icons joined by hops, with labels underneath.

Hop in the cloud

Design on the desktop, in a browser, or with an AI assistant. Run anywhere, read from everywhere.

Hop was built to run in containers, which makes a cloud deployment the natural home for it. Four things make that practical: a designer you can use on the desktop or in a browser, pipelines you can build with an AI coding assistant, a runtime that fits AWS and Google Cloud, and a file layer that treats cloud storage like a local folder.

Hop GUI on the desktop, or Hop Web in a browser

The Hop GUI is a free desktop application for Windows, macOS, and Linux, and many engineers prefer it: a local install, fast, and usable offline. Hop Web is the same canvas served from a container, for teams that would rather open a browser than install anything. We run it inside your cloud account behind your single sign-on. Both work on the same project files in Git, so a pipeline drawn on a laptop and one drawn in the browser are interchangeable.

https://hop.yourcompany.com/uisigned in via SSOPROJECTcustomers/customers_load.hplnightly_load.hwfmetadata/connectionsrun configsENVIRONMENTproductionCustomers CSVFilter rowsLookup regionCustomers tablereviewerSame canvas, no install.Served from a container in your cloud account; projects come from Git.
Hop Web served from your own cloud account: the same canvas as the desktop Hop GUI, reached through a browser and protected by your sign-on.
  • Desktop Hop GUI for engineers who want a local install; same files, same Git repository
  • Hop Web for everyone else: no install, new team members productive on day one
  • Hop Web runs on Kubernetes, ECS, or Cloud Run behind your SSO, with Git as the source of truth

VFS: cloud storage as a filesystem

Hop's virtual file system lets any file transform or action read and write remote locations by URL. The same Text file input that reads a local CSV reads from an S3 bucket, a Google Cloud Storage bucket, an Azure container, or an SFTP server once the path changes. Credentials come from the secrets manager, not the pipeline.

File transformread or writeAmazon S3s3://bucket/pathGoogle Cloud Storagegs://bucket/pathAzure Blob Storageazure://container/pathSFTP and FTPsftp://host/pathHTTP and HTTPShttps://host/export.csvLocal and zipped filesfile:///data/in.csvChange the path, not the pipeline. Credentials come from the secrets manager at run time.
One file transform, six locations: with VFS the location is just the URL scheme in the path. Swap the path per environment and the pipeline itself does not change.

Build pipelines and workflows with an AI coding assistant

A Hop pipeline is a plain text file, and so is a workflow. That means an AI coding assistant such as Claude Code can draft one from a sentence, refactor fifty of them at once, or add the validation step you forgot, working in the same Git repository as everything else. The result opens on the Hop Web canvas for a visual check, and lands in production the same way any change does: a pull request, CI, and a promotion.

  • Describe the pipeline in plain language; the assistant writes the .hpl and .hwf files and the metadata they need
  • Bulk changes that would take days by hand, such as a new naming convention or an extra audit column, become one reviewed pull request
  • Round trip freely: open the generated pipeline in Hop Web, adjust it on the canvas, commit it back
  • Same guardrails as any change: lint, tests, and an engineer's review before it runs against real data
AI CODING ASSISTANT"Create a pipeline that readscustomers.csv from S3, rejectsbad emails, looks up the region,and loads Postgres nightly."Writing pipeline and workflow files ...customers_load.hpl, nightly_load.hwf, connection metadatacommitPULL REQUEST+ customers/customers_load.hpl+ customers/nightly_load.hwf+ metadata/rdbms/warehouse.jsonlint and unit tests passedevaluation set passedreview requested: data engineerreviewHOP WEB: customers_load.hplS3 CSVFilterLookupPostgresReviewed on the canvas, then approved.Adjust visually and commit back if needed.merge, CI builds the image, promote to test and then production. Same path as every change.
From a sentence to a running pipeline: the assistant writes Hop files into Git, CI checks them, an engineer reviews them on the Hop Web canvas, and the promotion path is unchanged.

  • On AWS

    Hop Server and Hop Web on ECS or EKS, task roles instead of access keys, S3 for landing and archive zones, RDS or Redshift as sources and targets, Secrets Manager for credentials, and CloudWatch for logs and alarms.

  • On Google Cloud

    Hop on GKE or Cloud Run with workload identity, Cloud Storage buckets for files, BigQuery and Cloud SQL as sources and targets, Secret Manager for credentials, and Cloud Logging and Monitoring for observability. Beam runners on Dataflow when a job needs to scale out.

Azure works the same way with AKS or Container Apps, Blob Storage, Key Vault, and Azure Monitor. The pattern is what matters: managed identity, managed storage, managed secrets, and Hop in containers between them.

Part of a bigger stack

Hop is a complement, not a replacement

Few teams run a single data tool, and we do not suggest they should. Hop earns its place as the open source integration and orchestration layer next to the platforms you already use: it does the ingestion, file handling, and cross-tool coordination that every stack needs, while Databricks, Snowflake, BigQuery, and the rest do the heavy compute and analytics they are built for.

SOURCESOPEN SOURCE LAYERPLATFORMSSFTP drops and filesLegacy and operational DBsSaaS APIsKafka topicsrows, filesAPACHE HOPingest and validateland in storagetrigger platform jobscoordinate and notifyworkflows in Git, containers in your cloud,credentials from your secrets managerno license, no lock-injobs, filesDatabricksrun jobs via API, read resultsSnowflake, BigQuerybulk load, run SQLCloud storage: S3, GCS, ADLSland and archive filesdbt, Airflowcall them, or be calledresults, status, and row counts come back for checks and deliveryHeavy compute stays on the platform. Hop owns the movement, the checks, and the sequence around it.
Hop as the open source layer between the sources and the platforms: it ingests and lands data, triggers work in Databricks, Snowflake, or BigQuery, and coordinates the steps and notifications around them. The platforms keep the heavy compute.

Where the line sits

Hop does

  • Reaching sources the big platforms do not reach easily: SFTP drops, legacy databases, SaaS APIs, spreadsheets, and files
  • Landing, validating, and normalizing data into cloud storage or a warehouse
  • Coordinating steps that span tools: load, then run a job, then check the result, then notify
  • Operational and file-heavy work that is awkward or expensive to do inside a compute platform

The platforms do

  • Large-scale transformation, machine learning, and analytics on data that has already landed
  • Warehouse modeling and serving layers, often with dbt on top
  • Streaming backbones and event buses that Hop reads from and writes to

  • With Databricks

    Hop ingests from the awkward sources, lands raw files in S3, ADLS, or Cloud Storage, and triggers a Databricks job through its REST API from a workflow action, then reads results back over JDBC for downstream delivery. Databricks keeps the heavy compute and ML; Hop keeps the glue open source.

  • With Snowflake and BigQuery

    Hop bulk-loads into Snowflake and BigQuery with their native loaders, runs SQL in either, and handles the file movement around them. Warehouse modeling stays in SQL and dbt, orchestrated from the same Hop workflow that loaded the data.

  • With Airflow, dbt, and Kafka

    Airflow can call Hop pipelines as containerized tasks, or Hop workflows can schedule themselves and call dbt when a load finishes. Hop's Kafka transforms consume and produce events, so streaming and batch share one set of pipelines and one Git repository.

Why the split pays off

  • The integration layer carries no license cost and no lock-in, so you can change platforms later without rewriting the plumbing
  • Proprietary compute is spent on compute, not on parsing files or polling an FTP server
  • One place to see the whole flow: a Hop workflow shows the load, the Databricks job, the dbt run, and the alert on a single canvas
  • Everything in Git, reviewed and promoted the same way, regardless of which platform a step talks to

Questions

Apache Hop, briefly

Does Apache Hop replace Databricks, Snowflake, or BigQuery?
No, and it should not try to. Hop is the open source integration and orchestration layer: it reaches the sources those platforms do not, lands and validates data, triggers jobs in them, and coordinates the steps around them. The platforms keep the large-scale compute, machine learning, and analytics. Most of our clients run Hop alongside at least one of them.
What is Apache Hop?
Apache Hop is an open source data orchestration and integration platform from the Apache Software Foundation. Pipelines and workflows are designed visually on a canvas and stored as plain text files, which makes Hop a natural fit for Git, containers, and cloud-native operations.
What is the difference between a Hop pipeline and a Hop workflow?
A pipeline moves data: its transforms read, change, and write rows, and they all run at once. A workflow decides what runs and when: its actions execute in order from Start, and hops between them carry a condition for success, failure, or either. Workflows are what you schedule; pipelines do the data work inside them.
Can Apache Hop run in containers and in the cloud?
Yes. Hop was built to run in containers. We run Hop Server and Hop Web on Kubernetes, Amazon ECS, or Cloud Run, with credentials from a secrets manager, and Hop's virtual file system reads and writes S3, Google Cloud Storage, Azure Blob Storage, and SFTP by URL.
Can an AI coding assistant build Hop pipelines?
Yes. Because a pipeline is a plain .hpl file and a workflow a plain .hwf file, an assistant such as Claude Code can draft one from a description, refactor many at once, or add a missing step, all inside the same Git repository. The result opens on the Hop Web canvas for review and reaches production through the usual pull request and CI path.
What does an Apache Hop engagement with Avizing look like?
Fixed scope with clear milestones: assess what you run today, design the target architecture and security model, deploy in slices with rollback ready, then hand over with monitoring, documentation, and training. Ongoing support is optional.

Next step

Free Consultation

Tell us about your data platform, or the one you wish you had. A short call is usually enough to say whether we can help and what it would take.

Get in touch(877) 881-6238