Using Apache NiFi for Change Data Capture (CDC)

‍

What is NiFi?

‍

Apache NiFi is an open-source data integration and flow management tool designed to automate the movement of data between systems. It provides a visual interface to design, monitor, and control data pipelines, supporting hundreds of processors for routing, transformation, and system integration. NiFi excels at:

- Data flow automation - defining how data moves between sources and destinations.

- Protocol mediation - connecting heterogeneous systems through standard connectors.

- Data transformation - enriching, filtering, or shaping records in real-time.

- Operational control - enabling fine-grained monitoring, throttling, and error handling.

NiFi is often deployed in environments where teams need to build flexible data flows without deep programming expertise, while maintaining operational transparency and reliability.

‍

CDC in NiFi: How It Works

‍

Change Data Capture (CDC) is the process of detecting and delivering changes from a source database (inserts, updates, deletes) in real time. While NiFi is powerful as a data flow engine, it does not natively implement log-based CDC for databases like Db2, Oracle, or SQL Server... Instead, it provides two main approaches:

‍

1. Query-based incremental fetch
‍

Using processors like QueryDatabaseTable or GenerateTableFetch, NiFi polls the source via JDBC and retrieves new or updated rows based on a high-watermark column (e.g., last updated timestamp, sequence ID). This works, but:

- Deletes are not captured unless modeled in the schema.

- Latency is tied to polling frequency.

- It increases load on the source system.

‍

2. Log-based CDC through external tools
‍

For true, non-intrusive CDC, NiFi integrates with external CDC engines such as:

- Qlik Replicate (streaming changes into Kafka, then consumed from Kafka with NiFi)

- IBM InfoSphere CDC / IIDR (streaming into MQ or Kafka)

- Popsink (directly consumed from Pospink thanks to Kafka-like consumer endpoints)

With these setups, NiFi acts as a downstream consumer, taking CDC streams from a dedicated engine and routing them to other systems.

‍

Popsink for CDC

‍

Popsink is a real-time data replication platform purpose-built for CDC across modern and legacy systems, including challenging environments like IBM i (AS/400), Oracle, and z/OS. Popsink captures changes directly from transaction logs or journals, ensuring:

- Low impact on production systems (no SQL polling).

- Full fidelity (inserts, updates, deletes, schema evolution).

- High throughput and low latency suitable for mission-critical workloads.

Popsink exposes these CDC streams through a Kafka-compatible API, meaning that any tool or framework that speaks Kafka (including NiFi) can subscribe to changes without extra middleware.

‍

NiFi + Popsink: A Practical CDC Pipeline

‍

In this model:

1. Popsink captures changes from the source system in real time.

2. These changes are published to a Kafka-compatible Popsink endpoint.

3. NiFi consumes the CDC events using the ConsumeKafka processors.

4. NiFi then applies transformations, routing, or delivery into downstream systems (e.g., Snowflake, S3, Elasticsearch, APIs).

This combination offers the best of both worlds:

- Popsink ensures accurate, low-latency CDC extraction.

- NiFi provides flexible orchestration and delivery across your data ecosystem.

‍

Conclusion

‍

Apache NiFi is a versatile tool for managing and transforming data flows, but it is not designed as a CDC engine. To implement CDC effectively, pairing NiFi with a specialized CDC solution is key. Popsink provides exactly that – non-intrusive, log-based CDC from complex enterprise systems while exposing a Kafka-compatible stream NiFi can consume natively.

For teams already invested in NiFi, integrating Popsink means you can keep NiFi’s flexibility for flow orchestration while ensuring your CDC pipelines are reliable, high-performance, and production-grade.

‍