README.md

Not using GitHub? Find me on LinkedIn!

debezium-example

Debezium Example | Postgres ➡️ Elasticsearch

Schema

                   +-------------+
                   |             |
                   |  PostgreSQL |
                   |             |
                   +------+------+
                          |
                          |
                          |
          +---------------v------------------+
          |                                  |
          |           Kafka Connect          |
          |    (Debezium, ES connectors)     |
          |                                  |
          +---------------+------------------+
                          |
                          |
                          |
                          |
                  +-------v--------+
                  |                |
                  | Elasticsearch  |
                  |                |
                  +----------------+

We are using Docker Compose to deploy the following components:

• PostgreSQL
• Kafka
  • ZooKeeper
  • Kafka Broker
  • Kafka Connect with Debezium and Elasticsearch Connectors
• Elasticsearch

Install Java 11 & Maven

• https://jdk.java.net/
• https://maven.apache.org/

Kafka Streams Foreign Key Joins

This example demonstrates how two Debezium change data topics can be joined via Kafka Streams, using the new foreign key join feature in Apache Kafka 2.4 (KIP-213).

The source database contains two tables, customers and addresses, with a foreign key relationship from the latter to the former, i.e. a customer can have multiple addresses.

cd aggregator-service
mvn clean package

Run docker-compose.yml file

./scripts/start.sh

Open http://127.0.0.1:9021/

Deploy configs

Install ElasticSearch Head & query data

https://chrome.google.com/webstore/detail/elasticsearch-head/ffmkiejjmecolpfloofpjologoblkegm

Documents

• Debezium Tutorial
• Confluent Hub
• KAFKA STREAMS

License

https://github.com/4pet-social