Replication Monitoring using Kafka UI

Overview

Kafka UI is a web-based interface for managing and monitoring Apache Kafka clusters. This topic provides setup instructions, basic commands, and guides for common operations.

Install Path

• Installation Directory: /opt/kafka-ui

• Executable JAR File: /opt/kafka-ui/kafka-ui.jar

• Systemd Service Name: kafka-ui

Basic Commands

• Start Kafka UI: sudo systemctl start kafka-ui

• Stop Kafka UI: sudo systemctl stop kafka-ui

• Restart Kafka UI: sudo systemctl restart kafka-ui

• Check Kafka UI Status: sudo systemctl status kafka-ui

• Enable Kafka UI to Start on Boot: sudo systemctl enable kafka-ui

• Disable Kafka UI from Starting on Boot: sudo systemctl disable kafka-ui

Getting Started with Kafka UI

1. Open your browser and go to: http://{IP Address}/kafkaui

2. Login is required using nginx authentication.

   Default values:

   • Username: admin

   • Password: hubble

3. You should see the dashboard listing all configured Kafka clusters.

How to Configure or Set up a New Cluster

1. Add the Cluster Details: Create a new cluster in the Dashboard section by specifying:

   • Cluster Name

   • Environment

   • Cluster ID or Endpoint

     

2. Add Kafka Broker Details: Under the Bootstrap Server section, provide the Kafka broker URL(s) (for example, broker1:9092,broker2:9092).

   

3. Add Schema Registry Details: In the Schema Registry section, enter the Schema Registry URL (example, http://localhost:8081) and any authentication credentials, if required.

   

4. Add Kafka Connect Details: In the Kafka Connect section, provide the Kafka Connect REST URL (for example, http://localhost:8083) along with any authentication details if required.

   

How to View the Replication Stats in Kafka UI

1. Open Kafka UI in your browser.

2. Navigate to the Topics section.

3. Select the topic related to your replication task.

4. View key metrics such as:

   • Message count

   • Partition offsets

   • Consumer lag

   • Throughput

5. Optionally, go to the Consumers or Consumer Groups section to monitor which consumers are reading the replicated data and their current offsets.

This allows you to track the health and progress of your replication jobs in real time.

How to View the Kafka Connect Stats?

1. From the Kafka Connect section:

   • Select the connector.

   • Go to the Tasks tab.

   • Check for failing tasks.

2. Errors will also appear in:

   • The Status of connectors or tasks (for example, FAILED)

   • Logs from the Kafka Connect server (see logs section)

     

3. Open the connector you want to inspect.

   

Replication stats such as task status, partitions, and offsets will be visible in the connector overview.

How to View the Failed Topic/Table Stats

Note: In this setup, if message processing fails, the messages are automatically routed to the DLQ (Dead Letter Queue) topic.

1. You can find the original topic name and error details in the headers section of each message in the DLQ. This helps trace back the source and diagnose the issue.

   

2. Errors typically surface in Dead Letter Topics (DLTs), which you can inspect like normal topics.

   Tip: To troubleshoot failures, check the messages in the DLQ (Dead Letter Queue) topic. In the Headers section of each message, you can usually find the exception message or error details that caused the failure.

   

How to Clean up the Records in Brokers?

Important: Use with caution. Cleanup is typically managed by Kafka retention policies, and manual deletion may lead to data loss if not handled carefully.

1. Open Kafka UI in your browser.

2. Go to the “Topics” section and select the topic you want to clean.

3. Click on the Actions or Settings menu next to the topic name.

4. Choose Clear Messages to remove all records from the topic.

When to Clean:

Manual cleanup is recommended in scenarios such as:

• Re-replicating tables without primary key columns in insert mode, where duplicate records may be generated.

• Resetting a topic before a fresh load to ensure no stale or inconsistent data remains.

• Clearing error topics after the issues have been resolved and logged.

Refer to Insert Mode Replication With Kafka (Oracle EBS) for details.

How to Clean up the Records in Brokers

Important: Use with caution. Cleanup is typically managed by Kafka retention policies.

• To manually delete a topic:

  1. Go to the Topics tab.

  2. Select the topic you want to delete.

  3. Click the Delete button (if enabled in Kafka broker and UI config).

• Alternatively, using Kafka CLI:

  kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic your-topic-name

How to Update a Property in a Connector (for example, batch.size)

1. Go to the Connect Clusters section.

2. Open the connector you want to modify.

   

3. Click the Edit Configuration (pencil icon).

   

4. Update the desired property, for example:

   {

   "batch.size": "5000",

   "connection.password": "******"

   }

   Note: Ensure you update the connection.password field when making any property changes, as the characters “******” will be treated as a new password if not modified.

5. Save for the changes to apply.

How to View the Error Logs of Kafka UI

• Although Kafka UI does not directly expose backend logs, you can view them using journalctl with the following command:

  sudo journalctl -u kafka-ui -f

  This command displays real-time logs from the Kafka UI systemd service.

• For debugging Kafka Connect-related errors, check the logs of your Kafka Connect instance.

  These logs are typically found in /logs/connect.log or via Docker container logs if Kafka Connect is running in a container.