Kafka Maintenance Instructions

To manage disk usage and ensure optimal performance, it is important to configure log retention policies for Kafka Brokers, Kafka Connect, and Qlik Replicate. This guide walks you through updating the relevant configuration files and restarting services accordingly.

• Kafka Broker Log Retention Policy

• Kafka Connect Log Retention Policy

• Qlik Replicate Log Retention Policy

Kafka Broker Log Retention Policy

1. Navigate to the Kafka configuration directory:

   cd /etc/kafka

2. Open the `server.properties` file:

   vi server.properties

3. Locate and update one of the following properties to define the log retention period.

   Property	Description
   log.retention.hours	How many hours to keep a log file (default: 168 hours = 7 days)
   log.retention.minutes	Used instead of hours if defined
   log.retention.ms	Used instead of hours/minutes if defined

   Note: Use only one of these properties to avoid conflicts.

   

4. Repeat steps 2–3 for `server-1.properties` and `server-2.properties`.

5. Restart Kafka services using the Sink Connector.

   

Kafka Connect Log Retention Policy

Kafka Connect uses the Log4j framework for logging. To configure log retention:

1. Edit the Log4j configuration file:

   vi /etc/kafka/log4j.properties

2. Add or update the following properties to retain logs for 3 days:

   log4j.appender.kafkaAppender.MaxBackupIndex=3

   log4j.appender.controllerAppender.MaxBackupIndex=3

   log4j.appender.stateChangeAppender.MaxBackupIndex=3

   log4j.appender.requestAppender.MaxBackupIndex=3

   log4j.appender.cleanerAppender.MaxBackupIndex=3

   log4j.appender.authorizerAppender.MaxBackupIndex=3

3. Edit the Kafka Connect-specific Log4j file:

   vi /etc/kafka/connect-log4j.properties

4. Add the following property:

   log4j.appender.connectAppender.MaxBackupIndex=3

5. Restart all Kafka services using the Sink Connector.

   

Qlik Replicate Log Retention Policy

Qlik Replicate log retention is managed through the Qlik UI. Refer to Qlik Replication Documentation.

1. Open the Qlik Replicate console.

2. Navigate to the logging configuration section.

   

   

3. Set the desired retention period for logs (for example, 3 days). Save the configuration.

4. After applying the changes, logs from older database connections will be automatically removed based on the new settings.