Skip to main content

Troubleshooting Common Errors in Hubble Accelerator v5.0.x

This topic outlines the troubleshooting instructions for common errors encountered in Hubble Accelerator v5.0.x.

Brokers Issue with Kafka as Destination End-Point in Qlik Task

Brokers Not Running

  • Check if brokers are up and running using the following commands to verify ports:

    sudo systemctl status confluent-kafka;
    fuser 9093/tcp
    fuser 9094/tcp

Broker Ports Assigned to Different Service

  • If you do not see three brokers in the Broker section of Kafka UI but see Thread IDs when running:

    fuser 9092/tcp
    fuser 9093/tcp
    fucer 9094/tcp

  • Navigate to /etc/kafka/ and open:

    • server.properties

    • server-1.properties

    • server-2.properties

  • Find and change the listeners to new port addresses

    • listeners = PLAINTEXT://:9092 in server.properties

    • listeners = PLAINTEXT://:9093 in server-1.properties

    • listeners = PLAINTEXT://:9094 in server-2.properties

  • Start the services using the steps mentioned in the Using Kafka Services API topic in the Hubble Accelerator User Guide.

Kafka Brokers Stopped Responding and Restart Does Not Help

  • Check if you can see three brokers in Kafka UI.

  • In Sink Connector API, execute /api/Kafka/HealthCheck and verify confluent-kafka service is up and running.

  • If Kafka brokers are not up and running, try restarting using API call.

  • If Kafka brokers are still not running, run the following steps manually in the Linux terminal:

    sudo systemctl stop confluent-kafka;
    kafka-server-stop
    sudo systemctl stop confluent-zookeeper
    rm -rf /var/lib/zookeeper/version-2/*;
    rm -rf /var/lib/kafka/*;
    rm -rf /var/lib/kafka-1/*;
    rm -rf /var/lib/kafka-2/*;
    rm -rf /var/lib/kafka-logs-1/*;
    rm -rf /var/lib/kafka-logs-2/*;
    rm -rf /var/log/kafka-server-Custome-deta*;
    rm -rf /var/log/kafka/*;

  • Use the Swagger API to restart the services and then verify their status.

Sink Connector Not Loading

If Swagger is inaccessible from port 5002 and browser returns the following error:

or

  • Run the following command to start:

    sudo systemctl start hubble-sinkconnector

  • Check the status.

Common Issues with Kafka-Connect and How to View Logs

  • If you are unable to view Kafka Connect Tab in Kafka UI, add Sink Connectors using any of the Sink Connect Requests from Swagger and run the API calls.

  • The Kafka UI connectors you have posted will be visible here and will be identified by the Qlik task name.

  • Check if confluent-Kafka-connect is running using Sink connector by executing the Get request /api/Kafka/HealthCheck. It should return a response as "running".

  • Kafka-connect logs can be found at /var/log/kafka/connect.log.

How to Check Qlik, Vector, Kafka Brokers, Kafka-Connect, Sink Connector, Zookeeper Are Up and Running

  • Qlik

    • Run the following command in the Linux terminal:

      /etc/init.d/areplicate status

    • To start, run:

      /etc/init.d/areplicate start

  • Vector

    • Run the following command to view status:

      systemctl status actian-vectorVW

    • Output should be the following:

    • To start, run:

      systemctl start actian-vectorVW

  • Kafka Brokers, Schema Registry, Zookeeper, and Kafka Connect

    • Check status using the Get request /api/Kafka/HealthCheck.

    • To start services that are not running, use start or restart for service status and execute Get request /api/Kafka/KafkaService.

Special Character Exceptions When Global Rules Are Missing in Replication Task

In Qlik Replicate, if you notice log messages indicating special character exceptions, add Global Rule for handling special characters by following steps outlined in the Support Special Characters in Table and Column Names via Global Rules in Qlik Replication topic in the Hubble Accelerator User Guide.

Buffer Underflow Exception

  • Check logs in the file /var/log/kafka/connect.log.

  • If there is no Primary Key defined for the table (topic name), configure the primary key and try again.

QSF Memory Issue

  • Check logs in the file /var/log/kafka/connect.log.

  • If the error message indicates a memory issue, increase Vector QSF memory by following steps outlined in the Installing Hubble Accelerator topic in the Hubble Accelerator User Guide.

Java OutOfMemory Heap Space Issue

  • View logs at /var/log/kafka/connect.log.

  • If you see an error similar to java.lang.OutOfMemoryError: Java heap space, increase heap space by following steps outlined in the Installing Hubble Accelerator topic in the Hubble Accelerator User Guide.

Performance Degradation Due to High Load on Specific Tables

The Sink Connector API is designed to distribute data load across connectors based on table volume. However, if the default distribution is insufficient, especially for high-transaction tables, performance issues may arise.

To resolve this, consider isolating high-volume tables into separate tasks to improve load handling and system responsiveness.

For detailed steps on creating and managing such tasks, refer to the Migrating Existing Qlik Tasks To Kafka topic in the Hubble Accelerator User Guide.

Error: "Auto Evolution is Disabled" When Altering Existing Table or Column Definitions in Qlik Replicate with Kafka Sink Connector

Issue

When modifying the schema of an existing table (e.g., adding/removing columns), you may encounter the following error in the Kafka UI:

"Auto Evolution is disabled"

This typically occurs when the schema in the source has changed, but the sink connector is not configured to handle schema evolution automatically.

Resolution Steps

  1. Stop the Failed Qlik Replicate Task.

    • Navigate to the Qlik Replicate UI.

    • Locate the failed task.

    • Click the Stop button to halt the replication process.

  2. Delete the Existing Sink Connector task.

    Use the appropriate API endpoint to delete the sink connector associated with the failed task.

  3. Recreate the Sink Connector.

    Use the Qlik task JSON file (exported before the schema change) to recreate the sink connector. This file contains metadata about the tables and replication configuration.

    • API URL:

      http://{AcceleratorHost}:5002/Swagger/index.html

    • Required Parameters:

      1. SourceDatabaseType: Choose from SQL, Oracle, or DB2.

      2. SourceDatabaseConnectionString: Use the standard format provided in the API documentation.

      3. TargetDatabaseUserName: Username for the Actian Vector target database.

      4. TargetDatabasePassword: Password for the target database.

      5. NumberOfKafkaConnector: Default is 5 (parallel processing of 5 tables at a time).

    • Request Body:

      Paste the exported Qlik task JSON file here. This will guide the API in recreating the necessary tables and connectors.

    • Action:

      Click Execute to:

      1. Drop and recreate the target tables.

      2. Re-establish the sink connectors for Kafka-to-Actian Vector replication.

  4. Reload the Task in Qlik Replicate.

    • Go back to the Qlik Replicate UI.

    • Use the Reload option (as shown in the UI) to restart the task with the updated schema.

Adding a New Table to an Existing Qlik Replicate Task

Perform the following steps to add a new table to an existing Qlik Replicate task:

  1. Edit the Qlik Task

    1. Open the Qlik Replicate UI.

    2. Navigate to the required task and click Edit.

    3. Add the new table and save the changes.

    4. Do not start the task yet.

  2. Update the Sink Connector

    1. Go to the Sink Connector API (http://{AcceleratorHost}:5002/Swagger/index.html).

    2. Delete the existing sink connector for the task.

    3. Use the Qlik task JSON file (exported before migration) to recreate the sink connector.

  3. Recreate Tables and Sink Connectors

    • If the target tables do not exist, use the API to create them along with the sink connectors.

      • Provide the following in the API:

        • SourceDatabaseType (SQL, ORACLE, DB2)

        • SourceDatabaseConnectionString

        • TargetDatabaseUserName and Password

        • NumberOfKafkaConnector (default: 5)

        • Paste the Qlik task JSON in the request body

      • Click Execute to complete the setup.

  4. Reload the Task

    Return to the Qlik Replicate UI and use the Reload option to apply the changes and start replication.

Data Loss or Missing Rows During Replication in Qlik Replicate with Kafka Sink Connector

Issue

In the event of data loss or missing rows during replication from Kafka to the target database, follow these steps to recover and reprocess the failed or missing data.

Resolution Steps

  1. Stop the Qlik Replicate Task

    • Open the Qlik Replicate UI.

    • Identify the affected task and click Stop to halt replication.

  2. Stop Kafka Services

    • Navigate to the Sink Connector UI.

    • Use the following API endpoint to stop all Kafka services:

      GET /api/Kafka/KafkaService

  3. Start Kafka Services

    Once all services are stopped, restart them using the same endpoint:

    GET /api/Kafka/KafkaService

  4. Reload the Target in Qlik Replicate

    • Return to the Qlik Replicate UI.

    • Use the Reload Target option to reprocess and sync the data, ensuring any missing or failed rows are captured.

Required Information to Be Shared with Engineering Team

If none of the above resolutions work or if new errors are encountered, reach out to the engineering team with the following mandatory details:

  • All files in /var/log/kafka/

  • Export Task JSON file from Qlik where the error is encountered.

  • Complete log file from Qlik Replicate shown in log viewer.

  • Vector log files (errlog.log usually found at /opt/Actian/VectorVW/ingres/files/errlog.log).

Was this article helpful?

We're sorry to hear that.