Skip to main content

Using SinkConnector API

Three APIs are provided to check the health of Kafka services and change their status:

API Access

To use the SinkConnect APIs, open the Swagger UI in your browser:

Swagger UI: http://{IP Address}/Swagger/index.html

Replace {IP Address} with the actual IP address of your SinkConnector service.

SinkConfiguration

This API is used to create target tables (drop and re-create if they exist) and create the sink connector to replicate data from Kafka to the target.

Steps:

  1. Select the source database type (SQL, Oracle, DB2) from which data will be replicated.

  2. Provide the connection string of the source database (a standard template is provided in the API).

  3. Enter the target database username and password.

  4. Specify the number of Kafka connectors to be created (default is 5). This allows connectors to run in parallel, similar to Qlik (5 tables at a time).

  5. In the request body section, post the Qlik task JSON file. This file helps determine which tables need to be created in the target database.

  6. Click Execute. This API will create all the tables in the target database and also create the sink connectors to replicate data from Kafka to the Actian Vector database.

AddOnlySinkConfiguration

This API is used to create the sink connector to replicate data from Kafka to the target (it does not create target tables).

Steps:

  1. Enter the target database username and password.

  2. Specify the number of Kafka connectors to be created (default is 5). This allows connectors to run in parallel, similar to Qlik (5 tables at a time).

  3. In the request body section, post the Qlik task JSON file. This file helps distribute the tables into different Kafka connectors for data replication.

  4. Click Execute. This API will create the sink connectors to replicate data from Kafka to the Actian Vector database.

ClearKafkaTopicsAndTruncateTargetTables

This API endpoint clears messages from specified Kafka topics and truncates associated target database tables.

Caution: Providing TaskName will remove all Kafka topics matching the pattern {TaskName}.*. Use with caution as this may affect other active or in-progress tasks.

Note: The database details are used to truncate tables on the target Actian Vector database.

RemoveSinkConnector

This API is used to remove existing sink connectors.

Provide the Qlik task name used when creating the sink connector. This will delete all sink connectors associated with that task name.

Troubleshooting

  1. Check Log Files:

    • Navigate to the folder: /opt/insightsoftware/hubble-accelerator/5.0.x/hubble-sinkconnector/logs/

    • Review the log files for any issues.

  2. Schema Reading Issues: When reading schema details from the source database, connection failures may occur due to network issues. If this happens, rerun the execution.

  3. Table Creation Issues: If there are issues creating tables in the target database, the sink connectors will not be created. Correct these errors and try again.

  4. Sink Connector Creation Issues:

    • Issues may arise when creating the sink connector, especially if the existing connector is still processing messages from Kafka. When posting the sink connector, the existing connector is deleted and recreated.

    • Wait until all messages are processed or stop all Kafka services and restart them.

  5. Restart the service: After addressing the issues, restart the service as needed.

Was this article helpful?

We're sorry to hear that.