Insert Mode Replication With Kafka (Oracle EBS)

Handle Tables Without Primary Keys in Qlik Replication

Tables were segregated based on their unique indexes by referring to the table DDLs and the reference documentation available in ETRM 12.2.2. This approach ensures accurate identification of uniquely identifiable records, aligning with Oracle's structural definitions and best practices.

Segregated List

Contact insightsoftware Support for the latest segregated list.

Replication Tasks

Dedicated Qlik tasks were created to replicate the segregated tables identified based on unique index analysis. These tasks are designed to ensure efficient and reliable data replication, aligning with the structure and integrity defined in the source systems.

Steps to follow for Fresh Insert Mode Run

Create Qlik Task:
- Initialize a new Qlik task to load data into the destination.
- Create a new endpoint to ensure proper data flow and connectivity between Qlik and Kafka.
Swagger API:
- Create the necessary tables and post the sink connector configurations. Use a maximum of three Kafka connectors for this process to ensure optimal performance and maintainability.
Navigate to Kafka UI, select the Kafka Connect tab, and open the newly created Sink Connector tasks.
- Change the sink connector configuration values for all the newly created sink connectors:
  - Insert Mode
  - Delete Enable Mode
  - PK Mode
  - tasks.max Replication
```
"tasks.max": "1",
```
```
"insert.mode": "insert",
```
```
"delete.enabled": "false",
```
```
"pk.mode": "none",
```
Run the Qlik task.

Re-run the Qlik Task in Insert Mode

Clear the messages in all the topics (i.e., the tables being replicated).

To ensure that there is no residual or duplicated data in the system, clear the messages in all the topics corresponding to the tables being replicated. This step involves:
- Identifying the relevant Kafka topics that hold the data for the tables being replicated.
- Using Kafka UI or Kafka command-line tools, delete or clear the messages from these topics to reset their state.
- This action is necessary because when running the replication in insert mode, it is necessary to ensure that:
  - Any previously unprocessed or partially processed data does not get replicated again.
  - Avoid duplication of data in the target system as we are performing a fresh replication.
By clearing the messages, it is ensured that the replication process only picks up new records and accurately processes the data from the source system into the destination without reprocessing old or redundant entries.
Truncate the tables in the target database.

To ensure that the target database tables are cleared and ready for fresh data, truncate the relevant tables. This step is crucial when replicating data in insert mode because:
- Truncating the tables removes all existing data from the target tables, ensuring that there is no conflict with previously replicated records.
- This action ensures that only the newly replicated data from the source system will be inserted into the target tables, preventing any old or stale data from being part of the replication process.
- Helps to identify the target tables that will receive the replicated data.
- Use your preferred database management tool or SQL commands to truncate the tables. Sample SQL for truncating:
  
  TRUNCATE TABLE table_name;
Run the Qlik task to start the data loading process into the destination, ensuring that all configurations are properly applied.