I have an input dataset whose records will be appended to an existing database. Prior to being appended, the data will go through heavy, time-intensive processing. I want to filter out records from the input dataset which already exist in the database to reduce processing time.
The difference between the input and database are illustrated here:
This an overview of the kind of process I am looking at. The input data will eventually feed into the database.
My current solution involves using a Matcher transformer on the combined database and input, then filtering the NotMatched result using a FeatureTypeFilter to retain only the input records.
Is there a more efficient way to obtain the difference features?
أكثر...
The difference between the input and database are illustrated here:
This an overview of the kind of process I am looking at. The input data will eventually feed into the database.
My current solution involves using a Matcher transformer on the combined database and input, then filtering the NotMatched result using a FeatureTypeFilter to retain only the input records.
Is there a more efficient way to obtain the difference features?
أكثر...