We have been trying to figure out a way to remove duplicates for years now.
For example, many loggers just resubmit signs (just in case).
This is an ever increasing junk data problem, so we had to tackle it.
We are hopeful that we have a solution! This week hopefully. Performance tests underway.