Using with rules

It's not designed for it - it doesn't break the logic out into the if-then steps that you'd expect to use with rules.

Rules (in my experience) process single actions in single entities based on conditions.
The dedupe module (at the moment) compares many items at once.

Though I guess it could be refactored to "find anything similar to node X" and expose that as a node action. However, it's so short that adding enough rules structure and API support would triple the module size. It would be a different thing.