Show if there are 'new' comments on a view of terms?

I have comments on nodes that are categorised by terms. I need to show a view of the terms and if there are any new comments in any of the nodes that term is tagged with.

So if I have the following:

Term1
-NodeA (has new comments)
-NodeB (has new comments)
Term2
-NodeC (no new comments)
-NodeD (no new comments)
Term3
-NodeE (no new comments)
-NodeF (has new comments)
Term4 (has no nodes)

I need my view to look like this:

Term1 – new comments!
Term2
Term3 – new comments!
Term4

First I tried making a view of comments to include the new comments field, and I was going to use relationships to show the terms. However if a term has no nodes or comments (Term4 in my example) I need the term to still be displayed.

So I then tried making a view of the terms. I added the relationships 'Taxonomy term: Content with term' and the field 'Content: New comments' which uses this relationship. This does show the number of new comments as expected. It gives me duplicated results (which I may be able to fix with grouping fields) but unexpectedly it also hides the terms that have no comments or nodes.

Am I going about this the right way and if so how can I stop the 'empty' terms being hidden?