'One Time' setting not behaving as expected

I'm not sure if I've got this right, but my assumption is that with the message persistence set to 'One Time' (and using 'new message alert') if you close the notification then on further page reloads the notification should not show up again (until the next new message).

This isn't what I'm observing, however. Every page reload sees the notification reappear, regardless of whether or not I close the notification.