Selected custom theme but default is used

I am sure I am missing something fairly basic here. I created a custom error theme and choose that theme (for 404) in the interface at:

/admin/settings/customerror

All seems, ok, but when I get to a 404 page on the site, I see my site default theme. I also set the path for 404 in error reporting to: customerror/404. I have cleared my site cache and confirmed the theme works fine at the /admin/blocks page. In addition I set a "test" message for 404 and indeed it shows up so I have no idea how to begin troubleshooting this or how to fix it.

Edit:
One thing I noticed, If I go to the url: /customerror/404 I see my custom error theme but if I go to a made up error page like /error-page, then it does not work as described above. I got the impression this would work for any non valid url that was a 404.

Thanks.