Turning Site Offline Doesn't Occur Until Cache is Cleared

If you have aggressive performance turned on, and attempt to turn the site on maintenance mode, anonymous users can still visit the site. They are still able to visit the site until you clear the cache.

In order to fix this, we'll have to clear the page cache when the site maintenance is toggled.