Wheel keeps spinning on every action

When I add a fieldable panels pane, my blue wheel keeps spinning no matter what button I click. This is true for 'Finish', 'Cancel' but also upload button when I add an image field.
But the panes are saved and are displayed at admin/structure/panels/entity/manage/my-pane.
There is probably a refresh issue.
This happens both within the in-place editor and through the normal panels interface.