Uninstall appears to lack permission to remove folders

So I'm testing this very promising module. Running a fresh install of Drupal 6.5 with only this module manually added.
I select event and image modules to test it, it installs fine (after I add the ssh module).

Next up, uninstall.

Only image is available for uninstall, even though I never activated either of them.
I select image and hit uninstall and this is what meets me:

* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/sv.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/sl.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/pt-br.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/pl.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/nl.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/it.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/image.pot.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/hu.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/he.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/fr.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/fi.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/eu.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/es.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/de.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/da.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/cs.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/ca.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/ar.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/translations/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/tests/image.test.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/tests/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/README.txt.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/LICENSE.txt.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/INSTALL.txt.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/image.views.inc.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/image.module.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/image.install.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/image.info.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/image.imagemagick.inc.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/image.admin.inc.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_im_advanced/translations/image_im_advanced.pot.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_im_advanced/translations/de.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_im_advanced/translations/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_im_advanced/image_im_advanced.module.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_im_advanced/image_im_advanced.install.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_im_advanced/image_im_advanced.info.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_im_advanced/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/translations/image_import.pot.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/translations/image_import-module.sl.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/translations/image_import-module.hu.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/translations/de.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/translations/da.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/translations/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/tests/image_import.test.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/tests/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/image_import.module.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/image_import.install.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/image_import.info.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/image_import.admin.inc.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_import/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/translations/image_gallery.pot.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/translations/image_gallery-module.sl.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/translations/image_gallery-module.sk.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/translations/image_gallery-module.hu.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/translations/image_gallery-module.he.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/translations/image_gallery-module.es.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/translations/de.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/translations/da.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/translations/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/tests/image_gallery.test.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/tests/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/image_gallery.module.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/image_gallery.install.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/image_gallery.info.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/image_gallery.css.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_gallery/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/translations/image_attach.pot.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/translations/image_attach-module.sl.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/translations/image_attach-module.hu.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/translations/de.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/translations/da.po.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/translations/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/image_attach.views.inc.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/image_attach.module.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/image_attach.install.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/image_attach.info.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/image_attach.css.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/image_attach/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/contrib/.
* Could not remove /home/pluginmanager/drupal6/sites/all/modules/image/.

I browse to admin/build/modules/ and image is still around. I head over to the server, looking manually at the folders and all the files are still around.

I'm guessing that this is because permissions are lacking removal rights - but I might be wrong. All I know is that when I installed Drupal I had to give W permissions to folders that Drupal wanted to manipulate in some way. When I look at the server I can see that those permissions are not set that way on the image folder.