I added that easily, just in js/media_insert.js in line 12:

$('<a class="button media-insert" href="#">' + Drupal.t('Insert') + '</a>')

That improves appearance making it a button like those from media module.

Also, it would be better to have media tag inserted at cursor point, not the end and I'm not that good to know how to do it.

As many others, I'd like to see adding of Title and Alt attributes as this patch enables them #1343022: Adding alt and title attributes from fields to image markup. I wanted to change your code to add those fields but I couldn't as classes are needed to find them in DOM.

I'm looking to change the media module to add classes to alt and title so they can be caught with script to produce something like this

[[{"type":"media","view_mode":"media_preview","fid":"3","attributes":{"alt":"Alt text","title":"Title text","class":"media-image","typeof":"foaf:Image"}}]]