Documentation for To execute operations: options?

No documentation about that at the moment, though I plan to write some.

A PHP script can only do a limited amount of processing before it hits the memory and execution limit.
So if you invoke an operation directly, the number of items you can handle depends on your server setup.
If you're deleting nodes, one server might be able to delete 100 nodes in one go, another a thousand, another only 10....

Batch API goes around the memory and execution limit, so you can handle as many items as you want (select 100 000 rows in VBO, click delete, and it will work). You get a progress bar and it does its thing in the background. However, for a smaller amount of items (that the server could handle in one go), this approach is slower.

Related feature request: #1184844: Provide an execute type that switches from direct execution to batch api based on the number of selected items.

There is now a job queue option besides the batch api and direct execution. What would be the difference between batch api and job queue?

And it is documented here:

http://drupal.org/node/1203388

The point of the job queue is to delay execution. You execute a VBO operation and it fills a queue.
Then a runner executes a queue (usually on cron). Good for lengthy operations that don't need to happen right now, like uploads.
The many options and ways of executing a queue also make it attracting.