How to define my own 960 grid ?

If you have the alpha them enabled and you create a new omega child theme, you should see a slew of files in /sites/all/themes/yourtheme/css/.

You can modify the numeric values in each of those files, starting with yourtheme-alpha-default-normal.css. I created an 880-px grid based on a 16-column 960 grid (minus 5px in each column). This gave me a very slim look and feel with 40 extra pixels of white space in each margin.

Here's the formula I used:
(column number * (column width + (2*margin))) - (2*margin)

For the 880 pixel grid with 35-pixel columns and 10-pixel margins, this turned out to be
(n*(35+(2*10))) - (2*10)
or more simply
(n*55) - 20

Below is the code for the re-written column widths. I've also attached the full CSS file which you can use for reference. This should give you an idea of how to modify all the remaining measurements.

I'd love to see how to best implement this without overwriting the 960 grid, so that you can use a custom grid *and* the default 960 grid in a site.

/*
 * ==========================================================
 * Columns
 * ==========================================================
 */

.container-16 .grid-1 {
  width: 35px;
}

.container-16 .grid-2 {
  width: 90px;
}

.container-16 .grid-3 {
  width: 145px;
}

.container-16 .grid-4 {
  width: 200px;
}

.container-16 .grid-5 {
  width: 255px;
}

.container-16 .grid-6 {
  width: 310px;
}

.container-16 .grid-7 {
  width: 365px;
}

.container-16 .grid-8 {
  width: 420px;
}

.container-16 .grid-9 {
  width: 475px;
}

.container-16 .grid-10 {
  width: 530px;
}

.container-16 .grid-11 {
  width: 585px;
}

.container-16 .grid-12 {
  width: 640px;
}

.container-16 .grid-13 {
  width: 695px;
}

.container-16 .grid-14 {
  width: 750px;
}

.container-16 .grid-15 {
  width: 805px;
}

.container-16 .grid-16 {
  width: 860px;
}

I have both Alpha and Omega enabled (along with the omega tools module).

If you do create a custom grid, then I strongly urge you to create a child subtheme, and make all your edits in the subtheme, not in Alpha or Omega. You'll need to have the omega_tools module installed to create a subtheme the easy/automatic way.

In answer to your question : maybe you don't need to have Alpha enabled at all -- it's not clear in the theming documentation if Alpha is merely a library of source code to borrow from for your subtheme.

I too was struggling with this, and then found this very clear writeup in the Omega wiki: Creating and Setting up a Custom Grid for Omega 3.x.

By the way, you don't need to have any Alpha or Omega themes enabled except for your actual subtheme.