To create a Custom Filter, you can do these steps:

• Make sure you have the Custom Filter module installed
• Go to Custom Filters (/admin/settings/customfilter) (6.x & 7.x)
• Click Add Filter tab

Now there should be a form asking some properties. Give this new filter:

• Name, as an identifier for this filter
• Description, some sort of description to describe what this filter is about and what this filter does. This information is used only in the Custom Filter administration page.
• Tips (short), short tips to be shown in the node creation form, below body field.
• Tips (full), longer tips to be shown in Compose Tips. That is the page when you click more information about formatting options.
• Cache to determine the node should be cached. When this option is on, filtering will be cached. If you need a filter that must be processed everytime the node loads (e.g. filter that shows current date/time, or filter that generates random numbers), you should turn this option off.

Replacement rules

The just created filter won't do anything. Before our new filter could do any filtering, first we must add one or some replacement rules. To add replacement rules, you can do these steps:

• Go to Custom Filter
• Click on the name of our new filter (click on the name, not on edit. Edit is for editing properties of the filter).
• Click Add Rule.

Now there should be a form. Here you can specify:

• Name, name for this replacement rule.
• Pattern, pattern to be found.
• PHP Code, an option to tell Drupal about the replacement text.
• Replacement Text.
• Weight.
• Description.

Matching original strings

On the "Edit rule" page, there is a note at the bottom that explains that you can use parts of the string you're matching in the replacement text. To use this, wrap parts of your regular expression in the "Pattern" box inside brackets. For example: /<ref="([^"\r\n]*)">([^<]*)(<\/ref>)/i

The first original string that will be matched in this pattern is this bit: [^"\r\n]* which is used to match anything that is not a double-quote, line break or new line. The second piece to be matched is [^<]* which matches anything before the next <.

You can now refer to each of these using the $n and ${n} notation given in the explanation text. $1 will refer to the first original string and $2 will refer to the second.
Note: I'm not sure what the difference is between the two notations, $n and ${n}. Perhaps it helps with things like quotation marks?

Using created filter in Input Format

Nested rule and how it works

Using cache option