Regular Expressions

Authors:	Tom Dunham
Date:	2009-03-31

A mini-language to express patterns in string data (which can then be used to search a string for that pattern)
Very powerful
Very easy to abuse

Links on regular expressions (including the origin of that quote)

And there is a good book

Mastering Regular Expressions Powerful Techniques for Perl and Other Tools By Jeffrey E. F. Friedl ISBN 10: 1-56592-257-3 | ISBN 13: 9781565922570

Syntax

Regular expressions are a mixture of literal characters and meta-characters.

Letters and numbers match themselves
- the expression test will match the string test
- test → test
. matches any character (except newline)
- t..t → test, and tent, text, that, tilt...

^ and $ match the beginning and end of the string respectively
- ^t..t → turtler, twattle, twitch, twitched, twitted...
- t..t$ → surtout, unrepentant, volitant, unimportant, whitest...
- ^t..t$ → tent, test, text, tret
[ and ] define a character class - a set of characters that can match
- t[aou]t → tat, tot, tut...
- t[aou]t ↛ toot, tout, taut (as the character class matches only one character)
- (↛ means "will not match")

Characters
- \d matches digits
- \w matches word characters
Whitespace
- \s matches a whitespace character (tabs, spaces, newlines)
- \n matches a newline
- \b matches a word boundary (word, space, word)

You can repeat matches with * and +

Regular expressions are provided by the re module:

>>> import re
>>> re.findall(r"\bto[aeiou]+t\b", allwords)
['toit', 'toot', 'tout']

findall will return a list of all non-overlapping matches in a string.

>>> import re
>>> m = re.search(r"\bto[aeiou]+t\b", allwords)
>>> m.start()
916486
>>> m.end()
916490
>>> allwords[m.start():m.end()]
'toit'

See handout

Using the wordlist CROSSWD.TXT,

Find all words ending in oligy
Find words the start with dys
Write an expression that matches 'gelatine', 'gemstone', 'gendarme', 'gene', 'genome', 'genuine', 'geophone', and 'germane'
The file rebase_allenz.txt contains a header, a number of records and a set of references. Extract the references

You can extract data from a string that matches a regular expression by putting the area you are interested in in a group.

Matching happens from left to right, and every term matches as much as possible (it's greedy). Be careful with .* and .+

"^t(.+)(.*)t$" The second group will never match anything

You can change this behavior, and force a non-greedy match using a ?

^t(.+?)(.*)t$ Matches and extracts:

turnout u,rnou
turnspit u,rnspi
turret u,rre
tut u,
twangiest w,angies

It can be useful to change the way matches are performed, you do this by passing flags:

re.search(r"\b.*oligy\b", w, re.IGNORECASE)
re.search(r"\b.*oligy\b", w, re.IGNORECASE|re.DOTALL)

The IGNORECASE flag makes the expression case-insensitive, the DOTALL flag makes . match everything, including newlines.

See handout

Extract the eight field names from the header of rebase_allenz.txt
Extract the enzyme names and recognition sequences from rebase_allenz.txt. Remove the point of cleavage information, and produce a list of pairs (tuples with 2 elements) containing (enzyme name, (list of sites)). Print those enzymes with more than one site (there are six).

regular expressions don't find overlapping matches
DNA is double-stranded and can be searched in both directions, which regular expressions don't understand
DNA sequences can be very big

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

-- Jamie Zawinski

Some more regex links

An online regex builder http://www.txt2re.com