Tokenisation

for each line:

• If it contains '<num>' remove all whitespace
• If it contains '<DOCID' or '<DOCNO' or any case variant do nothing.
• Else:
  • insert space to separate SGML tags from PCDATA
  • insert space before punctuation (.!?,:;)
  • replace entities &gt;, &lt; and &amp; by >, < and & and enclose them with space
  • replace signs, brackets and quotes with space (+#()[]@$%^&*_\={}"|'`~`)

Then any sequence of whitespace separates tokens. A token starting with < and ending with > is considered to be a SGML tag. Note that tags cannot contain whitespace because any such tag would have been split into at least two tokens during tokenisation.

Non-breakable space (\xa0 in ISO 8859-1) seems not to be treated as whitespace. Some tokens in the Finnish corpus start with this character. Furtunatly, the other steps accepted non-breakable space as a token character as well.