Class Tokenizer

This class takes a set of highlighting rules, and creates a tokenizer out of them. For more information, see the wiki on extending highlighters.

Hierarchy (View Summary)

Tokenizer
- Tokenizer

Index

Constructors

Properties

matchMappings regExps reportError splitRegex states

Methods

$applyToken $arrayTokens $setMaxTokenCount createSplitterRegexp getLineTokens removeCapturingGroups

Constructors

constructor

new Tokenizer(rules: any): Tokenizer
Constructs a new tokenizer based on the given rules and flags.
Parameters
- rules: any
  The highlighting rules
Returns Tokenizer
Inherited from Tokenizer.constructor
- Defined in src/tokenizer.js:14

Properties

matchMappings

matchMappings: {}

regExps

regExps: {}

reportError

reportError: (msg: any, data: any) => void

splitRegex

splitRegex: RegExp

states

states: any

Methods

$applyToken

$applyToken(str: string): Token[]
Parameters
- str: string
Returns Token[]
Inherited from Tokenizer.$applyToken
- Defined in src/tokenizer.js:116

$arrayTokens

$arrayTokens(str: string): string | Token[]
Parameters
- str: string
Returns string | Token[]
Inherited from Tokenizer.$arrayTokens
- Defined in src/tokenizer.js:140

$setMaxTokenCount

$setMaxTokenCount(m: number): void
Parameters
- m: number
Returns void
Inherited from Tokenizer.$setMaxTokenCount
- Defined in src/tokenizer.js:108

createSplitterRegexp

createSplitterRegexp(src: string, flag: string): RegExp
Parameters
- src: string
- flag: string
Returns RegExp
Inherited from Tokenizer.createSplitterRegexp
- Defined in src/tokenizer.js:175

getLineTokens

getLineTokens(line: any, startState: any): any
Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.
Parameters
- line: any
- startState: any
Returns any
Overrides Tokenizer.getLineTokens
- Defined in src/tokenizer_dev.js:17

removeCapturingGroups

removeCapturingGroups(src: string): string
Parameters
- src: string
Returns string
Inherited from Tokenizer.removeCapturingGroups
- Defined in src/tokenizer.js:163