Class Tokenizer

This class takes a set of highlighting rules, and creates a tokenizer out of them. For more information, see the wiki on extending highlighters.

Hierarchy (view full)

Tokenizer
- Tokenizer

Index

Constructors

Properties

matchMappings regExps reportError splitRegex states

Methods

$applyToken $arrayTokens $setMaxTokenCount createSplitterRegexp getLineTokens removeCapturingGroups

Constructors

constructor

new Tokenizer(rules): Tokenizer
Constructs a new tokenizer based on the given rules and flags.
Parameters
- rules: any
  The highlighting rules
Returns Tokenizer
Inherited from Tokenizer.constructor
- Defined in src/tokenizer.js:14

Properties

matchMappings

matchMappings: {}

Type declaration

regExps

regExps: {}

Type declaration

reportError

reportError: ((msg, data) => void)

Type declaration

- (msg, data): void
- Parameters
  - msg: any
  - data: any
  Returns void

splitRegex

splitRegex: RegExp

states

states: any

Methods

$applyToken

$applyToken(str): Token[]
Parameters
- str: string
Returns Token[]
Inherited from Tokenizer.$applyToken
- Defined in src/tokenizer.js:116

$arrayTokens

$arrayTokens(str): string | Token[]
Parameters
- str: string
Returns string | Token[]
Inherited from Tokenizer.$arrayTokens
- Defined in src/tokenizer.js:140

$setMaxTokenCount

$setMaxTokenCount(m): void
Parameters
- m: number
Returns void
Inherited from Tokenizer.$setMaxTokenCount
- Defined in src/tokenizer.js:108

createSplitterRegexp

createSplitterRegexp(src, flag): RegExp
Parameters
- src: string
- flag: string
Returns RegExp
Inherited from Tokenizer.createSplitterRegexp
- Defined in src/tokenizer.js:175

getLineTokens

getLineTokens(line, startState): any
Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.
Parameters
- line: any
- startState: any
Returns any
Overrides Tokenizer.getLineTokens
- Defined in src/tokenizer_dev.js:17

removeCapturingGroups

removeCapturingGroups(src): string
Parameters
- src: string
Returns string
Inherited from Tokenizer.removeCapturingGroups
- Defined in src/tokenizer.js:163

Generated using TypeDoc