Class Tokenizer

constructor

• new Tokenizer(rules): Tokenizer

• Constructs a new tokenizer based on the given rules and flags.

  Parameters

  • rules: any

    The highlighting rules

  Returns Tokenizer

matchMappings

regExps

reportError

splitRegex

states

$applyToken

• $applyToken(str): Token[]

• Parameters

  • str: string

  Returns Token[]

$arrayTokens

• $arrayTokens(str): string | Token[]

• Parameters

  • str: string

  Returns string | Token[]

$setMaxTokenCount

• $setMaxTokenCount(m): void

• Parameters

  • m: number

  Returns void

createSplitterRegexp

• createSplitterRegexp(src, flag): RegExp

• Parameters

  • src: string

  • flag: string

  Returns RegExp

getLineTokens

• getLineTokens(line, startState): {
      state: string | string[];
      tokens: Token[];
  }

• Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.

  Parameters

  • line: string

  • startState: string | string[]

  Returns {
      state: string | string[];
      tokens: Token[];
  }

  • state: string | string[]

  • tokens: Token[]

removeCapturingGroups

• removeCapturingGroups(src): string

• Parameters

  • src: string

  Returns string