Class Tokenizer

constructor

• new Tokenizer(rules: any): Tokenizer

  Constructs a new tokenizer based on the given rules and flags.

  Parameters

  • rules: any

    The highlighting rules

  Returns Tokenizer

matchMappings

regExps

reportError

splitRegex

states

$applyToken

• $applyToken(str: string): Token[]

  Parameters

  • str: string

  Returns Token[]

$arrayTokens

• $arrayTokens(str: string): string | Token[]

  Parameters

  • str: string

  Returns string | Token[]

$setMaxTokenCount

• $setMaxTokenCount(m: number): void

  Parameters

  • m: number

  Returns void

createSplitterRegexp

• createSplitterRegexp(src: string, flag: string): RegExp

  Parameters

  • src: string
  • flag: string

  Returns RegExp

getLineTokens

• getLineTokens(
      line: string,
      startState: string | string[],
  ): { state: string | string[]; tokens: Token[] }

  Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.

  Parameters

  • line: string
  • startState: string | string[]

  Returns { state: string | string[]; tokens: Token[] }

removeCapturingGroups

• removeCapturingGroups(src: string): string

  Parameters

  • src: string

  Returns string