Charlie Marsh 86ccdcc9d9 Add support for multi-character operator tokens to SimpleTokenizer (#6563) ...

## Summary

Allows for proper lexing of tokens like `->`.

The main challenge is to ensure that our forward and backwards
representations are the same for cases like `===`. Specifically, we want
that to lex as `==` followed by `=` regardless of whether it's a
forwards or backwards lex. To do so, we identify the range of the
sequential characters (the full span of `===`), lex it forwards, then
return the last token.

## Test Plan

`cargo test`

2023-08-16 09:09:19 -04:00

..

src

Add support for multi-character operator tokens to SimpleTokenizer (#6563)

2023-08-16 09:09:19 -04:00

Cargo.toml

Remove some extraneous newlines in Cargo.toml (#6577)

2023-08-14 23:39:41 +00:00