Dylan f2b7c82534 Handle t-string prefixes in SimpleTokenizer (#20578) ...

The simple tokenizer is meant to skip strings, but it was recording a
`Name` token for t-strings (from the `t`). This PR fixes that.

2025-09-25 14:33:37 -05:00

..

simple_tokenizer__comment_containing_single_quoted_string.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__comment_containing_triple_quoted_string.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__empty_string_literal.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__fstring.snap

Handle t-string prefixes in SimpleTokenizer (#20578)

2025-09-25 14:33:37 -05:00

simple_tokenizer__identifier_ending_in_non_start_char.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__identifier_starting_with_string_kind.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__ignore_word_with_only_id_continuing_chars.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__Reverse.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__single_quoted_multiline_string_containing_comment.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__single_quoted_multiline_string_implicit_concatenation.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__string_followed_by_multiple_comments.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__string_with_byte_kind.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__string_with_double_escaped_backslash.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__string_with_escaped_quote.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__string_with_invalid_kind.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__string_with_kind.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_bogus.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_comma.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_continuation.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_eq.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_invalid_operators.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_multichar.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_not_eq.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_operators.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_parentheses.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_slash.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_substring.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tokenize_trivia.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tricky_unicode.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__triple_quoted_multiline_string_containing_comment.snap

Update insta snapshots (#14366)

2024-11-15 19:31:15 +01:00

simple_tokenizer__tstring.snap

Handle t-string prefixes in SimpleTokenizer (#20578)

2025-09-25 14:33:37 -05:00