Dhruv Manilawala 2567e14b7a Lexer should consider BOM for the start offset (#11732) ...

## Summary

This PR fixes a bug where the lexer didn't consider the BOM into the
start offset.

fixes: #11731

## Test Plan

Add multiple test cases which involves BOM character in the source for
the lexer and verify the snapshot.

2024-06-04 08:45:46 +00:00

..

ruff_python_parser__lexer__tests__assignment.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__bom_with_offset_edge.snap

Lexer should consider BOM for the start offset (#11732)

2024-06-04 08:45:46 +00:00

ruff_python_parser__lexer__tests__bom_with_offset.snap

Lexer should consider BOM for the start offset (#11732)

2024-06-04 08:45:46 +00:00

ruff_python_parser__lexer__tests__bom.snap

Lexer should consider BOM for the start offset (#11732)

2024-06-04 08:45:46 +00:00

ruff_python_parser__lexer__tests__comment_until_mac_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__comment_until_unix_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__comment_until_windows_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__dedent_after_whitespace.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__double_dedent_with_mac_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__double_dedent_with_tabs_mac_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__double_dedent_with_tabs_unix_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__double_dedent_with_tabs_windows_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__double_dedent_with_unix_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__double_dedent_with_windows_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__emoji_identifier.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__empty_fstrings.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__empty_ipython_escape_command.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__escape_unicode_name.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_comments.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_conversion.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_escape_braces.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_escape_raw.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_escape.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_expression_multiline.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_multiline.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_named_unicode_raw.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_named_unicode.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_nested.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_parentheses.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_prefix.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_single_quote_escape_mac_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_single_quote_escape_unix_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_single_quote_escape_windows_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_with_format_spec.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_with_ipy_escape_command.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_with_lambda_expression.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_with_multiline_format_spec.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_with_named_expression.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring_with_nul_char.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__fstring.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__indentation_with_mac_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__indentation_with_unix_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__indentation_with_windows_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__invalid_leading_zero_big.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__invalid_leading_zero_small.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__ipython_escape_command_assignment.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__ipython_escape_command_indentation.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_mac_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_unix_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_windows_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_with_mac_eol_and_eof.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_with_unix_eol_and_eof.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__ipython_escape_command_line_continuation_with_windows_eol_and_eof.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__ipython_escape_command.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__ipython_help_end_escape_command.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__line_comment_empty.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__line_comment_long.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__line_comment_single_whitespace.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__line_comment_whitespace.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__logical_newline_line_comment.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__match_softkeyword_in_notebook.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__newline_in_brackets_mac_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__newline_in_brackets_unix_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__newline_in_brackets_windows_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__non_logical_newline_in_string_continuation.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__numbers.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__operators.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__string_continuation_with_mac_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__string_continuation_with_unix_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__string_continuation_with_windows_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__string.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__tet_too_low_dedent.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__triple_quoted_mac_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__triple_quoted_unix_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__lexer__tests__triple_quoted_windows_eol.snap

Maintain synchronicity between the lexer and the parser (#11457)

2024-06-03 18:23:50 +05:30

ruff_python_parser__string__tests__backspace_alias.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__bell_alias.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__carriage_return_alias.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__character_tabulation_with_justification_alias.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__delete_alias.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__dont_panic_on_8_in_octal_escape.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__double_quoted_byte.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__escape_alias.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__escape_char_in_byte_literal.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__escape_octet.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__form_feed_alias.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__fstring_constant_range.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__fstring_escaped_character.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__fstring_escaped_newline.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__fstring_line_continuation.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__fstring_parse_self_documenting_base_more.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__fstring_parse_self_documenting_base.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__fstring_parse_self_documenting_format.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__fstring_unescaped_newline.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__hts_alias.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__invalid_byte_literal_error.snap

Replace LALRPOP parser with hand-written parser (#10036)

2024-04-18 17:57:39 +05:30

ruff_python_parser__string__tests__invalid_unicode_literal.snap

Replace LALRPOP parser with hand-written parser (#10036)

2024-04-18 17:57:39 +05:30

ruff_python_parser__string__tests__invalid_unicode_name_error.snap

Replace LALRPOP parser with hand-written parser (#10036)

2024-04-18 17:57:39 +05:30

ruff_python_parser__string__tests__missing_unicode_lbrace_error.snap

Replace LALRPOP parser with hand-written parser (#10036)

2024-04-18 17:57:39 +05:30

ruff_python_parser__string__tests__missing_unicode_rbrace_error.snap

Replace LALRPOP parser with hand-written parser (#10036)

2024-04-18 17:57:39 +05:30

ruff_python_parser__string__tests__parse_empty_fstring.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_f_string_concat_1.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_f_string_concat_2.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_f_string_concat_3.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_f_string_concat_4.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_fstring_equals.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_fstring_nested_concatenation_string_spec.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_fstring_nested_spec.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_fstring_nested_string_spec.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_fstring_not_equals.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_fstring_not_nested_spec.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_fstring_self_doc_prec_space.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_fstring_self_doc_trailing_space.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_fstring_yield_expr.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_fstring.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_string_concat.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_string_triple_quotes_with_kind.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_u_f_string_concat_1.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_u_f_string_concat_2.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_u_string_concat_1.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__parse_u_string_concat_2.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__raw_byte_literal_1.snap

Fix typo in string_token_flags.rs (#10476)

2024-03-19 17:43:08 +00:00

ruff_python_parser__string__tests__raw_byte_literal_2.snap

Fix typo in string_token_flags.rs (#10476)

2024-03-19 17:43:08 +00:00

ruff_python_parser__string__tests__raw_fstring.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__single_quoted_byte.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__string_parser_escaped_mac_eol.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__string_parser_escaped_unix_eol.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__string_parser_escaped_windows_eol.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00

ruff_python_parser__string__tests__triple_quoted_raw_fstring.snap

Track casing of r-string prefixes in the tokenizer and AST (#10314)

2024-03-18 17:18:04 +00:00