Uros Bizjak
3378323bbb
locking/lockref: Use try_cmpxchg64 in CMPXCHG_LOOP macro
...
Use try_cmpxchg64 instead of cmpxchg64 in CMPXCHG_LOOP macro.
x86 CMPXCHG instruction returns success in ZF flag, so this
change saves a compare after cmpxchg (and related move instruction
in front of cmpxchg). The main loop of lockref_get improves from:
13: 48 89 c1 mov %rax,%rcx
16: 48 c1 f9 20 sar $0x20,%rcx
1a: 83 c1 01 add $0x1,%ecx
1d: 48 89 ce mov %rcx,%rsi
20: 89 c1 mov %eax,%ecx
22: 48 89 d0 mov %rdx,%rax
25: 48 c1 e6 20 shl $0x20,%rsi
29: 48 09 f1 or %rsi,%rcx
2c: f0 48 0f b1 4d 00 lock cmpxchg %rcx,0x0(%rbp)
32: 48 39 d0 cmp %rdx,%rax
35: 75 17 jne 4e <lockref_get+0x4e>
to:
13: 48 89 ca mov %rcx,%rdx
16: 48 c1 fa 20 sar $0x20,%rdx
1a: 83 c2 01 add $0x1,%edx
1d: 48 89 d6 mov %rdx,%rsi
20: 89 ca mov %ecx,%edx
22: 48 c1 e6 20 shl $0x20,%rsi
26: 48 09 f2 or %rsi,%rdx
29: f0 48 0f b1 55 00 lock cmpxchg %rdx,0x0(%rbp)
2f: 75 02 jne 33 <lockref_get+0x33>
[ Michael Ellerman and Mark Rutland confirm that code generation on
powerpc and arm64 respectively is also ok, even though they do not
have a native arch_try_cmpxchg() implementation, and rely on the
default fallback case - Linus ]
Signed-off-by: Uros Bizjak <ubizjak@gmail.com >
Tested-by: Michael Ellerman <mpe@ellerman.id.au >
Tested-by: Mark Rutland <mark.rutland@arm.com >
Cc: Peter Zijlstra <peterz@infradead.org >
Cc: Thomas Gleixner <tglx@linutronix.de >
Cc: Waiman.Long@hp.com
Cc: paulmck@linux.vnet.ibm.com
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org >
2022-05-26 09:52:53 -07:00
..
2020-08-10 01:32:59 +09:00
2022-03-21 16:02:36 -07:00
2022-05-09 17:20:37 -07:00
2021-07-08 11:48:20 -07:00
2022-05-17 10:01:40 -06:00
2020-08-09 14:10:26 -07:00
2022-04-08 14:20:36 -10:00
2020-12-15 22:46:19 -08:00
2021-09-08 11:50:26 -07:00
2022-03-03 10:47:52 +12:00
2021-07-08 11:48:20 -07:00
2022-03-08 15:20:21 -08:00
2021-07-08 11:48:20 -07:00
2022-02-13 16:50:06 -08:00
2021-04-14 23:04:44 +10:00
2021-10-19 23:44:30 +08:00
2020-08-10 01:32:59 +09:00
2020-12-29 15:36:49 -08:00
2021-09-24 16:13:35 -07:00
2021-11-18 13:16:22 -08:00
2021-10-18 12:28:52 -07:00
2019-05-24 17:39:02 +02:00
2019-05-24 17:39:02 +02:00
2020-11-18 14:15:17 -06:00
2022-01-20 08:52:55 +02:00
2021-10-13 14:54:13 -05:00
2019-05-30 11:26:32 -07:00
2021-12-13 10:56:09 +01:00
2021-10-01 16:52:48 -04:00
2021-05-06 19:24:12 -07:00
2020-10-16 13:25:14 -06:00
2022-03-23 19:00:33 -07:00
2019-05-21 10:50:45 +02:00
2021-11-06 14:08:17 -07:00
2020-06-11 15:14:53 +02:00
2019-06-05 17:37:16 +02:00
2018-08-16 12:14:42 -07:00
2022-05-19 23:46:10 +02:00
2019-05-24 17:27:11 +02:00
2021-07-08 11:48:22 -07:00
2018-11-30 07:22:05 +01:00
2020-08-20 15:45:14 -04:00
2019-06-19 17:09:55 +02:00
2021-06-23 16:41:41 -06:00
2021-05-05 16:07:40 +02:00
2019-05-24 17:39:02 +02:00
2021-10-01 16:52:48 -04:00
2019-10-18 15:01:57 +02:00
2021-11-06 13:30:41 -07:00
2019-06-19 17:09:06 +02:00
2021-01-21 14:06:00 -07:00
2021-05-06 19:24:12 -07:00
2019-06-19 17:09:06 +02:00
2022-01-31 11:21:43 +11:00
2022-01-31 11:21:43 +11:00
2022-03-07 12:48:35 -07:00
2022-03-07 12:48:35 -07:00
2019-06-19 17:09:06 +02:00
2019-06-19 17:09:06 +02:00
2020-06-18 17:26:43 +10:00
2021-08-19 09:02:55 +09:00
2021-06-22 13:56:43 +02:00
2022-05-13 12:35:17 +02:00
2021-07-01 11:06:05 -07:00
2020-01-31 10:30:40 -08:00
2021-07-01 11:06:06 -07:00
2021-05-06 19:24:12 -07:00
2021-07-01 11:06:06 -07:00
2021-10-19 23:44:30 +08:00
2021-11-08 16:55:26 -08:00
2020-07-31 11:49:08 +02:00
2021-08-13 14:09:32 -10:00
2021-09-23 09:25:59 +02:00
2020-11-20 14:45:33 +11:00
2022-04-22 21:30:57 +02:00
2021-10-21 13:01:25 +02:00
2020-10-16 11:11:20 -07:00
2021-04-16 16:10:37 -07:00
2020-12-15 22:46:15 -08:00
2021-09-30 21:24:06 -04:00
2020-12-15 22:46:15 -08:00
2021-01-03 20:05:18 -05:00
2020-10-16 11:11:22 -07:00
2020-04-15 21:36:41 +01:00
2020-01-08 16:59:19 +00:00
2022-01-15 08:47:31 -08:00
2022-01-15 08:47:31 -08:00
2021-10-18 07:49:39 -04:00
2022-03-07 12:48:35 -07:00
2022-01-15 08:47:31 -08:00
2019-10-14 15:04:00 -07:00
2020-11-18 14:15:17 -06:00
2022-04-27 10:57:33 -07:00
2019-05-13 11:07:33 +02:00
2020-10-20 14:39:37 -07:00
2019-05-21 10:50:45 +02:00
2019-05-21 10:50:45 +02:00
2019-06-19 17:09:56 +02:00
2020-08-14 19:56:57 -07:00
2022-02-21 10:16:39 -05:00
2022-04-13 21:32:21 +02:00
2019-05-30 11:26:32 -07:00
2019-05-24 17:27:11 +02:00
2021-08-19 09:02:55 +09:00
2022-05-22 11:32:30 -07:00
2022-05-25 11:32:53 -07:00
2022-03-24 19:06:48 -07:00
2022-04-22 12:32:04 +02:00
2022-03-22 15:57:11 -07:00
2020-09-28 12:14:08 +01:00
2022-04-22 12:32:03 +02:00
2021-07-08 11:48:20 -07:00
2019-06-19 17:09:08 +02:00
2021-12-28 11:26:18 +01:00
2022-04-05 15:39:19 +02:00
2022-01-20 08:52:53 +02:00
2021-05-19 15:05:11 +02:00
2020-10-16 11:11:19 -07:00
2021-08-13 18:37:38 +02:00
2022-01-20 08:52:53 +02:00
2021-07-08 11:48:20 -07:00
2022-04-05 13:32:27 -06:00
2019-06-05 17:37:06 +02:00
2021-12-04 10:56:24 +01:00
2022-05-26 09:52:53 -07:00
2022-03-11 10:42:56 +01:00
2021-02-17 17:31:06 -06:00
2021-04-16 16:10:37 -07:00
2019-05-24 17:39:02 +02:00
2022-05-22 11:32:30 -07:00
2018-10-16 13:45:44 +02:00
2021-10-18 12:28:52 -07:00
2019-05-21 10:50:45 +02:00
2020-09-26 10:33:57 -07:00
2019-05-24 17:39:02 +02:00
2019-05-21 10:50:45 +02:00
2021-07-08 11:48:20 -07:00
2022-04-22 21:30:57 +02:00
2019-07-03 16:57:18 +02:00
2021-12-24 14:54:29 -08:00
2019-05-21 10:50:45 +02:00
2021-07-08 11:48:20 -07:00
2021-08-08 13:00:20 +01:00
2022-02-27 09:29:02 -08:00
2021-09-17 13:52:17 +01:00
2021-04-16 16:10:37 -07:00
2021-07-01 11:06:05 -07:00
2021-09-19 17:13:35 -07:00
2021-05-06 19:24:12 -07:00
2019-05-21 10:50:45 +02:00
2022-05-18 02:20:17 -04:00
2019-05-21 11:28:45 +02:00
2019-05-21 10:50:45 +02:00
2022-05-22 11:32:30 -07:00
2021-04-16 16:10:37 -07:00
2022-05-18 15:53:52 +02:00
2019-06-05 17:37:16 +02:00
2019-09-25 17:51:39 -07:00
2020-08-12 10:58:00 -07:00
2022-02-06 11:05:28 +00:00
2021-07-08 11:48:20 -07:00
2021-07-08 11:48:20 -07:00
2022-03-21 20:01:34 -06:00
2021-11-09 10:02:50 -08:00
2021-07-08 11:48:20 -07:00
2021-08-24 19:52:40 -03:00
2019-08-08 07:45:01 -06:00
2022-01-18 13:03:55 +01:00
2019-09-24 15:54:09 -07:00
2022-05-18 15:53:52 +02:00
2022-04-06 10:11:48 +02:00
2021-05-19 10:51:40 +02:00
2022-03-17 20:17:18 -07:00
2022-04-06 10:55:50 +02:00
2022-03-21 08:13:04 -07:00
2019-05-30 11:26:32 -07:00
2022-02-13 16:50:06 -08:00
2021-09-25 08:20:49 -07:00
2022-04-19 21:58:47 +02:00
2022-04-19 21:58:47 +02:00
2021-06-18 11:43:09 +02:00
2022-01-15 08:47:31 -08:00
2021-07-08 11:48:20 -07:00
2020-08-12 10:58:00 -07:00
2019-07-01 19:34:46 -07:00
2022-04-01 00:27:52 +02:00
2019-05-21 10:50:45 +02:00
2021-05-13 16:14:45 +02:00
2022-03-17 20:17:14 -07:00
2021-01-18 11:03:26 +01:00
2020-12-15 22:46:16 -08:00
2022-01-20 08:52:54 +02:00
2018-11-30 12:13:15 -08:00
2021-07-01 11:06:03 -07:00
2022-03-03 12:47:33 -05:00
2018-10-15 16:31:29 -04:00
2021-11-06 13:30:33 -07:00
2022-04-04 14:29:20 -06:00
2022-03-29 15:13:36 -07:00
2021-10-26 17:23:46 -04:00
2020-05-08 18:18:12 +01:00
2021-06-25 11:31:03 -06:00
2022-02-25 09:36:06 +01:00
2018-10-16 13:45:44 +02:00
2022-01-20 08:52:54 +02:00
2020-03-06 11:56:59 +01:00
2019-05-21 10:50:45 +02:00
2020-06-15 13:32:11 -07:00
2021-10-27 13:40:14 +02:00
2021-12-06 16:04:44 -08:00
2021-03-24 15:16:09 -07:00
2021-09-06 11:04:03 -07:00
2021-09-08 11:50:26 -07:00
2019-06-05 17:36:37 +02:00
2019-06-05 17:36:37 +02:00
2021-07-01 11:06:05 -07:00
2019-04-08 16:44:21 -06:00
2022-05-03 10:15:07 +02:00
2022-01-20 08:52:55 +02:00
2019-10-16 14:56:21 +02:00
2021-11-06 13:30:37 -07:00
2022-03-28 19:25:11 -04:00
2021-07-19 11:39:28 +02:00
2019-10-03 12:12:23 -04:00
2021-02-17 14:08:01 +01:00
2020-08-12 10:58:00 -07:00
2020-11-18 14:15:17 -06:00
2020-04-07 10:43:43 -07:00
2022-03-23 19:00:35 -07:00
2021-02-05 11:03:47 -08:00
2019-05-24 17:39:02 +02:00
2020-10-16 11:11:22 -07:00
2020-03-23 17:01:47 +01:00
2022-05-19 16:54:15 +02:00
2022-04-22 15:35:40 -04:00
2020-08-12 10:58:00 -07:00