mirror/linux
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge tag 'trace-v5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace

Browse Source 
Pull tracing updates from Steven Rostedt:

 - Added option for per CPU threads to the hwlat tracer

 - Have hwlat tracer handle hotplug CPUs

 - New tracer: osnoise, that detects latency caused by interrupts,
   softirqs and scheduling of other tasks.

 - Added timerlat tracer that creates a thread and measures in detail
   what sources of latency it has for wake ups.

 - Removed the "success" field of the sched_wakeup trace event. This has
   been hardcoded as "1" since 2015, no tooling should be looking at it
   now. If one exists, we can revert this commit, fix that tool and try
   to remove it again in the future.

 - tgid mapping fixed to handle more than PID_MAX_DEFAULT pids/tgids.

 - New boot command line option "tp_printk_stop", as tp_printk causes
   trace events to write to console. When user space starts, this can
   easily live lock the system. Having a boot option to stop just after
   boot up is useful to prevent that from happening.

 - Have ftrace_dump_on_oops boot command line option take numbers that
   match the numbers shown in /proc/sys/kernel/ftrace_dump_on_oops.

 - Bootconfig clean ups, fixes and enhancements.

 - New ktest script that tests bootconfig options.

 - Add tracepoint_probe_register_may_exist() to register a tracepoint
   without triggering a WARN*() if it already exists. BPF has a path
   from user space that can do this. All other paths are considered a
   bug.

 - Small clean ups and fixes

* tag 'trace-v5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace: (49 commits)
  tracing: Resize tgid_map to pid_max, not PID_MAX_DEFAULT
  tracing: Simplify & fix saved_tgids logic
  treewide: Add missing semicolons to __assign_str uses
  tracing: Change variable type as bool for clean-up
  trace/timerlat: Fix indentation on timerlat_main()
  trace/osnoise: Make 'noise' variable s64 in run_osnoise()
  tracepoint: Add tracepoint_probe_register_may_exist() for BPF tracing
  tracing: Fix spelling in osnoise tracer "interferences" -> "interference"
  Documentation: Fix a typo on trace/osnoise-tracer
  trace/osnoise: Fix return value on osnoise_init_hotplug_support
  trace/osnoise: Make interval u64 on osnoise_main
  trace/osnoise: Fix 'no previous prototype' warnings
  tracing: Have osnoise_main() add a quiescent state for task rcu
  seq_buf: Make trace_seq_putmem_hex() support data longer than 8
  seq_buf: Fix overflow in seq_buf_putmem_hex()
  trace/osnoise: Support hotplug operations
  trace/hwlat: Support hotplug operations
  trace/hwlat: Protect kdata->kthread with get/put_online_cpus
  trace: Add timerlat tracer
  trace: Add osnoise tracer
  ...
This commit is contained in:
Linus Torvalds
2021-07-03 11:13:22 -07:00
parent a70bb580bf 4030a6e6a6
commit 757fa80f4e
74 changed files with 4464 additions and 394 deletions
Download Patch File Download Diff File Expand all files Collapse all files

62
kernel/trace/Kconfig
View File

@@ -356,6 +356,68 @@ config HWLAT_TRACER
file. Every time a latency is greater than tracing_thresh, it will
be recorded into the ring buffer.
config OSNOISE_TRACER
bool "OS Noise tracer"
select GENERIC_TRACER
help
In the context of high-performance computing (HPC), the Operating
System Noise (osnoise) refers to the interference experienced by an
application due to activities inside the operating system. In the
context of Linux, NMIs, IRQs, SoftIRQs, and any other system thread
can cause noise to the system. Moreover, hardware-related jobs can
also cause noise, for example, via SMIs.
The osnoise tracer leverages the hwlat_detector by running a similar
loop with preemption, SoftIRQs and IRQs enabled, thus allowing all
the sources of osnoise during its execution. The osnoise tracer takes
note of the entry and exit point of any source of interferences,
increasing a per-cpu interference counter. It saves an interference
counter for each source of interference. The interference counter for
NMI, IRQs, SoftIRQs, and threads is increased anytime the tool
observes these interferences' entry events. When a noise happens
without any interference from the operating system level, the
hardware noise counter increases, pointing to a hardware-related
noise. In this way, osnoise can account for any source of
interference. At the end of the period, the osnoise tracer prints
the sum of all noise, the max single noise, the percentage of CPU
available for the thread, and the counters for the noise sources.
In addition to the tracer, a set of tracepoints were added to
facilitate the identification of the osnoise source.
The output will appear in the trace and trace_pipe files.
To enable this tracer, echo in "osnoise" into the current_tracer
file.
config TIMERLAT_TRACER
bool "Timerlat tracer"
select OSNOISE_TRACER
select GENERIC_TRACER
help
The timerlat tracer aims to help the preemptive kernel developers
to find sources of wakeup latencies of real-time threads.
The tracer creates a per-cpu kernel thread with real-time priority.
The tracer thread sets a periodic timer to wakeup itself, and goes
to sleep waiting for the timer to fire. At the wakeup, the thread
then computes a wakeup latency value as the difference between
the current time and the absolute time that the timer was set
to expire.
The tracer prints two lines at every activation. The first is the
timer latency observed at the hardirq context before the
activation of the thread. The second is the timer latency observed
by the thread, which is the same level that cyclictest reports. The
ACTIVATION ID field serves to relate the irq execution to its
respective thread execution.
The tracer is build on top of osnoise tracer, and the osnoise:
events can be used to trace the source of interference from NMI,
IRQs and other threads. It also enables the capture of the
stacktrace at the IRQ context, which helps to identify the code
path that can cause thread delay.
config MMIOTRACE
bool "Memory mapped IO tracing"
depends on HAVE_MMIOTRACE_SUPPORT && PCI

1
kernel/trace/Makefile
View File

@@ -58,6 +58,7 @@ obj-$(CONFIG_IRQSOFF_TRACER) += trace_irqsoff.o
obj-$(CONFIG_PREEMPT_TRACER) += trace_irqsoff.o
obj-$(CONFIG_SCHED_TRACER) += trace_sched_wakeup.o
obj-$(CONFIG_HWLAT_TRACER) += trace_hwlat.o
obj-$(CONFIG_OSNOISE_TRACER) += trace_osnoise.o
obj-$(CONFIG_NOP_TRACER) += trace_nop.o
obj-$(CONFIG_STACK_TRACER) += trace_stack.o
obj-$(CONFIG_MMIOTRACE) += trace_mmiotrace.o

3
kernel/trace/bpf_trace.c
View File

@@ -1842,7 +1842,8 @@ static int __bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *
if (prog->aux->max_tp_access > btp->writable_size)
return -EINVAL;
return tracepoint_probe_register(tp, (void *)btp->bpf_func, prog);
return tracepoint_probe_register_may_exist(tp, (void *)btp->bpf_func,
prog);
}
int bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *prog)

2
kernel/trace/ring_buffer.c
View File

@@ -3391,7 +3391,7 @@ static void check_buffer(struct ring_buffer_per_cpu *cpu_buffer,
case RINGBUF_TYPE_PADDING:
if (event->time_delta == 1)
break;
/* fall through */
fallthrough;
case RINGBUF_TYPE_DATA:
ts += event->time_delta;
break;

265
kernel/trace/trace.c
View File

@@ -87,6 +87,7 @@ void __init disable_tracing_selftest(const char *reason)
/* Pipe tracepoints to printk */
struct trace_iterator *tracepoint_print_iter;
int tracepoint_printk;
static bool tracepoint_printk_stop_on_boot __initdata;
static DEFINE_STATIC_KEY_FALSE(tracepoint_printk_key);
/* For tracers that don't implement custom flags */
@@ -197,12 +198,12 @@ __setup("ftrace=", set_cmdline_ftrace);
static int __init set_ftrace_dump_on_oops(char *str)
{
if (*str++ != '=' || !*str) {
if (*str++ != '=' || !*str || !strcmp("1", str)) {
ftrace_dump_on_oops = DUMP_ALL;
return 1;
}
if (!strcmp("orig_cpu", str)) {
if (!strcmp("orig_cpu", str) || !strcmp("2", str)) {
ftrace_dump_on_oops = DUMP_ORIG;
return 1;
}
@@ -257,6 +258,13 @@ static int __init set_tracepoint_printk(char *str)
}
__setup("tp_printk", set_tracepoint_printk);
static int __init set_tracepoint_printk_stop(char *str)
{
tracepoint_printk_stop_on_boot = true;
return 1;
}
__setup("tp_printk_stop_on_boot", set_tracepoint_printk_stop);
unsigned long long ns2usecs(u64 nsec)
{
nsec += 500;
@@ -1683,8 +1691,7 @@ static ssize_t trace_seq_to_buffer(struct trace_seq *s, void *buf, size_t cnt)
unsigned long __read_mostly tracing_thresh;
static const struct file_operations tracing_max_lat_fops;
#if (defined(CONFIG_TRACER_MAX_TRACE) || defined(CONFIG_HWLAT_TRACER)) && \
defined(CONFIG_FSNOTIFY)
#ifdef LATENCY_FS_NOTIFY
static struct workqueue_struct *fsnotify_wq;
@@ -2185,8 +2192,15 @@ void tracing_reset_all_online_cpus(void)
}
}
/*
* The tgid_map array maps from pid to tgid; i.e. the value stored at index i
* is the tgid last observed corresponding to pid=i.
*/
static int *tgid_map;
/* The maximum valid index into tgid_map. */
static size_t tgid_map_max;
#define SAVED_CMDLINES_DEFAULT 128
#define NO_CMDLINE_MAP UINT_MAX
static arch_spinlock_t trace_cmdline_lock = __ARCH_SPIN_LOCK_UNLOCKED;
@@ -2459,24 +2473,41 @@ void trace_find_cmdline(int pid, char comm[])
preempt_enable();
}
static int *trace_find_tgid_ptr(int pid)
{
/*
* Pairs with the smp_store_release in set_tracer_flag() to ensure that
* if we observe a non-NULL tgid_map then we also observe the correct
* tgid_map_max.
*/
int *map = smp_load_acquire(&tgid_map);
if (unlikely(!map || pid > tgid_map_max))
return NULL;
return &map[pid];
}
int trace_find_tgid(int pid)
{
if (unlikely(!tgid_map || !pid || pid > PID_MAX_DEFAULT))
return 0;
int *ptr = trace_find_tgid_ptr(pid);
return tgid_map[pid];
return ptr ? *ptr : 0;
}
static int trace_save_tgid(struct task_struct *tsk)
{
int *ptr;
/* treat recording of idle task as a success */
if (!tsk->pid)
return 1;
if (unlikely(!tgid_map || tsk->pid > PID_MAX_DEFAULT))
ptr = trace_find_tgid_ptr(tsk->pid);
if (!ptr)
return 0;
tgid_map[tsk->pid] = tsk->tgid;
*ptr = tsk->tgid;
return 1;
}
@@ -2730,9 +2761,45 @@ trace_event_buffer_lock_reserve(struct trace_buffer **current_rb,
if (!tr->no_filter_buffering_ref &&
(trace_file->flags & (EVENT_FILE_FL_SOFT_DISABLED | EVENT_FILE_FL_FILTERED)) &&
(entry = this_cpu_read(trace_buffered_event))) {
/* Try to use the per cpu buffer first */
/*
* Filtering is on, so try to use the per cpu buffer first.
* This buffer will simulate a ring_buffer_event,
* where the type_len is zero and the array[0] will
* hold the full length.
* (see include/linux/ring-buffer.h for details on
* how the ring_buffer_event is structured).
*
* Using a temp buffer during filtering and copying it
* on a matched filter is quicker than writing directly
* into the ring buffer and then discarding it when
* it doesn't match. That is because the discard
* requires several atomic operations to get right.
* Copying on match and doing nothing on a failed match
* is still quicker than no copy on match, but having
* to discard out of the ring buffer on a failed match.
*/
int max_len = PAGE_SIZE - struct_size(entry, array, 1);
val = this_cpu_inc_return(trace_buffered_event_cnt);
if ((len < (PAGE_SIZE - sizeof(*entry) - sizeof(entry->array[0]))) && val == 1) {
/*
* Preemption is disabled, but interrupts and NMIs
* can still come in now. If that happens after
* the above increment, then it will have to go
* back to the old method of allocating the event
* on the ring buffer, and if the filter fails, it
* will have to call ring_buffer_discard_commit()
* to remove it.
*
* Need to also check the unlikely case that the
* length is bigger than the temp buffer size.
* If that happens, then the reserve is pretty much
* guaranteed to fail, as the ring buffer currently
* only allows events less than a page. But that may
* change in the future, so let the ring buffer reserve
* handle the failure in that case.
*/
if (val == 1 && likely(len <= max_len)) {
trace_event_setup(entry, type, trace_ctx);
entry->array[0] = len;
return entry;
@@ -5172,6 +5239,8 @@ int trace_keep_overwrite(struct tracer *tracer, u32 mask, int set)
int set_tracer_flag(struct trace_array *tr, unsigned int mask, int enabled)
{
int *map;
if ((mask == TRACE_ITER_RECORD_TGID) ||
(mask == TRACE_ITER_RECORD_CMD))
lockdep_assert_held(&event_mutex);
@@ -5194,10 +5263,19 @@ int set_tracer_flag(struct trace_array *tr, unsigned int mask, int enabled)
trace_event_enable_cmd_record(enabled);
if (mask == TRACE_ITER_RECORD_TGID) {
if (!tgid_map)
tgid_map = kvcalloc(PID_MAX_DEFAULT + 1,
sizeof(*tgid_map),
GFP_KERNEL);
if (!tgid_map) {
tgid_map_max = pid_max;
map = kvcalloc(tgid_map_max + 1, sizeof(*tgid_map),
GFP_KERNEL);
/*
* Pairs with smp_load_acquire() in
* trace_find_tgid_ptr() to ensure that if it observes
* the tgid_map we just allocated then it also observes
* the corresponding tgid_map_max value.
*/
smp_store_release(&tgid_map, map);
}
if (!tgid_map) {
tr->trace_flags &= ~TRACE_ITER_RECORD_TGID;
return -ENOMEM;
@@ -5609,37 +5687,16 @@ static const struct file_operations tracing_readme_fops = {
static void *saved_tgids_next(struct seq_file *m, void *v, loff_t *pos)
{
int *ptr = v;
int pid = ++(*pos);
if (*pos || m->count)
ptr++;
(*pos)++;
for (; ptr <= &tgid_map[PID_MAX_DEFAULT]; ptr++) {
if (trace_find_tgid(*ptr))
return ptr;
}
return NULL;
return trace_find_tgid_ptr(pid);
}
static void *saved_tgids_start(struct seq_file *m, loff_t *pos)
{
void *v;
loff_t l = 0;
int pid = *pos;
if (!tgid_map)
return NULL;
v = &tgid_map[0];
while (l <= *pos) {
v = saved_tgids_next(m, v, &l);
if (!v)
return NULL;
}
return v;
return trace_find_tgid_ptr(pid);
}
static void saved_tgids_stop(struct seq_file *m, void *v)
@@ -5648,9 +5705,14 @@ static void saved_tgids_stop(struct seq_file *m, void *v)
static int saved_tgids_show(struct seq_file *m, void *v)
{
int pid = (int *)v - tgid_map;
int *entry = (int *)v;
int pid = entry - tgid_map;
int tgid = *entry;
seq_printf(m, "%d %d\n", pid, trace_find_tgid(pid));
if (tgid == 0)
return SEQ_SKIP;
seq_printf(m, "%d %d\n", pid, tgid);
return 0;
}
@@ -6135,7 +6197,7 @@ static int __tracing_resize_ring_buffer(struct trace_array *tr,
ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
unsigned long size, int cpu_id)
{
int ret = size;
int ret;
mutex_lock(&trace_types_lock);
@@ -7529,6 +7591,91 @@ static const struct file_operations snapshot_raw_fops = {
#endif /* CONFIG_TRACER_SNAPSHOT */
/*
* trace_min_max_write - Write a u64 value to a trace_min_max_param struct
* @filp: The active open file structure
* @ubuf: The userspace provided buffer to read value into
* @cnt: The maximum number of bytes to read
* @ppos: The current "file" position
*
* This function implements the write interface for a struct trace_min_max_param.
* The filp->private_data must point to a trace_min_max_param structure that
* defines where to write the value, the min and the max acceptable values,
* and a lock to protect the write.
*/
static ssize_t
trace_min_max_write(struct file *filp, const char __user *ubuf, size_t cnt, loff_t *ppos)
{
struct trace_min_max_param *param = filp->private_data;
u64 val;
int err;
if (!param)
return -EFAULT;
err = kstrtoull_from_user(ubuf, cnt, 10, &val);
if (err)
return err;
if (param->lock)
mutex_lock(param->lock);
if (param->min && val < *param->min)
err = -EINVAL;
if (param->max && val > *param->max)
err = -EINVAL;
if (!err)
*param->val = val;
if (param->lock)
mutex_unlock(param->lock);
if (err)
return err;
return cnt;
}
/*
* trace_min_max_read - Read a u64 value from a trace_min_max_param struct
* @filp: The active open file structure
* @ubuf: The userspace provided buffer to read value into
* @cnt: The maximum number of bytes to read
* @ppos: The current "file" position
*
* This function implements the read interface for a struct trace_min_max_param.
* The filp->private_data must point to a trace_min_max_param struct with valid
* data.
*/
static ssize_t
trace_min_max_read(struct file *filp, char __user *ubuf, size_t cnt, loff_t *ppos)
{
struct trace_min_max_param *param = filp->private_data;
char buf[U64_STR_SIZE];
int len;
u64 val;
if (!param)
return -EFAULT;
val = *param->val;
if (cnt > sizeof(buf))
cnt = sizeof(buf);
len = snprintf(buf, sizeof(buf), "%llu\n", val);
return simple_read_from_buffer(ubuf, cnt, ppos, buf, len);
}
const struct file_operations trace_min_max_fops = {
.open = tracing_open_generic,
.read = trace_min_max_read,
.write = trace_min_max_write,
};
#define TRACING_LOG_ERRS_MAX 8
#define TRACING_LOG_LOC_MAX 128
@@ -9532,6 +9679,8 @@ static __init int tracer_init_tracefs(void)
return 0;
}
fs_initcall(tracer_init_tracefs);
static int trace_panic_handler(struct notifier_block *this,
unsigned long event, void *unused)
{
@@ -9952,7 +10101,7 @@ void __init trace_init(void)
trace_event_init();
}
__init static int clear_boot_tracer(void)
__init static void clear_boot_tracer(void)
{
/*
* The default tracer at boot buffer is an init section.
@@ -9962,26 +10111,21 @@ __init static int clear_boot_tracer(void)
* about to be freed.
*/
if (!default_bootup_tracer)
return 0;
return;
printk(KERN_INFO "ftrace bootup tracer '%s' not registered.\n",
default_bootup_tracer);
default_bootup_tracer = NULL;
return 0;
}
fs_initcall(tracer_init_tracefs);
late_initcall_sync(clear_boot_tracer);
#ifdef CONFIG_HAVE_UNSTABLE_SCHED_CLOCK
__init static int tracing_set_default_clock(void)
__init static void tracing_set_default_clock(void)
{
/* sched_clock_stable() is determined in late_initcall */
if (!trace_boot_clock && !sched_clock_stable()) {
if (security_locked_down(LOCKDOWN_TRACEFS)) {
pr_warn("Can not set tracing clock due to lockdown\n");
return -EPERM;
return;
}
printk(KERN_WARNING
@@ -9991,8 +10135,21 @@ __init static int tracing_set_default_clock(void)
"on the kernel command line\n");
tracing_set_clock(&global_trace, "global");
}
}
#else
static inline void tracing_set_default_clock(void) { }
#endif
__init static int late_trace_init(void)
{
if (tracepoint_printk && tracepoint_printk_stop_on_boot) {
static_key_disable(&tracepoint_printk_key.key);
tracepoint_printk = 0;
}
tracing_set_default_clock();
clear_boot_tracer();
return 0;
}
late_initcall_sync(tracing_set_default_clock);
#endif
late_initcall_sync(late_trace_init);

35
kernel/trace/trace.h
View File

@@ -45,6 +45,8 @@ enum trace_type {
TRACE_BLK,
TRACE_BPUTS,
TRACE_HWLAT,
TRACE_OSNOISE,
TRACE_TIMERLAT,
TRACE_RAW_DATA,
TRACE_FUNC_REPEATS,
@@ -290,7 +292,8 @@ struct trace_array {
struct array_buffer max_buffer;
bool allocated_snapshot;
#endif
#if defined(CONFIG_TRACER_MAX_TRACE) || defined(CONFIG_HWLAT_TRACER)
#if defined(CONFIG_TRACER_MAX_TRACE) || defined(CONFIG_HWLAT_TRACER) \
|| defined(CONFIG_OSNOISE_TRACER)
unsigned long max_latency;
#ifdef CONFIG_FSNOTIFY
struct dentry *d_max_latency;
@@ -438,6 +441,8 @@ extern void __ftrace_bad_type(void);
IF_ASSIGN(var, ent, struct bprint_entry, TRACE_BPRINT); \
IF_ASSIGN(var, ent, struct bputs_entry, TRACE_BPUTS); \
IF_ASSIGN(var, ent, struct hwlat_entry, TRACE_HWLAT); \
IF_ASSIGN(var, ent, struct osnoise_entry, TRACE_OSNOISE);\
IF_ASSIGN(var, ent, struct timerlat_entry, TRACE_TIMERLAT);\
IF_ASSIGN(var, ent, struct raw_data_entry, TRACE_RAW_DATA);\
IF_ASSIGN(var, ent, struct trace_mmiotrace_rw, \
TRACE_MMIO_RW); \
@@ -668,15 +673,15 @@ void update_max_tr_single(struct trace_array *tr,
struct task_struct *tsk, int cpu);
#endif /* CONFIG_TRACER_MAX_TRACE */
#if (defined(CONFIG_TRACER_MAX_TRACE) || defined(CONFIG_HWLAT_TRACER)) && \
defined(CONFIG_FSNOTIFY)
#if (defined(CONFIG_TRACER_MAX_TRACE) || defined(CONFIG_HWLAT_TRACER) \
|| defined(CONFIG_OSNOISE_TRACER)) && defined(CONFIG_FSNOTIFY)
#define LATENCY_FS_NOTIFY
#endif
#ifdef LATENCY_FS_NOTIFY
void latency_fsnotify(struct trace_array *tr);
#else
static inline void latency_fsnotify(struct trace_array *tr) { }
#endif
#ifdef CONFIG_STACKTRACE
@@ -1945,4 +1950,22 @@ static inline bool is_good_name(const char *name)
return true;
}
/*
* This is a generic way to read and write a u64 value from a file in tracefs.
*
* The value is stored on the variable pointed by *val. The value needs
* to be at least *min and at most *max. The write is protected by an
* existing *lock.
*/
struct trace_min_max_param {
struct mutex *lock;
u64 *val;
u64 *min;
u64 *max;
};
#define U64_STR_SIZE 24 /* 20 digits max */
extern const struct file_operations trace_min_max_fops;
#endif /* _LINUX_KERNEL_TRACE_H */

27
kernel/trace/trace_boot.c
View File

@@ -225,14 +225,37 @@ static void __init
trace_boot_init_events(struct trace_array *tr, struct xbc_node *node)
{
struct xbc_node *gnode, *enode;
bool enable, enable_all = false;
const char *data;
node = xbc_node_find_child(node, "event");
if (!node)
return;
/* per-event key starts with "event.GROUP.EVENT" */
xbc_node_for_each_child(node, gnode)
xbc_node_for_each_child(gnode, enode)
xbc_node_for_each_child(node, gnode) {
data = xbc_node_get_data(gnode);
if (!strcmp(data, "enable")) {
enable_all = true;
continue;
}
enable = false;
xbc_node_for_each_child(gnode, enode) {
data = xbc_node_get_data(enode);
if (!strcmp(data, "enable")) {
enable = true;
continue;
}
trace_boot_init_one_event(tr, gnode, enode);
}
/* Event enablement must be done after event settings */
if (enable) {
data = xbc_node_get_data(gnode);
trace_array_set_clr_event(tr, data, NULL, true);
}
}
/* Ditto */
if (enable_all)
trace_array_set_clr_event(tr, NULL, NULL, true);
}
#else
#define trace_boot_enable_events(tr, node) do {} while (0)

41
kernel/trace/trace_entries.h
View File

@@ -360,3 +360,44 @@ FTRACE_ENTRY(func_repeats, func_repeats_entry,
__entry->count,
FUNC_REPEATS_GET_DELTA_TS(__entry))
);
FTRACE_ENTRY(osnoise, osnoise_entry,
TRACE_OSNOISE,
F_STRUCT(
__field( u64, noise )
__field( u64, runtime )
__field( u64, max_sample )
__field( unsigned int, hw_count )
__field( unsigned int, nmi_count )
__field( unsigned int, irq_count )
__field( unsigned int, softirq_count )
__field( unsigned int, thread_count )
),
F_printk("noise:%llu\tmax_sample:%llu\thw:%u\tnmi:%u\tirq:%u\tsoftirq:%u\tthread:%u\n",
__entry->noise,
__entry->max_sample,
__entry->hw_count,
__entry->nmi_count,
__entry->irq_count,
__entry->softirq_count,
__entry->thread_count)
);
FTRACE_ENTRY(timerlat, timerlat_entry,
TRACE_TIMERLAT,
F_STRUCT(
__field( unsigned int, seqnum )
__field( int, context )
__field( u64, timer_latency )
),
F_printk("seq:%u\tcontext:%d\ttimer_latency:%llu\n",
__entry->seqnum,
__entry->context,
__entry->timer_latency)
);

3
kernel/trace/trace_events_hist.c
View File

@@ -2434,12 +2434,12 @@ create_field_var_hist(struct hist_trigger_data *target_hist_data,
char *subsys_name, char *event_name, char *field_name)
{
struct trace_array *tr = target_hist_data->event_file->tr;
struct hist_field *event_var = ERR_PTR(-EINVAL);
struct hist_trigger_data *hist_data;
unsigned int i, n, first = true;
struct field_var_hist *var_hist;
struct trace_event_file *file;
struct hist_field *key_field;
struct hist_field *event_var;
char *saved_filter;
char *cmd;
int ret;
@@ -5232,6 +5232,7 @@ static void unregister_field_var_hists(struct hist_trigger_data *hist_data)
cmd = hist_data->field_var_hists[i]->cmd;
ret = event_hist_trigger_func(&trigger_hist_cmd, file,
"!hist", "hist", cmd);
WARN_ON_ONCE(ret < 0);
}
}

3
kernel/trace/trace_events_trigger.c
View File

@@ -916,7 +916,8 @@ void unpause_named_trigger(struct event_trigger_data *data)
/**
* set_named_trigger_data - Associate common named trigger data
* @data: The trigger data of a named trigger to unpause
* @data: The trigger data to associate
* @named_data: The common named trigger to be associated
*
* Named triggers are sets of triggers that share a common set of
* trigger data. The first named trigger registered with a given name

584
kernel/trace/trace_hwlat.c
View File

@@ -34,7 +34,7 @@
* Copyright (C) 2008-2009 Jon Masters, Red Hat, Inc. <jcm@redhat.com>
* Copyright (C) 2013-2016 Steven Rostedt, Red Hat, Inc. <srostedt@redhat.com>
*
* Includes useful feedback from Clark Williams <clark@redhat.com>
* Includes useful feedback from Clark Williams <williams@redhat.com>
*
*/
#include <linux/kthread.h>
@@ -54,20 +54,33 @@ static struct trace_array *hwlat_trace;
#define DEFAULT_SAMPLE_WIDTH 500000 /* 0.5s */
#define DEFAULT_LAT_THRESHOLD 10 /* 10us */
/* sampling thread*/
static struct task_struct *hwlat_kthread;
static struct dentry *hwlat_sample_width; /* sample width us */
static struct dentry *hwlat_sample_window; /* sample window us */
static struct dentry *hwlat_thread_mode; /* hwlat thread mode */
enum {
MODE_NONE = 0,
MODE_ROUND_ROBIN,
MODE_PER_CPU,
MODE_MAX
};
static char *thread_mode_str[] = { "none", "round-robin", "per-cpu" };
/* Save the previous tracing_thresh value */
static unsigned long save_tracing_thresh;
/* NMI timestamp counters */
static u64 nmi_ts_start;
static u64 nmi_total_ts;
static int nmi_count;
static int nmi_cpu;
/* runtime kthread data */
struct hwlat_kthread_data {
struct task_struct *kthread;
/* NMI timestamp counters */
u64 nmi_ts_start;
u64 nmi_total_ts;
int nmi_count;
int nmi_cpu;
};
struct hwlat_kthread_data hwlat_single_cpu_data;
DEFINE_PER_CPU(struct hwlat_kthread_data, hwlat_per_cpu_data);
/* Tells NMIs to call back to the hwlat tracer to record timestamps */
bool trace_hwlat_callback_enabled;
@@ -96,11 +109,24 @@ static struct hwlat_data {
u64 sample_window; /* total sampling window (on+off) */
u64 sample_width; /* active sampling portion of window */
int thread_mode; /* thread mode */
} hwlat_data = {
.sample_window = DEFAULT_SAMPLE_WINDOW,
.sample_width = DEFAULT_SAMPLE_WIDTH,
.thread_mode = MODE_ROUND_ROBIN
};
static struct hwlat_kthread_data *get_cpu_data(void)
{
if (hwlat_data.thread_mode == MODE_PER_CPU)
return this_cpu_ptr(&hwlat_per_cpu_data);
else
return &hwlat_single_cpu_data;
}
static bool hwlat_busy;
static void trace_hwlat_sample(struct hwlat_sample *sample)
{
struct trace_array *tr = hwlat_trace;
@@ -136,7 +162,9 @@ static void trace_hwlat_sample(struct hwlat_sample *sample)
void trace_hwlat_callback(bool enter)
{
if (smp_processor_id() != nmi_cpu)
struct hwlat_kthread_data *kdata = get_cpu_data();
if (!kdata->kthread)
return;
/*
@@ -145,15 +173,24 @@ void trace_hwlat_callback(bool enter)
*/
if (!IS_ENABLED(CONFIG_GENERIC_SCHED_CLOCK)) {
if (enter)
nmi_ts_start = time_get();
kdata->nmi_ts_start = time_get();
else
nmi_total_ts += time_get() - nmi_ts_start;
kdata->nmi_total_ts += time_get() - kdata->nmi_ts_start;
}
if (enter)
nmi_count++;
kdata->nmi_count++;
}
/*
* hwlat_err - report a hwlat error.
*/
#define hwlat_err(msg) ({ \
struct trace_array *tr = hwlat_trace; \
\
trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, msg); \
})
/**
* get_sample - sample the CPU TSC and look for likely hardware latencies
*
@@ -163,6 +200,7 @@ void trace_hwlat_callback(bool enter)
*/
static int get_sample(void)
{
struct hwlat_kthread_data *kdata = get_cpu_data();
struct trace_array *tr = hwlat_trace;
struct hwlat_sample s;
time_type start, t1, t2, last_t2;
@@ -175,9 +213,8 @@ static int get_sample(void)
do_div(thresh, NSEC_PER_USEC); /* modifies interval value */
nmi_cpu = smp_processor_id();
nmi_total_ts = 0;
nmi_count = 0;
kdata->nmi_total_ts = 0;
kdata->nmi_count = 0;
/* Make sure NMIs see this first */
barrier();
@@ -197,7 +234,7 @@ static int get_sample(void)
outer_diff = time_to_us(time_sub(t1, last_t2));
/* This shouldn't happen */
if (outer_diff < 0) {
pr_err(BANNER "time running backwards\n");
hwlat_err(BANNER "time running backwards\n");
goto out;
}
if (outer_diff > outer_sample)
@@ -209,7 +246,7 @@ static int get_sample(void)
/* Check for possible overflows */
if (total < last_total) {
pr_err("Time total overflowed\n");
hwlat_err("Time total overflowed\n");
break;
}
last_total = total;
@@ -225,7 +262,7 @@ static int get_sample(void)
/* This shouldn't happen */
if (diff < 0) {
pr_err(BANNER "time running backwards\n");
hwlat_err(BANNER "time running backwards\n");
goto out;
}
@@ -247,15 +284,15 @@ static int get_sample(void)
ret = 1;
/* We read in microseconds */
if (nmi_total_ts)
do_div(nmi_total_ts, NSEC_PER_USEC);
if (kdata->nmi_total_ts)
do_div(kdata->nmi_total_ts, NSEC_PER_USEC);
hwlat_data.count++;
s.seqnum = hwlat_data.count;
s.duration = sample;
s.outer_duration = outer_sample;
s.nmi_total_ts = nmi_total_ts;
s.nmi_count = nmi_count;
s.nmi_total_ts = kdata->nmi_total_ts;
s.nmi_count = kdata->nmi_count;
s.count = count;
trace_hwlat_sample(&s);
@@ -273,7 +310,6 @@ out:
}
static struct cpumask save_cpumask;
static bool disable_migrate;
static void move_to_next_cpu(void)
{
@@ -281,15 +317,13 @@ static void move_to_next_cpu(void)
struct trace_array *tr = hwlat_trace;
int next_cpu;
if (disable_migrate)
return;
/*
* If for some reason the user modifies the CPU affinity
* of this thread, then stop migrating for the duration
* of the current test.
*/
if (!cpumask_equal(current_mask, current->cpus_ptr))
goto disable;
goto change_mode;
get_online_cpus();
cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
@@ -300,7 +334,7 @@ static void move_to_next_cpu(void)
next_cpu = cpumask_first(current_mask);
if (next_cpu >= nr_cpu_ids) /* Shouldn't happen! */
goto disable;
goto change_mode;
cpumask_clear(current_mask);
cpumask_set_cpu(next_cpu, current_mask);
@@ -308,8 +342,9 @@ static void move_to_next_cpu(void)
sched_setaffinity(0, current_mask);
return;
disable:
disable_migrate = true;
change_mode:
hwlat_data.thread_mode = MODE_NONE;
pr_info(BANNER "cpumask changed while in round-robin mode, switching to mode none\n");
}
/*
@@ -328,7 +363,8 @@ static int kthread_fn(void *data)
while (!kthread_should_stop()) {
move_to_next_cpu();
if (hwlat_data.thread_mode == MODE_ROUND_ROBIN)
move_to_next_cpu();
local_irq_disable();
get_sample();
@@ -351,178 +387,380 @@ static int kthread_fn(void *data)
return 0;
}
/**
* start_kthread - Kick off the hardware latency sampling/detector kthread
/*
* stop_stop_kthread - Inform the hardware latency sampling/detector kthread to stop
*
* This kicks the running hardware latency sampling/detector kernel thread and
* tells it to stop sampling now. Use this on unload and at system shutdown.
*/
static void stop_single_kthread(void)
{
struct hwlat_kthread_data *kdata = get_cpu_data();
struct task_struct *kthread;
get_online_cpus();
kthread = kdata->kthread;
if (!kthread)
goto out_put_cpus;
kthread_stop(kthread);
kdata->kthread = NULL;
out_put_cpus:
put_online_cpus();
}
/*
* start_single_kthread - Kick off the hardware latency sampling/detector kthread
*
* This starts the kernel thread that will sit and sample the CPU timestamp
* counter (TSC or similar) and look for potential hardware latencies.
*/
static int start_kthread(struct trace_array *tr)
static int start_single_kthread(struct trace_array *tr)
{
struct hwlat_kthread_data *kdata = get_cpu_data();
struct cpumask *current_mask = &save_cpumask;
struct task_struct *kthread;
int next_cpu;
if (hwlat_kthread)
return 0;
/* Just pick the first CPU on first iteration */
get_online_cpus();
cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
put_online_cpus();
next_cpu = cpumask_first(current_mask);
if (kdata->kthread)
goto out_put_cpus;
kthread = kthread_create(kthread_fn, NULL, "hwlatd");
if (IS_ERR(kthread)) {
pr_err(BANNER "could not start sampling thread\n");
put_online_cpus();
return -ENOMEM;
}
/* Just pick the first CPU on first iteration */
cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
if (hwlat_data.thread_mode == MODE_ROUND_ROBIN) {
next_cpu = cpumask_first(current_mask);
cpumask_clear(current_mask);
cpumask_set_cpu(next_cpu, current_mask);
}
sched_setaffinity(kthread->pid, current_mask);
kdata->kthread = kthread;
wake_up_process(kthread);
out_put_cpus:
put_online_cpus();
return 0;
}
/*
* stop_cpu_kthread - Stop a hwlat cpu kthread
*/
static void stop_cpu_kthread(unsigned int cpu)
{
struct task_struct *kthread;
kthread = per_cpu(hwlat_per_cpu_data, cpu).kthread;
if (kthread)
kthread_stop(kthread);
per_cpu(hwlat_per_cpu_data, cpu).kthread = NULL;
}
/*
* stop_per_cpu_kthreads - Inform the hardware latency sampling/detector kthread to stop
*
* This kicks the running hardware latency sampling/detector kernel threads and
* tells it to stop sampling now. Use this on unload and at system shutdown.
*/
static void stop_per_cpu_kthreads(void)
{
unsigned int cpu;
get_online_cpus();
for_each_online_cpu(cpu)
stop_cpu_kthread(cpu);
put_online_cpus();
}
/*
* start_cpu_kthread - Start a hwlat cpu kthread
*/
static int start_cpu_kthread(unsigned int cpu)
{
struct task_struct *kthread;
char comm[24];
snprintf(comm, 24, "hwlatd/%d", cpu);
kthread = kthread_create_on_cpu(kthread_fn, NULL, cpu, comm);
if (IS_ERR(kthread)) {
pr_err(BANNER "could not start sampling thread\n");
return -ENOMEM;
}
cpumask_clear(current_mask);
cpumask_set_cpu(next_cpu, current_mask);
sched_setaffinity(kthread->pid, current_mask);
hwlat_kthread = kthread;
per_cpu(hwlat_per_cpu_data, cpu).kthread = kthread;
wake_up_process(kthread);
return 0;
}
/**
* stop_kthread - Inform the hardware latency sampling/detector kthread to stop
*
* This kicks the running hardware latency sampling/detector kernel thread and
* tells it to stop sampling now. Use this on unload and at system shutdown.
*/
static void stop_kthread(void)
#ifdef CONFIG_HOTPLUG_CPU
static void hwlat_hotplug_workfn(struct work_struct *dummy)
{
if (!hwlat_kthread)
return;
kthread_stop(hwlat_kthread);
hwlat_kthread = NULL;
struct trace_array *tr = hwlat_trace;
unsigned int cpu = smp_processor_id();
mutex_lock(&trace_types_lock);
mutex_lock(&hwlat_data.lock);
get_online_cpus();
if (!hwlat_busy || hwlat_data.thread_mode != MODE_PER_CPU)
goto out_unlock;
if (!cpumask_test_cpu(cpu, tr->tracing_cpumask))
goto out_unlock;
start_cpu_kthread(cpu);
out_unlock:
put_online_cpus();
mutex_unlock(&hwlat_data.lock);
mutex_unlock(&trace_types_lock);
}
static DECLARE_WORK(hwlat_hotplug_work, hwlat_hotplug_workfn);
/*
* hwlat_cpu_init - CPU hotplug online callback function
*/
static int hwlat_cpu_init(unsigned int cpu)
{
schedule_work_on(cpu, &hwlat_hotplug_work);
return 0;
}
/*
* hwlat_read - Wrapper read function for reading both window and width
* @filp: The active open file structure
* @ubuf: The userspace provided buffer to read value into
* @cnt: The maximum number of bytes to read
* @ppos: The current "file" position
*
* This function provides a generic read implementation for the global state
* "hwlat_data" structure filesystem entries.
* hwlat_cpu_die - CPU hotplug offline callback function
*/
static ssize_t hwlat_read(struct file *filp, char __user *ubuf,
size_t cnt, loff_t *ppos)
static int hwlat_cpu_die(unsigned int cpu)
{
char buf[U64STR_SIZE];
u64 *entry = filp->private_data;
u64 val;
int len;
stop_cpu_kthread(cpu);
return 0;
}
if (!entry)
static void hwlat_init_hotplug_support(void)
{
int ret;
ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "trace/hwlat:online",
hwlat_cpu_init, hwlat_cpu_die);
if (ret < 0)
pr_warn(BANNER "Error to init cpu hotplug support\n");
return;
}
#else /* CONFIG_HOTPLUG_CPU */
static void hwlat_init_hotplug_support(void)
{
return;
}
#endif /* CONFIG_HOTPLUG_CPU */
/*
* start_per_cpu_kthreads - Kick off the hardware latency sampling/detector kthreads
*
* This starts the kernel threads that will sit on potentially all cpus and
* sample the CPU timestamp counter (TSC or similar) and look for potential
* hardware latencies.
*/
static int start_per_cpu_kthreads(struct trace_array *tr)
{
struct cpumask *current_mask = &save_cpumask;
unsigned int cpu;
int retval;
get_online_cpus();
/*
* Run only on CPUs in which hwlat is allowed to run.
*/
cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
for_each_online_cpu(cpu)
per_cpu(hwlat_per_cpu_data, cpu).kthread = NULL;
for_each_cpu(cpu, current_mask) {
retval = start_cpu_kthread(cpu);
if (retval)
goto out_error;
}
put_online_cpus();
return 0;
out_error:
put_online_cpus();
stop_per_cpu_kthreads();
return retval;
}
static void *s_mode_start(struct seq_file *s, loff_t *pos)
{
int mode = *pos;
mutex_lock(&hwlat_data.lock);
if (mode >= MODE_MAX)
return NULL;
return pos;
}
static void *s_mode_next(struct seq_file *s, void *v, loff_t *pos)
{
int mode = ++(*pos);
if (mode >= MODE_MAX)
return NULL;
return pos;
}
static int s_mode_show(struct seq_file *s, void *v)
{
loff_t *pos = v;
int mode = *pos;
if (mode == hwlat_data.thread_mode)
seq_printf(s, "[%s]", thread_mode_str[mode]);
else
seq_printf(s, "%s", thread_mode_str[mode]);
if (mode != MODE_MAX)
seq_puts(s, " ");
return 0;
}
static void s_mode_stop(struct seq_file *s, void *v)
{
seq_puts(s, "\n");
mutex_unlock(&hwlat_data.lock);
}
static const struct seq_operations thread_mode_seq_ops = {
.start = s_mode_start,
.next = s_mode_next,
.show = s_mode_show,
.stop = s_mode_stop
};
static int hwlat_mode_open(struct inode *inode, struct file *file)
{
return seq_open(file, &thread_mode_seq_ops);
};
static void hwlat_tracer_start(struct trace_array *tr);
static void hwlat_tracer_stop(struct trace_array *tr);
/**
* hwlat_mode_write - Write function for "mode" entry
* @filp: The active open file structure
* @ubuf: The user buffer that contains the value to write
* @cnt: The maximum number of bytes to write to "file"
* @ppos: The current position in @file
*
* This function provides a write implementation for the "mode" interface
* to the hardware latency detector. hwlatd has different operation modes.
* The "none" sets the allowed cpumask for a single hwlatd thread at the
* startup and lets the scheduler handle the migration. The default mode is
* the "round-robin" one, in which a single hwlatd thread runs, migrating
* among the allowed CPUs in a round-robin fashion. The "per-cpu" mode
* creates one hwlatd thread per allowed CPU.
*/
static ssize_t hwlat_mode_write(struct file *filp, const char __user *ubuf,
size_t cnt, loff_t *ppos)
{
struct trace_array *tr = hwlat_trace;
const char *mode;
char buf[64];
int ret, i;
if (cnt >= sizeof(buf))
return -EINVAL;
if (copy_from_user(buf, ubuf, cnt))
return -EFAULT;
if (cnt > sizeof(buf))
cnt = sizeof(buf);
buf[cnt] = 0;
val = *entry;
mode = strstrip(buf);
len = snprintf(buf, sizeof(buf), "%llu\n", val);
ret = -EINVAL;
return simple_read_from_buffer(ubuf, cnt, ppos, buf, len);
}
/**
* hwlat_width_write - Write function for "width" entry
* @filp: The active open file structure
* @ubuf: The user buffer that contains the value to write
* @cnt: The maximum number of bytes to write to "file"
* @ppos: The current position in @file
*
* This function provides a write implementation for the "width" interface
* to the hardware latency detector. It can be used to configure
* for how many us of the total window us we will actively sample for any
* hardware-induced latency periods. Obviously, it is not possible to
* sample constantly and have the system respond to a sample reader, or,
* worse, without having the system appear to have gone out to lunch. It
* is enforced that width is less that the total window size.
*/
static ssize_t
hwlat_width_write(struct file *filp, const char __user *ubuf,
size_t cnt, loff_t *ppos)
{
u64 val;
int err;
err = kstrtoull_from_user(ubuf, cnt, 10, &val);
if (err)
return err;
/*
* trace_types_lock is taken to avoid concurrency on start/stop
* and hwlat_busy.
*/
mutex_lock(&trace_types_lock);
if (hwlat_busy)
hwlat_tracer_stop(tr);
mutex_lock(&hwlat_data.lock);
if (val < hwlat_data.sample_window)
hwlat_data.sample_width = val;
else
err = -EINVAL;
for (i = 0; i < MODE_MAX; i++) {
if (strcmp(mode, thread_mode_str[i]) == 0) {
hwlat_data.thread_mode = i;
ret = cnt;
}
}
mutex_unlock(&hwlat_data.lock);
if (err)
return err;
if (hwlat_busy)
hwlat_tracer_start(tr);
mutex_unlock(&trace_types_lock);
return cnt;
*ppos += cnt;
return ret;
}
/**
* hwlat_window_write - Write function for "window" entry
* @filp: The active open file structure
* @ubuf: The user buffer that contains the value to write
* @cnt: The maximum number of bytes to write to "file"
* @ppos: The current position in @file
*
* This function provides a write implementation for the "window" interface
* to the hardware latency detector. The window is the total time
* in us that will be considered one sample period. Conceptually, windows
* occur back-to-back and contain a sample width period during which
* actual sampling occurs. Can be used to write a new total window size. It
* is enforced that any value written must be greater than the sample width
* size, or an error results.
/*
* The width parameter is read/write using the generic trace_min_max_param
* method. The *val is protected by the hwlat_data lock and is upper
* bounded by the window parameter.
*/
static ssize_t
hwlat_window_write(struct file *filp, const char __user *ubuf,
size_t cnt, loff_t *ppos)
{
u64 val;
int err;
err = kstrtoull_from_user(ubuf, cnt, 10, &val);
if (err)
return err;
mutex_lock(&hwlat_data.lock);
if (hwlat_data.sample_width < val)
hwlat_data.sample_window = val;
else
err = -EINVAL;
mutex_unlock(&hwlat_data.lock);
if (err)
return err;
return cnt;
}
static const struct file_operations width_fops = {
.open = tracing_open_generic,
.read = hwlat_read,
.write = hwlat_width_write,
static struct trace_min_max_param hwlat_width = {
.lock = &hwlat_data.lock,
.val = &hwlat_data.sample_width,
.max = &hwlat_data.sample_window,
.min = NULL,
};
static const struct file_operations window_fops = {
.open = tracing_open_generic,
.read = hwlat_read,
.write = hwlat_window_write,
/*
* The window parameter is read/write using the generic trace_min_max_param
* method. The *val is protected by the hwlat_data lock and is lower
* bounded by the width parameter.
*/
static struct trace_min_max_param hwlat_window = {
.lock = &hwlat_data.lock,
.val = &hwlat_data.sample_window,
.max = NULL,
.min = &hwlat_data.sample_width,
};
static const struct file_operations thread_mode_fops = {
.open = hwlat_mode_open,
.read = seq_read,
.llseek = seq_lseek,
.release = seq_release,
.write = hwlat_mode_write
};
/**
* init_tracefs - A function to initialize the tracefs interface files
*
@@ -546,18 +784,25 @@ static int init_tracefs(void)
hwlat_sample_window = tracefs_create_file("window", 0640,
top_dir,
&hwlat_data.sample_window,
&window_fops);
&hwlat_window,
&trace_min_max_fops);
if (!hwlat_sample_window)
goto err;
hwlat_sample_width = tracefs_create_file("width", 0644,
top_dir,
&hwlat_data.sample_width,
&width_fops);
&hwlat_width,
&trace_min_max_fops);
if (!hwlat_sample_width)
goto err;
hwlat_thread_mode = trace_create_file("mode", 0644,
top_dir,
NULL,
&thread_mode_fops);
if (!hwlat_thread_mode)
goto err;
return 0;
err:
@@ -569,18 +814,22 @@ static void hwlat_tracer_start(struct trace_array *tr)
{
int err;
err = start_kthread(tr);
if (hwlat_data.thread_mode == MODE_PER_CPU)
err = start_per_cpu_kthreads(tr);
else
err = start_single_kthread(tr);
if (err)
pr_err(BANNER "Cannot start hwlat kthread\n");
}
static void hwlat_tracer_stop(struct trace_array *tr)
{
stop_kthread();
if (hwlat_data.thread_mode == MODE_PER_CPU)
stop_per_cpu_kthreads();
else
stop_single_kthread();
}
static bool hwlat_busy;
static int hwlat_tracer_init(struct trace_array *tr)
{
/* Only allow one instance to enable this */
@@ -589,7 +838,6 @@ static int hwlat_tracer_init(struct trace_array *tr)
hwlat_trace = tr;
disable_migrate = false;
hwlat_data.count = 0;
tr->max_latency = 0;
save_tracing_thresh = tracing_thresh;
@@ -608,7 +856,7 @@ static int hwlat_tracer_init(struct trace_array *tr)
static void hwlat_tracer_reset(struct trace_array *tr)
{
stop_kthread();
hwlat_tracer_stop(tr);
/* the tracing threshold is static between runs */
last_tracing_thresh = tracing_thresh;
@@ -637,6 +885,8 @@ __init static int init_hwlat_tracer(void)
if (ret)
return ret;
hwlat_init_hotplug_support();
init_tracefs();
return 0;

2059
kernel/trace/trace_osnoise.c Normal file
View File

File diff suppressed because it is too large Load Diff

119
kernel/trace/trace_output.c
View File

@@ -1202,7 +1202,6 @@ trace_hwlat_print(struct trace_iterator *iter, int flags,
return trace_handle_return(s);
}
static enum print_line_t
trace_hwlat_raw(struct trace_iterator *iter, int flags,
struct trace_event *event)
@@ -1232,6 +1231,122 @@ static struct trace_event trace_hwlat_event = {
.funcs = &trace_hwlat_funcs,
};
/* TRACE_OSNOISE */
static enum print_line_t
trace_osnoise_print(struct trace_iterator *iter, int flags,
struct trace_event *event)
{
struct trace_entry *entry = iter->ent;
struct trace_seq *s = &iter->seq;
struct osnoise_entry *field;
u64 ratio, ratio_dec;
u64 net_runtime;
trace_assign_type(field, entry);
/*
* compute the available % of cpu time.
*/
net_runtime = field->runtime - field->noise;
ratio = net_runtime * 10000000;
do_div(ratio, field->runtime);
ratio_dec = do_div(ratio, 100000);
trace_seq_printf(s, "%llu %10llu %3llu.%05llu %7llu",
field->runtime,
field->noise,
ratio, ratio_dec,
field->max_sample);
trace_seq_printf(s, " %6u", field->hw_count);
trace_seq_printf(s, " %6u", field->nmi_count);
trace_seq_printf(s, " %6u", field->irq_count);
trace_seq_printf(s, " %6u", field->softirq_count);
trace_seq_printf(s, " %6u", field->thread_count);
trace_seq_putc(s, '\n');
return trace_handle_return(s);
}
static enum print_line_t
trace_osnoise_raw(struct trace_iterator *iter, int flags,
struct trace_event *event)
{
struct osnoise_entry *field;
struct trace_seq *s = &iter->seq;
trace_assign_type(field, iter->ent);
trace_seq_printf(s, "%lld %llu %llu %u %u %u %u %u\n",
field->runtime,
field->noise,
field->max_sample,
field->hw_count,
field->nmi_count,
field->irq_count,
field->softirq_count,
field->thread_count);
return trace_handle_return(s);
}
static struct trace_event_functions trace_osnoise_funcs = {
.trace = trace_osnoise_print,
.raw = trace_osnoise_raw,
};
static struct trace_event trace_osnoise_event = {
.type = TRACE_OSNOISE,
.funcs = &trace_osnoise_funcs,
};
/* TRACE_TIMERLAT */
static enum print_line_t
trace_timerlat_print(struct trace_iterator *iter, int flags,
struct trace_event *event)
{
struct trace_entry *entry = iter->ent;
struct trace_seq *s = &iter->seq;
struct timerlat_entry *field;
trace_assign_type(field, entry);
trace_seq_printf(s, "#%-5u context %6s timer_latency %9llu ns\n",
field->seqnum,
field->context ? "thread" : "irq",
field->timer_latency);
return trace_handle_return(s);
}
static enum print_line_t
trace_timerlat_raw(struct trace_iterator *iter, int flags,
struct trace_event *event)
{
struct timerlat_entry *field;
struct trace_seq *s = &iter->seq;
trace_assign_type(field, iter->ent);
trace_seq_printf(s, "%u %d %llu\n",
field->seqnum,
field->context,
field->timer_latency);
return trace_handle_return(s);
}
static struct trace_event_functions trace_timerlat_funcs = {
.trace = trace_timerlat_print,
.raw = trace_timerlat_raw,
};
static struct trace_event trace_timerlat_event = {
.type = TRACE_TIMERLAT,
.funcs = &trace_timerlat_funcs,
};
/* TRACE_BPUTS */
static enum print_line_t
trace_bputs_print(struct trace_iterator *iter, int flags,
@@ -1442,6 +1557,8 @@ static struct trace_event *events[] __initdata = {
&trace_bprint_event,
&trace_print_event,
&trace_hwlat_event,
&trace_osnoise_event,
&trace_timerlat_event,
&trace_raw_data_event,
&trace_func_repeats_event,
NULL

24
kernel/trace/trace_sched_wakeup.c
View File

@@ -26,9 +26,9 @@ static struct task_struct *wakeup_task;
static int wakeup_cpu;
static int wakeup_current_cpu;
static unsigned wakeup_prio = -1;
static int wakeup_rt;
static int wakeup_dl;
static int tracing_dl = 0;
static bool wakeup_rt;
static bool wakeup_dl;
static bool tracing_dl;
static arch_spinlock_t wakeup_lock =
(arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED;
@@ -498,7 +498,7 @@ static void __wakeup_reset(struct trace_array *tr)
{
wakeup_cpu = -1;
wakeup_prio = -1;
tracing_dl = 0;
tracing_dl = false;
if (wakeup_task)
put_task_struct(wakeup_task);
@@ -572,9 +572,9 @@ probe_wakeup(void *ignore, struct task_struct *p)
* another task until the first one wakes up.
*/
if (dl_task(p))
tracing_dl = 1;
tracing_dl = true;
else
tracing_dl = 0;
tracing_dl = false;
wakeup_task = get_task_struct(p);
@@ -685,8 +685,8 @@ static int wakeup_tracer_init(struct trace_array *tr)
if (wakeup_busy)
return -EBUSY;
wakeup_dl = 0;
wakeup_rt = 0;
wakeup_dl = false;
wakeup_rt = false;
return __wakeup_tracer_init(tr);
}
@@ -695,8 +695,8 @@ static int wakeup_rt_tracer_init(struct trace_array *tr)
if (wakeup_busy)
return -EBUSY;
wakeup_dl = 0;
wakeup_rt = 1;
wakeup_dl = false;
wakeup_rt = true;
return __wakeup_tracer_init(tr);
}
@@ -705,8 +705,8 @@ static int wakeup_dl_tracer_init(struct trace_array *tr)
if (wakeup_busy)
return -EBUSY;
wakeup_dl = 1;
wakeup_rt = 0;
wakeup_dl = true;
wakeup_rt = false;
return __wakeup_tracer_init(tr);
}

33
kernel/tracepoint.c
View File

@@ -273,7 +273,8 @@ static void tracepoint_update_call(struct tracepoint *tp, struct tracepoint_func
* Add the probe function to a tracepoint.
*/
static int tracepoint_add_func(struct tracepoint *tp,
struct tracepoint_func *func, int prio)
struct tracepoint_func *func, int prio,
bool warn)
{
struct tracepoint_func *old, *tp_funcs;
int ret;
@@ -288,7 +289,7 @@ static int tracepoint_add_func(struct tracepoint *tp,
lockdep_is_held(&tracepoints_mutex));
old = func_add(&tp_funcs, func, prio);
if (IS_ERR(old)) {
WARN_ON_ONCE(PTR_ERR(old) != -ENOMEM);
WARN_ON_ONCE(warn && PTR_ERR(old) != -ENOMEM);
return PTR_ERR(old);
}
@@ -343,6 +344,32 @@ static int tracepoint_remove_func(struct tracepoint *tp,
return 0;
}
/**
* tracepoint_probe_register_prio_may_exist - Connect a probe to a tracepoint with priority
* @tp: tracepoint
* @probe: probe handler
* @data: tracepoint data
* @prio: priority of this function over other registered functions
*
* Same as tracepoint_probe_register_prio() except that it will not warn
* if the tracepoint is already registered.
*/
int tracepoint_probe_register_prio_may_exist(struct tracepoint *tp, void *probe,
void *data, int prio)
{
struct tracepoint_func tp_func;
int ret;
mutex_lock(&tracepoints_mutex);
tp_func.func = probe;
tp_func.data = data;
tp_func.prio = prio;
ret = tracepoint_add_func(tp, &tp_func, prio, false);
mutex_unlock(&tracepoints_mutex);
return ret;
}
EXPORT_SYMBOL_GPL(tracepoint_probe_register_prio_may_exist);
/**
* tracepoint_probe_register_prio - Connect a probe to a tracepoint with priority
* @tp: tracepoint
@@ -366,7 +393,7 @@ int tracepoint_probe_register_prio(struct tracepoint *tp, void *probe,
tp_func.func = probe;
tp_func.data = data;
tp_func.prio = prio;
ret = tracepoint_add_func(tp, &tp_func, prio);
ret = tracepoint_add_func(tp, &tp_func, prio, true);
mutex_unlock(&tracepoints_mutex);
return ret;
}