mirror
/
linux
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
linux/include/net
History
Ilias Apalodimas 6a5bcd84e8 page_pool: Allow drivers to hint on SKB recycling 
Up to now several high speed NICs have custom mechanisms of recycling
the allocated memory they use for their payloads.
Our page_pool API already has recycling capabilities that are always
used when we are running in 'XDP mode'. So let's tweak the API and the
kernel network stack slightly and allow the recycling to happen even
during the standard operation.
The API doesn't take into account 'split page' policies used by those
drivers currently, but can be extended once we have users for that.

The idea is to be able to intercept the packet on skb_release_data().
If it's a buffer coming from our page_pool API recycle it back to the
pool for further usage or just release the packet entirely.

To achieve that we introduce a bit in struct sk_buff (pp_recycle:1) and
a field in struct page (page->pp) to store the page_pool pointer.
Storing the information in page->pp allows us to recycle both SKBs and
their fragments.
We could have skipped the skb bit entirely, since identical information
can bederived from struct page. However, in an effort to affect the free path
as less as possible, reading a single bit in the skb which is already
in cache, is better that trying to derive identical information for the
page stored data.

The driver or page_pool has to take care of the sync operations on it's own
during the buffer recycling since the buffer is, after opting-in to the
recycling, never unmapped.

Since the gain on the drivers depends on the architecture, we are not
enabling recycling by default if the page_pool API is used on a driver.
In order to enable recycling the driver must call skb_mark_for_recycle()
to store the information we need for recycling in page->pp and
enabling the recycling bit, or page_pool_store_mem_info() for a fragment.

Co-developed-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Co-developed-by: Matteo Croce <mcroce@microsoft.com>
Signed-off-by: Matteo Croce <mcroce@microsoft.com>
Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
 		2021-06-07 14:11:47 -07:00
..
9p 9p: apply review requests for fid refcounting 2020-11-19 17:21:34 +01:00
bluetooth Bluetooth: Allow Microsoft extension to indicate curve validation 2021-04-08 12:26:34 +02:00
caif net: caif: add proper error handling 2021-06-03 15:05:06 -07:00
iucv net/af_iucv: don't track individual TX skbs for TRANS_HIPER sockets 2021-01-28 20:36:21 -08:00
netfilter Merge ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net 2021-06-07 13:01:52 -07:00
netns ipv6: Add a sysctl to control multipath hash fields 2021-05-18 13:27:32 -07:00
nfc NFC: nci: fix memory leak in nci_allocate_device 2021-05-17 13:56:29 -07:00
phonet treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 336 2019-06-05 17:37:07 +02:00
sctp sctp: do asoc update earlier in sctp_sf_do_dupcook_b 2021-04-30 15:06:34 -07:00
tc_act net/sched: act_vlan: Fix modify to allow 0 2021-06-01 16:54:42 -07:00
6lowpan.h 6lowpan: Replace zero-length array with flexible-array member 2020-02-28 14:51:30 +01:00
Space.h
…
act_api.h net: sched: fix err handler in tcf_action_init() 2021-04-08 13:47:33 -07:00
addrconf.h net: bridge: mcast: fix broken length + header check for MRDv6 Adv. 2021-04-27 14:02:06 -07:00
af_ieee802154.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 174 2019-05-30 11:26:41 -07:00
af_rxrpc.h afs: Don't truncate iter during data fetch 2021-04-23 10:17:26 +01:00
af_unix.h unix: uses an atomic type for scm files accounting 2020-02-28 12:12:53 -08:00
af_vsock.h vsock: add local transport support in the vsock core 2019-12-11 15:01:23 -08:00
ah.h
…
arp.h net: avoid potential false sharing in neighbor related code 2019-11-06 16:14:48 -08:00
atmclip.h
…
ax25.h ax25: fix possible use-after-free 2019-01-23 11:18:00 -08:00
ax88796.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500 2019-06-19 17:09:55 +02:00
bareudp.h bareudp: Reverted support to enable & disable rx metadata collection 2020-07-21 18:30:47 -07:00
bond_3ad.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 90 2019-05-24 17:37:53 +02:00
bond_alb.h bonding/alb: Add helper functions to get the xmit slave 2020-05-01 12:15:37 -07:00
bond_options.h bonding: add an option to specify a delay between peer notifications 2019-07-04 12:30:48 -07:00
bonding.h net/bonding: Declare TLS RX device offload support 2021-01-18 20:48:40 -08:00
bpf_sk_storage.h bpf: struct sock is declared twice in bpf_sk_storage header 2021-03-26 17:43:55 +01:00
busy_poll.h net, xdp, xsk: fix __sk_mark_napi_id_once napi_id error 2020-12-01 15:51:19 +01:00
calipso.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 13 2019-05-21 11:28:45 +02:00
cfg80211-wext.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500 2019-06-19 17:09:55 +02:00
cfg80211.h mac80211: properly handle A-MSDUs that start with an RFC 1042 header 2021-05-11 20:13:03 +02:00
cfg802154.h cfg802154: Replace zero-length array with flexible-array member 2020-02-29 14:39:08 +01:00
checksum.h saner calling conventions for csum_and_copy_..._user() 2020-08-20 15:45:15 -04:00
cipso_ipv4.h cipso: Remove unused inline functions 2020-07-15 07:45:24 -07:00
cls_cgroup.h bpf: Allow to retrieve cgroup v1 classid from v2 hooks 2020-03-27 19:40:38 -07:00
codel.h
…
codel_impl.h
…
codel_qdisc.h
…
compat.h compat: always include linux/compat.h from net/compat.h 2020-11-23 13:31:54 -08:00
datalink.h
…
dcbevent.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 201 2019-05-30 11:29:52 -07:00
dcbnl.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 201 2019-05-30 11:29:52 -07:00
devlink.h devlink: Allow setting parent node of rate objects 2021-06-02 14:08:37 -07:00
dn.h
…
dn_dev.h
…
dn_fib.h net: dn_fib: Replace zero-length array with flexible-array member 2020-02-29 21:52:20 -08:00
dn_neigh.h
…
dn_nsp.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 24 2019-05-21 11:52:39 +02:00
dn_route.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 24 2019-05-21 11:52:39 +02:00
dsa.h net: dsa: free skb->cb usage in core driver 2021-04-27 14:10:15 -07:00
dsfield.h ipv6: Annotate bitwise IPv6 dsfield pointer cast 2019-12-16 16:09:44 -08:00
dst.h net: Consolidate common blackhole dst ops 2021-03-10 12:24:18 -08:00
dst_cache.h
…
dst_metadata.h
…
dst_ops.h net/dst: use a smaller percpu_counter batch for dst entries accounting 2020-05-08 21:33:33 -07:00
erspan.h erspan: Add type I version 0 support. 2020-05-05 13:23:29 -07:00
esp.h ESP: Export esp_output_fill_trailer function 2020-02-19 13:52:32 +01:00
espintcp.h xfrm: espintcp: save and call old ->sk_destruct 2020-04-20 07:34:16 +02:00
ethoc.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500 2019-06-19 17:09:55 +02:00
failover.h
…
fib_notifier.h ipv6: Remove old route notifications and convert listeners 2019-12-24 22:37:30 -08:00
fib_rules.h fib: use indirect call wrappers in the most common fib_rules_ops 2020-07-28 17:42:31 -07:00
firewire.h
…
flow.h flow: remove spi key from flowi struct 2021-04-19 12:25:11 +02:00
flow_dissector.h flow_dissector: constify raw input data argument 2021-03-14 14:46:32 -07:00
flow_offload.h net: flow_offload: add FLOW_ACTION_PPPOE_PUSH 2021-03-24 12:48:39 -07:00
fou.h
…
fq.h net/fq_impl: do not maintain a backlog-sorted list of flows 2021-01-21 13:33:45 +01:00
fq_impl.h net/fq_impl: do not maintain a backlog-sorted list of flows 2021-01-21 13:33:45 +01:00
garp.h treewide: Use sizeof_field() macro 2019-12-09 10:36:44 -08:00
gen_stats.h net_sched: extend packet counter to 64bit 2019-11-05 18:20:55 -08:00
genetlink.h mptcp: avoid lock_fast usage in accept path 2021-02-12 16:31:46 -08:00
geneve.h net: Move the definition of the default Geneve udp port to public header file 2019-03-22 12:09:31 -07:00
gre.h ip_gre: add csum offload support for gre header 2021-01-29 20:39:14 -08:00
gro.h gro: add combined call_gro_receive() + INDIRECT_CALL_INET() helper 2021-03-18 19:51:12 -07:00
gro_cells.h
…
gtp.h
…
gue.h GUE: Fix a typo 2020-06-22 21:12:44 -07:00
hwbm.h net: hwbm: if CONFIG_NET_HWBM unset, make stub functions static 2019-10-25 16:24:32 -07:00
icmp.h net: icmp: pass zeroed opts from icmp{,v6}_ndo_send before sending 2021-02-23 11:29:52 -08:00
ieee80211_radiotap.h mac80211: add radiotap flag to assure frames are not reordered 2020-11-06 11:01:01 +01:00
ieee802154_netdev.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 174 2019-05-30 11:26:41 -07:00
if_inet6.h mld: add mc_lock for protecting per-interface mld data 2021-03-26 15:14:56 -07:00
ife.h net: ife: drop include of module.h from net/ife.h 2019-04-22 21:50:53 -07:00
ila.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
inet6_connection_sock.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
inet6_hashtables.h net: Track socket refcounts in skb_steal_sock() 2020-03-30 13:45:04 -07:00
inet_common.h bpf: Allow rewriting to ports under ip_unprivileged_port_start 2021-01-27 18:18:15 -08:00
inet_connection_sock.h tcp: relookup sock for RST+ACK packets handled by obsolete req sock 2021-03-15 14:34:29 -07:00
inet_ecn.h inet_ecn: Use csum16_add() helper for IP_ECN_set_* helpers 2020-12-14 18:38:58 -08:00
inet_frag.h inet: frags: batch fqdir destroy works 2020-12-12 15:08:54 -08:00
inet_hashtables.h tcp: fix race condition when creating child sockets from syncookies 2020-11-23 16:32:33 -08:00
inet_sock.h inet: remove inet_sk_copy_descendant() 2020-08-26 07:33:19 -07:00
inet_timewait_sock.h tcp: honor SO_PRIORITY in TIME_WAIT state 2019-09-27 12:05:02 +02:00
inetpeer.h net: ipv4: use a dedicated counter for icmp_v4 redirect packets 2019-02-08 21:50:15 -08:00
ip.h inet: constify inet_sdif() argument 2020-11-10 17:56:54 -08:00
ip6_checksum.h tcp: remove indirect calls for icsk->icsk_af_ops->send_check 2020-06-20 17:47:53 -07:00
ip6_fib.h IPv6: Add "offload failed" indication to routes 2021-02-08 16:47:03 -08:00
ip6_route.h net: allow user to set metric on default route learned via Router Advertisement 2021-01-26 18:39:45 -08:00
ip6_tunnel.h ip6_tunnel: allow not to count pkts on tstats by passing dev as NULL 2019-06-18 20:48:45 -04:00
ip_fib.h ipv4: Add a sysctl to control multipath hash fields 2021-05-18 13:27:32 -07:00
ip_tunnels.h Merge https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-11-19 19:08:46 -08:00
ip_vs.h netfilter: move handlers to net/ip_vs.h 2021-02-04 18:37:57 -08:00
ipcomp.h
…
ipconfig.h
…
ipv6.h ipv6: Add a sysctl to control multipath hash fields 2021-05-18 13:27:32 -07:00
ipv6_frag.h ipv6: Remove dependency of ipv6_frag_thdr_truncated on ipv6 module 2020-11-19 10:49:50 -08:00
ipv6_stubs.h ipv6: add ipv6_dev_find to stubs 2021-03-30 13:29:39 -07:00
ipx.h bonding/alb: properly access headers in bond_alb_xmit() 2020-02-05 14:28:09 +01:00
iw_handler.h
…
kcm.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500 2019-06-19 17:09:55 +02:00
l3mdev.h l3mdev: add infrastructure for table to VRF mapping 2020-06-20 17:22:22 -07:00
lag.h
…
lapb.h net: lapb: Make "lapb_t1timer_running" able to detect an already running timer 2021-03-23 14:14:50 -07:00
lib80211.h
…
llc.h
…
llc_c_ac.h
…
llc_c_ev.h
…
llc_c_st.h
…
llc_conn.h llc: fix sk_buff leak in llc_conn_service() 2019-10-08 13:23:05 -07:00
llc_if.h
…
llc_pdu.h
…
llc_s_ac.h
…
llc_s_ev.h
…
llc_s_st.h
…
llc_sap.h
…
lwtunnel.h net: add net available in build_state 2020-03-29 22:30:57 -07:00
mac80211.h mac80211: make ieee80211_vif_to_wdev work when the vif isn't in the driver 2021-04-19 12:03:13 +02:00
mac802154.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 174 2019-05-30 11:26:41 -07:00
macsec.h net: macsec: add support for getting offloaded stats 2020-03-26 20:17:36 -07:00
mip6.h net: mip6: Replace zero-length array with flexible-array member 2020-03-02 11:16:27 -08:00
mld.h mld: add new workqueues for process mld events 2021-03-26 15:14:56 -07:00
mpls.h net: Make mpls_entry_encode() available for generic users 2020-05-29 21:20:20 -07:00
mpls_iptunnel.h net: mpls: Replace zero-length array with flexible-array member 2020-02-28 12:08:37 -08:00
mptcp.h mptcp: use mptcp_addr_info in mptcp_out_options 2021-04-07 14:09:39 -07:00
mrp.h treewide: Use sizeof_field() macro 2019-12-09 10:36:44 -08:00
ncsi.h
…
ndisc.h ipv6: ndisc: adjust ndisc_ifinfo_sysctl_change prototype 2020-08-24 06:40:07 -07:00
neighbour.h net: Exempt multicast addresses from five-second neighbor lifetime 2020-11-13 14:24:39 -08:00
net_failover.h
…
net_namespace.h net: dccp: use net_generic storage 2021-04-09 16:34:56 -07:00
net_ratelimit.h
…
netevent.h
…
netlabel.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 13 2019-05-21 11:28:45 +02:00
netlink.h treewide: rename nla_strlcpy to nla_strscpy. 2020-11-16 08:08:54 -08:00
netprio_cgroup.h netprio: use css ID instead of cgroup ID 2019-11-12 08:18:03 -08:00
netrom.h net: netrom: Fix error cleanup path of nr_proto_init 2019-04-11 13:59:49 -07:00
nexthop.h nexthop: Rename artifacts related to legacy multipath nexthop groups 2021-03-28 17:53:39 -07:00
nl802154.h
…
nsh.h
…
p8022.h
…
page_pool.h page_pool: Allow drivers to hint on SKB recycling 2021-06-07 14:11:47 -07:00
pie.h pie: realign comment 2020-03-04 13:25:55 -08:00
ping.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
pkt_cls.h net: zero-initialize tc skb extension on allocation 2021-05-25 15:36:42 -07:00
pkt_sched.h net: sched: fix tx action rescheduling issue during deactivation 2021-05-14 15:05:46 -07:00
pptp.h
…
protocol.h net: Remove the member netns_ok 2021-05-17 15:29:35 -07:00
psample.h psample: Add additional metadata attributes 2021-03-14 15:00:43 -07:00
psnap.h
…
raw.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
rawv6.h
…
red.h sch_red: fix off-by-one checks in red_check_params() 2021-03-25 17:40:43 -07:00
regulatory.h net/wireless: regulatory.h: drop duplicate word in comment 2020-07-31 09:24:23 +02:00
request_sock.h tcp: bpf: Optionally store mac header in TCP_SAVE_SYN 2020-08-24 14:35:00 -07:00
rose.h
…
route.h lsm,selinux: pass flowi_common instead of flowi to the LSM hooks 2020-11-23 18:36:21 -05:00
rpl.h net: ipv6: Use struct_size() helper and kcalloc() 2020-06-23 20:27:09 -07:00
rsi_91x.h
…
rtnetlink.h ipv6: report errors for iftoken via netlink extack 2021-04-08 13:52:36 -07:00
rtnh.h net: Rename net/nexthop.h net/rtnh.h 2019-04-22 21:47:25 -07:00
sch_generic.h net: sched: fix packet stuck problem for lockless qdisc 2021-05-14 15:05:46 -07:00
scm.h fs: Move __scm_install_fd() to __receive_fd() 2020-07-13 11:03:44 -07:00
secure_seq.h
…
seg6.h seg6: fix seg6_validate_srh() to avoid slab-out-of-bounds 2020-06-04 15:39:32 -07:00
seg6_hmac.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
seg6_local.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
selftests.h net: selftest: fix build issue if INET is disabled 2021-04-28 14:06:45 -07:00
slhc_vj.h
…
smc.h net/smc: introduce CHID callback for ISM devices 2020-09-28 15:19:03 -07:00
snmp.h net/tls: add skeleton of MIB statistics 2019-10-05 16:29:00 -07:00
sock.h sock: expose so_timestamping options for mptcp 2021-06-04 14:08:09 -07:00
sock_reuseport.h net: sock_reuseport: Replace zero-length array with flexible-array member 2020-02-29 21:52:19 -08:00
stp.h
…
strparser.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500 2019-06-19 17:09:55 +02:00
switchdev.h net: bridge: switchdev: include local flag in FDB notifications 2021-04-16 15:15:45 -07:00
tcp.h tcp: export timestamp helpers for mptcp 2021-06-04 14:08:09 -07:00
tcp_states.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
timewait_sock.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152 2019-05-30 11:26:32 -07:00
tipc.h
…
tls.h net/tls: Fix use-after-free after the TLS device goes down and up 2021-06-01 15:58:05 -07:00
tls_toe.h net/tls: rename tls_hw_* functions tls_toe_* 2019-10-04 14:07:07 -07:00
transp_v6.h tcp: move ipv4_specific to tcp include file 2020-06-23 20:10:15 -07:00
tso.h net: tso: cache transport header length 2020-06-18 20:46:23 -07:00
tun_proto.h
…
udp.h skmsg: Pass psock pointer to ->psock_update_sk_prot() 2021-04-12 17:34:27 +02:00
udp_tunnel.h udp: call udp_encap_enable for v6 sockets when enabling encap 2021-02-04 18:37:14 -08:00
udplite.h
…
vsock_addr.h vsock: remove include/linux/vm_sockets.h file 2019-11-14 18:12:17 -08:00
vxlan.h net: sched: only keep the available bits when setting vxlan md->gbp 2020-09-14 16:49:39 -07:00
wext.h
…
x25.h net/x25: add new state X25_STATE_5 2019-12-09 10:28:43 -08:00
x25device.h
…
xdp.h net, veth: Alloc skb in bulk for ndo_xdp_xmit 2021-02-04 01:00:07 +01:00
xdp_priv.h page_pool: do not release pool until inflight == 0. 2019-11-16 12:39:10 -08:00
xdp_sock.h bpf, xdp: Make bpf_redirect_map() a map operation 2021-03-10 01:06:34 +01:00
xdp_sock_drv.h xsk: Introduce batched Tx descriptor interfaces 2020-11-17 22:07:40 +01:00
xfrm.h xfrm: Fix NULL pointer dereference on policy lookup 2021-03-24 10:00:24 +01:00
xsk_buff_pool.h xsk: Fix race in SKB mode transmit with shared cq 2020-12-18 16:10:21 +01:00