Jens Axboe 428f138268 io_uring/napi: ensure napi polling is aborted when work is available ...

While testing io_uring NAPI with DEFER_TASKRUN, I ran into slowdowns and
stalls in packet delivery. Turns out that while
io_napi_busy_loop_should_end() aborts appropriately on regular
task_work, it does not abort if we have local task_work pending.

Move io_has_work() into the private io_uring.h header, and gate whether
we should continue polling on that as well. This makes NAPI polling on
send/receive work as designed with IORING_SETUP_DEFER_TASKRUN as well.

Fixes: 8d0c12a80c ("io-uring: add napi busy poll support")
Signed-off-by: Jens Axboe <axboe@kernel.dk>

2024-02-14 13:01:25 -07:00

..

advise.c

io_uring: always go async for unsupported fadvise flags

2023-01-29 15:18:26 -07:00

advise.h

…

alloc_cache.h

io_uring: use mempool KASAN hook

2023-12-29 11:58:41 -08:00

cancel.c

io_uring/cancel: don't default to setting req->work.cancel_seq

2024-02-08 13:27:06 -07:00

cancel.h

io_uring/cancel: don't default to setting req->work.cancel_seq

2024-02-08 13:27:06 -07:00

epoll.c

io_uring: undeprecate epoll_ctl support

2023-05-26 20:22:41 -06:00

epoll.h

…

fdinfo.c

io_uring/fdinfo: remove need for sqpoll lock for thread/pid retrieval

2023-11-15 06:35:46 -07:00

fdinfo.h

…

filetable.c

io_uring: drop any code related to SCM_RIGHTS

2023-12-19 12:36:34 -07:00

filetable.h

io_uring: expand main struct io_kiocb flags to 64-bits

2024-02-08 13:27:03 -07:00

fs.c

io_uring/fs: consider link->flags when getting path for LINKAT

2023-11-20 09:01:42 -07:00

fs.h

…

futex.c

io_uring: add support for vectored futex waits

2023-09-29 02:37:08 -06:00

futex.h

io_uring: add support for vectored futex waits

2023-09-29 02:37:08 -06:00

io_uring.c

io_uring/napi: ensure napi polling is aborted when work is available

2024-02-14 13:01:25 -07:00

io_uring.h

io_uring/napi: ensure napi polling is aborted when work is available

2024-02-14 13:01:25 -07:00

io-wq.c

io-wq: fully initialize wqe before calling cpuhp_state_add_instance_nocalls()

2023-10-05 14:11:18 -06:00

io-wq.h

io_uring: break out of iowq iopoll on teardown

2023-09-07 09:02:27 -06:00

kbuf.c

io_uring/kbuf: cleanup passing back cflags

2024-02-08 13:27:06 -07:00

kbuf.h

io_uring/kbuf: cleanup passing back cflags

2024-02-08 13:27:06 -07:00

Makefile

io-uring: add napi busy poll support

2024-02-09 11:54:19 -07:00

msg_ring.c

io_uring: use io_file_from_index in io_msg_grab_file

2023-06-20 09:36:22 -06:00

msg_ring.h

io_uring: get rid of double locking

2022-12-07 06:47:13 -07:00

napi.c

io_uring/napi: ensure napi polling is aborted when work is available

2024-02-14 13:01:25 -07:00

napi.h

io_uring: add register/unregister napi function

2024-02-09 11:54:32 -07:00

net.c

io_uring/net: fix sr->len for IORING_OP_RECV with MSG_WAITALL and buffers

2024-02-01 06:42:36 -07:00

net.h

io_uring: Add KASAN support for alloc_caches

2023-04-03 07:16:14 -06:00

nop.c

…

nop.h

…

notif.c

io_uring/notif: add constant for ubuf_info flags

2023-04-15 14:21:04 -06:00

notif.h

io_uring/notif: add constant for ubuf_info flags

2023-04-15 14:21:04 -06:00

opdef.c

io_uring: add support for ftruncate

2024-02-09 09:04:39 -07:00

opdef.h

io_uring/rw: mark readv/writev as vectored in the opcode definition

2023-09-21 12:00:46 -06:00

openclose.c

io_uring: enable audit and restrict cred override for IORING_OP_FIXED_FD_INSTALL

2024-01-23 15:25:14 -07:00

openclose.h

io_uring/openclose: add support for IORING_OP_FIXED_FD_INSTALL

2023-12-12 07:42:57 -07:00

poll.c

io-uring: add napi busy poll support

2024-02-09 11:54:19 -07:00

poll.h

io_uring/rw: ensure poll based multishot read retries appropriately

2024-01-28 20:37:11 -07:00

refs.h

…

register.c

io_uring: add register/unregister napi function

2024-02-09 11:54:32 -07:00

register.h

io_uring/register: move io_uring_register(2) related code to register.c

2023-12-19 08:54:20 -07:00

rsrc.c

io_uring: drop any code related to SCM_RIGHTS

2023-12-19 12:36:34 -07:00

rsrc.h

io_uring: Don't include af_unix.h.

2024-02-12 19:02:11 -07:00

rw.c

io_uring: Don't include af_unix.h.

2024-02-12 19:02:11 -07:00

rw.h

io_uring/rw: add separate prep handler for fixed read/write

2023-11-06 07:43:16 -07:00

slist.h

io_uring: silence variable ‘prev’ set but not used warning

2023-03-09 10:10:58 -07:00

splice.c

splice: return type ssize_t from all helpers

2023-12-12 16:19:59 +01:00

splice.h

…

sqpoll.c

io-uring: add sqpoll support for napi busy poll

2024-02-09 11:54:28 -07:00

sqpoll.h

io_uring/sqpoll: fix io-wq affinity when IORING_SETUP_SQPOLL is used

2023-08-16 13:40:28 -06:00

statx.c

io_uring: for requests that require async, force it

2023-01-29 15:18:26 -07:00

statx.h

…

sync.c

io_uring: for requests that require async, force it

2023-01-29 15:18:26 -07:00

sync.h

…

tctx.c

io_uring: Add io_uring_setup flag to pre-register ring fd and never install it

2023-05-16 08:06:00 -06:00

tctx.h

io_uring: simplify __io_uring_add_tctx_node

2022-10-07 12:25:30 -06:00

timeout.c

io_uring: never overflow io_aux_cqe

2023-08-11 10:42:57 -06:00

timeout.h

io_uring: remove unused return from io_disarm_next

2022-09-21 13:15:01 -06:00

truncate.c

io_uring: add support for ftruncate

2024-02-09 09:04:39 -07:00

truncate.h

io_uring: add support for ftruncate

2024-02-09 09:04:39 -07:00

uring_cmd.c

io_uring: Don't include af_unix.h.

2024-02-12 19:02:11 -07:00

uring_cmd.h

io_uring: Remove unnecessary BUILD_BUG_ON

2023-05-04 08:19:05 -06:00

waitid.c

io_uring: add IORING_OP_WAITID support

2023-09-21 12:04:45 -06:00

waitid.h

io_uring: add IORING_OP_WAITID support

2023-09-21 12:04:45 -06:00

xattr.c

io_uring: use file_mnt_idmap helper

2024-02-06 19:55:14 -07:00

xattr.h

…