Checking whether the ui and render threads are preempted
during frame drawing,then adjusting the prority and core
selection if they are preempted,because we expect the
threads to be executed first.By introducing the
sched_mode parameter, we can check the prev thread
preemption status in the hook, and judging important
threads for executing business logic.
Bug: 285166029
Change-Id: I6af31dff4c9032940c7f1c991a25a49ebbeac7a8
Signed-off-by: Dezhi Huang <huangdezhi@hihonor.com>
Now that the branch is used to create production GKI
images, need to institute ACK DrNo for all commits.
The DrNo approvers are in the android-mainline branch
at /OWNERS_DrNo.
Bug: 287162457
Signed-off-by: Matthias Maennich <maennich@google.com>
Change-Id: Id5bb83d7add5f314df6816c1c51b4bf2d8018e79
commit f7abf14f0001a5a47539d9f60bbdca649e43536b upstream.
For some unknown reason the introduction of the timer_wait_running callback
missed to fixup posix CPU timers, which went unnoticed for almost four years.
Marco reported recently that the WARN_ON() in timer_wait_running()
triggers with a posix CPU timer test case.
Posix CPU timers have two execution models for expiring timers depending on
CONFIG_POSIX_CPU_TIMERS_TASK_WORK:
1) If not enabled, the expiry happens in hard interrupt context so
spin waiting on the remote CPU is reasonably time bound.
Implement an empty stub function for that case.
2) If enabled, the expiry happens in task work before returning to user
space or guest mode. The expired timers are marked as firing and moved
from the timer queue to a local list head with sighand lock held. Once
the timers are moved, sighand lock is dropped and the expiry happens in
fully preemptible context. That means the expiring task can be scheduled
out, migrated, interrupted etc. So spin waiting on it is more than
suboptimal.
The timer wheel has a timer_wait_running() mechanism for RT, which uses
a per CPU timer-base expiry lock which is held by the expiry code and the
task waiting for the timer function to complete blocks on that lock.
This does not work in the same way for posix CPU timers as there is no
timer base and expiry for process wide timers can run on any task
belonging to that process, but the concept of waiting on an expiry lock
can be used too in a slightly different way:
- Add a mutex to struct posix_cputimers_work. This struct is per task
and used to schedule the expiry task work from the timer interrupt.
- Add a task_struct pointer to struct cpu_timer which is used to store
a the task which runs the expiry. That's filled in when the task
moves the expired timers to the local expiry list. That's not
affecting the size of the k_itimer union as there are bigger union
members already
- Let the task take the expiry mutex around the expiry function
- Let the waiter acquire a task reference with rcu_read_lock() held and
block on the expiry mutex
This avoids spin-waiting on a task which might not even be on a CPU and
works nicely for RT too.
Fixes: ec8f954a40 ("posix-timers: Use a callback for cancel synchronization on PREEMPT_RT")
Reported-by: Marco Elver <elver@google.com>
Change-Id: Ic069585c15bc968dec3c2b99cc70256f56a70b32
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Marco Elver <elver@google.com>
Tested-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/r/87zg764ojw.ffs@tglx
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit bccf9fe296)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
* keystone/mirror-android14-6.1: (154 commits)
UPSTREAM: maple_tree: make maple state reusable after mas_empty_area()
Revert "Revert "mm/mmap: regression fix for unmapped_area{_topdown}""
FROMLIST: scsi: ufs: ufs-mediatek: Set UFSHCD_QUIRK_MCQ_BROKEN_RTC quirk
FROMLIST: scsi: ufs: ufs-mediatek: Set UFSHCD_QUIRK_MCQ_BROKEN_INTR quirk
FROMLIST: scsi: ufs: core: Add host quirk UFSHCD_QUIRK_MCQ_BROKEN_RTC
FROMLIST: scsi: ufs: core: Add host quirk UFSHCD_QUIRK_MCQ_BROKEN_INTR
ANDROID: GKI: Add symbols and update symbol list for Unisoc
UPSTREAM: mailbox: mailbox-test: fix a locking issue in mbox_test_message_write()
UPSTREAM: mailbox: mailbox-test: Fix potential double-free in mbox_test_message_write()
ANDROID: dma-buf: support users to change dma_buf.name
ANDROID: set CONFIG_IKHEADERS=m for gki_defconfig.
ANDROID: mm: Avoid merging cma with others
ANDROID: cpufreq: times: record fast switch frequency transitions
ANDROID: fix kernelci build failure in vmscan.c
ANDROID: ABI: Update symbol list for Exynos SoC
ANDROID: gki_defconfig: enable NVME
ANDROID: ABI: Update symbols to unisoc whitelist for the scheduler
UPSTREAM: usb: gadget: uvc: queue empty isoc requests if no video buffer is available
ANDROID: GKI: Update symbol list for xiaomi
ANDROID: vendor_hooks:vendor hook for madvise_cold_or_pageout_pte_range.
...
Change-Id: Id90d7b744ff4e84934edfc8c276d775acddd9ab4
Signed-off-by: keystone-kernel-automerger <keystone-kernel-automerger@google.com>
Export cgroup_add_legacy_cftypes and a helper function to allow vendor module to expose additional files in the memory cgroup hierarchy.
Bug: 192052083
Signed-off-by: Liujie Xie <xieliujie@oppo.com>
(cherry picked from commit f41a95eadca98506e627b21f5cc73332bba4d95c)
(cherry picked from commit bf24c43b7f90290d2ac6f8163b43ab00f8f820b9)
Change-Id: Ie2b936b3e77c7ab6d740d1bb6d70e03c70a326a7
Signed-off-by: lvwenhuan <lvwenhuan@oppo.com>
With hooks below, we can mark a lock-owned thread with an identifiable flag, which can protect it from being preempted by some other unimportant threads, and then waiter will be wakeup more quickly.
https://android-review.googlesource.com/c/kernel/common/+/2183353
but now we find an issue like this one:
static inline void __up_write(struct rw_semaphore *sem)
{
...
// Step 1. we clear flag.
trace_android_vh_record_rwsem_lock_starttime(current, 0);
// Step 2. owner may be preempted by unimportant threads.
rwsem_clear_owner(sem);
...
// Step 3. wake up waiter, but it's too later.
if (unlikely(tmp & RWSEM_FLAG_WAITERS))
rwsem_wake(sem);
}
This patch will clear protect-flag after waking up waiters.
Bug: 286024926
Change-Id: I71f8b6a7d8a01336fd36b8267c2cb5edab65bd11
Signed-off-by: xieliujie <xieliujie@oppo.com>
Add ANDROID_OEM_DATA to struct rq, which is used to implement oem's
scheduler tuning.
Bug: 188899490
Change-Id: I1904b4fd83effc4b309bfb98811e9718398504f4
Signed-off-by: Liangliang Li <liliangliang@vivo.com>
(cherry picked from commit 8fdd8e4bb5e0fc1af6ec2eec73280c2ffe569a37)
Add encryption support to bootloader based hibernation. This will
encrypt the hibernation snapshot image before it is written to the swap
partition.
Bug: 279879797
Change-Id: I07046ad7fb848fc62258871ab41b3e03246c40dc
Signed-off-by: Shreyas K K <quic_shrekk@quicinc.com>
Add vendor hooks to disable randomization of swap slot allocation for
swap partition used for saving hibernation image. Another level of
randomization of swap slots takes place at the firmware level as well
in order to address the wear leveling for UFS/MMC devices, so this
vendor hook checks if a block device represents the swap partition being
used for saving hibernation image, if yes, the swap slot allocation for
such partition is serialized at kernel level.
There is a performance advantage of reading contiguous pages of hibernation
image, it makes the restore logic of hibernation image simpler and faster
as there are no seeks involved in the secondary storage to read multiple
contiguous pages of the image.
Bug: 279879797
Change-Id: I8258b5166d8c6952fe9eb91a5a9826f33b836f00
Signed-off-by: Vivek Kumar <quic_vivekuma@quicinc.com>
Signed-off-by: Shreyas K K <quic_shrekk@quicinc.com>
To add encryption support to bootloader based hibernation, export
symbols snapshot_get_image_size and alloc_swapdev_block.
These symbols can be used by vendor implementation to be called before
and after storing the snapshot image.
Bug: 279879797
Change-Id: I0d44bf833a97fce5bc5213712b2b2523a9e22607
Signed-off-by: Shreyas K K <quic_shrekk@quicinc.com>
vma->lock being part of the vm_area_struct causes performance regression
during page faults because during contention its count and owner fields
are constantly updated and having other parts of vm_area_struct used
during page fault handling next to them causes constant cache line
bouncing. Fix that by moving the lock outside of the vm_area_struct.
All attempts to keep vma->lock inside vm_area_struct in a separate cache
line still produce performance regression especially on NUMA machines.
Smallest regression was achieved when lock is placed in the fourth cache
line but that bloats vm_area_struct to 256 bytes.
Considering performance and memory impact, separate lock looks like the
best option. It increases memory footprint of each VMA but that can be
optimized later if the new size causes issues. Note that after this
change vma_init() does not allocate or initialize vma->lock anymore. A
number of drivers allocate a pseudo VMA on the stack but they never use
the VMA's lock, therefore it does not need to be allocated. The future
drivers which might need the VMA lock should use
vm_area_alloc()/vm_area_free() to allocate the VMA.
Link: https://lkml.kernel.org/r/20230227173632.3292573-34-surenb@google.com
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit c7f8f31c00d187a2c71a241c7f2bd6aa102a4e6f)
Bug: 161210518
Change-Id: I0c4dff518b74e1a9ea2b40636b436a122841564d
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
call_rcu() can take a long time when callback offloading is enabled. Its
use in the vm_area_free can cause regressions in the exit path when
multiple VMAs are being freed.
Because exit_mmap() is called only after the last mm user drops its
refcount, the page fault handlers can't be racing with it. Any other
possible user like oom-reaper or process_mrelease are already synchronized
using mmap_lock. Therefore exit_mmap() can free VMAs directly, without
the use of call_rcu().
Expose __vm_area_free() and use it from exit_mmap() to avoid possible
call_rcu() floods and performance regressions caused by it.
Link: https://lkml.kernel.org/r/20230227173632.3292573-33-surenb@google.com
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 0d2ebf9c3f7822e7ba3e4792ea3b6b19aa2da34a)
Bug: 161210518
Change-Id: I4fbf3ef38fdb22a3c80dcc61125ec21d2c426100
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Assert there are no holders of VMA lock for reading when it is about to be
destroyed.
Link: https://lkml.kernel.org/r/20230227173632.3292573-21-surenb@google.com
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit f2e13784c16a98e269b3111ac02ae44446dd589c)
Bug: 161210518
Change-Id: I67fc318fd08575c9bcc54fd00b0304246b0cd279
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Introduce per-VMA locking. The lock implementation relies on a per-vma
and per-mm sequence counters to note exclusive locking:
- read lock - (implemented by vma_start_read) requires the vma
(vm_lock_seq) and mm (mm_lock_seq) sequence counters to differ.
If they match then there must be a vma exclusive lock held somewhere.
- read unlock - (implemented by vma_end_read) is a trivial vma->lock
unlock.
- write lock - (vma_start_write) requires the mmap_lock to be held
exclusively and the current mm counter is assigned to the vma counter.
This will allow multiple vmas to be locked under a single mmap_lock
write lock (e.g. during vma merging). The vma counter is modified
under exclusive vma lock.
- write unlock - (vma_end_write_all) is a batch release of all vma
locks held. It doesn't pair with a specific vma_start_write! It is
done before exclusive mmap_lock is released by incrementing mm
sequence counter (mm_lock_seq).
- write downgrade - if the mmap_lock is downgraded to the read lock, all
vma write locks are released as well (effectivelly same as write
unlock).
Link: https://lkml.kernel.org/r/20230227173632.3292573-13-surenb@google.com
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 5e31275cc997f8ec5d9e8d65fe9840ebed89db19)
Bug: 161210518
Change-Id: I5e0db53a4b5562e59dd031fabbae4f97acc1bce1
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
This prepares for page faults handling under VMA lock, looking up VMAs
under protection of an rcu read lock, instead of the usual mmap read lock.
Link: https://lkml.kernel.org/r/20230227173632.3292573-11-surenb@google.com
Signed-off-by: Michel Lespinasse <michel@lespinasse.org>
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 20cce633f4254cc0df39665449726e3172518f6c)
Bug: 161210518
Change-Id: Ic1262d00de9dc84ed2cc506b3bae7ef714efdfaa
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Replace direct modifications to vma->vm_flags with calls to modifier
functions to be able to track flag changes and to keep vma locking
correctness.
[akpm@linux-foundation.org: fix drivers/misc/open-dice.c, per Hyeonggon Yoo]
Link: https://lkml.kernel.org/r/20230126193752.297968-5-surenb@google.com
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Mike Rapoport (IBM) <rppt@kernel.org>
Acked-by: Sebastian Reichel <sebastian.reichel@collabora.com>
Reviewed-by: Liam R. Howlett <Liam.Howlett@Oracle.com>
Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Arjun Roy <arjunroy@google.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: David Rientjes <rientjes@google.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Greg Thelen <gthelen@google.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jann Horn <jannh@google.com>
Cc: Joel Fernandes <joelaf@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: Laurent Dufour <ldufour@linux.ibm.com>
Cc: Lorenzo Stoakes <lstoakes@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Minchan Kim <minchan@google.com>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Peter Oskolkov <posk@google.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Punit Agrawal <punit.agrawal@bytedance.com>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Shakeel Butt <shakeelb@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: Song Liu <songliubraving@fb.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 1c71222e5f2393b5ea1a41795c67589eea7e3490)
Bug: 161210518
Change-Id: Ifc352b487db109adab17dd33a83f5c7e68c0bbc6
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
To simplify the usage of VM_LOCKED_CLEAR_MASK in vm_flags_clear(), replace
it with VM_LOCKED_MASK bitmask and convert all users.
Link: https://lkml.kernel.org/r/20230126193752.297968-4-surenb@google.com
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Mike Rapoport (IBM) <rppt@kernel.org>
Reviewed-by: Davidlohr Bueso <dave@stgolabs.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Arjun Roy <arjunroy@google.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: David Howells <dhowells@redhat.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Greg Thelen <gthelen@google.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jann Horn <jannh@google.com>
Cc: Joel Fernandes <joelaf@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: Laurent Dufour <ldufour@linux.ibm.com>
Cc: Liam R. Howlett <Liam.Howlett@Oracle.com>
Cc: Lorenzo Stoakes <lstoakes@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Minchan Kim <minchan@google.com>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Peter Oskolkov <posk@google.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Punit Agrawal <punit.agrawal@bytedance.com>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Sebastian Reichel <sebastian.reichel@collabora.com>
Cc: Shakeel Butt <shakeelb@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: Song Liu <songliubraving@fb.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit e430a95a04efc557bc4ff9b3035c7c85aee5d63f)
Bug: 161210518
Change-Id: I17bbcc01a133511dbfaf3d82fbc4b25ecdd0b376
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Patch series "introduce vm_flags modifier functions", v4.
This patchset was originally published as a part of per-VMA locking [1]
and was split after suggestion that it's viable on its own and to
facilitate the review process. It is now a preprequisite for the next
version of per-VMA lock patchset, which reuses vm_flags modifier functions
to lock the VMA when vm_flags are being updated.
VMA vm_flags modifications are usually done under exclusive mmap_lock
protection because this attrubute affects other decisions like VMA merging
or splitting and races should be prevented. Introduce vm_flags modifier
functions to enforce correct locking.
This patch (of 7):
Convert vma assignment in vm_area_dup() to a memcpy() to prevent compiler
errors when we add a const modifier to vma->vm_flags.
Link: https://lkml.kernel.org/r/20230126193752.297968-1-surenb@google.com
Link: https://lkml.kernel.org/r/20230126193752.297968-2-surenb@google.com
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Mike Rapoport (IBM) <rppt@kernel.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Arjun Roy <arjunroy@google.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: David Rientjes <rientjes@google.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Greg Thelen <gthelen@google.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jann Horn <jannh@google.com>
Cc: Joel Fernandes <joelaf@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: Laurent Dufour <ldufour@linux.ibm.com>
Cc: Liam R. Howlett <Liam.Howlett@Oracle.com>
Cc: Lorenzo Stoakes <lstoakes@gmail.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Minchan Kim <minchan@google.com>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Peter Oskolkov <posk@google.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Punit Agrawal <punit.agrawal@bytedance.com>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Shakeel Butt <shakeelb@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: Song Liu <songliubraving@fb.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Will Deacon <will@kernel.org>
Cc: Sebastian Reichel <sebastian.reichel@collabora.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
(cherry picked from commit 06e78b614e3780f9ac32056f2861159fd19d9702)
Bug: 161210518
Change-Id: I8f0b92d5ed602d087316e99b29f59d75a644d0e4
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Try to mitigate potential future driver core api changes by adding
padding to a number of core internal scheduler structures.
Bug: 151154716
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I0ef2f8dd5f3259dcf443c5045aa1e8505ed78a76
commit: da07d2f9c153e457e845d4dcfdd13568d71d18a4 upstream.
Traversing the Perf Domains requires rcu_read_lock() to be held and is
conditional on sched_energy_enabled(). Ensure right protections applied.
Also skip capacity inversion detection for our own pd; which was an
error.
Fixes: 44c7b80bffc3 ("sched/fair: Detect capacity inversion")
Reported-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Change-Id: I1c81232e94a38a68e39cf73f8893f185e268928d
Signed-off-by: Qais Yousef (Google) <qyousef@layalina.io>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
Link: https://lore.kernel.org/r/20230112122708.330667-3-qyousef@layalina.io
(cherry picked from commit da07d2f9c153e457e845d4dcfdd13568d71d18a4)
Signed-off-by: Qais Yousef (Google) <qyousef@layalina.io>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit d362a03d92)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
commit: aa69c36f31aadc1669bfa8a3de6a47b5e6c98ee8 upstream.
We do consider thermal pressure in util_fits_cpu() for uclamp_min only.
With the exception of the biggest cores which by definition are the max
performance point of the system and all tasks by definition should fit.
Even under thermal pressure, the capacity of the biggest CPU is the
highest in the system and should still fit every task. Except when it
reaches capacity inversion point, then this is no longer true.
We can handle this by using the inverted capacity as capacity_orig in
util_fits_cpu(). Which not only addresses the problem above, but also
ensure uclamp_max now considers the inverted capacity. Force fitting
a task when a CPU is in this adverse state will contribute to making the
thermal throttling last longer.
Change-Id: I3c1a4758eb387c5c8e0ba68751840517fff4ae64
Signed-off-by: Qais Yousef <qais.yousef@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20220804143609.515789-10-qais.yousef@arm.com
(cherry picked from commit aa69c36f31aadc1669bfa8a3de6a47b5e6c98ee8)
Signed-off-by: Qais Yousef (Google) <qyousef@layalina.io>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit 799c7301de)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
commit: 44c7b80bffc3a657a36857098d5d9c49d94e652b upstream.
Check each performance domain to see if thermal pressure is causing its
capacity to be lower than another performance domain.
We assume that each performance domain has CPUs with the same
capacities, which is similar to an assumption made in energy_model.c
We also assume that thermal pressure impacts all CPUs in a performance
domain equally.
If there're multiple performance domains with the same capacity_orig, we
will trigger a capacity inversion if the domain is under thermal
pressure.
The new cpu_in_capacity_inversion() should help users to know when
information about capacity_orig are not reliable and can opt in to use
the inverted capacity as the 'actual' capacity_orig.
Change-Id: I23a98594e6b24697eb33d9bbfb8dd46a12bfa050
Signed-off-by: Qais Yousef <qais.yousef@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lore.kernel.org/r/20220804143609.515789-9-qais.yousef@arm.com
(cherry picked from commit 44c7b80bffc3a657a36857098d5d9c49d94e652b)
Signed-off-by: Qais Yousef (Google) <qyousef@layalina.io>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
(cherry picked from commit fe1c982958)
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
* keystone/mirror-android14-6.1: (49 commits)
ANDROID: ABI: update symbol list related to meminfo for galaxy
ANDROID: vendor_hooks: add hooks for extra memory
BACKPORT: FROMLIST: ovl: get_acl: Fix null pointer dereference at realinode in rcu-walk mode
BACKPORT: FROMLIST: ovl: ovl_permission: Fix null pointer dereference at realinode in rcu-walk mode
BACKPORT: FROMLIST: ovl: Let helper ovl_i_path_real() return the realinode
ANDROID: GKI: Update symbol list for xiaomi
ANDROID: vendor_hooks:vendor hook for control memory dirty rate
ANDROID: f2fs: enable cleancache
FROMGIT: userfaultfd: fix regression in userfaultfd_unmap_prep()
FROMGIT: BACKPORT: mm: avoid rewalk in mmap_region
FROMGIT: maple_tree: clear up index and last setting in single entry tree
FROMGIT: maple_tree: add mas_prev_range() and mas_find_range_rev interface
FROMGIT: maple_tree: introduce mas_prev_slot() interface
FROMGIT: maple_tree: relocate mas_rewalk() and mas_rewalk_if_dead()
FROMGIT: maple_tree: add mas_next_range() and mas_find_range() interfaces
FROMGIT: maple_tree: introduce mas_next_slot() interface
FROMGIT: maple_tree: try harder to keep active node with mas_prev()
FROMGIT: maple_tree: try harder to keep active node after mas_next()
FROMGIT: BACKPORT: mm/mmap: change do_vmi_align_munmap() for maple tree iterator changes
FROMLIST: BACKPORT: maple_tree: Refine mas_preallocate() node calculations
...
Change-Id: I83b39b4475c96e1bc80eb237846d625924aad6c4
Signed-off-by: keystone-kernel-automerger <keystone-kernel-automerger@google.com>
Haibo Li reported:
| Unable to handle kernel paging request at virtual address
| ffffff802a0d8d7171
| Mem abort info⭕
| ESR = 0x9600002121
| EC = 0x25: DABT (current EL), IL = 32 bitsts
| SET = 0, FnV = 0 0
| EA = 0, S1PTW = 0 0
| FSC = 0x21: alignment fault
| Data abort info⭕
| ISV = 0, ISS = 0x0000002121
| CM = 0, WnR = 0 0
| swapper pgtable: 4k pages, 39-bit VAs, pgdp=000000002835200000
| [ffffff802a0d8d71] pgd=180000005fbf9003, p4d=180000005fbf9003,
| pud=180000005fbf9003, pmd=180000005fbe8003, pte=006800002a0d8707
| Internal error: Oops: 96000021 [#1] PREEMPT SMP
| Modules linked in:
| CPU: 2 PID: 45 Comm: kworker/u8:2 Not tainted
| 5.15.78-android13-8-g63561175bbda-dirty #1
| ...
| pc : kcsan_setup_watchpoint+0x26c/0x6bc
| lr : kcsan_setup_watchpoint+0x88/0x6bc
| sp : ffffffc00ab4b7f0
| x29: ffffffc00ab4b800 x28: ffffff80294fe588 x27: 0000000000000001
| x26: 0000000000000019 x25: 0000000000000001 x24: ffffff80294fdb80
| x23: 0000000000000000 x22: ffffffc00a70fb68 x21: ffffff802a0d8d71
| x20: 0000000000000002 x19: 0000000000000000 x18: ffffffc00a9bd060
| x17: 0000000000000001 x16: 0000000000000000 x15: ffffffc00a59f000
| x14: 0000000000000001 x13: 0000000000000000 x12: ffffffc00a70faa0
| x11: 00000000aaaaaaab x10: 0000000000000054 x9 : ffffffc00839adf8
| x8 : ffffffc009b4cf00 x7 : 0000000000000000 x6 : 0000000000000007
| x5 : 0000000000000000 x4 : 0000000000000000 x3 : ffffffc00a70fb70
| x2 : 0005ff802a0d8d71 x1 : 0000000000000000 x0 : 0000000000000000
| Call trace:
| kcsan_setup_watchpoint+0x26c/0x6bc
| __tsan_read2+0x1f0/0x234
| inflate_fast+0x498/0x750
| zlib_inflate+0x1304/0x2384
| __gunzip+0x3a0/0x45c
| gunzip+0x20/0x30
| unpack_to_rootfs+0x2a8/0x3fc
| do_populate_rootfs+0xe8/0x11c
| async_run_entry_fn+0x58/0x1bc
| process_one_work+0x3ec/0x738
| worker_thread+0x4c4/0x838
| kthread+0x20c/0x258
| ret_from_fork+0x10/0x20
| Code: b8bfc2a8 2a0803f7 14000007 d503249f (78bfc2a8) )
| ---[ end trace 613a943cb0a572b6 ]-----
The reason for this is that on certain arm64 configuration since
e35123d83e ("arm64: lto: Strengthen READ_ONCE() to acquire when
CONFIG_LTO=y"), READ_ONCE() may be promoted to a full atomic acquire
instruction which cannot be used on unaligned addresses.
Fix it by avoiding READ_ONCE() in read_instrumented_memory(), and simply
forcing the compiler to do the required access by casting to the
appropriate volatile type. In terms of generated code this currently
only affects architectures that do not use the default READ_ONCE()
implementation.
The only downside is that we are not guaranteed atomicity of the access
itself, although on most architectures a plain load up to machine word
size should still be atomic (a fact the default READ_ONCE() still relies
on itself).
BUG: 285794521
(cherry picked from commit 8dec88070d964bfeb4198f34cb5956d89dd1f557)
Reported-by: Haibo Li <haibo.li@mediatek.com>
Tested-by: Haibo Li <haibo.li@mediatek.com>
Cc: <stable@vger.kernel.org> # 5.17+
Change-Id: I16c9f83c3b4e28021a936249cafb1501760aa59d
Signed-off-by: Marco Elver <elver@google.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: 杨辉 <yanghui10@xiaomi.corp-partner.google.com>
* keystone/mirror-android14-6.1: (21 commits)
ANDROID: Add vendor hook to the effective_cpu_util
ANDROID: Update symbol list for mtk
ANDROID: ABI: update symbol list for galaxy
ANDROID: update symbol for unisoc vendor_hooks
ANDROID: thermal: Add vendor hook to check power range
ANDROID: thermal: Add vendor hook to get thermal zone device
ANDROID: thermal: Add hook for cpufreq thermal
UPSTREAM: mm,kfence: decouple kfence from page granularity mapping judgement
UPSTREAM: mm/page_alloc: fix potential deadlock on zonelist_update_seq seqlock
UPSTREAM: mm: page_alloc: skip regions with hugetlbfs pages when allocating 1G pages
UPSTREAM: usb: gadget: f_fs: Add unbind event before functionfs_unbind
UPSTREAM: mm: kmsan: handle alloc failures in kmsan_vmap_pages_range_noflush()
ANDROID: ABI: Update oplus symbol list
Revert "FROMGIT: usb: gadget: udc: core: Prevent redundant calls to pullup"
ANDROID: GKI: Update symbols to symbol list
ANDROID: vendor_hooks: Add hooks for binder proc transaction
UPSTREAM: mm: kmsan: handle alloc failures in kmsan_ioremap_page_range()
UPSTREAM: mm/huge_memory.c: warn with pr_warn_ratelimited instead of VM_WARN_ON_ONCE_FOLIO
UPSTREAM: mm/userfaultfd: fix uffd-wp handling for THP migration entries
UPSTREAM: mm/khugepaged: check again on anon uffd-wp during isolation
...
Change-Id: Ic294e740b17e6b1cda1767c3bfe29d15a3c9b8db
Signed-off-by: keystone-kernel-automerger <keystone-kernel-automerger@google.com>
android_rvh_effective_cpu_util:
To perform vendor-specific cpu util, it is used in EAS/schedutil/thermal.
The effective_cpu_util would be called when thermal calc the dynamic power,
it's non-atomic context, so set the hook be restricted.
Bug: 226686099
Test: build pass
Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
Change-Id: I6fd77f44ca4328f5ef37d96989aa2e08d65e29bb
* keystone/mirror-android14-6.1:
ANDROID: GKI: update xiaomi symbol list
BACKPORT: FROMGIT: Multi-gen LRU: fix workingset accounting
BACKPORT: FROMGIT: PCI: dwc: Wait for link up only if link is started
ANDROID: ABI: update symbol list for exynos
ANDROID: ufs: Improve MCQ err handling
ANDROID: ABI: Update oplus symbol list
ANDROID: GKI: sched: add rvh for new cfs task util
Change-Id: I9282031a1663da4db78e778f2657dc4eb400b116
Signed-off-by: keystone-kernel-automerger <keystone-kernel-automerger@google.com>
A vendor hook is added in post_init_entity_util_avg before
a new cfs task's util is attached to cfs_rq's util so that
vendors can gather and modify se's information to modify
scheduling behavior and DVFS as they want.
trace_android_rvh_new_task_stats is not a proper hook because
it is called after the task's util is attached to cfs_rq's util,
which means updating cfs_rq's sched_avg and DVFS request are done.
Bug: 184219858
Signed-off-by: Choonghoon Park <choong.park@samsung.com>
Change-Id: I2deaa93297f8464895978496c9838cdffaa35b7f
(cherry picked from commit 1eea1cbdd3)
* keystone/mirror-android14-6.1:
ANDROID: Disable BTI_KERNEL, enable UNWIND_PATCH_PAC_INTO_SCS
BACKPORT: arm64: implement dynamic shadow call stack for Clang
UPSTREAM: scs: add support for dynamic shadow call stacks
BACKPORT: arm64: unwind: add asynchronous unwind tables to kernel and modules
ANDROID: ABI: Update oplus symbol list
ANDROID: vendor_hooks: Add hook in shrink_node_memcgs
ANDROID: ashmem: Export is_ashmem_file
ANDROID: ABI: update symbol list related to show_mem for Exynos
ANDROID: vendor_hooks: Add hooks for memory when debug
ANDROID: GKI: Update symbol list for honor
ANDROID: mutex: Add vendor hook to init mutex oem data.
ANDROID: ABI: Update oplus symbol list
ANDROID: vendor_hook: add hooks in dm_bufio.c
Change-Id: Iabd85e02421856407ef797b2408adce83fc447af
Signed-off-by: keystone-kernel-automerger <keystone-kernel-automerger@google.com>
In order to allow arches to use code patching to conditionally emit the
shadow stack pushes and pops, rather than always taking the performance
hit even on CPUs that implement alternatives such as stack pointer
authentication on arm64, add a Kconfig symbol that can be set by the
arch to omit the SCS codegen itself, without otherwise affecting how
support code for SCS and compiler options (for register reservation, for
instance) are emitted.
Also, add a static key and some plumbing to omit the allocation of
shadow call stack for dynamic SCS configurations if SCS is disabled at
runtime.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
Tested-by: Sami Tolvanen <samitolvanen@google.com>
Link: https://lore.kernel.org/r/20221027155908.1940624-3-ardb@kernel.org
Signed-off-by: Will Deacon <will@kernel.org>
(cherry picked from commit 9beccca0984022a844850e32f0d7dd80d4a225de)
Bug: 283954062
Change-Id: I71ed23533124b071bd6bf5ab91b2af3bbf03b42b
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Add vendors hooks for recording memory used
Vendor modules allocate and manages the memory itself.
These memories might not be included in kernel memory
statistics. Also, detailed references and vendor-specific
information are managed only inside modules. When
various problems such as memory leaks occurs, these
information should be showed in real-time.
Bug: 182443489
Bug: 234407991
Bug: 277799025
Signed-off-by: Liujie Xie <xieliujie@oppo.com>
Change-Id: I62d8bb2b6650d8b187b433f97eb833ef0b784df1
Signed-off-by: Hyesoo Yu <hyesoo.yu@samsung.com>
Add hooks to init oem data of mutex.
Bug: 231527236
Signed-off-by: Dezhi Huang <huangdezhi@hihonor.com>
Change-Id: Id0aeac168e81bd3d88051657c32ba709c329dbdd
* keystone/mirror-android14-6.1: (37 commits)
ANDROID: GKI: Update symbol list for honor
ANDROID: mm: create vendor hooks for page alloc
ANDROID: ABI: Update oplus symbol list
ANDROID: vendor_hooks: Add hooks for cpufreq_acct_update_power
ANDROID: ABI: Update symbol list for Exynos SoC
ANDROID: ABI: Update oplus symbol list
ANDROID: vendor_hooks: Add hooks for account irqtime process tick
ANDROID: vendor_hooks: Add hooks to dup_task_struct
ANDROID: vendor_hooks: Add hooks to record the time of the process in various states
ANDROID: vendor_hooks: Add hooks for signal
ANDROID: power: wakeup_reason: change abort log
ANDROID: GKI: Update symbol list for xiaomi
ANDROID: psi: Add vendor hooks for PSI tracing
UPSTREAM: ext4: fix invalid free tracking in ext4_xattr_move_to_block()
FROMGIT: scsi: ufs: core: mcq: Fix &hwq->cq_lock deadlock issue
ANDROID: ABI: Update oplus symbol list
ANDROID: android: Export symbols for invoking cpufreq_update_util()
ANDROID: ABI: Update oplus symbol list
ANDROID: vendor_hooks: Export the tracepoints task_rename
ANDROID: Add macros to create reserved data fields to backport upstream changes
...
Change-Id: I5fac4bd1a75e6a202e52b2cfbe1e5c45d1aafd17
Signed-off-by: keystone-kernel-automerger <keystone-kernel-automerger@google.com>
Add a hook in irqtime_account_process_tick, which helps to get
information about the high load task.
Bug: 187904818
Change-Id: I644f7d66b09d047ca6b0a0fbd2915a6387c8c007
Signed-off-by: Liangliang Li <liliangliang@vivo.com>
(cherry picked from commit fe580539f6cec43ddb0d6ecfd39aa2f4e45754ca)
(cherry picked from commit bf3b384e0876a3111d114392b405ed269dcb9c9e)
Add hook to dup_task_struct for vendor data fields initialisation.
Bug: 188004638
Change-Id: I4b58604ee822fb8d1e0cc37bec72e820e7318427
Signed-off-by: Liangliang Li <liliangliang@vivo.com>
(cherry picked from commit f66d96b14aab5051fdf6b5054d87362c17a7b365)
(cherry picked from commit bafafe0ec46160573bef46d3d0f5d6c65fadaa3b)
These hooks will do the following works:
a) record the time of the process in various states
b) Make corresponding optimization strategies in different hooks
Bug: 205938967
Signed-off-by: Liujie Xie <xieliujie@oppo.com>
Change-Id: Ia3c47bbf0aadd17337ce18fd910343b1b8c3ef93
(cherry picked from commit a61d61bab7c3d221d5d2250eacab2e4e9f59b252)
Signed-off-by: Carlos Llamas <cmllamas@google.com>
(cherry picked from commit b7a1174cc5dfc68cf769547b91958df407e55981)
(cherry picked from commit e1f430a48702844c01933a925c92b077ba14ec3f)
Add hook to boost thread when process killed.
Bug: 237749933
Signed-off-by: xieliujie <xieliujie@oppo.com>
Change-Id: I7cc6f248397021f3a8271433144a0e582ed27cfa
(cherry picked from commit 709679142d583b0b7338d931fdd43b27b1bbf9e0)
(cherry picked from commit 226f36edddc3ad523dbb901ca3d5683f5f4e1707)
Add hooks to capture various per-zone memory stats when
a trigger threshold is hit.
Bug: 268290366
Change-Id: Ibe39263ddb05ffc3fa63b5225497a90c6480c8d7
Signed-off-by: Georgi Djakov <quic_c_gdjako@quicinc.com>
In order to update cpufreq, vendor modules invoke cpufreq_update_util(),
but when we build our modules, report error:
ERROR: modpost: "cpufreq_update_util_data" [xxx.ko] undefined!
Bug: 192218676
Signed-off-by: Liujie Xie <xieliujie@oppo.com>
Change-Id: Ib1da70229f04b08d8d812d065021dec0bf891e0e
(cherry picked from commit 8943a2e7a33e33fd89614ac83b33b30f8d8c6b96)
(cherry picked from commit d1bc61dd85e8d28f1df0e22e2d73aa9e99cb645e)
Export the tracepoint task_rename to identify specific new task，
to customize task's util for power and performance, or optimize
task schedule parameters.
Bug: 189985971
Change-Id: I3bb71eae316e3096d361e7b47012ba46ea4be509
Signed-off-by: Liujie Xie <xieliujie@oppo.com>
(cherry picked from commit 016d3f7b6986d0ca0acef18c59a47cf6eaa4f562)
(cherry picked from commit cfc14a391adb4d44d8186694b4884815bd85be6c)
Add a hook in account_process_tick, which help us to get information
about the high load task and the cpu they running on.
Bug: 183260319
Change-Id: I54162ce3c65bd69e08d2d4747e4d4883efe4c442
Signed-off-by: Liujie Xie <xieliujie@oppo.com>
this hook will allow to account tick in every third of more ticks
to save cpu time for accounting
Bug: 279549765
Change-Id: I5d18e0167fdce076d13aecc653dcf6387bcb25f2
Signed-off-by: Liujie Xie <xieliujie@oppo.com>
* keystone/mirror-android14-6.1: (63 commits)
ANDROID: GKI: update the ABI symbol list
ANDROID: freezer: Add vendor hook to freezer for GKI purpose.
ANDROID: freezer: export the freezer_cgrp_subsys for GKI purpose.
ANDROID: GKI: update the ABI symbol list
ANDROID: Add vendor hooks for binder perf tuning
ANDROID: Add vendor hooks to signal.
ANDROID: Update the ABI symbol list
ANDROID: page_pinner: add missing page_pinner_put_page
ANDROID: page_pinner: prevent pp_buffer uninitialized access
ANDROID: page_pinner: prevent pp_buffer access before initialization
ANDROID: mm: fix use-after free of page_ext in page_pinner
ANDROID: mm: introduce page_pinner
ANDROID: abi_gki_aarch64_qcom: Add gh_rm_register_platform_ops
ANDROID: gunyah: Sync remaining gunyah drivers with latest
ANDROID: gunyah: Sync with latest "mailbox: Add Gunyah message queue mailbox"
ANDROID: gunyah: Sync with latest "gunyah: Common types and error codes for Gunyah hypercalls"
ANDROID: gunyah: Sync with latest hypercalls
ANDROID: gunyah: Sync with latest documentation and UAPI
ANDROID: gunyah: Sync with latest "firmware: qcom_scm: Register Gunyah platform ops"
BACKPORT: firmware: qcom_scm: Use fixed width src vm bitmap
...
Change-Id: I8c82b8ddecaa7f30ff816b51ef08003059a97a6c
Signed-off-by: keystone-kernel-automerger <keystone-kernel-automerger@google.com>
Add the vendor hook to freezer.c so that OEM's logic can be executed
when the process is about to be frozen. We need to clear the flag for
some tasks and rebind task dependencies for optimization purposes.
Bug: 187458531
Bug: 281920779
Signed-off-by: heshuai1 <heshuai1@xiaomi.com>
Change-Id: Iea42fd9604d6b33ccd6502425416f0dd28eecebb
(cherry picked from commit a1580311c36ca28344b2f03b3c8a72d9f8db5bde)
Exporting the symbol freezer_cgrp_subsys, in that vendor module can
add can_attach & cancel_attach member function. It is vendor-specific
tuning.
Bug: 182496370
Bug: 281920779
Signed-off-by: Zhuguangqing <zhuguangqing@xiaomi.com>
Change-Id: I153682b9d1015eed3f048b45ea6495ebb8f3c261
(cherry picked from commit ee3f4d2821f5b2a794f0a1f5ed423f561a01adae)
(cherry picked from commit 8a90e4d4e555dd5484213c6fec5061958016a194)
This hook will allow us to get signal messages so that we can set
limitations for certain tasks and restore them when receiving important signals.
Bug: 184898838
Bug: 281920779
Signed-off-by: Zhuguangqing <zhuguangqing@xiaomi.com>
Change-Id: I83a28b0a6eb413976f4c57f2314d008ad792fa0d
(cherry picked from commit 58e3f869fc3fe84fb7062496ccd049db47f3ed7f)
