The function opencodes cpumask_nth(). The dedicated helper is faster
than an open for-loop.
Signed-off-by: Yury Norov [NVIDIA] <yury.norov@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Change the logic in the restore function to properly handle bank
exceptions.
The check for exceptions in the saved state should be performed before
conducting any other ringstat register checks.
If a bank was saved with an exception, the ringstat will have the
appropriate rp_halt/rp_exception bits set, causing the driver to exit
the restore process with an error. Instead, the restore routine should
first check the ringexpstat register, and if any exception was raised,
it should stop further checks and return without any error. In other
words, if a ring pair is in an exception state at the source, it should
be restored the same way at the destination but without raising an error.
Even though this approach might lead to losing the exception state
during migration, the driver will log the exception from the saved state
during the restore process.
Signed-off-by: Svyatoslav Pankratov <svyatoslav.pankratov@intel.com>
Fixes: bbfdde7d19 ("crypto: qat - add bank save and restore flows")
Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The commit ca88a2bdd4 ("crypto: qat - allow disabling SR-IOV VFs")
introduced an unnecessary change that prevented enabling SR-IOV when
IOMMU is disabled. In certain scenarios, it is desirable to enable
SR-IOV even in the absence of IOMMU. Thus, restoring the previous
functionality to allow VFs to be enumerated in the absence of IOMMU.
Fixes: ca88a2bdd4 ("crypto: qat - allow disabling SR-IOV VFs")
Signed-off-by: Ahsan Atta <ahsan.atta@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Reviewed-by: Michal Witwicki <michal.witwicki@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Cryptographic Accelerator Unit (CPT) support different engine groups, one
for asymmetric algorithms (only AE engines in this group), one for the
most common symmetric algorithms (all SE and all IE engines in this group),
and one for other symmetric algorithms (only SE engines in this group).
For symmetric engine (SE), we obtain the group number using
"MBOX_MSG_GET_ENG_GRP_NUM" mailbox. Let's follow a similar approach to
determine the group number for asymmetric engine (AE).
Signed-off-by: Amit Singh Tomar <amitsinght@marvell.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
By default, otx2_cpt_get_kcrypto_eng_grp_num() returns the engine group
number of SE engine type. Add an engine type parameter to support
retrieving the engine group number for different engine types.
Since otx2_cpt_get_kcrypto_eng_grp_num() always returns the kernel crypto
engine group number, rename it to otx2_cpt_get_eng_grp_num().
Signed-off-by: Amit Singh Tomar <amitsinght@marvell.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
There is a race condition/UAF in padata_reorder that goes back
to the initial commit. A reference count is taken at the start
of the process in padata_do_parallel, and released at the end in
padata_serial_worker.
This reference count is (and only is) required for padata_replace
to function correctly. If padata_replace is never called then
there is no issue.
In the function padata_reorder which serves as the core of padata,
as soon as padata is added to queue->serial.list, and the associated
spin lock released, that padata may be processed and the reference
count on pd would go away.
Fix this by getting the next padata before the squeue->serial lock
is released.
In order to make this possible, simplify padata_reorder by only
calling it once the next padata arrives.
Fixes: 16295bec63 ("padata: Generic parallelization/serialization interface")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The sha3 partial hash on s390 is in little-endian just like the
final hash. However the generic implementation produces native
or big-endian partial hashes.
Make s390 sha3 conform to that by doing the byte-swap on export
and import.
Reported-by: Ingo Franzki <ifranzki@linux.ibm.com>
Fixes: 6f90ba7065 ("crypto: s390/sha3 - Use API partial block handling")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The hmac export state needs to be one block-size bigger to account
for the ipad.
Reported-by: Ingo Franzki <ifranzki@linux.ibm.com>
Fixes: 08811169ac ("crypto: s390/hmac - Use API partial block handling")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
If an error occurs during queueing the engine load will never be
decremented. Fix this by moving the engine load adjustment into
the cleanup function.
Fixes: bf8f91e711 ("crypto: marvell - Add load balancing between engines")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
octeontx2 crypto driver allocates memory using kmalloc/kzalloc,
and uses this memory for dma (does dma_map_single()). It assumes
that kmalloc/kzalloc will return 128-byte aligned address. But
kmalloc/kzalloc returns 8-byte aligned address after below changes:
"9382bc44b5f5 arm64: allow kmalloc() caches aligned to the
smaller cache_line_size()
Memory allocated are used for following purpose:
- Input data or scatter list address - 8-Byte alignment
- Output data or gather list address - 8-Byte alignment
- Completion address - 32-Byte alignment.
This patch ensures all addresses are aligned as mentioned above.
Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com>
Cc: <stable@vger.kernel.org> # v6.8+
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
octeontx2 crypto driver allocates memory using kmalloc/kzalloc,
and uses this memory for dma (does dma_map_single()). It assumes
that kmalloc/kzalloc will return 128-byte aligned address. But
kmalloc/kzalloc returns 8-byte aligned address after below changes:
"9382bc44b5f5 arm64: allow kmalloc() caches aligned to the
smaller cache_line_size()
Memory allocated are used for following purpose:
- Input data or scatter list address - 8-Byte alignment
- Output data or gather list address - 8-Byte alignment
- Completion address - 32-Byte alignment.
This patch ensures all addresses are aligned as mentioned above.
Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com>
Cc: <stable@vger.kernel.org> # v6.5+
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
octeontx2 crypto driver allocates memory using kmalloc/kzalloc,
and uses this memory for dma (does dma_map_single()). It assumes
that kmalloc/kzalloc will return 128-byte aligned address. But
kmalloc/kzalloc returns 8-byte aligned address after below changes:
"9382bc44b5f5 arm64: allow kmalloc() caches aligned to the
smaller cache_line_size()"
Completion address should be 32-Byte alignment when loading
microcode.
Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com>
Cc: <stable@vger.kernel.org> # v6.5+
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
There is no point in setting the hash state after finalisation
since the hash state must never be used again. Remove that code.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The dc_data structure holds data required for handling compression
operations, such as overflow buffers. In this context, the use of
managed memory allocation APIs (devm_kzalloc() and devm_kfree())
is not necessary, as these data structures are freed and
re-allocated when a device is restarted in adf_dev_down() and
adf_dev_up().
Additionally, managed APIs automatically handle memory cleanup when the
device is detached, which can lead to conflicts with manual cleanup
processes. Specifically, if a device driver invokes the adf_dev_down()
function as part of the cleanup registered with
devm_add_action_or_reset(), it may attempt to free memory that is also
managed by the device's resource management system, potentially leading
to a double-free.
This might result in a warning similar to the following when unloading
the device specific driver, for example qat_6xxx.ko:
qat_free_dc_data+0x4f/0x60 [intel_qat]
qat_compression_event_handler+0x3d/0x1d0 [intel_qat]
adf_dev_shutdown+0x6d/0x1a0 [intel_qat]
adf_dev_down+0x32/0x50 [intel_qat]
devres_release_all+0xb8/0x110
device_unbind_cleanup+0xe/0x70
device_release_driver_internal+0x1c1/0x200
driver_detach+0x48/0x90
bus_remove_driver+0x74/0xf0
pci_unregister_driver+0x2e/0xb0
Use unmanaged memory allocation APIs (kzalloc_node() and kfree()) for
the dc_data structure. This ensures that memory is explicitly allocated
and freed under the control of the driver code, preventing manual
deallocation from interfering with automatic cleanup.
Fixes: 1198ae56c9 ("crypto: qat - expose deflate through acomp api for QAT GEN2")
Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The sunset period of SHA-1 is approaching [1] and FIPS 140 certificates
have a validity of 5 years. Any distros starting FIPS certification for
their kernels now would therefore most likely end up on the NIST
Cryptographic Module Validation Program "historical" list before their
certification expires.
While SHA-1 is technically still allowed until Dec. 31, 2030, it is
heavily discouraged by NIST and it makes sense to set .fips_allowed to
0 now for any crypto algorithms that reference it in order to avoid any
costly surprises down the line.
[1]: https://www.nist.gov/news-events/news/2022/12/nist-retires-sha-1-cryptographic-algorithm
Acked-by: Stephan Mueller <smueller@chronox.de>
Cc: Marcus Meissner <meissner@suse.de>
Cc: Jarod Wilson <jarod@redhat.com>
Cc: Neil Horman <nhorman@tuxdriver.com>
Cc: John Haxby <john.haxby@oracle.com>
Signed-off-by: Vegard Nossum <vegard.nossum@oracle.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use crypto_ahash_blocksize() and crypto_ahash_digestsize() helpers instead
of directly accessing 'struct ahash_alg' members.
Signed-off-by: Ovidiu Panait <ovidiu.panait.oss@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Remove `ivlen` field of `sun8i_cipher_req_ctx`, as it is not really useful.
The iv length returned by crypto_skcipher_ivsize() is already available
everywhere and can be used instead.
Signed-off-by: Ovidiu Panait <ovidiu.panait.oss@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
In sun8i_ce_cipher_unprepare(), dma_unmap_sg() is incorrectly called with
the number of entries returned by dma_map_sg(), rather than using the
original number of entries passed when mapping the scatterlist.
To fix this, stash the original number of entries passed to dma_map_sg()
in the request context.
Fixes: 0605fa0f78 ("crypto: sun8i-ce - split into prepare/run/unprepare")
Signed-off-by: Ovidiu Panait <ovidiu.panait.oss@gmail.com>
Acked-by: Corentin LABBE <clabbe.montjoie@gmail.com>
Tested-by: Corentin LABBE <clabbe.montjoie@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The mapped SG lists are written to hash_engine->ahash_src_addr which
has the size ASPEED_HASH_SRC_DMA_BUF_LEN. Since scatterlists are
not bound in size, make sure that size is not exceeded.
If the mapped SG list is larger than the buffer, simply iterate
over it as is done in the dma_prepare case.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Rather than failing a hash larger than ASPEED_CRYPTO_SRC_DMA_BUF_LEN,
just hash them over and over again until it's done.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
If a hash request fails due to a DMA mapping error, or if it is too
large to fit in the the driver buffer, use a fallback to do the hash
rather than failing.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use the Crypto API partial block handling.
Also switch to the generic export format.
Remove final function that is no longer used by the Crypto API.
Move final padding into aspeed_ahash_dma_prepare_sg.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Rather than processing a final as two separate updates, combine
them into one for the linear dma_prepare case.
This means that the total hash size is slightly reduced, but that
will be fixed up later by repeating the hash if necessary.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The only time when sham_final needs to be called in sham_finup
is when the finup request fits into the partial block. Move this
special handling into sham_update.
The comment about releaseing resources is non-sense. The Crypto
API does not mandate the use of final so the user could always go
away after an update and never come back. Therefore the driver
must not hold any resources after an update call.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Move the from-device DMA buffer to the front of the structure.
Sort the rest by size and alignment.
Keep the partial block buffer at the end.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The hmac implementation in aspeed simply duplicates what the new
ahash hmac template already does, namely construct ipad and opad
by hand and then adding them to the hash before feeding it to the
engine.
Remove them and just use the generic ahash hmac template.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Some drivers cannot have a fallback, e.g., because the key is held
in hardware. Allow these to be used with ahash by adding the bit
CRYPTO_ALG_NO_FALLBACK.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Tested-by: Harald Freudenberger <freude@linux.ibm.com>
The HKDF self-tests depend on the HMAC algorithms being registered.
HMAC is now registered at module_init, which put it at the same level as
HKDF. Move HKDF to late_initcall so that it runs afterwards.
Fixes: ef93f15628 ("Revert "crypto: run initcalls for generic implementations earlier"")
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Pull timer cleanup from Thomas Gleixner:
"The delayed from_timer() API cleanup:
The renaming to the timer_*() namespace was delayed due massive
conflicts against Linux-next. Now that everything is upstream finish
the conversion"
* tag 'timers-cleanups-2025-06-08' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
treewide, timers: Rename from_timer() to timer_container_of()
Pull x86 fixes from Thomas Gleixner:
"A small set of x86 fixes:
- Cure IO bitmap inconsistencies
A failed fork cleans up all resources of the newly created thread
via exit_thread(). exit_thread() invokes io_bitmap_exit() which
does the IO bitmap cleanups, which unfortunately assume that the
cleanup is related to the current task, which is obviously bogus.
Make it work correctly
- A lockdep fix in the resctrl code removed the clearing of the
command buffer in two places, which keeps stale error messages
around. Bring them back.
- Remove unused trace events"
* tag 'x86-urgent-2025-06-08' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
fs/resctrl: Restore the rdt_last_cmd_clear() calls after acquiring rdtgroup_mutex
x86/iopl: Cure TIF_IO_BITMAP inconsistencies
x86/fpu: Remove unused trace events
Pull timer fix from Thomas Gleixner:
"Add the missing seq_file forward declaration in the timer namespace
header"
* tag 'timers-urgent-2025-06-08' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
timens: Add struct seq_file forward declaration
Add initial DMR support, which required smarter RAPL probe
Fix AMD MSR RAPL energy reporting
Add RAPL power limit configuration output
Minor fixes
Signed-off-by: Len Brown <len.brown@intel.com>
For the RAPL package energy status counter, Intel and AMD share the same
perf_subsys and perf_name, but with different MSR addresses.
Both rapl_counter_arch_infos[0] and rapl_counter_arch_infos[1] are
introduced to describe this counter for different Vendors.
As a result, the perf counter is probed twice, and causes a failure in
in get_rapl_counters() because expected_read_size and actual_read_size
don't match.
Fix the problem by skipping the already probed counter.
Note, this is not a perfect fix. For example, if different
vendors/platforms use the same MSR value for different purpose, the code
can be fooled when it probes a rapl_counter_arch_infos[] entry that does
not belong to the running Vendor/Platform.
In a long run, better to put rapl_counter_arch_infos[] into the
platform_features so that this becomes Vendor/Platform specific.
Signed-off-by: Zhang Rui <rui.zhang@intel.com>
Signed-off-by: Len Brown <len.brown@intel.com>
platform_features->rapl_msrs describes the RAPL MSRs supported. While
RAPL Perf counters can be exposed from different kernel backend drivers,
e.g. RAPL MSR I/F driver, or RAPL TPMI I/F driver.
Thus, turbostat should first blindly probe all the available RAPL Perf
counters, and falls back to the RAPL MSR counters if they are listed in
platform_features->rapl_msrs.
With this, platforms that don't have RAPL MSRs can clear the
platform_features->rapl_msrs bits and use RAPL Perf counters only.
Signed-off-by: Zhang Rui <rui.zhang@intel.com>
Signed-off-by: Len Brown <len.brown@intel.com>
Increase the code readability by moving the no_perf/no_msr flag and the
cai->perf_name/cai->msr sanity checks into the counter probe functions.
No functional change.
Signed-off-by: Zhang Rui <rui.zhang@intel.com>
Signed-off-by: Len Brown <len.brown@intel.com>
probe_rapl_msr() is reused for probing RAPL MSR counters, cstate MSR
counters and MPERF/APERF/SMI MSR counters, thus its name is misleading.
Similar to add_perf_counter(), introduce add_msr_counter() to probe a
counter via MSR. Introduce wrapper function add_rapl_msr_counter() at
the same time to add extra check for Zero return value for specified
RAPL counters.
No functional change intended.
Signed-off-by: Zhang Rui <rui.zhang@intel.com>
Signed-off-by: Len Brown <len.brown@intel.com>
As the only caller of add_msr_perf_counter_(), add_msr_perf_counter()
just gives extra debug output on top. There is no need to keep both
functions.
Remove add_msr_perf_counter_() and move all the logic to
add_msr_perf_counter().
No functional change.
Signed-off-by: Zhang Rui <rui.zhang@intel.com>
Signed-off-by: Len Brown <len.brown@intel.com>
As the only caller of add_cstate_perf_counter_(),
add_cstate_perf_counter() just gives extra debug output on top. There is
no need to keep both functions.
Remove add_cstate_perf_counter_() and move all the logic to
add_cstate_perf_counter().
No functional change.
Signed-off-by: Zhang Rui <rui.zhang@intel.com>
Signed-off-by: Len Brown <len.brown@intel.com>
As the only caller of add_rapl_perf_counter_(), add_rapl_perf_counter()
just gives extra debug output on top. There is no need to keep both
functions.
Remove add_rapl_perf_counter_() and move all the logic to
add_rapl_perf_counter().
No functional change.
Signed-off-by: Zhang Rui <rui.zhang@intel.com>
Signed-off-by: Len Brown <len.brown@intel.com>
