An often overlooked aspect of the skcipher walker API is that an
error is not just indicated by a non-zero return value, but by the
fact that walk->nbytes is zero.
Thus it is an error to call skcipher_walk_done after getting back
walk->nbytes == 0 from the previous interaction with the walker.
This is because when walk->nbytes is zero the walker is left in
an undefined state and any further calls to it may try to free
uninitialised stack memory.
The arm64 ccm code has to deal with zero-length messages, and
it needs to process data even when walk->nbytes == 0 is returned.
It doesn't have this bug because there is an explicit check for
walk->nbytes != 0 prior to the skcipher_walk_done call.
However, the loop is still sufficiently different from the usual
layout and it appears to have been copied into other code which
then ended up with this bug. This patch rewrites it to follow the
usual convention of checking walk->nbytes.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Tested-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Lukas Bulwahn [Mon, 30 Jan 2023 07:31:09 +0000 (08:31 +0100)]
MAINTAINERS: repair file entry for STARFIVE TRNG DRIVER
Commit c388f458bc34 ("hwrng: starfive - Add TRNG driver for StarFive SoC")
adds the STARFIVE TRNG DRIVER section to MAINTAINERS, but refers to the
non-existing file drivers/char/hw_random/starfive-trng.c rather than to the
actually added file drivers/char/hw_random/jh7110-trng.c in this commit.
Hence, ./scripts/get_maintainer.pl --self-test=patterns complains about a
broken reference.
Repair this file entry in STARFIVE TRNG DRIVER.
Fixes: c388f458bc34 ("hwrng: starfive - Add TRNG driver for StarFive SoC") Signed-off-by: Lukas Bulwahn <lukas.bulwahn@gmail.com> Acked-by: Jia Jie Ho <jiajie.ho@starfivetech.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Herbert Xu [Fri, 27 Jan 2023 11:03:21 +0000 (19:03 +0800)]
hwrng: starfive - Enable compile testing
Enable compile testing for jh7110. Also remove the dependency on
HW_RANDOM.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Jia Jie Ho <jiajie.ho@starfivetech.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Linus Walleij [Wed, 25 Jan 2023 00:23:11 +0000 (01:23 +0100)]
crypto: ux500/hash - delete driver
It turns out we can just modify the newer STM32 HASH driver
to be used with Ux500 and now that we have done that, delete
the old and sparsely maintained Ux500 HASH driver.
Signed-off-by: Linus Walleij <linus.walleij@linaro.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Linus Walleij [Wed, 25 Jan 2023 00:23:10 +0000 (01:23 +0100)]
crypto: stm32/hash - Support Ux500 hash
The Ux500 has a hash block which is an ancestor to the STM32
hash block. With some minor code path additions we can
support also this variant in the STM32 driver. Differences:
- Ux500 only supports SHA1 and SHA256 (+/- MAC) so we split
up the algorithm registration per-algorithm and register
each algorithm along with its MAC variant separately.
- Ux500 does not have an interrupt to indicate that hash
calculation is complete, so we add code paths to handle
polling for completion if the interrupt is missing in the
device tree.
- Ux500 is lacking the SR status register, to check if an
operating is complete, we need to poll the HASH_STR_DCAL
bit in the HASH_STR register instead.
- Ux500 had the resulting hash at address offset 0x0c and
8 32bit registers ahead. We account for this with a special
code path when reading out the hash digest.
- Ux500 need a special bit set in the control register before
performing the final hash calculation on an empty message.
- Ux500 hashes on empty messages will be performed if the
above bit is set, but are incorrect. For this reason we
just make an inline synchronous hash using a fallback
hash.
Tested on the Ux500 Golden device with the extended tests.
Linus Walleij [Wed, 25 Jan 2023 00:23:09 +0000 (01:23 +0100)]
crypto: stm32/hash - Wait for idle before final CPU xmit
When calculating the hash using the CPU, right before the final
hash calculation, heavy testing on Ux500 reveals that it is wise
to wait for the hardware to go idle before calculating the
final hash.
The default test vectors mostly worked fine, but when I used the
extensive tests and stress the hardware I ran into this problem.
Linus Walleij [Wed, 25 Jan 2023 00:23:08 +0000 (01:23 +0100)]
crypto: stm32/hash - Use existing busy poll function
When exporting state we are waiting indefinitely in the same
was as the ordinary stm32_hash_wait_busy() poll-for-completion
function but without a timeout, which means we could hang in
an eternal loop. Fix this by waiting for completion like the
rest of the code.
Linus Walleij [Wed, 25 Jan 2023 00:23:07 +0000 (01:23 +0100)]
crypto: stm32/hash - Simplify code
We are passing (rctx->flags & HASH_FLAGS_FINUP) as indicator
for the final request but we already know this to be true since
we are in the (final) arm of an if-statement set from the same
flag. Just open-code it as true.
Linus Walleij [Wed, 25 Jan 2023 00:23:06 +0000 (01:23 +0100)]
dt-bindings: crypto: Let STM32 define Ux500 HASH
This adds device tree bindings for the Ux500 HASH block
as a compatible in the STM32 HASH bindings.
The Ux500 HASH binding has been used for ages in the kernel
device tree for Ux500 but was never documented, so fill in
the gap by making it a sibling of the STM32 HASH block,
which is what it is.
The relationship to the existing STM32 HASH block is pretty
obvious when looking at the register map, and I have written
patches to reuse the STM32 HASH driver on the Ux500.
The main difference from the outside is that the Ux500 HASH
lacks the interrupt line, so some special if-clauses are
needed to accomodate this in the binding.
Reviewed-by: Rob Herring <robh@kernel.org> Signed-off-by: Linus Walleij <linus.walleij@linaro.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Tom Lendacky [Mon, 23 Jan 2023 22:53:08 +0000 (16:53 -0600)]
crypto: ccp - Flush the SEV-ES TMR memory before giving it to firmware
Perform a cache flush on the SEV-ES TMR memory after allocation to prevent
any possibility of the firmware encountering an error should dirty cache
lines be present. Use clflush_cache_range() to flush the SEV-ES TMR memory.
Fixes: 97f9ac3db661 ("crypto: ccp - Add support for SEV-ES to the PSP driver") Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The ZLIB format (RFC 1950) is made of deflate compressed data surrounded
by a header and a footer. The QAT accelerators support only the deflate
algorithm, therefore the header and the footer need to be inserted in
software.
This adds logic in the QAT driver to support the ZLIB format. In
particular:
* Generalize the function qat_comp_alg_compress_decompress() to allow
skipping an initial region (header) of the source and/or destination
scatter lists.
* Add logic to register the qat_zlib_deflate algorithm into the acomp
framework.
* For ZLIB compression, skip the initial portion of the destination
buffer before sending the job to the QAT accelerator and insert the
ZLIB header and footer in the callback, after the QAT request has
been processed.
* For ZLIB decompression, parse the header in the input buffer
provided by the user and verify its validity before attempting the
decompression of the buffer with QAT. Then submit the buffer to QAT
for decompression. In the callback verify the correctness of the
footer by comparing the value of the ADLER produced by QAT with the
one in the destination buffer.
Signed-off-by: Lucas Segarra Fernandez <lucas.segarra.fernandez@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Extend qat_bl_sgl_to_bufl() to allow skipping the mapping of a region
of the source and the destination scatter lists starting from byte
zero.
This is to support the ZLIB format (RFC 1950) in the qat driver.
The ZLIB format is made of deflate compressed data surrounded by a
header and a footer. The QAT accelerators support only the deflate
algorithm, therefore the header should not be mapped since it is
inserted in software.
Signed-off-by: Lucas Segarra Fernandez <lucas.segarra.fernandez@intel.com> Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
While the akcipher API as a whole is designed to be called only
from thread context, its completion path is still called from
softirq context as usual. Therefore we must not use GFP_KERNEL
on that path.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Herbert Xu [Sun, 22 Jan 2023 08:07:37 +0000 (16:07 +0800)]
crypto: xts - Handle EBUSY correctly
As it is xts only handles the special return value of EINPROGRESS,
which means that in all other cases it will free data related to the
request.
However, as the caller of xts may specify MAY_BACKLOG, we also need
to expect EBUSY and treat it in the same way. Otherwise backlogged
requests will trigger a use-after-free.
Fixes: 8083b1bf8163 ("crypto: xts - add support for ciphertext stealing") Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Previously the child skcipher request was stored on the stack and
therefore needed to be zeroed. As it is now dynamically allocated
we no longer need to do so.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
ye xingchen [Thu, 19 Jan 2023 07:36:49 +0000 (15:36 +0800)]
crypto: aspeed - Use devm_platform_get_and_ioremap_resource()
Convert platform_get_resource(), devm_ioremap_resource() to a single
call to devm_platform_get_and_ioremap_resource(), as this is exactly
what this function does.
Signed-off-by: ye xingchen <ye.xingchen@zte.com.cn> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Yang Yingliang [Thu, 19 Jan 2023 01:48:59 +0000 (09:48 +0800)]
crypto: aspeed - change aspeed_acry_akcipher_algs to static
aspeed_acry_akcipher_algs is only used in aspeed-acry.c now,
change it to static.
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com> Acked-by: Andrew Jeffery <andrew@aj.id.au> Reviewed-by: Neal Liu <neal_liu@aspeedtech.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Vladis Dronov [Tue, 17 Jan 2023 17:20:06 +0000 (18:20 +0100)]
crypto: testmgr - disallow certain DRBG hash functions in FIPS mode
According to FIPS 140-3 IG, section D.R "Hash Functions Acceptable for
Use in the SP 800-90A DRBGs", modules certified after May 16th, 2023
must not support the use of: SHA-224, SHA-384, SHA512-224, SHA512-256,
SHA3-224, SHA3-384. Disallow HMAC and HASH DRBGs using SHA-384 in FIPS
mode.
Signed-off-by: Vladis Dronov <vdronov@redhat.com> Reviewed-by: Stephan Müller <smueller@chronox.de> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Taehee Yoo [Sun, 15 Jan 2023 12:15:36 +0000 (12:15 +0000)]
crypto: x86/aria-avx512 - fix build failure with old binutils
The minimum version of binutils for kernel build is currently 2.23 and
it doesn't support GFNI.
So, it fails to build the aria-avx512 if the old binutils is used.
aria-avx512 requires GFNI, so it should not be allowed to build if the
old binutils is used.
The AS_AVX512 and AS_GFNI are added to the Kconfig to disable build
aria-avx512 if the old binutils is used.
Fixes: c970d42001f2 ("crypto: x86/aria - implement aria-avx512") Reported-by: Jan Beulich <jbeulich@suse.com> Signed-off-by: Taehee Yoo <ap420073@gmail.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Taehee Yoo [Sun, 15 Jan 2023 12:15:35 +0000 (12:15 +0000)]
crypto: x86/aria-avx2 - fix build failure with old binutils
The minimum version of binutils for kernel build is currently 2.23 and
it doesn't support GFNI.
So, it fails to build the aria-avx2 if the old binutils is used.
The code using GFNI is an optional part of aria-avx2.
So, it disables GFNI part in it when the old binutils is used.
Fixes: 37d8d3ae7a58 ("crypto: x86/aria - implement aria-avx2") Reported-by: Jan Beulich <jbeulich@suse.com> Signed-off-by: Taehee Yoo <ap420073@gmail.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Taehee Yoo [Sun, 15 Jan 2023 12:15:34 +0000 (12:15 +0000)]
crypto: x86/aria-avx - fix build failure with old binutils
The minimum version of binutils for kernel build is currently 2.23 and
it doesn't support GFNI.
So, it fails to build the aria-avx if the old binutils is used.
The code using GFNI is an optional part of aria-avx.
So, it disables GFNI part in it when the old binutils is used.
In order to check whether the using binutils is supporting GFNI or not,
AS_GFNI is added.
Fixes: ba3579e6e45c ("crypto: aria-avx - add AES-NI/AVX/x86_64/GFNI assembler implementation of aria cipher") Reported-by: Jan Beulich <jbeulich@suse.com> Signed-off-by: Taehee Yoo <ap420073@gmail.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Herbert Xu [Fri, 13 Jan 2023 10:27:51 +0000 (18:27 +0800)]
crypto: seqiv - Handle EBUSY correctly
As it is seqiv only handles the special return value of EINPROGERSS,
which means that in all other cases it will free data related to the
request.
However, as the caller of seqiv may specify MAY_BACKLOG, we also need
to expect EBUSY and treat it in the same way. Otherwise backlogged
requests will trigger a use-after-free.
Fixes: 0a270321dbf9 ("[CRYPTO] seqiv: Add Sequence Number IV Generator") Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Herbert Xu [Fri, 13 Jan 2023 10:24:09 +0000 (18:24 +0800)]
crypto: essiv - Handle EBUSY correctly
As it is essiv only handles the special return value of EINPROGERSS,
which means that in all other cases it will free data related to the
request.
However, as the caller of essiv may specify MAY_BACKLOG, we also need
to expect EBUSY and treat it in the same way. Otherwise backlogged
requests will trigger a use-after-free.
Fixes: be1eb7f78aa8 ("crypto: essiv - create wrapper template...") Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Acked-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Lukas Bulwahn [Fri, 13 Jan 2023 07:47:15 +0000 (08:47 +0100)]
crypto: atmel-i2c - avoid defines prefixed with CONFIG
Defines prefixed with "CONFIG" should be limited to proper Kconfig options,
that are introduced in a Kconfig file.
Here, a definition for the driver's configuration zone is named
CONFIG_ZONE. Rename this local definition to CONFIGURATION_ZONE to avoid
defines prefixed with "CONFIG".
No functional change.
Signed-off-by: Lukas Bulwahn <lukas.bulwahn@gmail.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Lukas Bulwahn [Wed, 11 Jan 2023 12:02:03 +0000 (13:02 +0100)]
crypto: hisilicon - remove redundant config PCI dependency for some CRYPTO_DEV_HISI configs
While reviewing dependencies in some Kconfig files, I noticed the redundant
dependency "depends on PCI && PCI_MSI". The config PCI_MSI has always,
since its introduction, been dependent on the config PCI. So, it is
sufficient to just depend on PCI_MSI, and know that the dependency on PCI
is implicitly implied.
Reduce the dependencies of configs CRYPTO_DEV_HISI_SEC2,
CRYPTO_DEV_HISI_QM, CRYPTO_DEV_HISI_ZIP and CRYPTO_DEV_HISI_HPRE.
No functional change and effective change of Kconfig dependendencies.
Linus Walleij [Tue, 10 Jan 2023 19:43:07 +0000 (20:43 +0100)]
crypto: stm32 - Use accelerated readsl/writesl
When reading or writing crypto buffers the inner loops can
be replaced with readsl and writesl which will on ARM result
in a tight assembly loop, speeding up encryption/decryption
a little bit. This optimization was in the Ux500 driver so
let's carry it over to the STM32 driver.
Cc: Maxime Coquelin <mcoquelin.stm32@gmail.com> Cc: Alexandre Torgue <alexandre.torgue@foss.st.com> Cc: Lionel Debieve <lionel.debieve@foss.st.com> Signed-off-by: Linus Walleij <linus.walleij@linaro.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
crypto: aspeed - Replace zero-length array with flexible-array member
Zero-length arrays are deprecated[1] and we are moving towards
adopting C99 flexible-array members instead. So, replace zero-length
array declaration in struct aspeed_sham_ctx with flex-array
member.
This helps with the ongoing efforts to tighten the FORTIFY_SOURCE
routines on memcpy() and help us make progress towards globally
enabling -fstrict-flex-arrays=3 [2].
The /dma/dma0chan0 sysfs file is not removed since dma_chan object
has been released in ccp_dma_release() before releasing dma device.
A correct procedure would be: release dma channels first => unregister
dma device => release ccp dma object.
Link: https://bugzilla.kernel.org/show_bug.cgi?id=216888 Fixes: 68dbe80f5b51 ("crypto: ccp - Release dma channels before dmaengine unrgister") Tested-by: Vladis Dronov <vdronov@redhat.com> Signed-off-by: Koba Ko <koba.ko@canonical.com> Reviewed-by: Vladis Dronov <vdronov@redhat.com> Acked-by: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Ard Biesheuvel [Wed, 14 Dec 2022 17:19:57 +0000 (18:19 +0100)]
crypto: aead - fix inaccurate documentation
The AEAD documentation conflates associated data and authentication
tags: the former (along with the ciphertext) is authenticated by the
latter. Fix the doc accordingly.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Ard Biesheuvel [Wed, 14 Dec 2022 17:19:56 +0000 (18:19 +0100)]
crypto: tcrypt - include larger key sizes in RFC4106 benchmark
RFC4106 wraps AES in GCM mode, and can be used with larger key sizes
than 128/160 bits, just like AES itself. So add these to the tcrypt
recipe so they will be benchmarked as well.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Ard Biesheuvel [Wed, 14 Dec 2022 17:19:55 +0000 (18:19 +0100)]
crypto: arm64/gcm - add RFC4106 support
Add support for RFC4106 ESP encapsulation to the accelerated GCM
implementation. This results in a ~10% speedup for IPsec frames of
typical size (~1420 bytes) on Cortex-A53.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Samuel Holland [Sat, 31 Dec 2022 22:01:44 +0000 (16:01 -0600)]
crypto: sun8i-ce - Add TRNG clock to the D1 variant
At least the D1 variant requires a separate clock for the TRNG.
Without this clock enabled, reading from /dev/hwrng reports:
sun8i-ce 3040000.crypto: DMA timeout for TRNG (tm=96) on flow 3
Experimentation shows that the necessary clock is the SoC's internal
RC oscillator. This makes sense, as noise from the oscillator can be
used as a source of entropy.
Signed-off-by: Samuel Holland <samuel@sholland.org> Reviewed-by: Jernej Skrabec <jernej.skrabec@gmail.com> Acked-by: Corentin Labbe <clabbe.montjoie@gmail.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Samuel Holland [Sat, 31 Dec 2022 22:01:43 +0000 (16:01 -0600)]
dt-bindings: crypto: sun8i-ce: Add compatible for D1
D1 has a crypto engine similar to the one in other Allwinner SoCs.
Like H6, it has a separate MBUS clock gate.
It also requires the internal RC oscillator to be enabled for the TRNG
to return data, presumably because noise from the oscillator is used as
an entropy source. This is likely the case for earlier variants as well,
but it really only matters for H616 and newer SoCs, as H6 provides no
way to disable the internal oscillator.
Signed-off-by: Samuel Holland <samuel@sholland.org> Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org> Acked-by: Corentin Labbe <clabbe.montjoie@gmail.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Ard Biesheuvel [Mon, 2 Jan 2023 10:18:46 +0000 (11:18 +0100)]
crypto: skcipher - Use scatterwalk (un)map interface for dst and src buffers
The skcipher walk API implementation avoids scatterwalk_map() for
mapping the source and destination buffers, and invokes kmap_atomic()
directly if the buffer in question is not in low memory (which can only
happen on 32-bit architectures). This avoids some overhead on 64-bit
architectures, and most notably, permits the skcipher code to run with
preemption enabled.
Now that scatterwalk_map() has been updated to use kmap_local(), none of
this is needed, so we can simply use scatterwalk_map/unmap instead.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Danny Tsen [Mon, 5 Dec 2022 00:34:57 +0000 (19:34 -0500)]
crypto: p10-aes-gcm - Supporting functions for ghash
This perl code is taken from the OpenSSL project and added gcm_init_htable function
used in the p10-aes-gcm-glue.c code to initialize hash table. gcm_hash_p8 is used
to hash encrypted data blocks.
Signed-off-by: Danny Tsen <dtsen@linux.ibm.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Danny Tsen [Mon, 5 Dec 2022 00:34:56 +0000 (19:34 -0500)]
crypto: p10-aes-gcm - Supporting functions for AES
This code is taken from CRYPTOGAMs[1]. The following functions are used,
aes_p8_set_encrypt_key is used to generate AES round keys and aes_p8_encrypt is used
to encrypt single block.
Signed-off-by: Danny Tsen <dtsen@linux.ibm.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Taehee Yoo [Sun, 1 Jan 2023 09:12:52 +0000 (09:12 +0000)]
crypto: x86/aria - implement aria-avx512
aria-avx512 implementation uses AVX512 and GFNI.
It supports 64way parallel processing.
So, byteslicing code is changed to support 64way parallel.
And it exports some aria-avx2 functions such as encrypt() and decrypt().
AVX and AVX2 have 16 registers.
They should use memory to store/load state because of lack of registers.
But AVX512 supports 32 registers.
So, it doesn't require store/load in the s-box layer.
It means that it can reduce overhead of store/load in the s-box layer.
Also code become much simpler.
Benchmark with modprobe tcrypt mode=610 num_mb=8192, i3-12100:
ARIA-AVX512(128bit and 256bit)
testing speed of multibuffer ecb(aria) (ecb-aria-avx512) encryption
tcrypt: 1 operation in 1504 cycles (1024 bytes)
tcrypt: 1 operation in 4595 cycles (4096 bytes)
tcrypt: 1 operation in 1763 cycles (1024 bytes)
tcrypt: 1 operation in 5540 cycles (4096 bytes)
testing speed of multibuffer ecb(aria) (ecb-aria-avx512) decryption
tcrypt: 1 operation in 1502 cycles (1024 bytes)
tcrypt: 1 operation in 4615 cycles (4096 bytes)
tcrypt: 1 operation in 1759 cycles (1024 bytes)
tcrypt: 1 operation in 5554 cycles (4096 bytes)
ARIA-AVX2 with GFNI(128bit and 256bit)
testing speed of multibuffer ecb(aria) (ecb-aria-avx2) encryption
tcrypt: 1 operation in 2003 cycles (1024 bytes)
tcrypt: 1 operation in 5867 cycles (4096 bytes)
tcrypt: 1 operation in 2358 cycles (1024 bytes)
tcrypt: 1 operation in 7295 cycles (4096 bytes)
testing speed of multibuffer ecb(aria) (ecb-aria-avx2) decryption
tcrypt: 1 operation in 2004 cycles (1024 bytes)
tcrypt: 1 operation in 5956 cycles (4096 bytes)
tcrypt: 1 operation in 2409 cycles (1024 bytes)
tcrypt: 1 operation in 7564 cycles (4096 bytes)
Signed-off-by: Taehee Yoo <ap420073@gmail.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Taehee Yoo [Sun, 1 Jan 2023 09:12:51 +0000 (09:12 +0000)]
crypto: x86/aria - implement aria-avx2
aria-avx2 implementation uses AVX2, AES-NI, and GFNI.
It supports 32way parallel processing.
So, byteslicing code is changed to support 32way parallel.
And it exports some aria-avx functions such as encrypt() and decrypt().
There are two main logics, s-box layer and diffusion layer.
These codes are the same as aria-avx implementation.
But some instruction are exchanged because they don't support 256bit
registers.
Also, AES-NI doesn't support 256bit register.
So, aesenclast and aesdeclast are used twice like below:
vextracti128 $1, ymm0, xmm6;
vaesenclast xmm7, xmm0, xmm0;
vaesenclast xmm7, xmm6, xmm6;
vinserti128 $1, xmm6, ymm0, ymm0;
Benchmark with modprobe tcrypt mode=610 num_mb=8192, i3-12100:
ARIA-AVX2 with GFNI(128bit and 256bit)
testing speed of multibuffer ecb(aria) (ecb-aria-avx2) encryption
tcrypt: 1 operation in 2003 cycles (1024 bytes)
tcrypt: 1 operation in 5867 cycles (4096 bytes)
tcrypt: 1 operation in 2358 cycles (1024 bytes)
tcrypt: 1 operation in 7295 cycles (4096 bytes)
testing speed of multibuffer ecb(aria) (ecb-aria-avx2) decryption
tcrypt: 1 operation in 2004 cycles (1024 bytes)
tcrypt: 1 operation in 5956 cycles (4096 bytes)
tcrypt: 1 operation in 2409 cycles (1024 bytes)
tcrypt: 1 operation in 7564 cycles (4096 bytes)
ARIA-AVX with GFNI(128bit and 256bit)
testing speed of multibuffer ecb(aria) (ecb-aria-avx) encryption
tcrypt: 1 operation in 2761 cycles (1024 bytes)
tcrypt: 1 operation in 9390 cycles (4096 bytes)
tcrypt: 1 operation in 3401 cycles (1024 bytes)
tcrypt: 1 operation in 11876 cycles (4096 bytes)
testing speed of multibuffer ecb(aria) (ecb-aria-avx) decryption
tcrypt: 1 operation in 2735 cycles (1024 bytes)
tcrypt: 1 operation in 9424 cycles (4096 bytes)
tcrypt: 1 operation in 3369 cycles (1024 bytes)
tcrypt: 1 operation in 11954 cycles (4096 bytes)
Signed-off-by: Taehee Yoo <ap420073@gmail.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Taehee Yoo [Sun, 1 Jan 2023 09:12:50 +0000 (09:12 +0000)]
crypto: x86/aria - do not use magic number offsets of aria_ctx
aria-avx assembly code accesses members of aria_ctx with magic number
offset. If the shape of struct aria_ctx is changed carelessly,
aria-avx will not work.
So, we need to ensure accessing members of aria_ctx with correct
offset values, not with magic numbers.
It adds ARIA_CTX_enc_key, ARIA_CTX_dec_key, and ARIA_CTX_rounds in the
asm-offsets.c So, correct offset definitions will be generated.
aria-avx assembly code can access members of aria_ctx safely with
these definitions.
Signed-off-by: Taehee Yoo <ap420073@gmail.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
David Rientjes [Fri, 30 Dec 2022 22:18:46 +0000 (14:18 -0800)]
crypto: ccp - Avoid page allocation failure warning for SEV_GET_ID2
For SEV_GET_ID2, the user provided length does not have a specified
limitation because the length of the ID may change in the future. The
kernel memory allocation, however, is implicitly limited to 4MB on x86 by
the page allocator, otherwise the kzalloc() will fail.
When this happens, it is best not to spam the kernel log with the warning.
Simply fail the allocation and return ENOMEM to the user.
Fixes: d6112ea0cb34 ("crypto: ccp - introduce SEV_GET_ID2 command") Reported-by: Andy Nguyen <theflow@google.com> Reported-by: Peter Gonda <pgonda@google.com> Suggested-by: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: David Rientjes <rientjes@google.com> Acked-by: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Nicolai Stange [Thu, 29 Dec 2022 21:17:10 +0000 (22:17 +0100)]
crypto: testmgr - allow ecdsa-nist-p256 and -p384 in FIPS mode
The kernel provides implementations of the NIST ECDSA signature
verification primitives. For key sizes of 256 and 384 bits respectively
they are approved and can be enabled in FIPS mode. Do so.
Nicolai Stange [Thu, 29 Dec 2022 21:17:09 +0000 (22:17 +0100)]
crypto: testmgr - disallow plain ghash in FIPS mode
ghash may be used only as part of the gcm(aes) construction in FIPS
mode. Since commit d6097b8d5d55 ("crypto: api - allow algs only in specific
constructions in FIPS mode") there's support for using spawns which by
itself are marked as non-approved from approved template instantiations.
So simply mark plain ghash as non-approved in testmgr to block any attempts
of direct instantiations in FIPS mode.
Nicolai Stange [Thu, 29 Dec 2022 21:17:08 +0000 (22:17 +0100)]
crypto: testmgr - disallow plain cbcmac(aes) in FIPS mode
cbcmac(aes) may be used only as part of the ccm(aes) construction in FIPS
mode. Since commit d6097b8d5d55 ("crypto: api - allow algs only in specific
constructions in FIPS mode") there's support for using spawns which by
itself are marked as non-approved from approved template instantiations.
So simply mark plain cbcmac(aes) as non-approved in testmgr to block any
attempts of direct instantiations in FIPS mode.
Vladis Dronov [Thu, 29 Dec 2022 21:17:07 +0000 (22:17 +0100)]
crypto: s390/aes - drop redundant xts key check
xts_fallback_setkey() in xts_aes_set_key() will now enforce key size
rule in FIPS mode when setting up the fallback algorithm keys, which
makes the check in xts_aes_set_key() redundant or unreachable. So just
drop this check.
xts_fallback_setkey() now makes a key size check in xts_verify_key():
Vladis Dronov [Thu, 29 Dec 2022 21:17:06 +0000 (22:17 +0100)]
crypto: xts - drop xts_check_key()
xts_check_key() is obsoleted by xts_verify_key(). Over time XTS crypto
drivers adopted the newer xts_verify_key() variant, but xts_check_key()
is still used by a number of drivers. Switch drivers to use the newer
xts_verify_key() and make a couple of cleanups. This allows us to drop
xts_check_key() completely and avoid redundancy.
Signed-off-by: Vladis Dronov <vdronov@redhat.com> Reviewed-by: Eric Biggers <ebiggers@google.com> Reviewed-by: Nicolai Stange <nstange@suse.de> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Nicolai Stange [Thu, 29 Dec 2022 21:17:05 +0000 (22:17 +0100)]
crypto: xts - restrict key lengths to approved values in FIPS mode
According to FIPS 140-3 IG C.I., only (total) key lengths of either
256 bits or 512 bits are allowed with xts(aes). Make xts_verify_key() to
reject anything else in FIPS mode.
As xts(aes) is the only approved xts() template instantiation in FIPS mode,
the new restriction implemented in xts_verify_key() effectively only
applies to this particular construction.
Herbert Xu [Wed, 28 Dec 2022 11:03:32 +0000 (19:03 +0800)]
crypto: caam - Avoid GCC memset bug warning
Certain versions of gcc don't like the memcpy with a NULL dst
(which only happens with a zero length). This only happens
when debugging is enabled so add an if clause to work around
these warnings.
A similar warning used to be generated by sparse but that was
fixed years ago.
Herbert Xu [Tue, 27 Dec 2022 14:27:39 +0000 (15:27 +0100)]
lib/mpi: Fix buffer overrun when SG is too long
The helper mpi_read_raw_from_sgl sets the number of entries in
the SG list according to nbytes. However, if the last entry
in the SG list contains more data than nbytes, then it may overrun
the buffer because it only allocates enough memory for nbytes.
Fixes: 2d4d1eea540b ("lib/mpi: Add mpi sgl helpers") Reported-by: Roberto Sassu <roberto.sassu@huaweicloud.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Reviewed-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Commit 453de3eb08c4 ("crypto: ux500/cryp - delete driver") removes the
config CRYPTO_DEV_UX500_CRYP, but leaves an obsolete reference in the
dependencies of config CRYPTO_DEV_UX500_DEBUG.
Remove that obsolete reference, and adjust the description while at it.
Ard Biesheuvel [Tue, 13 Dec 2022 16:13:10 +0000 (17:13 +0100)]
crypto: scatterwalk - use kmap_local() not kmap_atomic()
kmap_atomic() is used to create short-lived mappings of pages that may
not be accessible via the kernel direct map. This is only needed on
32-bit architectures that implement CONFIG_HIGHMEM, but it can be used
on 64-bit other architectures too, where the returned mapping is simply
the kernel direct address of the page.
However, kmap_atomic() does not support migration on CONFIG_HIGHMEM
configurations, due to the use of per-CPU kmap slots, and so it disables
preemption on all architectures, not just the 32-bit ones. This implies
that all scatterwalk based crypto routines essentially execute with
preemption disabled all the time, which is less than ideal.
So let's switch scatterwalk_map/_unmap and the shash/ahash routines to
kmap_local() instead, which serves a similar purpose, but without the
resulting impact on preemption on architectures that have no need for
CONFIG_HIGHMEM.
Cc: Eric Biggers <ebiggers@kernel.org> Cc: Herbert Xu <herbert@gondor.apana.org.au> Cc: "Elliott, Robert (Servers)" <elliott@hpe.com> Signed-off-by: Ard Biesheuvel <ardb@kernel.org> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Sergiu Moga [Wed, 7 Dec 2022 13:59:55 +0000 (15:59 +0200)]
crypto: atmel - Add capability case for the 0x600 SHA and AES IP versions
In order for the driver to be made aware of the capabilities of the SHA
and AES IP versions 0x600 , such as those present on the SAM9X60 SoC's,
add a corresponding switch case to the capability method of the respective
drivers. Without this, besides the capabilities not being correctly set,
the self tests may hang since the driver is endlessly waiting for a
completion to be set by a never occurring DMA interrupt handler.
Signed-off-by: Sergiu Moga <sergiu.moga@microchip.com> Acked-by: Nicolas Ferre <nicolas.ferre@microchip.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Eric Biggers [Tue, 20 Dec 2022 05:40:41 +0000 (21:40 -0800)]
crypto: x86/ghash - use le128 instead of u128
The u128 struct type is going away, so make ghash-clmulni-intel use
le128 instead. Note that the field names a and b swapped, as they were
backwards with u128. (a is meant to be high-order and b low-order.)
Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Eric Biggers [Tue, 20 Dec 2022 05:40:40 +0000 (21:40 -0800)]
crypto: x86/ghash - fix unaligned access in ghash_setkey()
The key can be unaligned, so use the unaligned memory access helpers.
Fixes: 8ceee72808d1 ("crypto: ghash-clmulni-intel - use C implementation for setkey()") Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Tianjia Zhang [Wed, 21 Dec 2022 07:32:32 +0000 (15:32 +0800)]
crypto: arm64/sm4 - fix possible crash with CFI enabled
The SM4 CCM/GCM assembly functions for encryption and decryption is
called via indirect function calls. Therefore they need to use
SYM_TYPED_FUNC_START instead of SYM_FUNC_START to cause its type hash
to be emitted when the kernel is built with CONFIG_CFI_CLANG=y.
Otherwise, the code crashes with a CFI failure (if the compiler didn't
happen to optimize out the indirect call).
Fixes: 67fa3a7fdf80 ("crypto: arm64/sm4 - add CE implementation for CCM mode") Fixes: ae1b83c7d572 ("crypto: arm64/sm4 - add CE implementation for GCM mode") Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
treewide: Convert del_timer*() to timer_shutdown*()
Due to several bugs caused by timers being re-armed after they are
shutdown and just before they are freed, a new state of timers was added
called "shutdown". After a timer is set to this state, then it can no
longer be re-armed.
The following script was run to find all the trivial locations where
del_timer() or del_timer_sync() is called in the same function that the
object holding the timer is freed. It also ignores any locations where
the timer->function is modified between the del_timer*() and the free(),
as that is not considered a "trivial" case.
This was created by using a coccinelle script and the following
commands:
Linus Torvalds [Fri, 23 Dec 2022 22:44:08 +0000 (14:44 -0800)]
Merge tag 'spi-fix-v6.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi
Pull spi fix from Mark Brown:
"One driver specific change here which handles the case where a SPI
device for some reason tries to change the bus speed during a message
on fsl_spi hardware, this should be very unusual"
* tag 'spi-fix-v6.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi:
spi: fsl_spi: Don't change speed while chipselect is active
Linus Torvalds [Fri, 23 Dec 2022 22:38:00 +0000 (14:38 -0800)]
Merge tag 'regulator-fix-v6.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator
Pull regulator fixes from Mark Brown:
"Two core fixes here, one for a long standing race which some Qualcomm
systems have started triggering with their UFS driver and another
fixing a problem with supply lookup introduced by the fixes for devm
related use after free issues that were introduced in this merge
window"
* tag 'regulator-fix-v6.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator:
regulator: core: fix deadlock on regulator enable
regulator: core: Fix resolve supply lookup issue
Linus Torvalds [Fri, 23 Dec 2022 20:00:24 +0000 (12:00 -0800)]
Merge tag 'hardening-v6.2-rc1-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux
Pull kernel hardening fixes from Kees Cook:
- Fix CFI failure with KASAN (Sami Tolvanen)
- Fix LKDTM + CFI under GCC 7 and 8 (Kristina Martsenko)
- Limit CONFIG_ZERO_CALL_USED_REGS to Clang > 15.0.6 (Nathan
Chancellor)
- Ignore "contents" argument in LoadPin's LSM hook handling
- Fix paste-o in /sys/kernel/warn_count API docs
- Use READ_ONCE() consistently for oops/warn limit reading
* tag 'hardening-v6.2-rc1-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux:
cfi: Fix CFI failure with KASAN
exit: Use READ_ONCE() for all oops/warn limit reads
security: Restrict CONFIG_ZERO_CALL_USED_REGS to gcc or clang > 15.0.6
lkdtm: cfi: Make PAC test work with GCC 7 and 8
docs: Fix path paste-o for /sys/kernel/warn_count
LoadPin: Ignore the "contents" argument of the LSM hooks
* tag 'pstore-v6.2-rc1-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux:
pstore: Properly assign mem_type property
pstore: Make sure CONFIG_PSTORE_PMSG selects CONFIG_RT_MUTEXES
pstore: Switch pmsg_lock to an rt_mutex to avoid priority inversion
Linus Torvalds [Fri, 23 Dec 2022 19:44:20 +0000 (11:44 -0800)]
Merge tag 'dma-mapping-2022-12-23' of git://git.infradead.org/users/hch/dma-mapping
Pull dma-mapping fixes from Christoph Hellwig:
"Fix up the sound code to not pass __GFP_COMP to the non-coherent DMA
allocator, as it copes with that just as badly as the coherent
allocator, and then add a check to make sure no one passes the flag
ever again"
* tag 'dma-mapping-2022-12-23' of git://git.infradead.org/users/hch/dma-mapping:
dma-mapping: reject GFP_COMP for noncoherent allocations
ALSA: memalloc: don't use GFP_COMP for non-coherent dma allocations
Linus Torvalds [Fri, 23 Dec 2022 19:39:18 +0000 (11:39 -0800)]
Merge tag '9p-for-6.2-rc1' of https://github.com/martinetd/linux
Pull 9p updates from Dominique Martinet:
- improve p9_check_errors to check buffer size instead of msize when
possible (e.g. not zero-copy)
- some more syzbot and KCSAN fixes
- minor headers include cleanup
* tag '9p-for-6.2-rc1' of https://github.com/martinetd/linux:
9p/client: fix data race on req->status
net/9p: fix response size check in p9_check_errors()
net/9p: distinguish zero-copy requests
9p/xen: do not memcpy header into req->rc
9p: set req refcount to zero to avoid uninitialized usage
9p/net: Remove unneeded idr.h #include
9p/fs: Remove unneeded idr.h #include
* tag 'drm-next-2022-12-23' of git://anongit.freedesktop.org/drm/drm: (39 commits)
drm/amdgpu: grab extra fence reference for drm_sched_job_add_dependency
drm/amdgpu: enable VCN DPG for GC IP v11.0.4
drm/amdgpu: skip mes self test after s0i3 resume for MES IP v11.0
drm/amd/pm: correct the fan speed retrieving in PWM for some SMU13 asics
drm/amd/pm: bump SMU13.0.0 driver_if header to version 0x34
drm/amdgpu: skip MES for S0ix as well since it's part of GFX
drm/amd/pm: avoid large variable on kernel stack
drm/amdkfd: Fix double release compute pasid
drm/amdkfd: Fix kfd_process_device_init_vm error handling
drm/amd/pm: update SMU13.0.0 reported maximum shader clock
drm/amd/pm: correct SMU13.0.0 pstate profiling clock settings
drm/amd/pm: enable GPO dynamic control support for SMU13.0.7
drm/amd/pm: enable GPO dynamic control support for SMU13.0.0
drm/amdgpu: revert "generally allow over-commit during BO allocation"
drm/amdgpu: Remove unnecessary domain argument
drm/amdgpu: Fix size validation for non-exclusive domains (v4)
drm/amdgpu: Check if fru_addr is not NULL (v2)
drm/i915/ttm: consider CCS for backup objects
drm/i915/migrate: fix corner case in CCS aux copying
drm/amdgpu: rework reserved VMID handling
...
Linus Torvalds [Fri, 23 Dec 2022 18:49:45 +0000 (10:49 -0800)]
Merge tag 'mips_6.2_1' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux
Pull MIPS fixes from Thomas Bogendoerfer:
"Fixes due to DT changes"
* tag 'mips_6.2_1' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux:
MIPS: dts: bcm63268: Add missing properties to the TWD node
MIPS: ralink: mt7621: avoid to init common ralink reset controller
Linus Torvalds [Fri, 23 Dec 2022 18:45:00 +0000 (10:45 -0800)]
Merge tag 'mm-hotfixes-stable-2022-12-22-14-34' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull hotfixes from Andrew Morton:
"Eight fixes, all cc:stable. One is for gcov and the remainder are MM"
* tag 'mm-hotfixes-stable-2022-12-22-14-34' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm:
gcov: add support for checksum field
test_maple_tree: add test for mas_spanning_rebalance() on insufficient data
maple_tree: fix mas_spanning_rebalance() on insufficient data
hugetlb: really allocate vma lock for all sharable vmas
kmsan: export kmsan_handle_urb
kmsan: include linux/vmalloc.h
mm/mempolicy: fix memory leak in set_mempolicy_home_node system call
mm, mremap: fix mremap() expanding vma with addr inside vma
Luca Stefani [Thu, 22 Dec 2022 13:10:49 +0000 (14:10 +0100)]
pstore: Properly assign mem_type property
If mem-type is specified in the device tree
it would end up overriding the record_size
field instead of populating mem_type.
As record_size is currently parsed after the
improper assignment with default size 0 it
continued to work as expected regardless of the
value found in the device tree.
Simply changing the target field of the struct
is enough to get mem-type working as expected.
John Stultz [Wed, 21 Dec 2022 05:18:55 +0000 (05:18 +0000)]
pstore: Make sure CONFIG_PSTORE_PMSG selects CONFIG_RT_MUTEXES
In commit 76d62f24db07 ("pstore: Switch pmsg_lock to an rt_mutex
to avoid priority inversion") I changed a lock to an rt_mutex.
However, its possible that CONFIG_RT_MUTEXES is not enabled,
which then results in a build failure, as the 0day bot detected:
https://lore.kernel.org/linux-mm/202212211244.TwzWZD3H-lkp@intel.com/
Thus this patch changes CONFIG_PSTORE_PMSG to select
CONFIG_RT_MUTEXES, which ensures the build will not fail.
Cc: Wei Wang <wvw@google.com> Cc: Midas Chien<midaschieh@google.com> Cc: Connor O'Brien <connoro@google.com> Cc: Kees Cook <keescook@chromium.org> Cc: Anton Vorontsov <anton@enomsg.org> Cc: Colin Cross <ccross@android.com> Cc: Tony Luck <tony.luck@intel.com> Cc: kernel test robot <lkp@intel.com> Cc: kernel-team@android.com Fixes: 76d62f24db07 ("pstore: Switch pmsg_lock to an rt_mutex to avoid priority inversion") Reported-by: kernel test robot <lkp@intel.com> Signed-off-by: John Stultz <jstultz@google.com> Signed-off-by: Kees Cook <keescook@chromium.org> Link: https://lore.kernel.org/r/20221221051855.15761-1-jstultz@google.com
Sami Tolvanen [Thu, 22 Dec 2022 22:57:47 +0000 (22:57 +0000)]
cfi: Fix CFI failure with KASAN
When CFI_CLANG and KASAN are both enabled, LLVM doesn't generate a
CFI type hash for asan.module_ctor functions in translation units
where CFI is disabled, which leads to a CFI failure during boot when
do_ctors calls the affected constructors:
CFI failure at do_basic_setup+0x64/0x90 (target:
asan.module_ctor+0x0/0x28; expected type: 0xa540670c)
Specifically, this happens because CFI is disabled for
kernel/cfi.c. There's no reason to keep CFI disabled here anymore, so
fix the failure by not filtering out CC_FLAGS_CFI for the file.
Note that https://reviews.llvm.org/rG3b14862f0a96 fixed the issue
where LLVM didn't emit CFI type hashes for any sanitizer constructors,
but now type hashes are emitted correctly for TUs that use CFI.
projects / people / aperard / linux.git / log