tree 25566beb1ca5919d3ef4bd04e73e9bacb677ac0e
parent 8434b4373058018ee5b79f06f2cafd6b7c0c15e1
author Heiko Stuebner <heiko.stuebner@theobroma-systems.com> 1594155446 +0200
committer Tom Rini <trini@konsulko.com> 1594243306 -0400

lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()

When calculating rrtmp/rr rsa_gen_key_prop() tries to make
(((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and
(((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[]
with rlen being num_bits * 2

On a 4096bit key this comes down to to 257 uint32_t elements
in rr and 256 elements in rrtmp but with the current allocation
rr and rrtmp only have 129 uint32_t elements.

On 2048bit keys this works by chance as the defined max_rsa_size=4096
allocates a suitable number of elements, but with an actual 4096bit key
this results in other memory parts getting overwritten.

So as suggested by Heinrich Schuchardt just use the actual bit-size
of the key as base for the size calculation, in turn making the code
compatible to any future keysizes.

Suggested-by: Heinrich Schuchardt <xypron.debian@gmx.de>
Signed-off-by: Heiko Stuebner <heiko.stuebner@theobroma-systems.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

rrtmp needs 2 + (((*prop)->num_bits * 2) >> 5) array elements.

Reviewed-by: Heinrich Schuchardt <xypron.glpk@gmx.de>