Incorrect Failure Handling in OpenSSL RSA KEM RSASVE Encapsulation

CVE-2026-31790 is an OpenSSL vulnerability in RSA KEM RSASVE encapsulation. Applications using EVP_PKEY_encapsulate() with RSA/RSASVE can disclose stale or uninitialized contents of a caller-provided ciphertext buffer to a malicious peer when encapsulation is attempted with an attacker-supplied invalid RSA public key that has not been validated first. The root cause is incorrect failure handling around RSA_public_encrypt(): the affected code checks only whether the return value is non-zero, even though RSA_public_encrypt() returns the number of bytes written on success and -1 on error. Because -1 is non-zero, an encryption failure can be misinterpreted as success, output lengths can be set, and the caller may transmit buffer contents as though a valid KEM ciphertext had been produced. The issue affects OpenSSL 3.0, 3.3, 3.4, 3.5, and 3.6, and the corresponding FIPS modules in 3.0, 3.1, 3.3, 3.4, 3.5, and 3.6 are also affected.