c - How to encrypt data using RSA, with SHA-256 as hash function and MGF1 as mask generating function?

Question

I was doing some experiments with cryptography. Now I have the public key of receiver and i want to encrypt some data and pass to the receiver.

I want to use RSAES-OAEP algorithm. with SHA-256 as hash function and MGF1 as mask generation function.

I want do this using openssl. I found a function RSA_public_encrypt() with this function we can specify the padding. One of the padding option available was

RSA_PKCS1_OAEP_PADDING
EME-OAEP as defined in PKCS #1 v2.0 with SHA-1 , MGF1 .

they are using sha-1.

I want to reconfigure the function to use SHA256 as hash function ans MGF1 as hash function. How can I do it ?

Answer 1

The following excerpt allows using OAEP with SHA256 for both the MGF and hash function. Tested with OpenSSL 1.0.2L

int flags = CMS_BINARY | CMS_PARTIAL | CMS_KEY_PARAM;
cms = CMS_encrypt(NULL, in, cipher, flags)
ri = CMS_add1_recipient_cert(cms, cert, flags);
pctx = CMS_RecipientInfo_get0_pkey_ctx(ri);
EVP_PKEY_CTX_set_rsa_padding(pctx, RSA_PKCS1_OAEP_PADDING);
EVP_PKEY_CTX_set_rsa_oaep_md(pctx, EVP_sha256());
EVP_PKEY_CTX_set_rsa_mgf1_md(pctx, EVP_sha256());