The explanation about RSA found in the body of the question, if it is what you really mean, shows that the question formulated in the title of the question is incorrect. Besides, it would be good to know why you are asking for alternatives, that is, what's wrong with OpenSSL. Please see my comment to the question.
It's hard to understand what you don't understand, but you really need to understand the subject. So, first of all, please see:
Public-key cryptography — Wikipedia, the free encyclopedia[
^] (first of all!),
RSA (cryptosystem) — Wikipedia, the free encyclopedia[
^].
Let me reiterate it: this forum is not the code-finding service. What you need to do is: try to understand what you really want and why, learn some basics of cryptography, learn the possibilities cryptography offers and then correct your understanding of what you want. When this is done, you will be more familiar with the terminology, which will help you to perform Web search and find available implementations.
Possible alternatives include cryptlib, Crypto++ and LibRSA:
https://www.cs.auckland.ac.nz/~pgut001/cryptlib[
^],
Crypto++ Library 5.6.3 — a Free C++ Class Library of Cryptographic Schemes[
^],
https://code.google.com/archive/p/librsa[
^].
—SA
Submit an article or tip