The question makes no sense, due to one simple reason: converting RGBA to RGB means exactly that: loosing alpha (which is "A", pixel opacity).
Besides, perhaps you need to know that AVI is badly obsolete; I don't know anyone who still creates new files of this format. How about something more reasonable?
Anyway, I have no idea what do you want to achieve, but I can give you some general advice.
First of all, you have to understand that things are more complicated: there is a number of different
media container types, which are roughly equivalent to the concept of "format"; each container supports some subset of
media streams of different (many, really many types, each in different versions) working with different kinds of codecs; a number of stream combine in the container with
multiplexing:
Digital container format — Wikipedia, the free encyclopedia[
^],
Comparison of video container formats — Wikipedia, the free encyclopedia[
^],
List of open-source codecs — Wikipedia, the free encyclopedia[
^],
List of codecs — Wikipedia, the free encyclopedia[
^].
Here is my advice: understand all that but better don't try to dig in it with your own code. Instead, use some available product. I would highly recommend FFMpeg or libav library, open-source products:
FFmpeg — Wikipedia, the free encyclopedia[
^],
FFmpeg[
^],
libavcodec — Wikipedia, the free encyclopedia[
^],
Libav[
^].
You can either use the available utilities, by spawning a separate process, or you can link them in your product as library; you always can link them in C++, as they are C libraries.
The products cover nearly everything; I don't know more comprehensive product to cover many different standards, types of conversions and other features.
—SA