A key question that's relevant here, I believe: are you deaf, yourself, and brought-up in a home where sign-language was used, or are you a non-deaf person who has studied sign-language ? Do you have any practical experience working with, or being around, deaf people ?
I think the first thing you should realize is that a formal, highly evolved, sign language, like American Sign Language (ASL), is a really another language in itself with all the richness and subtlety of any of the world's languages. ASL may have begun as an "imitative" language mapping English to gesture, but it has since evolved to a level far beyond that.
There are significant differences between children born-deaf, who are raised by parents who can hear (and perhaps learned some ASL after their children were born), and children born-deaf raised by deaf parents who communicated via ASL. For the second category, ASL is as much their "mother tongue" as your "mother tongue."
And now, the cumulative efforts of several generations of "native speakers" of ASL, have developed ASL further and further, probably beyond what any hearing person can imagine, except through "artistic" translations.
The project you describe, unless you limit it to a very small subset of common goal-oriented communication tasks, would be as enormous a project as real-time translation of Chinese to English.
I suggest you start here: [
^] to get a sense of what ASL is.
I also suggest you read Dr. Steven Pinker's (MIT Professor of Linguistics) book "The Language Instinct" to orient yourself to modern linguistic theory which has shaped profoundly by neuro-science, and to begin to inform yourself of the innate structure in the human child (what Chomsky calls the "Language Acquisition Device" ... LAD ...) that, in a unique developmental window of time, triggers a phenomenally rapid cascade of learning/internalizing complex grammar in the growing child's life: which is what creates a "native speaker" of a "mother tongue."
And, what's stopping you from using Google to examine how professional translators of speech to ASL learn, and are credentialed ? Aren't those rare people the "models" you would use as a basis for any conceptual experimentation with the idea of a computerized avatar ?
good luck !
Submit an article or tip