I think you need to start breaking your problem into smaller, manageable units of work
1) Split the text into words/token, where tokens may be space, punctuation)
2) look at using a RTF 'Builder' to build the RTF Text - eg
RichText Builder (StringBuilder for RTF)[
^]
3) for each word (possibly ignoring simple words), spell-check the word
4) if the spell-check says the word is bad, add the word with it's underline sequence using the RTFBuilder else add the word, remembering to add the space/punc tokens
5) at the end of the words (that) builder has a ToString(), that can then be used as the source for the RichTextBox
This is an iterative process - (3) for example may break down into a number of sub-steps, but you could create a simple dictionary/list of good words and test with your words first