count
is a local variable, it exists only until the method exits and is recreated and reinitialised each time the method is called by a new event.
You need to make it art of your class declaration by moving it outside the method:
Public Class Form1
Private count As Integer = 0
Private Sub RichTextBox1_KeyDown(ByVal sender As Object, ByVal e As KeyEventArgs)
If (e.KeyCode = Keys.Back) Then
count = count + 1
ElseIf (e.KeyCode = Keys.Enter) Then
MessageBox.Show(count)
End If
End Sub
End Class