Encoding streamReader to hebrew - vb.net

Question

5.00/5 (1 vote)

See more:

hey guys, s.o.s!!!
i'm developing a program which conect to my company support email (gmail) using winsock
(System.Net.Sockets). i am new regarding to this subject. aniway, i manage to connect to the
e-mail address and pass threw the ssl. i can see how many inbox i have and how many unread.
now, the problem is in the encoding of the text. i tryed using utf-7,8,32 and what-so-ever and still nothing. my main problem is that the emails is written in Hebrew and the encoding returns me all kind of wierd leters (gibrish). i am putting the function which handle the reading and the StreamReader here. pleaseeee help, thanks!

VB

Dim m_buffer() As Byte
Dim m_sslStream As SslStream

    Sub GetEmails(ByVal Server_Command As String)
        'Dim m_buffer() As Byte = System.Text.Encoding.ASCII.GetBytes(Server_Command.ToCharArray())
        Dim m_buffer() As Byte = System.Text.Encoding.GetEncoding("iso-8859-8").GetBytes(Server_Command.ToCharArray())
        Dim stream_Reader As StreamReader
        Dim TxtLine As String = ""
        Try
            m_sslStream.Write(m_buffer, 0, m_buffer.Length)
            stream_Reader = New StreamReader(m_sslStream)
            Do While stream_Reader.Peek() <> -1
                TxtLine += "***********" & vbNewLine & stream_Reader.ReadLine() & vbNewLine
            Loop
            TextBox1.Text = TxtLine
        Catch ex As Exception
            MsgBox(ex.Message)
        End Try
    End Sub

[After long discussions below, I think comprehensive answers are finally ready by now, Solutions 1-2 — SA]

Posted 10-Aug-14 7:51am

oronsultan

Updated 12-Aug-14 9:04am

Sergey Alexandrovich Kryukov

v3

Add a Solution

Comments

Sergey Alexandrovich Kryukov 10-Aug-14 14:10pm

Who told you that the input is in so outdated encoding as ISO-8859-8? It's not very likely. You need to know input encoding exactly, or just try all which can apply. As to UTF-s, such encodings as UTF-16 come in two variants UTF-16LE and UTF-16GE (different endianess)...
Besides, Hebrew text in some encoding could be further encoded to represent ASCII (does input visually look like ASCII?)... And so on...
—SA

oronsultan 10-Aug-14 14:15pm

the "ISO-8859-8" is only one of my trials. i tryed "Windows-1255" also. can u tell me how to encode it currectly?

Sergey Alexandrovich Kryukov 10-Aug-14 14:31pm

Why "to encode"? We don't know who to decode, that's the problem...
—SA

oronsultan 10-Aug-14 16:01pm

so i understand that u the expert of this month and a legend in "code project" but so far u didnt answerd my question (althaugh u dont have to). all i am asking is how can i read my hebrew emails in a clear way? pleaee, answer this and not arround this. if u can attach a code it will be great.

Sergey Alexandrovich Kryukov 10-Aug-14 16:52pm

I cannot. The only problem is that we are not aware of the way the data was encrypted in first place. With e-mail: such thing used to happen: different encoding layer one on top of another; because some agent in a middle assumes some encoding and recode text based on that assumption. It creates a big mess. Even if you sent me some code sample, if would be difficult to me because I don't read Hebrew.

However... could you read and post the sample of the mail headers? In case of multi-part mail, the headers of the parts containing text...

—SA

oronsultan 10-Aug-14 17:06pm

first of all, thanks for your reply.
seconed, i hope i understood u well. here is an example of encrypted hebrew headers:
"Y2hla2luZ9eQ15HXkteT15TXldeW15fXmNeZMTAyMDMNCg=="
as u can see, its not hebrew.
third, i want to understand what u saying: in order to encode the email into hebrew, i need to figure out the source of encrypted text? i am confused... please take it easy on me, thie subject is new for me.

Sergey Alexandrovich Kryukov 10-Aug-14 21:57pm

I did not say "figure out the source of encrypted text". If somebody told me something like that, I would be confused myself, would ask for clarification.
What you show is not a header; I don't know what is that. Header starts from a new line, is separated by ": " from its value; blank space at the beginning of a line means continuation of a header value from a previous line. I would be interested to see "Content-type" headers, maybe "Content-Transfer-Encoding", "Content-Disposition"...

I have another idea. If you could have the whole content of the mail message block, exactly as it arrived (how did you obtain the data, anyway?). Then you could save it in a file and named it as *.eml, and open it in any mail client (MS Outlook Express is good, but Outlook probably not, a very good option is open-source Mozilla Trunderbird) and see what happens. In other words: is there any ways to see the and read the mail messages?

I just notices SSL. If, by some reason, you got some package in the middle, not decrypted from SSL, without the key, the data could be considered as totally lost.

—SA

oronsultan 11-Aug-14 1:31am

dear SA, here is the full email as i see it:
+OK message follows
Delivered-To: minisoft365@gmail.com
Received: by 10.60.138.134 with SMTP id qq6csp59048oeb;
Sun, 10 Aug 2014 06:59:18 -0700 (PDT)
Return-Path: <oronsultan@gmail.com>
Received-SPF: pass (google.com: domain of oronsultan@gmail.com designates 10.182.137.195 as permitted sender) client-ip=10.182.137.195
Authentication-Results: mr.google.com;
spf=pass (google.com: domain of oronsultan@gmail.com designates 10.182.137.195 as permitted sender) smtp.mail=oronsultan@gmail.com;
dkim=pass header.i=@gmail.com
X-Received: from mr.google.com ([10.182.137.195])
by 10.182.137.195 with SMTP id qk3mr23322017obb.5.1407679158835 (num_hops = 1);
Sun, 10 Aug 2014 06:59:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:date:message-id:subject:from:to:content-type;
bh=GiapFM9Ie1h9ObO0ZnJK9hU3mvtfNKhqOkGm3iLhnig=;
b=oQuJ959hlOKTI3hpIMaScfxUWVxVX5hakoF6XC5HD+ZG7Z/KD0LIiJY0Q6Jt8VnTeL
p3OanHvllBAvM0DXvgDHnpLzmwfIJCJHVgEtbMeKNMDB2Y+4pCfFDS1SNk4tMiEdjcgW
lNfJX2MFlwrUpu4uJipJksJ6JQnklNgNA0R5sdnF5c8kIqJPML2vWGIITJsIzM1zWRgm
ywOpOlHKRfSNfsPS6b8Pl37iBM9uiKCadQ7pvOp91LvsnyisEN3qQdhlj5q7G1rCQNcD
agoCZRq0EPFdFCsQVXRAQ1eop2nUBmGg1iscRjxT4AtK639VA9MTKjBQye7aLrtVPtzc
h8uw==
MIME-Version: 1.0
X-Received: by 10.182.137.195 with SMTP id qk3mr23322017obb.5.1407679158830;
Sun, 10 Aug 2014 06:59:18 -0700 (PDT)
Received: by 10.76.153.200 with HTTP; Sun, 10 Aug 2014 06:59:18 -0700 (PDT)
Date: Sun, 10 Aug 2014 16:59:18 +0300
Message-ID: <cacgdqkv7rsec10s_r7xghzt5zkcxdmpo1pgp0fct=kkh+cou0a@mail.gmail.com>
Subject: =?UTF-8?B?15HXk9eZ16fXlA==?=
From: =?UTF-8?B?15DXldeo15XXnyDXodeV15zXmNef?= <oronsultan@gmail.com>
To: oron sultan <minisoft365@gmail.com>
Content-Type: multipart/alternative; boundary=001a11c361f6384c8e050046d84b

--001a11c361f6384c8e050046d84b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: base64

Y2hla2luZ9eQ15HXkteT15TXldeW15fXmNeZMTAyMDMNCg==
--001a11c361f6384c8e050046d84b
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: base64

PGRpdiBkaXI9InJ0bCI+Y2hla2luZ9eQ15HXkteT15TXldeW15fXmNeZMTAyMDM8L2Rpdj4NCg==
--001a11c361f6384c8e050046d84b--
.
if you could explain your last sentence i will appriciate it.

Sergey Alexandrovich Kryukov 11-Aug-14 2:36am

Ah, I did not know you have it all. The way you put your question suggested that information on encodings and other detail was lost. And forget my "last sentence" — the message is not encrypted. No problem at all; I answered your question. If you need to clarify something, don't hesitate to ask some follow-up questions.
—SA

oronsultan 11-Aug-14 3:04am

what should i put instead of the "inputFileName"?

Sergey Alexandrovich Kryukov 11-Aug-14 3:29am

I already answered. Look carefully. Input file is not the file of the string you decode. Read the same file, but first break it into the parts using empty line and the value of "boundary" (read it first from appropriate header). Then, for each part, read the headers and see how to read the content of each part. And only then, extract content of the part. And then see by header. I described the case when you base64-decode text to get UTF-8 data. This fragment (after base64-decoding) will go to UTF8.GetChars to get you a string. And so on...

In other words, parse everything combining it with Encoding operations and base64 decoding (there are more different presentations of data in mail message packages).

Or use OpenPop.NET; it has a demo application parsing a message, probably, from POP3, but, for study purposes, you can easily modify it to parse from EML files...

—SA

oronsultan 11-Aug-14 6:35am

Dear SA,
i am very sorry to trouble you once again. during the last hours i tryed over and over to find the answer according to your soloution and got nothing. i have difficult to understand all the process. for a start, am i suppose to divide the income email to many part according to the encoding of them? does one have a utf-8 when other is base64? i tryed to built a function according to the MSDN link you gave me and nothing. can you please, please, please show me how the code need to look like? i am loosing my self here.
anyway, this is my result so far:
"+OK message follows
Delivered-To: minisoft365@gmail.com
Received: by 10.60.138.134 with SMTP id qq6csp59048oeb;
Sun, 10 Aug 2014 06:59:18 -0700 (PDT)
Return-Path: <oronsultan@gmail.com>
Received-SPF: pass (google.com: domain of oronsultan@gmail.com designates 10.182.137.195 as permitted sender) client-ip=10.182.137.195
Authentication-Results: mr.google.com;
spf=pass (google.com: domain of oronsultan@gmail.com designates 10.182.137.195 as permitted sender) smtp.mail=oronsultan@gmail.com;
dkim=pass header.i=@gmail.com
X-Received: from mr.google.com ([10.182.137.195])
by 10.182.137.195 with SMTP id qk3mr23322017obb.5.1407679158835 (num_hops = 1);
Sun, 10 Aug 2014 06:59:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:date:message-id:subject:from:to:content-type;
bh=GiapFM9Ie1h9ObO0ZnJK9hU3mvtfNKhqOkGm3iLhnig=;
b=oQuJ959hlOKTI3hpIMaScfxUWVxVX5hakoF6XC5HD+ZG7Z/KD0LIiJY0Q6Jt8VnTeL
p3OanHvllBAvM0DXvgDHnpLzmwfIJCJHVgEtbMeKNMDB2Y+4pCfFDS1SNk4tMiEdjcgW
lNfJX2MFlwrUpu4uJipJksJ6JQnklNgNA0R5sdnF5c8kIqJPML2vWGIITJsIzM1zWRgm
ywOpOlHKRfSNfsPS6b8Pl37iBM9uiKCadQ7pvOp91LvsnyisEN3qQdhlj5q7G1rCQNcD
agoCZRq0EPFdFCsQVXRAQ1eop2nUBmGg1iscRjxT4AtK639VA9MTKjBQye7aLrtVPtzc
h8uw==
MIME-Version: 1.0
X-Received: by 10.182.137.195 with SMTP id qk3mr23322017obb.5.1407679158830;
Sun, 10 Aug 2014 06:59:18 -0700 (PDT)
Received: by 10.76.153.200 with HTTP; Sun, 10 Aug 2014 06:59:18 -0700 (PDT)
Date: Sun, 10 Aug 2014 16:59:18 +0300
Message-ID: <cacgdqkv7rsec10s_r7xghzt5zkcxdmpo1pgp0fct=kkh+cou0a@mail.gmail.com>
Subject: =?UTF-8?B?15HXk9eZ16fXlA==?=
From: =?UTF-8?B?15DXldeo15XXnyDXodeV15zXmNef?= <oronsultan@gmail.com>
To: oron sultan <minisoft365@gmail.com>
Content-Type: multipart/alternative; boundary=001a11c361f6384c8e050046d84b

--001a11c361f6384c8e050046d84b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: base64

Y2hla2luZ9eQ15HXkteT15TXldeW15fXmNeZMTAyMDMNCg==
--001a11c361f6384c8e050046d84b
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: base64

PGRpdiBkaXI9InJ0bCI+Y2hla2luZ9eQ15HXkteT15TXldeW15fXmNeZMTAyMDM8L2Rpdj4NCg==
--001a11c361f6384c8e050046d84b--
.
"
and this is my functions (only the first one work and active):
Sub GetEmails(ByVal Server_Command As String)
Dim m_buffer() As Byte = System.Text.Encoding.ASCII.GetBytes(Server_Command.ToCharArray())
Dim stream_Reader As StreamReader
Dim TxtLine As String = ""
Try
m_sslStream.Write(m_buffer, 0, m_buffer.Length)
stream_Reader = New StreamReader(m_sslStream, Encoding.UTF8)
Do While stream_Reader.Peek() <> -1
TxtLine += stream_Reader.ReadLine() & vbNewLine
Loop
TextBox1.Text = TxtLine
Catch ex As Exception
MsgBox(ex.Message)
End Try
End Sub
Public Sub EncodeWithString(x As String)
Dim inFile As System.IO.FileStream
Dim binaryData() As Byte

Try
inFile = New System.IO.FileStream(x, System.IO.FileMode.Open, System.IO.FileAccess.Read)
ReDim binaryData(inFile.Length)
Dim bytesRead As Long = inFile.Read(binaryData, 0, inFile.Length)
inFile.Close()
Catch exp As System.Exception
' Error creating stream or r

Sergey Alexandrovich Kryukov 11-Aug-14 11:25am

This is the key: the parts are encoded differently. This trick would be impossible if say, one part was UTF-8, another would be UTF-16. But this never happens. First of all, base64 can be combined with UTFs. Why is this possible? Because the whole file is reduced to either to ASCII or to UTF-8 (usually without BOM bytes). Both UTF-8 and ASCII are byte-oriented, only UTF-8 use more then one byte for certain characters and 1 for others. The stream can freely change from ASCII to UTF-8 and back without disturbing the readability of the file; you only need to know boundaries, but they are marked.

You case if even simpler: the whole file is reduced to ASCII; and you should read it as ASCII. Every part which is not ASCII is base64-encoded to make it ASCII. This is what I explained in my answer.

—SA

Sergey Alexandrovich Kryukov 11-Aug-14 11:34am

In your code, I have no idea what is Server_Command, why do you ever convert anything to char array and why do you write to ssl stream it as a buffer. Your question as about reading your mail, so I would prefer concentrating on reading it and getting some strings from its content. This is what was your original question. Let's concentrate on it without writing anything. Is your source of data some file, or you get the package directly from stream? In both cases, you obtain correct quite readable mail package for each message. Now, you have to read it as ASCII and break into parts, and then interpret each part. After interpretation, each part will be either string, or some binary stream (this is how a file is "attached").
In principles, some of the files may come in UTF-8, but this is not a usual case, but the fragment "charset=UTF-8" will be detected even if you read in ASCII... And, finally, you can read the file in UTF-8, just in case, because all ASCII files are read correctly as UTF-8.

—SA

oronsultan 12-Aug-14 3:33am

Dear Sergey,
first, i must give my appreciation regarding your attention toards my problem.
thanks to your guidence, i mangae (finally) to break the email to parts and handle each part. i also manage to convert the body of the email (the message itself) to hebrew and it looks great. however, in a wierd way, the "subject" part is a pickle. all i get in the subject, instead of hebrew is this:
"=?UTF-8?B?15HXk9eZ16fXlA==?="
my main thaught is that it is encoded in utf-8 so i tryed to convert it like this:
Dim byt As Byte() = System.Text.Encoding.UTF8.GetBytes(strData)
Convert.ToBase64String(byt)
and then:
System.Convert.FromBase64String(strData.Replace(vbCrLf, vbNullString))
and finally:
Encoding.UTF8.GetChars(DecodeBase64ToBytes(strData))
but no luck. can u please tell me how can i find the encoded source of the "subject" and how can i decode it to hebrew?
thanks again,
oron.

Sergey Alexandrovich Kryukov 12-Aug-14 10:40am

Ah, I forgot to tell you about this format. A while ago I easily found how to decode it. Please wait a bit; I hope after this part you will be done.
—SA

Sergey Alexandrovich Kryukov 12-Aug-14 11:20am

Okay, thank you for reminding. This is a different format defined for encoding headers, but the idea is very similar to the one for a part: encoding and transfer-encoding. I answered it in Solution 2.

I hope you are done now and will accept both solutions formally (green "Accept" buttons), will you?

In all cases, your follow-up questions will be welcome.

—SA

oronsultan 13-Aug-14 0:31am

that was amazing!!! you are great!!! thanks a thousand times!!!

Sergey Alexandrovich Kryukov 13-Aug-14 0:56am

You are very welcome, Oron.

By the way, I decided to up-vote the question, which I do very rarely. Unlike majority (unfortunately) "question" posts on this forum, not only this is a sensible question which makes sense (at least after discussion and clarifications), but you show the decent collaborative behavior and just adequate attention and efforts. It's useless to try to help people who don't try hard enough to help themselves.

Good luck, call again.
—SA

oronsultan 13-Aug-14 3:32am

I'm flattered. it was great reading this feedback coming from an expert in the field.
but still, The fat lady has yet to sing.
in solution num 2, i manage to encrypt the text but it seems that the new converted string is cut in the middle. for example, this RFC 2047:
"=?UTF-8?B?157Xmdeg15nXodeV16TXmCAtINee15fXnNen16og16rXnteZ15vXlCDXldep15nXqNeV?="
returns a good translation to hebrew but got cut after two anf a half word. Is problem familiar to you?
and here is the full code:
Info.SubjectData = Trim(Mid(S, I + 1)) 'stuff to structure
Dim str As String = Info.SubjectData
Dim x As Integer = str.IndexOf("=")
Dim fromName As String = str.Substring(x + 1, str.IndexOf("=", x + 1) - x - 1)
fromName = "=" & fromName & "="
str = System.Net.Mail.Attachment.CreateAttachmentFromString(String.Empty, Info.SubjectData).Name

Sergey Alexandrovich Kryukov 13-Aug-14 3:59am

Maybe you did not get it. This function works with the whole encoded string.
Try this:


string wholeThing =
"=?UTF-8?B?157Xmdeg15nXodeV16TXmCAtINee15fXnNen16og16rXnteZ15vXlCDXldep15nXqNeV?="; 
string value =
   System.Net.Mail.Attachment.CreateAttachmentFromString(
        String.Empty,
        wholeThing).Name;

In other words, no need to trim anything or parse. This method does it all for you; see also Solution 2; you could have misread the code sample a bit.

—SA

oronsultan 13-Aug-14 4:14am

did that. no good. let me parse it for u:
first, i remove the word "subject" from the next statment:
s = "Subject: =?UTF-8?B?157Xmdeg15nXodeV16TXmCAtINee15fXnNen16og16rXnteZ15vXlCDXldep15nXqNeV?="
and that goes like this:
Info.SubjectData = Trim(Mid(S, I + 1))
then, comes the next code that extract only the rfc 2047 text, add the "=" at the begining and the end and encode it:
Dim str As String = Info.SubjectData
Dim x As Integer = str.IndexOf("=")
Dim fromName As String = str.Substring(x + 1, str.IndexOf("=", x + 1) - x - 1)
fromName = "=" & fromName & "="
str = System.Net.Mail.Attachment.CreateAttachmentFromString(String.Empty, Info.SubjectData).Name
what is my problem?

Sergey Alexandrovich Kryukov 13-Aug-14 4:53am

You can find the problem under the debugger, but why trimming again? Just remove "Subject: ".Length characters at the beginning of s. By the way, does s itself work in this call? — just try it. Please remind me this issue a bit later...
—SA

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sergey Alexandrovich Kryukov · Accepted Answer · 2014-08-10T20:31:00

This is pretty simple. First, divide a message by parts. This header tells you some unique delimiter line:

Content-Type: multipart/alternative; boundary=001a11c361f6384c8e050046d84b

The delimiter is empty line followed by the value of "boundary".

Now, some headers are encoded with UTF-8, for example

Subject: =?UTF-8?B?15HXk9eZ16fXlA==?=

Three headers usually shown by a mail client will show:

From: אורון סולטן
To: oron sultan
Subject: בדיקה

Does it make sense to you?

Now, two parts at the end don't give any readable text, but I'll tell you how to read them. Look at those headers:

Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: base64

It means that the plain-text content is HTML formatted, written in UTF-8 and then base64-encoded to form an ASCII text, not readable but tolerable by legacy mail systems, for extra reliability. Indeed, this is what is usually done instead of directly using UTF-8. To get original text, just base64-decode it. This is how: http://msdn.microsoft.com/en-us/library/dhx0d524%28v=vs.110%29.aspx[^].

Before using trial-and-error approach, just inspect the message and learn if you see some unknown terms. Here is what happened: the misleading information in your question was the story of your attempting of different encoding, which suggests that you did not have information on encoding and could not see what's in the messages. In fact, you have all the headers, which makes the problem quite trivial.

If you want a comprehensive interpretation of each and every detail of a message, you can use the open-source library OpenPOP.NET:
http://sourceforge.net/projects/hpop[^].

This library is not very well written, but it thoroughly follows many standards like MIME, the information which is difficult to put together.

That's all.

—SA

Sergey Alexandrovich Kryukov · Accepted Answer · 2014-08-12T05:17:00

Oron Sultan asked:
…however, in a weird way, the "subject" part is a pickle. all i get in the subject, instead of Hebrew is this:
"=?UTF-8?B?15HXk9eZ16fXlA==?="…

Yes, this is a missing part. The remaining problem is the header values like "=?UTF-8?B?15HXk9eZ16fXlA==?=" (thanks for reminding me). Let me explain them.

This is the encoding defined by RFC 2047 and is used to represent any non-ASCII data as ASCII, in a value of a mail header: http://tools.ietf.org/html/rfc2047[^].

In this format, all string is sandwiched in pair of '=', and '?' delimit the following terms:
1) encoding (UTF-8), 2) transfer-encoding (B, "binary"), encoded text ("15HXk9eZ16fXlA=="). In your case, it means the same encoding technique as I described for a part in your multi-part sample: a UTF-8 text is base64-encoded. For other detail, please see RFC 2047.

Decoding of this form is available in OpenPOP.NET referenced above. But you can avoid using 3-rd parties (and doing it all by yourself, using Encoding and base64 algorithm after parsing the subject line by '=' and '?', which is also not so hard to do) and do it in a very simple way. Here is my "secret weapon": in .NET FCL, this string can be decoded as "attachment name". For example:

C#

string headerValue = System.Net.Mail.Attachment.CreateAttachmentFromString(
    string.Empty,
    "=?UTF-8?B?15HXk9eZ16fXlA==?=").Name;

Please see: http://msdn.microsoft.com/en-us/library/system.net.mail.attachment.createattachmentfromstring%28v=vs.110%29.aspx[^].

—SA