Pyrun_simplestring failing for special characters

Question

0.00/5 (No votes)

See more:

I was trying Basics embedding Python in C++ and when i run this code i get error
"SyntaxError: (unicode error) 'utf-8' codec can't decode byte 0xf4 in position 10
: invalid continuation byte". I am not able to figure out what's wrong in this. I encoded this txt on Pycharm it worked perfectly fine but not when i try to embed this with C++. Thanking in advance for the help

PyObject* pInt;

Py_Initialize();

PyRun_SimpleString("txt = u\"flag_for_Côte_d’Ivoire\"\nx = txt.encode()\nprint(x)");

Py_Finalize();

printf("\nPress any key to exit...\n");
if(!_getch()) _getch();
return 0;

if i only do

PyRun_SimpleString("txt = u\"flag_for_Côte_d’Ivoire\"")
it fails for that too. Python Version which i am using is Python 3.7.3

What I have tried:

I have different Encoding techniques. Couldn't find anything in Docs related to PyRun_simplestring.Python version i am using is Python3.7.3

Posted 9-May-19 0:23am

Member 13703287

Updated 9-May-19 23:09pm

Add a Solution

Comments

Richard MacCutchan 9-May-19 7:16am

Where is the string with the 0xf4 character?

Member 13703287 9-May-19 8:48am

"ô" this character is 0xf4. you can check https://www.rapidtables.com/convert/number/hex-to-ascii.html here

Richard MacCutchan 9-May-19 9:56am

Try changing the u prefix to u8.

Member 13703287 9-May-19 9:58am

I tried it is not supported in Windows i guess coz it is giving Invalid Syntax
error. I am working on VS 2008.

Richard MacCutchan 9-May-19 10:23am

VS2008 is very old now and does not support the latest C++ standard. You need to upgrade to 2017.

Member 13703287 9-May-19 10:39am

when used with Python 2.7 PyRun_SimpleString works perfectly fine and accepts same string whereas when i use with Python 3.7.3 it fails

Richard MacCutchan 9-May-19 11:54am

Sorry, but I am out of ideas on this.

Member 13703287 10-May-19 2:36am

Thank you for your help Richard. I tried with Python 3.0.0 and it failed which show this error or limitation was introduced with Python version 3.0.0 for embedding with C++

Richard MacCutchan 10-May-19 5:10am

See my solution below.

2 solutions

Solution 1

Convert it before the call. There's no point in doing it in the call because it buys you nothing and you're not controlling "that" environment (regarding assumptions about that encoding).

Quote:
When I discovered that the popular web development tool PHP has almost complete ignorance of character encoding issues, blithely using 8 bits for characters, making it darn near impossible to develop good international web applications, I thought, enough is enough.

The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) – Joel on Software[^]

Posted 9-May-19 6:16am

Gerry Schmitz

Comments

Member 13703287 10-May-19 5:24am

Thank you Gerry Schmitz for ur reply but converting before won't help. Pyrun_simplestring accepts only "simpleString"

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Richard MacCutchan · Accepted Answer · 2019-05-09T23:09:00

Well I cannot believe that this works in 2 but not in 3. So, I just tried this again with Python 3.7 and it works perfectly. What I did notice, which was different from my tests yesterday, was the addition of the line

#define PY_SSIZE_T_CLEAN

before the include of Python.h, as described at Introduction — Python 3.7.3 documentation[^].

So my source code now reads:

C++

#define PY_SSIZE_T_CLEAN
#include <Python.h>

int main()
{
//    PyObject* pInt;  -- not used

    Py_Initialize();

    PyRun_SimpleString("txt = u\"flag_for_Côte_d’Ivoire\"\nx = txt.encode()\nprint(x)");
    PyRun_SimpleString("txt = u\"flag_for_Côte_d’Ivoire\"\nprint(txt)"); // and text

    Py_Finalize();

    printf("\nPress any key to exit...\n");
}

And here is the output:

C:\Users\rjmac\Documents\Code\C++>pytest
b'flag_for_C\xc3\xb4te_d\xe2\x80\x99Ivoire'
flag_for_Côte_d’Ivoire

Press any key to exit...