Click here to Skip to main content
15,890,512 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I am running the following code, which gives me the tweets that contain the word cat, however at some points, I get an error

The code is:

Python
import tweepy
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
import json
import sentmod as s

#consumer key, consumer secret, access token, access secret.
ckey= "xxxxx"
csecret="xxxx"
atoken="xxxxx"
asecret="xxxxx"

class listener(StreamListener):
    
    def on_data(self, data):
        all_data = json.loads(data)
        tweet = all_data["text"]       
        sentiment_value, confidence = s.sentiment(tweet)
        tweet.encode('utf-8', 'ignore')
        if "RT" in tweet:
            pass
        else:
            tweets=open("tweets.txt","a",encoding="utf-8")
            tweets.write(tweet)
            tweets.write('\n')
            tweets.write(str(sentiment_value))
            tweets.write('\n')
            tweets.write(str(confidence))
            tweets.write('\n\n\n')
            tweets.close()
            print(tweet, sentiment_value, confidence)
            if confidence*100 >= 60:
                output = open("twitter-out.txt","a")
                output.write(sentiment_value)
                output.write('\n')
                output.close()
                return True


    def on_error(self, status):
        print(status)

auth = OAuthHandler(ckey, csecret)
auth.set_access_token(atoken, asecret)

twitterStream = Stream(auth, listener())
twitterStream.filter(track=['Cat'],languages=['en'])  #locations=[]





I get the following error after a few tweets:



print(tweet, sentiment_value, confidence)
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 44-44: Non-BMP character not supported in Tk

What I have tried:

decoding and encoding with utf-8 and utf-16, but did not work
Posted
Updated 8-Aug-19 11:15am

I suggest a google search would be your best route to go. Given that I am not about to run your code to see the tweet dump it creates, the highly likely issue is the data you are getting are emoji's (which i read in the first 5 minutes of googling) and are not supported in the UCS-2 codec.

python - 'UCS-2' codec can't encode characters in position 1050-1050 - Stack Overflow[^]

"Non-BMP character" Unicode error · Issue #16 · geduldig/TwitterAPI · GitHub[^]

Can't encode Non-BMP character · Issue #624 · tweepy/tweepy · GitHub[^]

UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 44-44: Non-BMP character not supported in Tk - Google Search[^]
 
Share this answer
 
Comments
Member 13647869 8-Feb-18 1:10am    
I know that. I have been googling for the past two days, but every time i get a method it doesn't work, its ok though thanks
David_Wimbley 8-Feb-18 2:52am    
If you knew that why didnt you include any of that in what you've tried. The first link "You could use a translation table to map everything outside of the BMP to the replacement character:" seems to be what would solve your problem yet your code does not reflect that.
Member 13647869 8-Feb-18 2:57am    
I knew that there are answers on google. ok
David_Wimbley 8-Feb-18 4:25am    
Then what are you looking for? That first link seems to be what would resolve your issue.
My problem was that I was encoding the tweets using utf-8, however, the correct encoding is utf-16be which I took from this link:
 
Share this answer
 

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900