Click here to Skip to main content
15,868,141 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
Since my data is too large, I use pd.read_csv('',chunksize=). I am using categorical_crossentropy as my loss function, however, on the last chunk, I have just one target. So I get the error:

You are passing a target array of shape (2110, 1) while using as loss categorical_crossentropy.

What I have tried:

Now I know I can use binary_crossentropy for that last chunk. So this is what I did:

X_train, X_test, y_train, y_test = train_test_split(train_data, train_labels, shuffle=True, test_size=0.3)
if y_train.shape[1] == 1:
   loss = 'binary_crossentropy'
else:
   loss = 'categorical_crossentropy'

When I do this, I get the error:

Python
IndexError: index 1 is out of bounds for axis 0 with size 1

My data is one-hot encoded. How can I resolve this error? Thanks
Posted
Updated 26-Mar-20 3:12am

1 solution

Wild guess:
Python
if y_train.shape[0] == 1:
 
Share this answer
 
Comments
Richard MacCutchan 26-Mar-20 9:20am    
Not so wild, more considered.
Anan Srivastava 26-Mar-20 9:20am    
Hi, thank you for your response but that can't be right. The error only occurs in the last chunk. y_train.shape[0] == 1 will give me 2110. Please let me know if I should share more of my code.
phil.o 26-Mar-20 9:31am    
Most of all, you should debug your code, because you are the only one having access to the data; this error clearly states a wrong index (1) into a collection of a single element; when indices are zero-based, the index of the first and only element is zero, not one.
Anan Srivastava 26-Mar-20 9:42am    
Thank you. Have been stuck at it for a while. Will try to figure it out and turn to help if I can't
phil.o 26-Mar-20 9:45am    
Seriously, you should try debugging :) This is incredibly useful, it is even absolutely mandatory if you are serious about development. Plus, this can be extremely fun.
We can provide some links eventually to get you on track with python debugging.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900