Add a second property to the viewmodel:
public CartData SelectedCartData
and bind the Selected property of the listview in XAML making sure to set the Mode to TwoWay. This way the binding is pulling data from the cartData collection and pushing the selected value back into the viewmodel. Once this happens, you can then call:
cartData.Remove(SelectedCartData);