You need to see what the underlying SQL being generated by entity framework is doing. Then, take that same sql and run it through sql server (assumingly) with query analyzer turned on to see where the bottle necks or missing indexes (if any) are.
To log the sql being generated by entity framework you'd need to do something like this. Assuming you are using log4net for example.
var log = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
db.Database.Log = log.Info;
db.Database.Log = Debug.WriteLine;
This would then output the generated sql to a log file or the debug output window.
From there, take that sql, run it in management studio, anaylze the query and adjust your linq query from there. Entity framework/linq is notorious for creating some nasty sql that doesn't make sense so I have a feeling a lot of your null checking you are doing in your query is generating a gigantic sql statement that is eating at your CPU.