If, as Shmuel Zang suggests, you have a handle to this file (i.e. a FileStream, StreamWriter, TextWriter etc) open, you won't be able to delete it until after you close the handle. So in the Closed event you should do something like
FileStream fs;
void FormClosed(object sender, EventArgs e){
fs.Close();
File.Delete(filename);
}
Note that (contrary to some of the solutions above) you should use the Form
Closed event for terminal tidy-up tasks like this, not FormClosing. FormClosing is called when a close
request is received (i.e. the user presses Alt+F4, or code calls Close), and any handler attached to the form can reject it and cause the form not to be closed after all. FormClosed is called when the form has actually been closed, and there is no chance it won't be closed. That's the point where you want to clear things up which you've created during the course of the application. (Well, either there or in your Main method after the Application.Run.)