You are creating links and therefore the text within the
a
tags will be shown as links. If you want to add separators, you have to print them outside the link tags:
$letters="abcdefghijklmnopqrstuvwxyz";
for ($i=0; $i<=35; $i++) {
$letter=substr($letters,$i,1);
if ($i)
echo " | ";
echo "<a href='babyNameFinder.php?initial=$letter'>$letter</a> ";
}