Sorry for not giving you a positive answer. The answer has to be negative, because you just demonstrated entering such domain where the FSM approach would be absurd, and also the domain of the non-testable algorithms.
Yes, many algorithms are not testable by definition, and many are testing-
infeasible. For some of such algorithms infeasible for testing, some stochastic testing may make sense, but not for all of them; and, anyway, there is no feasible exhaustive testing. To give one obvious example of non-testable algorithm, consider the problem of testing if some cryptographic algorithm is
cryptographically strong or not. Of course, if you know that the algorithm has the flaw, it could be theoretically possible to find out just one counter-example which is computationally feasible to demonstrate the flaw. But it such counter-example is not found, it is infeasible, just because the goal of the algorithm is to make reverse operation infeasible. Of course, the projects of testing of the cryptographic algorithms exist, but the result can be used as a proof only in the case when algorithm is successfully broken. And there are the algorithms which are not testable in principle.
But why do I think you are coming to this situation? First of all, let's consider FSM approach. FSM approach is really oriented to exhaustive testing. FSM is not "based on a variable". It is based on the set of states, and the set of transitions can be understood as the subset of Cartesian square of the set of states. But in practice, it should rather be a function over this subset. All the elements of the Cartesian square (maybe not only its subset) may carry some useful information; for example, the cell representing invalid transition may carry information explaining the reason for prohibiting the transition (for example, you should not move the door into open position when the lock is not open, you should not close the door when operator's hand is detected inside the machine). Something like that.
Now, consider the set of some Length values. The absurdity of FSM becomes especially apparent if you use floating-point value, such as double. Apparently, you have 2⁶⁴ values (some of them are semantically identical, such as different representations of NaN, but it does not change the picture). The Cartesian square will have 2⁶⁴*2⁶⁴ cells. If this is not enough, consider more realistic case, when the state is described with x, y, z, θ coordinates. Not enough? It could be easily some 100 degrees of freedom. I'll leave for your home exercise to calculate the number of states in such a FSM and the feasibility of testing. But the problem is not even the number of states, the idea is the principle: even though the set of floating-point values is formally a finite set, the idea of such data type is to represent the set of
real numbers, with some limited accuracy. That model is finite, but modeled object is not only infinite set, it is also a continuum (do you understand the difference? you need to). That makes the criteria of testing itself quite fuzzy. That's why many floating-point algorithms are not only infeasible for testing, but not testable in principle.
But will the model based on purely integer state values improve the situation? Yes, it can, but it won't remove the feasibility problem. I you can figure it out by yourself.
Please see also:
http://en.wikipedia.org/wiki/Test_data_generation#Infeasible_Paths[
^],
http://en.wikipedia.org/wiki/Test_plan[
^],
http://en.wikipedia.org/wiki/Test_data[
^],
http://en.wikipedia.org/wiki/Software_testing[
^].
So, even though I did not offer you a solution, I hope my post will help you to save time by not ramming the brick wall. You can address some stochastic testing approaches instead.
—SA