|
I have a numerical modeling code that runs into a stack overflow problem when the data set gets large. The overflow happens as I step into a Fortran DLL. Just before the DLL I create several large double arrays to hold the results coming back from the DLL. I suspect these large arrays are part of the problem.
Is it possible to get these arrays allocated on the heap? Would I have to wrap them in a class to achieve this?
|
|
|
|
|
An array *is* a class. Anything allocated with new is put on the heap.
Is the Fortran DLL an unmanaged DLL? If it is, are you sure your prototypes for the methods in the DLL are correct? How big are your arrays?
.45 ACP - because shooting twice is just silly
-----
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"The staggering layers of obscenity in your statement make it a work of art on so many levels." - J. Jystad, 2001
|
|
|
|
|
John Simmons / outlaw programmer wrote: How big are your arrays?
A man doesn't ask that of another man.
|
|
|
|
|
John Simmons / outlaw programmer wrote: Is the Fortran DLL an unmanaged DLL
I didn't know it was possible to have a managed Fortran DLL. Interesting...
Dybs
The shout of progress is not "Eureka!" it's "Strange... that's not what i expected". - peterchen
|
|
|
|
|
The Fortran DLL is unmanaged (Intel Fortran). The code works well for smaller problems, giving answers that are exactly the same as a 100% Fortran version of the code. I should state that this DLL is actually a C++ wrapper calling a Fortran static library.
I have 9 rank 4 arrays holding data in the C# part. For the failed run each arrays was of size 600 * 3 * 3 * 8 = 43200. So the 9 arrays would use about 3.1 MB. Going into the DLL the arrays are a little small, being only rank 3. The size for this run is 600 * 3 * 8 = 14400.
As I think about this, I realise there are several wrappers in the Fortran library also. These wrappers are used to modify the API to the Fortran. I guess each wrapper requires a stack frame. The nine large arrays are the output. I have just as many smaller arrays for input. As I go deeper into the wrappers, more hard coded input arrays are used.
The one time I followed the program flow, it crashed just inside one of the Fortran wrappers, before the first line of code.
BTW, I need to be able to run problems much larger than the run that failed.
|
|
|
|
|
When I was checking out this problem, I found this articla on Wikipedia:
http://en.wikipedia.org/wiki/Stack_overflow[^]
Looking at the examples, I see the following, where a large array size causes a stack overflow.
int main(void) {
double n[10000000];
return 0;
}
This is why I thought large array sizes could cause my problem.
|
|
|
|
|
|
OK, that makes sense. So it seems that the size of my arrays cannot directly lead to a stack overflow. But the stack overflow does occur when they become large. Somehow the two things are probably related.
Is there some way to check the size of items on the stack using the VS2008 IDE. I am not at work, so I cant look at the problem code, but I have some example code at home. I see in the debugger that I can look at Autos and Locals in their respective tabs, but I dont see a tab for the stack contents.
|
|
|
|
|
That is not how things get done normally in a managed application. Arrays are objects, hence allocated on the heap (one of the heaps actually, yours will be the "large object heap"). To allocate on the stack you would need the "stackalloc" keyword, which is not a good idea most of the time.
I strongly recommend you allocate all arrays on the managed side and pass them to the native/Fortran world as indicated in my article.
|
|
|
|
|
I am allocating all the arrays on the C# side, and they are all 1D arrays. What I am not doing is using the fixed keyword or the GCHandle class as outlined in your article. Do you think one of these techniques might solve my problem. I can test it out when I get to work on Monday.
|
|
|
|
|
if you have to go from C# to native, you need a pointer value; you simply cannot get a pointer value without using either fixed or GCHandle, unless you trust the automatic marshaling which AFAIK will allocate another array and copy all the data, something I never want to happen as it brings nothing (it doubles the memory footprint, wastes CPU cycles and trashes the caches).
|
|
|
|
|
OK, I did not realize there could be a problem with the pointers. As I have mentioned, the code works when passing small arrays. As a new C# programmer, I copied my interface to the DLL from existing code used at our company. I am fairly certain these other codes only pass relatively small arrays compared to what I will need to pass.
I see now that there are some fundamentals that I don't understand, such as automatic marshaling. Your "copy all the data" observation does not sound good. I always assumed that when I passed in an array, I was passing in a pointer. But from your article, it seems that this pointer can get moved around, and not be reliable? I have to put in more study time on this.
I am looking forward to testing yuor ideas on Monday.
I hope you understand that from the point of view of 34 years of Fortran programming, this is a headache. However, I understand the benefits to be gained by converting to a C# interface for our modeling subroutines. It is unfortunate that we don't have the resourses to convert all the Fortran, and eliminate the DLL.
|
|
|
|
|
Yes, the garbage collector "GC" (part of which runs on its own thread) may move around objects, and does so asynchronously to your own code execution; the managed world knows how to deal with that, unmanaged code does not. By using fixed or GCHandleType.Pinned you prevent such moves.
Warning: pinned objects temporarily reduce the GC's capabilities, so you should not keep objects pinned longer than necessary.
I suspect your code currently works for small arrays and not for larger ones because the automatic marshaling is somehow limited (maybe it does allocate a temp array on stack??); I do not know any details as I try to not rely on it (I do trust it when strings are involved and an ASCII/Unicode conversion may be in order).
I have been marshaling arrays of 100MB+ without any problem; once you get the hang of it, it works like a charm. I have lots of code in C (and SIMD assembly) that gets called to operate on managed objects, such as huge images.
|
|
|
|
|
Hi Luc,
Luc Pattyn wrote: AFAIK will allocate another array and copy all the data
Interesting - A problem in an untested part of my code waiting to happen, you may have just saved me many hours of head scratching! Do you have a source for this? (about to google now!)
DaveIf this helped, please vote & accept answer!
Binging is like googling, it just feels dirtier. (Pete O'Hanlon)
BTW, in software, hope and pray is not a viable strategy. (Luc Pattyn)
|
|
|
|
|
DaveyM69 wrote: Do you have a source for this?
I don't. The automatic marshaling, if it works, has to do something to protect native code against dangling pointers. I think I could devise a setup that proves me right or wrong. I might do that today...
|
|
|
|
|
Unless it copies everything (including all elements) by reference then it doesn't appear to create a copy.
I created a test C++ function that takes an array of ints as a parameter and I incremented all elements in the array. When I called this from C# and checked the values afterwards, all had been incremented - so it seems that the array is simply marshalled by reference (as I would expect as System.Array is a class) from managed to unmanaged.
DaveIf this helped, please vote & accept answer!
Binging is like googling, it just feels dirtier. (Pete O'Hanlon)
BTW, in software, hope and pray is not a viable strategy. (Luc Pattyn)
|
|
|
|
|
right.
|
|
|
|
|
Hi Dave,
sorry, I was wrong. My latest experiment[^] tells me automatic marshaling works fine.
There typically is no need to use unsafe and fixed keywords for passing arrays from managed to unmanaged world; only when the pointer(s) are kept around after the function returns, they may become invalid and GCHandle should be used.
|
|
|
|
|
are you using stackalloc? don't.
are you using P/Invoke and DllImport? if so, this[^] may help.
|
|
|
|
|
I am using DLLImport. I will check out your link.
|
|
|
|
|
Hi spainchaud,
sorry, I was wrong. My latest experiment[^] tells me automatic marshaling works fine.
There typically is no need to use unsafe and fixed keywords for passing arrays from managed to unmanaged world; only when the pointer(s) are kept around after the function returns, they may become invalid and GCHandle should be used.
|
|
|
|
|
I guess I will just have to put som more thought into my problem. Usually I come up with a solution, but it can take days to figure out some problems.
|
|
|
|
|
I have solved the problem. Thanks for everyone's help. The hint to the solution was the Wikipedia article on how to create a stack overflow. The example code they gave was:
int main(void) {
double n[10000000];
return 0;
}
When I got to work today, and started tracing the overflow, I found this code in my Fortran DLL.
COMPLEX*16 hxx(nstep*nrec*ntrn), hxy(nstep*nrec*ntrn), hxz(nstep*nrec*ntrn)
COMPLEX*16 hyx(nstep*nrec*ntrn), hyy(nstep*nrec*ntrn), hyz(nstep*nrec*ntrn)
COMPLEX*16 hzx(nstep*nrec*ntrn), hzy(nstep*nrec*ntrn), hzz(nstep*nrec*ntrn)
I realized I was putting these huge arrays on the stack. My new working code is:
COMPLEX*16, ALLOCATABLE :: hxx(:), hxy(:), hxz(:)
COMPLEX*16, ALLOCATABLE :: hyx(:), hyy(:), hyz(:)
COMPLEX*16, ALLOCATABLE :: hzx(:), hzy(:), hzz(:)
ALLOCATE(hxx(nstep*nrec*ntrn), hxy(nstep*nrec*ntrn), hxz(nstep*nrec*ntrn))
ALLOCATE(hyx(nstep*nrec*ntrn), hyy(nstep*nrec*ntrn), hyz(nstep*nrec*ntrn))
ALLOCATE(hzx(nstep*nrec*ntrn), hzy(nstep*nrec*ntrn), hzz(nstep*nrec*ntrn))
Thanks to the discussions here, I found the problem very quickly.
|
|
|
|
|
Hi all,
Hope someone can point me in the right direction.
I have a foreach loop that is pinging each COM port for a valid device that may be connected. I will eventually put this in a try-catch, but for now I'm just trying to determine how to create an array based upon the loop and then return that array to be used for generating a collection in a list box. Please see the code below: Also, a generic question...can a function return different data types eg. string, double, etc.
private string[] GetSerialDevice()
{
strPorts = SerialPort.GetPortNames();
foreach (string strPort in strPorts)
{
serialPort1 = new SerialPort(strPort, 115200, Parity.None, 8, StopBits.One);
serialPort1.Handshake = Handshake.None;
serialPort1.Close();
serialPort1.ReadTimeout = 500;
serialPort1.WriteTimeout = 500;
serialPort1.Open();
serialPort1.WriteLine("!001:USR1?\r");
string strUSR1Return = serialPort1.ReadTo("\r");
if (strUSR1Return == "+2.000000")
{
serialPort1.WriteLine("!001:SERL?\r");
string strSerialLow = serialPort1.ReadTo("\r");
serialPort1.WriteLine("!001:SERH?\r");
string strSerialHigh = serialPort1.ReadTo("\r");
double dblSerialNumber = double.Parse(strSerialHigh) * 65536 + double.Parse(strSerialLow);
Console.WriteLine("On" + " " + strPort + " " + "there is a valid amplifier with S/N "
+ dblSerialNumber.ToString());
}
}
serialPort1.Close();
return strPorts;
|
|
|
|
|
mprice214 wrote: can a function return different data types eg. string, double, etc.
yes why not.
What is your question ?
I am not getting exactly.
can you please more elaborate ?
Thank you
Life's Like a mirror. Smile at it & it smiles back at you.- P Pilgrim
So Smile Please
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
