How do I write software that run on a Grid?

Question

4.00/5 (1 vote)

See more:

I have access to my university's grid system (running on Sun Grid Engine 6.1 with 100+ nodes and 1000+ CPUs)

I have a program, which basically runs genetic algorithm on some dataset, written in C++. It currently executes in a single thread.

My question is.. do I need to parallelize the program on my own, perhaps, using MPI and OpenMP? Or is the parallelization done by the host engine itself?

If I have to parallelize on my own, how many threads should I create?

Thanks in advance

Posted 10-May-10 23:56pm

yusof_hardy

Updated 11-May-10 1:10am

v4

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Michel Godfroid · Answer 1 · 2010-05-11T00:24:00

Solution 1

Eh, it's a little more complicated than that.
First you'll need a parallellisation framework. OpenMPI works fine with Sun Grid Engine.
The number of threads is irrelevant, if you use the libraries well, the grid will be able to optimise the number of threads for you ( depending on current load, other running jobs, resources allocated to your program...).
You should plan on using as many threads as possible (your algorithms may have a limit on number of parallel tasks, of course), but adapt if fewer threads are available.

Posted 11-May-10 0:24am

Michel Godfroid

Comments

yusof_hardy 11-May-10 7:13am

Sorry I'm not familiar with the whole grid thing. Does it really matter which library I use (since the program is compiled anyway)? Does the engine only run interpreted scripts?

Michel Godfroid 11-May-10 7:38am

Your program in it's current state will run on the grid. Only it will use one processor and one thread, and will take as long as it does now.
In order to use the grid, you have to re-write your program so that the grid 'knows' how it can spread your program over it's processors. Informing the the grid about these capabilities is done through a parallellisation library (like OpenMPI). There you will define the parallel architecture, and how messages will be passed betwween grid cells). the whole subject is too large to treat here, but your University must have a Grid information center, where you will find literature and assistance in developing your programs. Colleagues/students who have used the grid before may also be helpful. I'm sorry but the whole subject is just too large to treat here.