Parallel Computing 
BWUniCluster 2.0 //bwForCluster

ADDITIVE Soft- und Hardware für Technik und Wissenschaft GmbH
Simone Szurmant
simone.szurmant@additive-net.de

Max-Planck-Str. 22b                          
D-61381 Friedrichsdorf  
Tel:   ++49-(0)6172-5905-30                   
http://www.additive-net.de

Short History of Parallel Computing with Mathematica

1999: Release of the Mathematica Parallel Computing Toolkit
2002: gridMathematica and Personal Grid Edition are stand-alone products

2008: Parallel Programming Tools were included in Mathematica

Parallel Computation Architecture in Mathematica

Mathematica provides a high-level, parallel language interface that allows you to connect multiple Mathematica kernels using the MathLink communication protocol. The parallel computing features are written almost entirely in the Wolfram language and are therefore machine-independent.

Every copy of Mathematica comes with all the components and tools to run and create parallel applications.

On any multicore computer system, Mathematica is able to run multiple parts of a computation concurrently and makes parallel computing easy enough that it can be used in seconds as a routine part of everyday work.

Getting Started

Using the graphical user interface

Mathematica’s graphical user interface provides a structured notebook view, help with entering commands, and extensive, interactive documentation.
The GUI can be started both on a local computer and on the bwUniCluster (not recommended, see page XX). However, a remote session with X-Forwarding requires patience.

Using a Mathematica script

Mathematica code can be scripted and run on a cluster using job schedulers such as MOAB, SLURM, or TORQUE.
To run a Mathematica script in batch mode on the bwUniCluster execute the following command:

math -script test.m

On a cluster usually other parameters have to be be specified before the script is sent, such as the required number of nodes or computation time.

Getting Started

When using the Mathematica frontend, you can find the status of all your currently running kernels in the evaluation menu.

Additionally, you can use the following commands to programmatically work with kernels:

The LaunchKernels command will explicitly launch all kernels that you have configured:

LaunchKernels[]

You can see a list of the running kernels via the Kernels command:

Kernels[]

You can get the number of running kernels via $KernelCount:

$KernelCount

Hello Parallel World

A first example may just demonstrate how to parallelize your computations. One very simple type of parallel program is to do a search. In the following example, one is added to a factorial and the result is tested to see if it is a prime number.

On a single core

t1 = AbsoluteTime[]; 
data = Table[PrimeQ[n! + 1], {n, 400, 550}]; 
timeLocal = AbsoluteTime[] - t1

Parallel Version: if you want to use the parallel functionality of Mathematica,  just wrap the regular Mathematica computation in Parallelize. Mathematica automatically uses the available cores on a machine depending on configuration and license.

t2=AbsoluteTime[];
data=Parallelize[Table[PrimeQ[n! + 1], {n, 400,550}]];
timeParallel=AbsoluteTime[]-t2

For some functions there are specialized functions for parallelization.

t2=AbsoluteTime[];
data=ParallelTable[PrimeQ[n! + 1], {n, 400,550}];
timeParallel=AbsoluteTime[]-t2

Hello Parallel World

Parallelize will automatically parallelize a computation if possible. It recognizes if the expression is a Mathematica function that operates on a list or long expression in a way that can be parallized.

Parallelize[(∑_(i=1)^250)i!]

Parallelize[Integrate[1/(y^3+1),{y,0,1}]]

A parallel computation may run more slowly than the non-parallel version if the computation takes less time than the overhead of parallelization.

ParallelTry

We can use ParallelTry to send multiple evaluations to kernels; this is useful when you want to pick a particular type of method, for example, but don't know which is the most efficient.

AbsoluteTiming[ParallelTry[
{#,FindMinimum[(E^(Sin[50 x]))+Sin[60 (E^y)]+Sin[70 Sin[x]]+Sin[Sin[80 y]]-Sin[10 (x+y)]+(1/4) ((x^2)+(y^2)),{x,y},Method->#]}&,{"Gradient","ConjugateGradient","InteriorPoint","QuasiNewton","Newton"}]]

Inside Automatic Parallelization: Values of Variables

Values of Variables defined on the local master kernel in a global Context are usually available for remote kernels - since Mathematica 10.3.

In order for a piece of code to run in parallel, all definitions it uses must be copied to the subkernels. 
This is called “distributing” the definitions and can be done manually by DistributeDefinitions. 
It is also done automatically by most parallel functions .

Automatic distribution is done only for $Context. This is controlled by $DistributedContexts.

$Context

$DistributedContexts

ctx`y=42;

ParallelEvaluate[ToString[ctx`y]]

Why not distribute definitions for all contexts? Because for a package to work properly, it may not be sufficient to just copy all definitions from its context. The package may do things upon loading which are more complex than issuing definitions. E.g. it may load LibraryFunctions. Thus packages are meant to be loaded on parallel kernels using ParallelNeeds instead of just copying their definitions over from the main kernel.

What can you do then?
Either

DistributeDefinitions[ctx`y]

ParallelEvaluate[ToString[ctx`y]]

to manually distribute the definitions of this one symbol, or

$DistributedContexts := {$Context, “ctx`”}

$DistributedContexts

to automatically distribute everything from the ctx` context.

a`g[x_]:={$KernelID,x}
b`g[x_]:={$KernelID,-x}

Parallelize[{a`g[1],b`g[1]},DistributedContexts->{"a`"}]

Visualizing the Mandelbrot Set

In this example we will take a look at how to visualize the Mandelbrot set. The Mandelbrot set is a set of points in the complex plane, the boundary of which forms a fractal.

First we define a function that returns the number of times it took for the set to converge at a given (x, y) coordinate.

mf[c_]:=Length[FixedPointList[((#)^2)+c &,c,20,SameTest->(Abs[#]>2.0 &)]]

Compute the data using Table:

AbsoluteTiming[datamf =Table[mf[x+y I],{x,-2,0.5, 0.002},{y,-1,1, 0.002}];]

We can use ArrayPlot to visualize the data:

ArrayPlot[datamf,ColorFunction-> Hue, ImageSize->400]

Visualizing the Mandelbrot Set: In parallel

To parallelize the process, Table can be replaced with ParallelTable as in the previous example:

DistributeDefinitions[mf];
AbsoluteTiming[dataParallel = ParallelTable[mf[x+y I],{x,-2,0.5, 0.002},{y,-1,1, 0.002}];]

The parallel functions in Mathematica support a Method option, which adjusts load balancing.  The automatic setting results in medium load balancing and is usually the best choice - it balances evaluation size and number of evaluations.

FinestGrained breaks the computation into the smallest possible subunits and CoarsestGrained breaks it into as many pieces as there are available kernels.

DistributeDefinitions[mf];
AbsoluteTiming[dataParallelC = ParallelTable[mf[x+y I],{x,-2,0.5, 0.002},{y,-1,1, 0.002},Method->"CoarsestGrained"];]

DistributeDefinitions[mf];
AbsoluteTiming[dataParallelF = ParallelTable[mf[x+y I],{x,-2,0.5, 0.002},{y,-1,1, 0.002},Method->"FinestGrained"];]

We can use ArrayPlot to visualize the data and check to see if we obtained the same results:

ArrayPlot[dataParallel,ColorFunction->Hue,ImageSize->400]

Inside Automatic Parallelization: Shared Variables

Mathematica uses independent kernels as parallel processors. These kernels do not share a common memory, and without shared variables the parallel evaluations are completly independent.

Mathematica provides functions that implement virtual shared memory for these remote kernels. 
If a variable a is shared, any kernel that reads the variable (simply by evaluating it), reads a common value that is maintained by the master kernel. Any kernel that changes the value of a, will modify the one global copy of the variable a.

SetSharedVariable[xs];

The drawback of a shared variable is that every access for read or write requires communication over the network, so it is slower than access to a local unshared variable.

In a situation where several concurrently running remote kernels access the same shared variable for reading and writing, there is no guarantee that the value of a variable is not changed by another process between the time you read a value and write a new value. Any other new value that another process wrote in the meantime would get overwritten.

xs=0;
ParallelDo[xs=xs+1,{10}];
xs

Mathematica provides the function CriticalSection[lck,expr] to acquire a lock, evaluate an expression and then release the lock. Once a process has acquired the lock, no other process can do so. The lock is released when the expression finishes evaluation.

xs=0;
ParallelDo[CriticalSection[xslock,xs=xs+1],{10}];
xs

Concurrency: Managing Parallel Processes

A process is simply a Mathematica expression being evaluated.

You can create any number of processes, many more than the number of available processors. Remaining processes will be queued and serviced when a processor becomes available.

The two basic commands are Parallel Submit to line up an expression for evaluating, and WaitAll to wait until a given process has finished.

jl = ParallelSubmit[1+1]

WaitAll[jl]

Concurrency: Managing Parallel Processes

eids = Table[ParallelSubmit[{i}, PrimeQ[i]], {i,9}]

The "Parallel `Developer`" context contains functions to control the process queue.

Needs["Parallel`Developer`"]

QueueRun[]

QueueRun[]

$QueueLength

WaitAll[eids]

Speed Up Computations with Parallel GPU Computing

Write code that will use the maximum available precision on the specific CUDA or OpenCL device.

Because the scheduler has not yet been running, all of them are still queued for evaluation.

code="
__kernel void julia_kernel(__global float *set, long width, long height, float cx, float cy) {
   long xIndex = get_global_id(0);
   long yIndex = get_global_id(1);
   long ii;

   float x = ZOOM_LEVEL*(width/2 - xIndex);
   float y = ZOOM_LEVEL*(height/2 - yIndex);
   float tmp;
   float c;

   if (xIndex < width && yIndex < height) {
       for (ii = 0; ii < MAX_ITERATIONS && x*x + y*y < BAILOUT; ii++) {
			tmp = x*x - y*y + cx;
			y = 2*x*y + cy;
			x = tmp;
		}
		c = log(0.1f + sqrt(x*x + y*y));
		set[xIndex + yIndex*width] = c;
    }
}
";

Compile and link the OpenCL code automatically in the Wolfram Language:

Needs["OpenCLLink`"]
{width,height}={512,512};
jset=OpenCLMemoryAllocate["Float",{height,width}];
JuliaCalculate=OpenCLFunctionLoad[code,"julia_kernel",{{"Float",_,"Output"},_Integer,_Integer, "Float", "Float"},{16,16},"Defines"->{"MAX_ITERATIONS"->10,"ZOOM_LEVEL"->"0.0050","BAILOUT"->"4.0"}];

Now you can build the interactive interface to dynamically call the OpenCL code:

Manipulate[
JuliaCalculate[jset, width, height, c[[1]], c[[2]], {width, height}];
ReliefPlot[OpenCLMemoryGet[jset], DataRange->{{-2.0, 2.0}, {-2.0, 2.0}}, ImageSize->256,ColorFunction->"SunsetColors"],
{{c, {0, 1}}, {-2, -2}, {2, 2}, Locator}]

bwUniCluster//bwForCluster

When using a cluster with a scheduler, capacities are often not directly available, but with a delay in time. 

Result: if you start a Mathematica GUI on the cluster only the main node you are logged on would be used for computation.
Check if you have access to job nodes by evaluating

ReadList["!scontrol show hostname $SLURM_JOB_NODELIST",String]

The following steps are necessary to use Mathematica on a managed cluster:
1. Open a new notebook
2. Enter a query for the SLURM job nodes 
3. Launch Mathematica compute kernels on them
4. Write Mathematica program code
5. Save your Mathematica script
6. Start your  batch job on the cluster

For steps 1 and 2 some gyus at the ADDITIVE tech support have written a small Mathematica package that queries the nodes and connects them to Mathematica.
The download is possible here:
https://soda.additive-net.de/index.php/s/QrdysS9CL4XaDkg

!This version is vor MOAB, please change “MOAB_NODELIST” to “SLURM_JOB_NODELIST”.

To use the package in Mathematica load it on start with the GET command,followed by its location (web, directory ..)

Get["https://soda.additive-net.de/index.php/s/QrdysS9CL4XaDkg/download/BWCluster.m"]

Usage:

LaunchClusterKernels["module load math/mathematica/13", 8]

You should adjust the version number to the one available in the cluster. 
The function starts ‘8’ Mathematica-Kernels on each node.
Your script should end with

CloseKernels[]

If you encounter problems with the Wolfram Language when trying to use Mathematica on the bwCluster, please do not hesitate to contact the ADDITIVE technical support team:support@additive-net.de

Further Ressources: Mathematica und ADDITIVE

For more information or to request a trial copy of Mathematica, please send an email to simone.szurmant@additive-net.de

ADDITIVE can help you with:

Installation und Configuration: Helping you install and configure gridMathematica software for your cluster, including a launch notebook for launching kernels on your cluster.

Training: Providing personal training “Grid Computing with Mathematica” .

Project Specific Support: discussion specific to your project, including parallelization and debugging strategies.

Further reading:

See the documentation for the parallel functionality of Mathematica by visiting: reference.wolfram.com/mathematica/ParallelTools/tutorial/Overview.html

Get more information about ADDITIVE by visiting:  www.additive-net.de

Get more information about gridMathematica by visiting:  www.wolfram.com/products/gridmathematica

ADDITIVE - Spezialisten für Berechnen, Visualisieren und Automatisieren

Auf unserer Webseite der ADDITIVE Academy finden Sie unsere Schulungsangebote:
http://academy.additive-net.de/de/schulungen/wolfram

Kontaktieren Sie uns bei Fragen bezüglich Mathematica, Training/Schulung und Anwendungsprogrammierung
http://www.additive-mathematica.de

ADDITIVE Soft- und Hardware für Technik und Wissenschaft GmbH
Max-Planck-Staße 22b, 61381 Friedrichsdorf
Vertrieb : 06172 - 5905 - 30 // mathematica@additive-net.de
Academy: 06172 - 5905 - 90 // academy@additive-net.de
Support : 06172 - 5905 - 20 // support@additive-net.de

({{, }})