[FIXED] How to print dsyev (or LAPACK in general) function progress?

Issue

I am using dsyev with Intel MKL in C.

I am diagonalizing a 20_000 x 20_000 real symmetric matrix of doubles.

I want to know how much time is left from the dsyev call, or roughly where it is.

I compile the source .c code using the Intel Link Line advisor.

Is there a way to do this?

Thank you!

Solution

No, LAPACK methods are not designed to do that.

You can create a performance model on the target machine and analyse the complexity of the function to predict the computational time taken by the function (approximate time). Alternatively you can reimplement the function using BLAS building block (not very simple).

Answered By - Jérôme Richard

Answer Checked By - Terry (PHPFixing Volunteer)

Read More

[FIXED] How to calculate time estimate for parallel tasks?

Issue

I need to calculate the total amount of time for a certain number of tasks to be completed. Details:

• 5 tasks total. Time estimates (in seconds) for each: [30, 10, 15, 20, 25]
• Concurrency: 3 tasks at a time

How can I calculate the total time it will take to process all tasks, given the concurrency? I know it will take at least as long as the longest task (25 seconds), but is there a formula/method to calculate a rough total estimate, that will scale with more tasks added?

Solution

If you don't mind making some approximations it could be quite simple. If the tasks take roughly the same time to complete, you could use the average of the tasks duration as a basis (here, 20 seconds).

Assuming that the system is always full of tasks, that task duration is small enough, that there are many tasks and that concurrency level is high enough, then:

estimated_duration = average_task_duration * nb_tasks / nb_workers

Where nb_workers is the number of concurrent threads.

Here is some Python code that shows the idea:

from random import random
from time import sleep, monotonic
from concurrent.futures import ThreadPoolExecutor

def task(i: int, duration: float):
  sleep(duration)

def main():
  nb_tasks = 20
  nb_workers = 3
  average_task_duration = 2.0
  expected_duration = nb_tasks * average_task_duration / nb_workers

  durations = [average_task_duration + (random() - 0.5) for _ in range(nb_tasks)]

  print(f"Starting work... Expected duration: {expected_duration:.2f} s")
  start = monotonic()
  with ThreadPoolExecutor(max_workers=nb_workers) as executor:
    for i, d in enumerate(durations):
      executor.submit(task, i, d)
  stop = monotonic()

  print(f"Elapsed: {(stop - start):.2f} s")


if __name__ == "__main__":
  main()

If these hypotheses cannot hold in your case, then you'd better use a Bin Packing algorithm as Jerôme suggested.

Answered By - Louis Lac

Answer Checked By - Candace Johnson (PHPFixing Volunteer)

Read More

[FIXED] how to cap kotlin coroutines maximum concurrency

Issue

I've got a Sequence (from File.walkTopDown) and I need to run a long-running operation on each of them. I'd like to use Kotlin best practices / coroutines, but I either get no parallelism, or way too much parallelism and hit a "too many open files" IO error.

File("/Users/me/Pictures/").walkTopDown()
    .onFail { file, ex -> println("ERROR: $file caused $ex") }
    .filter { ... only big images... }
    .map { file ->
        async { // I *think* I want async and not "launch"...
            ImageProcessor.fromFile(file)
        }
    }

This doesn't seem to run it in parallel, and my multi-core CPU never goes above 1 CPU's worth. Is there a way with coroutines to run "NumberOfCores parallel operations" worth of Deferred jobs?

I looked at Multithreading using Kotlin Coroutines which first creates ALL the jobs then joins them, but that means completing the Sequence/file tree walk completly bfore the heavy processing join step, and that seems... iffy! Splitting it into a collect and a process step means the collection could run way ahead of the processing.

val jobs = ... the Sequence above...
    .toSet()
println("Found ${jobs.size}")
jobs.forEach { it.await() }

Solution

I got it working with a Channel. But maybe I'm being redundant with your way?

val pipe = ArrayChannel<Deferred<ImageFile>>(20)
launch {
    while (!(pipe.isEmpty && pipe.isClosedForSend)) {
        imageFiles.add(pipe.receive().await())
    }
    println("pipe closed")
}
File("/Users/me/").walkTopDown()
        .onFail { file, ex -> println("ERROR: $file caused $ex") }
        .forEach { pipe.send(async { ImageFile.fromFile(it) }) }
pipe.close()

Answered By - Benjamin H

Answer Checked By - Robin (PHPFixing Admin)

Read More

[FIXED] What is the difference between concurrent programming and parallel programming?

Issue

What is the difference between concurrent programming and parallel programing? I asked google but didn't find anything that helped me to understand that difference. Could you give me an example for both?

For now I found this explanation: http://www.linux-mag.com/id/7411 - but "concurrency is a property of the program" vs "parallel execution is a property of the machine" isn't enough for me - still I can't say what is what.

Solution

If your program is using threads (concurrent programming), it's not necessarily going to be executed as such (parallel execution), since it depends on whether the machine can handle several threads.

Here's a visual example. Threads on a non-threaded machine:

        --  --  --
     /              \
>---- --  --  --  -- ---->>

Threads on a threaded machine:

     ------
    /      \
>-------------->>

The dashes represent executed code. As you can see, they both split up and execute separately, but the threaded machine can execute several separate pieces at once.

Answered By - Tor Valamo

Answer Checked By - Dawn Plyler (PHPFixing Volunteer)

Read More

[FIXED] What is the difference between concurrency and parallelism?

Issue

What is the difference between concurrency and parallelism?

Solution

Concurrency is when two or more tasks can start, run, and complete in overlapping time periods. It doesn't necessarily mean they'll ever both be running at the same instant. For example, multitasking on a single-core machine.

Parallelism is when tasks literally run at the same time, e.g., on a multicore processor.

---

Quoting Sun's Multithreaded Programming Guide:

• Concurrency: A condition that exists when at least two threads are making progress. A more generalized form of parallelism that can include time-slicing as a form of virtual parallelism.

• Parallelism: A condition that arises when at least two threads are executing simultaneously.

Answered By - RichieHindle

Answer Checked By - Katrina (PHPFixing Volunteer)

Read More

[FIXED] How to know when to stop a parallel foreach where the consumer is also the producer in C#

Issue

I am trying to process some items in a BlockingCollection in parallel using Parallel.ForEach(). When processing an item, it can generate 0-2 more items to process. The number of items to process will always eventually reach 0.

My issue is, as the consumer is also the producer (processing items can generate more items to process), I can't call BlockingCollection's CompleteAdding() when the BlockingCollection is empty as there could be other threads currently processing an item which will generate more items. Therefore I don't know how to let the BlockingCollection/Parallel.ForEach know it can exit.

Here is an example of the situation (modified for simplicity)

using System;
using System.Collections.Concurrent;
using System.Threading.Tasks;

namespace Example
{
    class Example
    {
        static void Main(string[] args)
        {
            var process = new BlockingCollection<int>() { 30 };

            var parallelOptions = new ParallelOptions() { MaxDegreeOfParallelism = Environment.ProcessorCount };

            Parallel.ForEach(process.GetConsumingEnumerable(), parallelOptions, item =>
            {
                if (item > 20)
                {
                    // Some add 2 items
                    process.Add(item - 1);
                    process.Add(item - 1);
                    Console.WriteLine($"process.Count: {process.Count} | item: {item} | Added: 2");
                }
                else if (item > 10)
                {
                    // Some add 1 item
                    process.Add(item-1);
                    Console.WriteLine($"process.Count: {process.Count}| item: {item} | Added: 1");
                }
                else
                {
                    // Some add 0 items
                    Console.WriteLine($"process.Count: {process.Count}| item: {item} | Added: 0");
                }
            });

            // Parallel.ForEach never exits
            Console.WriteLine("Completed Processing");

            Console.ReadKey();
        }
    }
}

I've tried modifying the MaxDegreeOfParallelism during the Parallel.ForEach to the minimum of the number of items to process and Environment.ProcessorCount but that doesn't do anything during the Parallel.ForEach.

I've also tried storing a count of the number of unprocessed items and performing a lock when updating this number on each thread. When the unprocessed items is 0 then I call the AddingCompleted method. This doesn't work either.

using System;
using System.Collections.Concurrent;
using System.Threading.Tasks;

namespace Example
{
    class Example
    {
        static void Main(string[] args)
        {
            var runningLock = new object();
            int running = 0;

            var process = new BlockingCollection<int>() { 30 };

            var parallelOptions = new ParallelOptions() { MaxDegreeOfParallelism = Environment.ProcessorCount };

            Parallel.ForEach(process.GetConsumingEnumerable(), parallelOptions, item =>
            {
                lock (runningLock)
                {
                    running++;
                }

                if (item > 20)
                {
                    // Some add 2 items
                    process.Add(item - 1);
                    process.Add(item - 1);
                    Console.WriteLine($"process.Count: {process.Count} | item: {item} | Added: 2 | running: {running}");
                }
                else if (item > 10)
                {
                    // Some add 1 item
                    process.Add(item - 1);
                    Console.WriteLine($"process.Count: {process.Count}| item: {item} | Added: 1 | running: {running}");
                }
                else
                {
                    // Some add 0 items
                    Console.WriteLine($"process.Count: {process.Count}| item: {item} | Added: 0 | running: {running}");
                }

                lock (runningLock)
                {
                    running--;

                    if (running == 0 && process.Count == 0)
                    {
                        Console.WriteLine($"Stopping | running: {running} | process.Count: {process.Count}");
                        process.CompleteAdding();
                    }
                }
            });

            // Parallel.ForEach never exits
            Console.WriteLine("Completed Processing");

            Console.ReadKey();
        }
    }
}

Should I use something else instead of Parallel.ForEach for this?

Also, when setting MaxDegreeOfParallelism to 1. If the initial item of the BlockingCollection is >= 27, it processes everything normally however, if it's <= 26, it stops processing items around 16? Also, higher MaxDegreeOfParallelism results in stopping processing items at a lower number.

using System;
using System.Collections.Concurrent;
using System.Threading.Tasks;

namespace Example
{
    class Example
    {
        static void Main(string[] args)
        {
            // Normal
            var process = new BlockingCollection<int>() { 27 };
            // Stops around 16
            //var process = new BlockingCollection<int>() { 26 };

            var parallelOptions = new ParallelOptions() { MaxDegreeOfParallelism = 1 };

            Parallel.ForEach(process.GetConsumingEnumerable(), parallelOptions, item =>
            {
                if (item > 20)
                {
                    // Some add 2 items
                    process.Add(item - 1);
                    process.Add(item - 1);
                    Console.WriteLine($"Process Size: {process.Count} | Current Num: {item} | Added: 2");
                }
                else if (item > 10)
                {
                    // Some add 1 item
                    process.Add(item - 1);
                    Console.WriteLine($"Process Size: {process.Count}| Current Num: {item} | Added: 1");
                }
                else
                {
                    // Some add 0 items
                    Console.WriteLine($"Process Size: {process.Count}| Current Num: {item} | Added: 0");
                }
            });

            // Parallel.ForEach never exits
            Console.WriteLine("Completed Processing");

            Console.ReadKey();
        }
    }
}

Here's the actual code if anyone prefers to look at the actual code instead of the abstracted version.

Solution

You were on the right track with this:

I've also tried storing a count of the number of unprocessed items and performing a lock when updating this number on each thread. When the unprocessed items is 0 then I call the AddingCompleted method.

The problem is that you actually are counting the number of active workers, not the number of unprocessed items. I.e. you only increment your counter when you start processing something, so there may be many other items in the queue not represented by that counter. To do the latter, what you need to do is increment a counter for every time you add something to the queue, and then decrement a counter for every time you finish processing something from the queue.

Now, had you tried that, you would have likely run into a different problem: by default, the Parallel.ForEach() method attempts to batch up items from the source. This does not work well with a source like BlockingCollection<T> which can block during enumeration, waiting on additional data. In your example, this leads to a deadlock where Parallel.ForEach() is waiting for more items before it will queue the most recent batch, while the BlockingCollection<T> is waiting for more items to be processed and thus cause more items to be queued.

With the ForEach() method waiting on the collection and the collection waiting on the ForEach() method, you get deadlock.

There is a fix for that though: you can provide ForEach() with a partitioner that is specifically configured to not buffer the data, but rather queue the work items immediately as they are retrieved.

Putting those two strategies together, you get a version of your code that looks something like this (with some minor changes to output that I had added for diagnostic purposes):

static void Main(string[] args)
{
    const int firstValue = 30;
    const int secondValues = 20;
    const int thirdValues = 10;

    var process = new BlockingCollection<int>() { firstValue };

    var parallelOptions = new ParallelOptions() { MaxDegreeOfParallelism = Environment.ProcessorCount };
    int totalItemCount = process.Count;

    OrderablePartitioner<int> partitioner = Partitioner.Create(process.GetConsumingEnumerable(), EnumerablePartitionerOptions.NoBuffering);

    Parallel.ForEach(partitioner, parallelOptions, (item, state, i) =>
    {
        string message;

        if (item > secondValues)
        {
            // Some add 2 items
            Interlocked.Add(ref totalItemCount, 2);
            process.Add(item - 1);
            process.Add(item - 1);
            message = $"{DateTime.Now.ToLongTimeString()}: process.Count: {process.Count} | item: {item} | Added: 2";
        }
        else if (item > thirdValues)
        {
            // Some add 1 item
            Interlocked.Increment(ref totalItemCount);
            process.Add(item - 1);
            message = $"{DateTime.Now.ToLongTimeString()}: process.Count: {process.Count}| item: {item} | Added: 1";
        }
        else
        {
            // Some add 0 items
            message = $"{DateTime.Now.ToLongTimeString()}: process.Count: {process.Count}| item: {item} | Added: 0";
        }

        int newCount = Interlocked.Decrement(ref totalItemCount);

        if (newCount == 0)
        {
            process.CompleteAdding();
        }

        Console.WriteLine($"{message} | newCount: {newCount} | i: {i}");
    });

    // Parallel.ForEach will exit
    Console.WriteLine("Completed Processing");    
    Console.ReadKey();
}

Answered By - Peter Duniho

Answer Checked By - Mildred Charles (PHPFixing Admin)

Read More

[FIXED] When PHP exec function creates a process, where is it queued so that it can be removed programmatically or from command line?

Issue

I have a PHP script that runs the following code:

exec("ls $image_subdir | parallel -j8 tesseract $image_subdir/{} /Processed/OCR/{.} -l eng pdf",$output, $result_code);

The code runs, however, even after I terminate the PHP script and close the browser, it continues to create the pdf files (thousands). It has been 24 hrs and it is still running. When I run a ps command, it only shows the 8 current processes that were created.

How can I find where all the pending ones are running and kill them? I believe I can simply restart Apache/PHP, but I would like to know where these pending processes are and how they can be down or controlled. It seemed originally that the code waited a minute while it executed the above code, then proceeded to the next line of code in the PHP script. So it appears that it created the jobs somewhere and then proceeded to the next line of code.

Is it perhaps something peculiar to the parallel command? Any information is very much appreciated. Thank you.

Solution

The jobs appear to have been produced by a perl process:

perl /usr/bin/parallel -j8 tesseract {...basically the code from the exec() function call in the php script}

perl was invoked either by the gnu parallel command or php's exec function. In any event, htop would not allow killing of process and did not produce any error or status and so it may be a permission problem preventing htop from killing the process. So it was done with sudo on the command line which ultimately killed the process and stopped any further processes creation from the original PHP exec() call.

Answered By - Michael

Answer Checked By - Mary Flores (PHPFixing Volunteer)

Read More

[FIXED] Why won't my cross platform test automation framework run in parallel?

Issue

I am currently rewriting the automated testing framework for my company's mobile testing. We are attempting to use an interface which is implemented by multiple Page Object Models dependent on the Operating System of the mobile device the application is being run on. I can get this framework to run sequentially and even create multiple threads but it will not run in parallel no matter what I do. Of Note, we use Appium and something called the DeviceCart/DeviceConnect which allows me to physically remote into multiple devices, thus this isn't running on a grid. With that said I will link my pertinent code (this is my second version of this same code, I wrote one with and one without using ThreadLocal)

This should instantiate a new driver with a new thread for each Test

public class TLDriverFactory {

    private ThreadLocal < AppiumDriver < MobileElement >> tlDriver = new ThreadLocal <>();

    public synchronized void setTLDriver(OS platform, String server, String udid, String bundleID) {

        switch (platform) {
        case IOS:
            tlDriver = ThreadLocal.withInitial(() -> {
                try {
                    return new IOSDriver < MobileElement > (new URL(server), DesiredCapsManager.getDesiredCapabilities(OS.IOS, udid, bundleID));
                } catch(MalformedURLException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
                return null;
            });
            break;
        case ANDROID:
            tlDriver = ThreadLocal.withInitial(() -> {
                try {
                    return new AndroidDriver < MobileElement > (new URL(server), DesiredCapsManager.getDesiredCapabilities(OS.ANDROID, udid, bundleID));
                } catch(MalformedURLException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
                return null;
            });
            break;
        default:
            break;
        }
    }

    public synchronized ThreadLocal < AppiumDriver < MobileElement >> getTLDriver() {
        return tlDriver;
    }
}

This handles browser capbilities

public class DesiredCapsManager {

    public static DesiredCapabilities getDesiredCapabilities(OS platform, String udid, String bundleID) {
        //Set DesiredCapabilities
        DesiredCapabilities capabilities = new DesiredCapabilities();

        capabilities.setCapability("deviceConnectUserName", "User@Name.com");
        capabilities.setCapability("deviceConnectApiKey", "API-Token-Here");
        capabilities.setCapability("udid", udid);
        capabilities.setCapability("platformName", platform);
        capabilities.setCapability("bundleID", bundleID);
        //IOS only Settings
        if (platform.equals(OS.IOS)) {
            capabilities.setCapability("automationName", "XCUITest");
        }
        else {
            //Android only Settings
            capabilities.setCapability("automationName", "appium");
        }

        return capabilities;
    }
}

This is the Base Test class from which every test inherits

public class BaseTest {

    protected AppiumDriver < MobileElement > driver;
    protected AppiumSupport.TLDriverFactory TLDriverFactory = new AppiumSupport.TLDriverFactory();

    public enum OS {
        ANDROID,
        IOS
    }

    @AfterMethod
    public synchronized void tearDown() throws Exception {
        driver.quit();
        TLDriverFactory.getTLDriver().remove();
    }
}

Here is the test case itself

public class Test_SignIn extends BaseTest {

    protected SignInPage signInPage;

    @Parameters(value = {
        "udid",
        "bundleID",
        "platform",
        "server"
    })
    @BeforeMethod
    public void setup(String udid, String bundleID, OS platform, String server) throws MalformedURLException,
    InterruptedException {
        //Set & Get ThreadLocal Driver
        TLDriverFactory.setTLDriver(platform, server, udid, bundleID);
        driver = TLDriverFactory.getTLDriver().get();
        Thread.sleep(5000);
        switch (platform) {
        case IOS:
            signInPage = new SignInPageIOS(driver);
            break;

        case ANDROID:
            signInPage = new SignInPageAndroid(driver);
            break;

        default:
            break;
        }

        System.out.println("Current Thread ID BeforeTest: " + Thread.currentThread().getName());
    }

    @Test
    public synchronized void Authenticate() throws Exception {
        System.out.println("Current Thread ID Test 1: " + Thread.currentThread().getName());
        signInPage.Login("Username", "Password");

    }
}

Here is the testng.xml file

   < !DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Test" parallel="tests" thread-count="4">
   <test name="SignIn" parallel ="instances" thread-count="2">
          <parameter name="udid" value="DeviceIdGoesHere" />
          <parameter name="bundleID" value="Environment.address.here" />
          <parameter name="platform" value="ANDROID" />
          <parameter name="server" value="http://deviceconnect/appium" />
          <classes>
              <class name="Test.Test_SignIn">
              </class>
          </classes>
   </test>
   <test name="SignIn2" parallel="instances" thread-count="2">
          <parameter name="udid" value="DeviceIdGoesHere" />
          <parameter name="bundleID" value="Environment.address.here" />
          <parameter name="platform" value="IOS" />
          <parameter name="server" value="http://deviceconnect/appium" />
          <classes>
              <class name="Test.Test_SignIn">
              </class>
          </classes>
   </test>
</suite>

What I'm looking for is if anyone can determine what mistake I've made or what the bottleneck is preventing the tests from running in parallel

Solution

Based on what you have shared so far, here's the cleaned-up and fixed code that should support your concurrency requirements.

The Driver factory class which is responsible for creation and clean-up of Appium driver instances for each and every thread, looks like below:

import io.appium.java_client.AppiumDriver;
import io.appium.java_client.MobileElement;
import io.appium.java_client.android.AndroidDriver;
import io.appium.java_client.ios.IOSDriver;

import java.net.MalformedURLException;
import java.net.URL;

public class TLDriverFactory {

    private static final ThreadLocal<AppiumDriver<MobileElement>> tlDriver = new ThreadLocal<>();

    public static void setTLDriver(BaseTest.OS platform, String server, String udid, String bundleID) throws MalformedURLException {

        System.out.println("Current Thread ID Driver Instantiation: " + Thread.currentThread().getName());

        AppiumDriver<MobileElement> driver;
        switch (platform) {
            case IOS:
                driver = new IOSDriver<>(new URL(server), DesiredCapsManager.getDesiredCapabilities(BaseTest.OS.IOS, udid, bundleID));
                break;

            default:
                driver = new AndroidDriver<>(new URL(server), DesiredCapsManager.getDesiredCapabilities(BaseTest.OS.ANDROID, udid, bundleID));
                break;
        }

        tlDriver.set(driver);
    }

    public static AppiumDriver<MobileElement> getTLDriver() {
        return tlDriver.get();
    }

    public static void cleanupTLDriver() {
        tlDriver.get().quit();
        tlDriver.remove();
    }
}

Here's how the BaseTest which I am guessing is supposed to be the base class for all tests, would look like

import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Parameters;

public class BaseTest {

    private static final ThreadLocal<SignInPage> signInPage = new ThreadLocal<>();

    public enum OS {
        ANDROID,
        IOS
    }

    @Parameters(value = {"udid", "bundleID", "platform", "server"})
    @BeforeMethod
    public void setup(String udid, String bundleID, OS platform, String server) throws Exception {
        //Set & Get ThreadLocal Driver
        TLDriverFactory.setTLDriver(platform, server, udid, bundleID);

        Thread.sleep(5000);
        SignInPage instance;
        switch (platform) {
            case IOS:
                instance = new SignInPageIOS(TLDriverFactory.getTLDriver());
                break;

            default:
                instance = new SignInPageAndroid(TLDriverFactory.getTLDriver());
                break;
        }

        System.out.println("Current Thread ID BeforeTest: " + Thread.currentThread().getName());
        signInPage.set(instance);
    }

    @AfterMethod
    public void tearDown() {
        System.out.println("Current Thread ID AfterTest: " + Thread.currentThread().getName());
        TLDriverFactory.cleanupTLDriver();
    }

    protected static SignInPage getPageForTest() {
        return signInPage.get();
    }

}

Here's how the constructor of your page classes would look like

import io.appium.java_client.AppiumDriver;
import io.appium.java_client.MobileElement;

public class SignInPageIOS extends SignInPage {
    public SignInPageIOS(AppiumDriver<MobileElement> tlDriver) {
        super(tlDriver);
    }
}

Here's how a typical test case could look like

import org.testng.annotations.Test;

public class Test_SignIn extends BaseTest {
    @Test
    public void authenticate() {
        //Get the instance of "SignInPage" for the current thread and then work with it.
        getPageForTest().Login("Username", "Password");
    }
}

Answered By - Krishnan Mahadevan

Answer Checked By - Willingham (PHPFixing Volunteer)

Read More

[FIXED] How to split set of vertices with Boost Graph?

Issue

I'm writing some algorithm in C++ for parallel graph coloring using Boost Graph and adjacency_list. I'm working with very big graph (the smallest has 32K vertices).

What I'm trying to do is to take the whole set of vertices, split them in parts and assign each part to a different thread and work in parallel, but I'm struggling with some passages.

The basic idea what this:

int step = g.m_vertices.size()/4;
int min = 0;

for(int i = 0; i < 4; i++){
    // call the function
}

And the function I call inside is something like that

for (vp = vertices(g); vp.first != vp.second; ++vp.first) {
    cout << *vp.first << endl;
}

So I have two questions:

1. g.m_vertices.size()/4; is the right solutions? If initially I have 10 vertices, then I remove some vertex in the middle (e.g. 4), only 6 vertices left (so this is the new size) but the index of the vertices go from 0 to 5 or from 0 to 9?

2. How can pass only a subset of vertices to vp instead of passing vertices(g)?

Solution

g.m_vertices.size()/4; is the right solutions?

That depends ONLY on your requirements.

If initially I have 10 vertices, then I remove some vertex in the middle (e.g. 4), only 6 vertices left (so this is the new size) but the index of the vertices go from 0 to 5 or from 0 to 9?

That depends on your graph model. You don't specify the type of your graph (I know, you do say which template, but not the template parameters). Assuming vecS for the Vertex container selector, then yes, after 4 removals, the vertex descriptors (and index) will be [0,6).

How can pass only a subset of vertices to vp instead of passing vertices(g)

Many ways.

1. you can std::for_each with a parallel execution policy
2. you can use openmp to create a parallel section from the plain loop
3. you can use filtered_graph adapter to create 4 "views" of the underlying graph and operate on those
4. you can use PBGL which is actually created for dealing with huge graphs. This has the added benefit that it works with threading/interprocess/inter-host communication, can coordinate algorithms across segments etc.
5. you can use sub_graphs; this is mainly (only) interesting if the way your graphs get built have a natural segmentation

None of the solutions are trivial. But, here's naive demo using filtered_graph:

Live On Compiler Explorer

#include <boost/graph/adjacency_list.hpp>
#include <boost/graph/filtered_graph.hpp>
#include <boost/graph/random.hpp>
#include <iostream>
#include <random>

using G = boost::adjacency_list<>;
using V = G::vertex_descriptor;

G make_graph() {
    G g;
    std::mt19937 prng(std::random_device{}());
    generate_random_graph(g, 32 * 1024 - (prng() % 37), 64 * 1024, prng);
    return g;
}

template <int NSegments, int Segment> struct SegmentVertices {
    std::hash<V> _h;
    bool operator()(V vd) const { return (_h(vd) % NSegments) == Segment; }
};

template <int N>
using Quart = boost::filtered_graph<G, boost::keep_all, SegmentVertices<4, N>>;

template <typename Graph>
void the_function(Graph const& g, std::string_view name)
{
    std::cout << name << " " << size(boost::make_iterator_range(vertices(g)))
            << " vertices\n";
}

int main()
{
    G g = make_graph();
    the_function(g, "full graph");

    Quart<0> f0(g, {}, {});
    Quart<1> f1(g, {}, {});
    Quart<2> f2(g, {}, {});
    Quart<3> f3(g, {}, {});

    the_function(f0, "f0");
    the_function(f1, "f1");
    the_function(f2, "f2");
    the_function(f3, "f3");
}

Printing e.g.

full graph 32766 vertices
f0 8192 vertices
f1 8192 vertices
f2 8191 vertices
f3 8191 vertices

Answered By - sehe

Answer Checked By - Terry (PHPFixing Volunteer)

Read More

[FIXED] How to run unit tests (MSTest) in parallel?

Issue

I am looking for ways to run test suites in parallel.

I am aware of .testrunconfig setting. This allows you to multiplex on the number of CPUs.

I want to run 1000 tests in parallel. This makes sense because I am testing a web service, so 90% of the time spent in a test is waiting for the service to respond.

Any ideas on how to pull this off ? The tests are written for VS, but I am open to running them outside of VS.

Later edit: the Visual Studio test team have added this in VS 2015 Update 1. See Mark Sowul's answer bellow.

Solution

Most of the answers on this page forget to mention that MSTest parallelizes tests in separate assemblies. You have to split your unittests into multiple .dll's to paralelize it.

But! The recent version - MSTest V2 - now CAN parallelize "in-assembly" (yay!) you just need to install a couple of nuget packages in your test project - TestFramework and TestAdapter - like described here https://blogs.msdn.microsoft.com/devops/2018/01/30/mstest-v2-in-assembly-parallel-test-execution/

And then simply add this to your test project

[assembly: Parallelize(Workers = 4, Scope = ExecutionScope.ClassLevel)]

EDIT: You can also disable parallel execution for a specific test using [DoNotParallelize] on a test method.

Answered By - Alex from Jitbit

Answer Checked By - Robin (PHPFixing Admin)

Read More