Sunday, March 14, 2010

Supporting multiple cores

With today's multi-core CPU's it seems that an application made with REAL Studio running on a single core is somewhat restricting. If you have a lot of data to process, large images to manipulate or other things that could happen in the background, it would seem that with a multi-core machine you could do this faster "if only REAL would make threads preemptive". We get a lot of requests for preemptive threads so that people can take advantage of multiple cores. In fact, this request is number 4 on the Top 20 list in our Feedback system.

It's been suggested that this should be "easy" to do. Just make threads preemptive (so that they will then run on any available core) and voila! Unfortunately, it's not that simple. Let's look at why the current threads are not preemptive. It's hard for you, the developer, to work with preemptive threads. Some languages, like Java, have language features built in to them to try and help make this less work, but it is still hard to get right and very hard to debug when you don't. Much of the framework would need updating to be thread safe and your application's user interface is not thread safe on any platform because the operating systems themselves don't have thread-safe user interface code. If you access something from a pre-emptive thread and that something is not thread-safe, there's a very good chance your application is going to crash. We have had to go to a lot of extra work just to make the threads you have today work without causing problems.

The REALbasic language already has functions like mutex and semaphores that help you make a multi-threaded program. But you have to protect every place you might set a value that could be shared by many threads. This would mean protecting any globals and pretty much anything that is not local to the method or the thread being executed. That's very different than what you have to do today and a lot more work. It's just not "simple" or "easy to use" the way most of REAL Studio is designed to be.

The end goal is to use all cores and thereby make your software more responsive, faster, or able to handle more data, do more things all at the same time. There's been a way to do this for as long as REAL Studio has supported building console applications. The design is to create a main application (GUI or not) and use other helper applications to run separate tasks. The main application and the helpers can communicate in any one of several ways: IPCSockets, TCPSockets, UDPSockets, XML, files, or just about any other way you can dream of. The upside to this way of solving the problem is you can design and implement the main application and the helpers independently and use the debugger to debug each independently. You can use any data in either program in the same way you always have. You don't have to worry about the framework being thread safe as the helper and main application run as completely separate processes with their own memory space. Most importantly, you can do this today.

In fact, some REAL Studio users are using this technique today. John Balestrieri used this technique for a large data processing application. You can read about it here:

and here:

I'm not going to say it's simple.You have to think about what portion of your application can be segmented out into a helper console application. You have to design the communications between the main and helper applications. You have to write, test and debug them. But you don't have to worry about variables changing mysteriously because some other thread changed the value. You don't have to use lots of mutex's or sempahores to block other threads from altering things when you least expect them. And you can use the entire framework that is available to console applications. If you need pictures and graphics that are not available in console applications, you can write faceless GUI applications. Last but not least, you can run as many instances of these helper applications as your computer (or all the computers available to you) can run.

If you think you need preemptive threads today, try the helper application approach and I think you'll be pleasantly surprised at how well it works.


Anonymous said...

We've been using this technique to utilize multiple cores for over a year now and it works quite well. What's really nice is that it scales to many CPUs if the child processes are designed appropriately.

Unknown said...

This approach would also move your architecture towards a model-view-controller -design which probably won't hurt anyway.

Aitch748 said...

The downside of this, at least until you get LLVM baked into REAL Studio, is that each executable has its own copy of the complete framework, so each console app is around a megabyte-and-a-half in size.

Might REAL Software consider a compiler option to compile the framework as a DLL called by each of your executables for a setup like this? (That would save disk space, though I don't know if each program would still require its own copy of the DLL within its process space.)

Karen said...

Having not done this myself, a perhaps naive seed of an idea occurs to me.

One thing that would help to encourage this type of architecture and increase the power of B would be to have RB build in more high level support for it.

Think of such an architecture supported in the IDE being able to be defined as "super project" where the main app and helper apps are "sub projects" which compile to separate apps in the same folder (or bundle on the mac) using the same libraries.

In the super project. Allow some "super global" classes, constants (and maybe properties) to be defined common to all sub projects.

Then provide language support for sending and receiving copies of instances o these classes between the helper apps, without is having to worry about serialization or communication protocols. ( that support would be why they would be different from external classes included in totally separate projects)

If super global properties are supported. put a transparent mechanism in to safely read and write them .

There are probably other things that would be useful but I can't think of any right now

While this would nor provide anything we could do ourselves (except using the same libraries). this would make using the helper app approach to use multiple core a lot more accessible to more RB users.

- Karen

Norman said...

if you compile all the helpers ass console apps with the same version of REALbasic then, for some platforms, its quite possible you can have only a single copy of the dylib / dll.
Console apps are organized slightly differently than GUI apps.

"Global" and "shared" are part of the problems this approach avoids.

Making it so processes have to use a very well defined API to share data back and forth is actually a good thing.
In fact a nice design would be to make it so that passing data back and forth was much like a database where you had to start transactions, do work, and then commit / rollback any changes.
Other processes would be insulated from changes to the data while the process was working, they'd not be able to read old data while the transaction was in progress, etc
I'd encourage you to read and the related paper from Sun that is mentioned.

Karen said...

Norm you said:
"Global" and "shared" are part of the problems this approach avoids.

I'm sorry I don't know the 'right' terminology but lacking a formal computer science background i don't know how to use the right terms.

I mean shared effectively but not in a simple concrete manor.

Global classes meaning having the same definition and that enabling the creation of a mechanism in the language to send instance data of such a global class between helper apps and create a clone of it in the receiving app.

Basically I'm saying are the components of the 'superproject' defines the API between helper apps in a way that is more familiar to most Rb users and takes most of the drugery out of it. making it more accessible.

Other processes would be insulated from changes to the data while the process was working, they'd not be able to read old data while the transaction was in progress, etc

I really do think you misread (my fault) the meaning behind what i said. I think i realize the issues in principle and I was not proposing anything different in principle, just in API to make it easier, faster and more approachable to most who are used to writing single apps.

Having the IDE and the language better help us to create, organize and code the architecture of multi-app (main/helper) projects with specific user friendly support would give a huge boost to RB's PERCEIVED power and RADness by many.

- Karen

Norman said...

Having the IDE help make the marshaling of data between the main and helper would be good (this is the API between the two and the act of sending and receiving the data between the two)

It's pretty much what Mars wrote in the post I mentioned.

Certainly not trivial - and it is possible for a person to do this manually at the moment. Doing it in a safe generic way in the IDE so you don't have to work so hard to set it up is a great goal - and a ton of work :)

Eric dLR said...

I've been using this technique to connect, disconnect and get data from a bluetooth serial connection. With a proper timeout and 'ping' device mechanism it helps dealing with undetected hardware disconnection and avoid the GUI to hang while connecting to the serial device.

Amando said...

It seems, in my opinion, my comments about LLVM compiler have opened a new thread on this wonderful blog.

Before going to the nitty gritty, I want to excuse me for not learning English as I would like. My skills are unfortunately not good and can't contribute as I would do in Spanish.

As I commented on my previous answer, previous post, I think RS needs to handle with Preemptive Threads the sooner as possible (My 1st choice, is about DLLs, BTW). While multicore architecture is something is still mastered by only top developers, I believe some work could be done to aid us in getting this new programming approach easier.

I have a project written using preemptive threads in C++ since 1999. Basically their approach is to handle a lot of CPU work in a thread. Debugging C++ threads is a nightmare, and writting for it as well. But basically the problem, IMHO, comes when Thread A changes a var that Thread B don't know has been changed. What I think is called a Deadlock.

Any variable inside the local scope is safe, but any other outside the thread itself must to be locked before changing it's value, as other thread can do some work on it as well. (Basically)

I am still trying to make my C++ thread code safe, but 80% of my code are about changing any single var or object instance that is global. I use criticalsections (Lock/Unlock) every time I change them. RealStudio could do this when using some var being global to avoid another thread doing wrong things. When some change is done globally, let the compiler do it automatically (optional).

Other problems could be warned by a new added feature that aids new developers in warning them that their code could be buggy.

Of course is not going to be an easy task. But in my opinion compilers tend to be very criptic about debugging or helping a non experience developer. The code can be run examining the code, before compiling and warning "Hey! You code is not safe... take this an this approach"

I don't pretend RealStudio to do all the work but it could be useful to help and warn about typical mistakes writing preemptive code. I am pretty sure no one will request that.

I am talking about console/services applications, as I know the problem with GUI apps. But it can be a beginning where we can create Dlls that use all cores and put RS on top of the list and be used by other compilers to do really complex things with the easy of use of RS.

Comms betwen GUI and Services could be done easier via some classes you can write for us ala Autodiscovery one.

My 0.2 cents... hope it helps!


Amando said...

Could not edit my previous message. Please ignore " I am pretty sure no one will request that." :)


Norman said...

If you're a member have a read of this. If you're not you can register for free and read it

Great article on "multithreading"

Norman said...

Actually here's another with some excellent ideas