Grid Engine

Beta2 of UGE 8.1 Is Available

Our team created beta2 packages of UGE 8.1. They are available now. We are waiting for your response!

In the meantime our team fixes known issues and writes new tests for our automated test environment. I am responsible for the Job Class enhancement and although there are already 47 additional tests for that functionality (2464 different tests scenarios) I still have something on my todo list. Due to that there is not much time for blogging in the moment - but stay tuned…


Univa Grid Engine User Group Summit

Join the engineers for the 2012 Univa Grid Engine User Group Summit in beautiful Regensburg, Germany.

Monday, October 1 2012
Summit Day 1- 8:30- 17:15
Tuesday October 2, 2012
Summit Day 2- 8:30- 16:15
Regensburg, Germany
Best Western Premier Hotel
Ziegetsdorfer Strasse 111 Regensburg - Bayern 93051

Find details and register for this event here


Beta Version of UGE 8.1 Will be Available Next Week

Today we have created beta-packages for Univa Grid Engine 8.1. Our test clusters will do a final check this weekend. If these tests are successful then the packages will be available next week.


Key Features of the Upcoming Univa Grid Engine 8.1 Release

Here is a summary of the key features of Univa Grid Engine 8.1 and corresponding links to blog entries or other blogs explaining the enhancements:

  1. Processor core and NUMA memory binding for jobs which enables applications to run consistently and faster.

  2. Job Classes describing how applications run in a cluster, slashing the time to onboard and manage workflow.

  3. Resource maps which define how hardware and software resources are ordered and used in the cluster helping to improve throughput and utilization of the cluster

  4. Improved Job Debugging and Diagnostics allowing administrators discover issues in less time

  5. New support for Postgres database job spooling that balances speed of submission with reliability in high volume clusters that have lots of small jobs

  6. Documented and tested integrations with common MPI environments allows for valuable time saved since Univa has done the integration work.

  7. And more …


Core and Memory Binding of Jobs in Univa Grid Engine 8.1

Execution nodes in Grid Engine clusters usually have multiple sockets and multiple cores with a hierarchy of different caches. This hardware architecture will provide performance benefit for jobs and therefore improve the overall throughput of a cluster if it is handled correctly.

Univa Grid Engine is not only aware of the underlaying hardware architecture of compute resources. It provides also the necessary semantics to give managers and users of a cluster full control where jobs should be executed and how they should be handled.

Especially Univa Grid Engine 8.1 is extremely powerful. Within this version of Univa Grid Engine the scheduler component is completely responsible for the socket and core selection. Due to that it is possible to guarantee core binding specific requests. This was different in UGE 8.0 and it is still in other available Grid Engine versions.

The scheduler is also aware of the memory allocation capabilities of the underlaying hardware. As result particular memory allocation strategies can be selected so that jobs and underlaying applications will have accelerated access to available memory. Also this feature is new in UGE 8.1.


Univa Grid Engine Free Trial

Univa Grid Engine is the next generation product that Sun Grid Engine and Oracle Grid Engine users have been waiting for. Customers save time and money through increased uptime, and with innovative feature and product evolution they can significantly reduce the total cost of ownership.

The trial version contains all features of the full version for clusters with up to 48 cores. Find more information here.


Submitting New Jobs Using Job Classes

Univa Grid Engine 8.1 provides a number of new submit switches that can also be used in combination with the qalter command to adjust jobs derived from a job class. These new switches can also be used to adjust conventionally submitted jobs and allow therefore additional operations that were not possible in the past.


Integration of Job Classes into the Existing System

Extensive use of job classes will have a positive impact on the cluster throughput in Univa Grid Engine 8.1 clusters. Reason for this is that job classes have been fully integrated into the core components of the system. For instance, the scheduling component can distinguish the different types of workloads easily. Also the algorithm in the scheduler that is responsible to find resources for a job was improved. Details are explained below.


Access Specifiers, Variants, Template Job Class

Now I will explain three additional aspects of job classes: Access specifiers - They allow deviation from a derived job class during or after submission of a job. Job class variants - They will help to keep the effort in configuring job classes small. The template job class - This job class might be used to enforce settings for all other job classes in a system.


Defining Job Classes as Templates for Jobs

Engineers improving Univa Grid Engine use an automated test environment during the development and during the test phase before a new version of Univa Grid Engine is released. This test suite automatically installs test clusters and runs several thousand test scenarios to see if functionality of Univa Grid Engine is broken.

For each bug that is fixed in the Univa Grid Engine code base a new test scenario is added to the test suite and also when new functionality is implemented then engineers have to create new tests to keep the test coverage high.

Today I have finished the 91st test scenario that tests job class functionality. Some of these new tests take care that the ownership of a job class is respected and some validate that the code is working correctly that should prevent users from deriving new jobs from job classes that do not allow this. Ownership and accessibility are two characteristics that have to be defined when new job classes are created in Univa Grid Engine 8.1.


Job Classes Introduction

Meanwhile I am working for more than 13 years in the team of engineers that is continuously improving and enhancing Grid Engine. Nevertheless I am often astonished how difficult and long-lasting it can be to design and implement new features. About seven months ago I started with a functional specification document describing a new object type called job class. A month later I was able to start with the implementation. Now, 319 modified files, 21.737 insertions and 5.673 deletions later, I am in the middle of the test phase. Our team created new alpha packages a week ago and test results indicate that we can release job classes with Univa Grid Engine 8.1 soon. It’s the right time to explain job classes in more detail with this and the upcoming blog entries.