UGE 8.1

Licenese Management with Univa License Orchestrator (2/2)

Following blog post gives an overview about Univa License Orchestrator architecture and how Univa Grid Engine clusters can be connected to the Univa License Orchestrator.

‘ULO
Read More...
Comments

Grid Engine for ARM

Here is a list of reactions on the announced ARM support for Grid Engine:

DataCenter Dynamics: Univa automates ARM infrastructure for Hadoop

insideHPC: Grid Engine for ARMv7 Opens the Door for Energy Efficient Datacenters of the Future

ZDNet: ARM in the datacenter gets another boost

Comments

Univa Grid Engine 8.1 Free Trial

‘Grid

Get Univa Grid Engine 8.1 here.

Comments

Beta2 of UGE 8.1 Is Available

Our team created beta2 packages of UGE 8.1. They are available now. We are waiting for your response!

In the meantime our team fixes known issues and writes new tests for our automated test environment. I am responsible for the Job Class enhancement and although there are already 47 additional tests for that functionality (2464 different tests scenarios) I still have something on my todo list. Due to that there is not much time for blogging in the moment - but stay tuned…

Comments

Beta Version of UGE 8.1 Will be Available Next Week

Today we have created beta-packages for Univa Grid Engine 8.1. Our test clusters will do a final check this weekend. If these tests are successful then the packages will be available next week.

Comments

Key Features of the Upcoming Univa Grid Engine 8.1 Release

Here is a summary of the key features of Univa Grid Engine 8.1 and corresponding links to blog entries or other blogs explaining the enhancements:

  1. Processor core and NUMA memory binding for jobs which enables applications to run consistently and faster.


  2. Job Classes describing how applications run in a cluster, slashing the time to onboard and manage workflow.


  3. Resource maps which define how hardware and software resources are ordered and used in the cluster helping to improve throughput and utilization of the cluster


  4. Improved Job Debugging and Diagnostics allowing administrators discover issues in less time


  5. New support for Postgres database job spooling that balances speed of submission with reliability in high volume clusters that have lots of small jobs


  6. Documented and tested integrations with common MPI environments allows for valuable time saved since Univa has done the integration work.

  7. And more …


Comments

Core and Memory Binding of Jobs in Univa Grid Engine 8.1

Execution nodes in Grid Engine clusters usually have multiple sockets and multiple cores with a hierarchy of different caches. This hardware architecture will provide performance benefit for jobs and therefore improve the overall throughput of a cluster if it is handled correctly.

Univa Grid Engine is not only aware of the underlaying hardware architecture of compute resources. It provides also the necessary semantics to give managers and users of a cluster full control where jobs should be executed and how they should be handled.

Especially Univa Grid Engine 8.1 is extremely powerful. Within this version of Univa Grid Engine the scheduler component is completely responsible for the socket and core selection. Due to that it is possible to guarantee core binding specific requests. This was different in UGE 8.0 and it is still in other available Grid Engine versions.

The scheduler is also aware of the memory allocation capabilities of the underlaying hardware. As result particular memory allocation strategies can be selected so that jobs and underlaying applications will have accelerated access to available memory. Also this feature is new in UGE 8.1.

Read More...
Comments

How can I find the error reason why queues/jobs go into error state

Grid Engine administrators sometimes have the problem that a bunch of queues switch into error state or that users complain about failing jobs where they cannot find the error reason.

Finding the root cause can be tricky because the startup process of a job itself is complicated. Many different parts of a real life UNIX environment are involved in this phase and they all have to work hand in hand. So what can be done to help Grid Engine users and administrators in this situation?

Read More...
Comments

Submitting New Jobs Using Job Classes

Univa Grid Engine 8.1 provides a number of new submit switches that can also be used in combination with the qalter command to adjust jobs derived from a job class. These new switches can also be used to adjust conventionally submitted jobs and allow therefore additional operations that were not possible in the past.

Read More...
Comments

Integration of Job Classes into the Existing System

Extensive use of job classes will have a positive impact on the cluster throughput in Univa Grid Engine 8.1 clusters. Reason for this is that job classes have been fully integrated into the core components of the system. For instance, the scheduling component can distinguish the different types of workloads easily. Also the algorithm in the scheduler that is responsible to find resources for a job was improved. Details are explained below.

Read More...
Comments

Access Specifiers, Variants, Template Job Class

Now I will explain three additional aspects of job classes: Access specifiers - They allow deviation from a derived job class during or after submission of a job. Job class variants - They will help to keep the effort in configuring job classes small. The template job class - This job class might be used to enforce settings for all other job classes in a system.

Read More...
Comments

Defining Job Classes as Templates for Jobs

Engineers improving Univa Grid Engine use an automated test environment during the development and during the test phase before a new version of Univa Grid Engine is released. This test suite automatically installs test clusters and runs several thousand test scenarios to see if functionality of Univa Grid Engine is broken.

For each bug that is fixed in the Univa Grid Engine code base a new test scenario is added to the test suite and also when new functionality is implemented then engineers have to create new tests to keep the test coverage high.

Today I have finished the 91st test scenario that tests job class functionality. Some of these new tests take care that the ownership of a job class is respected and some validate that the code is working correctly that should prevent users from deriving new jobs from job classes that do not allow this. Ownership and accessibility are two characteristics that have to be defined when new job classes are created in Univa Grid Engine 8.1.

Read More...
Comments

Job Classes Introduction

Meanwhile I am working for more than 13 years in the team of engineers that is continuously improving and enhancing Grid Engine. Nevertheless I am often astonished how difficult and long-lasting it can be to design and implement new features. About seven months ago I started with a functional specification document describing a new object type called job class. A month later I was able to start with the implementation. Now, 319 modified files, 21.737 insertions and 5.673 deletions later, I am in the middle of the test phase. Our team created new alpha packages a week ago and test results indicate that we can release job classes with Univa Grid Engine 8.1 soon. It’s the right time to explain job classes in more detail with this and the upcoming blog entries.

Read More...
Comments