Licenese Management with Univa License Orchestrator (2/2)

Following blog post gives an overview about Univa License Orchestrator architecture and how Univa Grid Engine clusters can be connected to the Univa License Orchestrator.

‘ULO

Understanding the Univa Grid Engine/Univa License Orchestrator Architecture

Preparing Univa Grid Engine so that it can be connected to Univa License Orchestrator

Before a Univa Grid Engine cluster can be connected to the Univa License Orchestrator instance some prerequisite steps are necessary:

  1. Update your Univa Grid Engine cluster if required.
    Make sure that your Univa Grid Engine installation supports Univa License Orchestrator. It might be required that you install an update before this is possible. For newer versions of Univa License Orchestrator you will find the requirements in the corresponding Installation Guide.
  2. Rename existing 'lo_' complexes.
    If you have complex variables in your cluster that start with the characters 'lo_' then rename them. The reason for this is that all license complexes that are reported by Univa License Orchestrator will start with that prefix. Those complexes are automatically managed by Univa License Orchestrator and must not be added, changed or deleted manually as soon as the Univa Grid Engine cluster is connected to the Univa License Orchestrator.
  3. Remove complexes that represent software licenses that are managed by Univa License Orchestrator
  4. Remove all complexes from your cluster that represent software licenses that will later on be reported by Univa License Orchestrator. Before the complexes can be removed it is required to remove all references from other objects like hosts, queues, resource quotas or the scheduler configuration that might exist in your cluster.

Enabling Univa License Orchestrator Components to Automate License Management

Univa License Orchestrator components need to be installed, configured and started before they will report license information to a Univa Grid Engine cluster and before a Univa Grid Engine cluster is able to send license requests to the Univa License Orchestrator. The list of steps below will only provide an overview on what is required. Detailed step by step instructions can be found in the 'Univa License Orchestrator Installation Guide' and 'Univa License Orchestrator Administration Guide'.

  1. Install Univa License Orchestrator
    Univa License Orchestrator needs to be installed. Part of this installation process is to define the root location ($LO_ROOT) where the binaries, scripts and configuration files should reside.
  2. Setup License Manager Communication
    Part of the Univa License Orchestrator setup process is it to tell it where license manager instances are located in the enterprise and how Univa License Orchestrator can talk with them. As soon as this is done Univa License Orchestrator will know the software license types, the maximum amount as well the currently available amount of licenses as they are registering in those license managers.
  3. Setup Univa Grid Engine Communication.
    To give a Univa Grid Engine cluster access to Univa License Orchestrator it is necessary to:
    • make the admin user of the Univa Grid Engine cluster also a submit user of Univa License Orchestrator.
    • $ loconf -as <uge_admin_user>
      
    • make the master host as well as all shadow hosts of the Univa Grid Engine cluster a management host of the Univa License Orchestrator.
    • $ for host in <qmaster_host><shadow_host><shadow_host2> ...
      % do 
      %    loconf -ah $host
      % done
      
    • add a Univa Grid Engine configuration object to Univa License Orchestrator (see qconf -augec). You have to specify the cluster name ($SGE_CLUSTER_NAME), master host (and all shadow hosts that might become master hosts) as well as the qmaster port ($SGE_QMASTER_PORT) of the Univa Grid Engine installation. This will allow the corresponding Univa Grid Engine cluster to communicate with Univa License Orchestrator.
    • $ qconf -sugec <a_cluster_name>
      clu_name           <a_cluster_name>
      clu_masterhosts    <list_of_master_and_shadow_hosts>
      clu_ugename        <uge_cluster_name>
      clu_port           <uge_qmaster_port>
      
    • define the variable 'lo_root' in the 'qmaster_params' of the Univa Grid Engine cluster and set it to the installation location of the Univa License Orchestrator ($LO_ROOT). The Univa Grid Engine qmaster and shadow hosts also requires the right to execute files located in subdirectories of $LO_ROOT/bin.
      $ qconf -sconf
      qmaster_params    lo_root=<lo_root_path> …
      ...
      

Activating the Univa Grid Engine/Univa License Orchestrator communication

  1. Manual Activation of Univa Grid Engine/Univa License Orchestrator communication.
    The Univa Grid Engine/Univa License Orchestrator communication is done by two components. One component is implemented as thread within the Univa Grid Engine qmaster process with the name 'lothread'. The second components is an application part of the Univa License Orchestrator installation and its name is 'lodial'. The 'lothread' has to be started manually and it will automatically start 'lodial' when required. To start 'lothread' use following Univa Grid Engine command:
    $ qconf -at 'lothread'
    
  2. Automate the start-up of the communication
    Please note that 'lothread' is not started automatically when the Univa Grid Engine qmaster process is restarted. To automate this it is necessary to modify the file $SGE_ROOT/$SGE_CELL/common/bootstrap file in the Univa Grid Engine cluster. If the file already contains a line that starts with 'lo_threads' then change its value to '1' and if the line is missing then add it. When this setting is present then Univa Grid Engine qmaster will automatically start the 'lothread' when it is started again.
    $ cat $SGE_ROOT/$SGE_CELL/common/bootstrap
    ...
    lo_threads              1
    

Management of Licenses within Univa License Orchestrator enabled Univa Grid Engine Clusters

Synchronization of License Information

When Univa Grid Engine is connected to a Univa License Orchestrator, then the 'lothread' and 'lodial' components synchronize information about software licenses that are known in Univa License Orchestrator. Within the Univa Grid Engine cluster, software licenses will automatically appear as consumable complexes where the primary name starts with the letters 'lo_'.

Those license complexes will be managed by the Univa License Orchestrator and must therefore not be deleted by Univa Grid Engine managers. It is also not possible to add other complexes where the name starts with 'lo_'. Modification of license complexes is also restricted. Managers can redefine the shortcut name, default request and urgency but any attempt to adjust other attributes will be ignored.

Additionally to the license complexes, Univa License Orchestrator will trigger changes the 'global' host object in the Univa Grid Engine cluster. Univa License Orchestrator will change the 'complex_values' so that it contains the maximum available amount of licenses for each software license. Univa License Orchestrator will also add 'load_values' that represent the amount of licenses that are currently available. Also those values will automatically be adjusted when they change.

Requesting Univa License Orchestrator Reported Licenses

Whenever a user wants to submit a job that requires a software license that is under the control of Univa License Orchestrator it has to specify a corresponding resource request additionally to all other resource requests and job submission parameters. Naturally, such license requests can also be specified in job classes or added in client or server JSV's to the specification of a job.

Jobs requesting Univa License Orchestrator managed software licenses will be handled slightly differently in the Univa Grid Engine system. Corresponding jobs might enter an additional jobs state: the l-state. When a job enters the l-state then Univa Grid Engine books all resources in the Univa Grid Engine cluster for this job and it sends a license request to the Univa License Orchestrator. As soon as a positive response for that license request arrives the job will be started and the job will change to the t-state (later r-state). Usually the response from Univa License Orchestrator is fast so that users might not even see that a job enters the l-state for a short time.

If there are multiple Univa Grid Engine clusters connected to a Univa License Orchestrator or if software licenses are also used outside the control of Univa Grid Engine then the likelihood increases that jobs remain in the l-state for a longer time. A job in l-state will wait for the license until:

  • the Univa Grid Engine system gets positive a response that the requested licenses are available.
  • the Univa Grid Engine system gets a response from Univa License Orchestrator that the license request will never get fulfilled.
  • the job will be rescheduled
  • the job will be deleted.

Influencing Scheduling Behaviour

Jobs with license request have to pass two schedulers so that they can be started. The Univa Grid Engine scheduler is responsible to assign all non-license resources whereas the Univa License Orchestrator scheduler is responsible for the assignment of software licenses only.

In the Univa Grid Engine system all license complexes can be used as any other complex. This means that they can be referenced in host or queue objects or as part of resource quota sets to define additional constraints that have to be adhered by the Univa Grid Engine scheduler. Also within the scheduler configuration itself license complexes might be used, e.g. to setup the urgency policy in the Univa Grid Engine policy scheme.

The Univa License Orchestrator system schedules license requests that are created by Univa Grid Engine. A license request is created for a job in the UGE system as soon as it enters the l-state. This license request inherits several attributes from the Univa Grid Engine job:

  • License requests are added as hard resource requests.
  • The submit user is the owner of the license request.
  • The project/department name will be the same if a corresponding object with the same name also exists in Univa License Orchestrator.
  • The account string will be copied.

Especially the user/project/department information can be used to influence the license scheduling in the Univa License Orchestrator cluster (e.g. by defining a share tree in the Univa License Orchestrator policy scheme).

License Scheduling between Multiple Clusters

License requests are derived from a variant of the 'LO_template' job class in the Univa License Orchestrator system. If a variant exists that has the same name as the cluster name where the license request is coming from ('LO_template.<cluster_name>') then this variant will be the template for the license request. If no such variant exists then the default variant ('LO_template.default') will be used. This mechanism allows Univa License Orchestrator managers to influence the license scheduling in the Univa License Orchestrator, e.g. by defining Resource Quote Sets that define limits for users of certain Univa Grid Engine clusters. It is also possible to change the Job Class variants itself to predefine settings for license requests.

Univa License Orchestrator license requests also have context variables attached that contains variables and values that are derived from the Univa Grid Engine cluster or Univa Grid Engine job. This job context contains the following information:

  • cluster name
  • user/group ID's and names of the Univa Grid Engine job owner
  • Univa Grid Engine job/task ID
  • project name
  • department

If the default settings of license requests are not sufficient then this additional information in combination with a JSV will allow adjusting the license requests. E.g. it is possible to change the project request to something like <cluster_name>_<project_name> instead of <project_name> only. This might be useful if different Univa Grid Engine clusters use the same project name for what are in reality different projects which should be distinguishable in Univa License Orchestrator.

blog comments powered by Disqus