Trees       Indices       Help   
Module gip_common
[hide private]
[frames] | no frames]

Module gip_common

source code

Common functions for GIP providers and plugins.

A set of general-purpose functions to help GIP plugin/provider authors write probes which are consistent and correct.

This module should generally follow PEP 8 coding guidelines.

Author: Brian Bockelman

Classes [hide private]
  VoMapper
This class maps a username to VO.
  FakeLogger
Super simple logger for python installs which don't have the logging package.
  _Constants
A convenience class for important constants.
  Attributes
Given a filename containing attributes, parse it into a dictionary.
Functions [hide private]
 
check_gip_location()
This function checks to make sure that GIP_LOCATION is set and exists.		 source code
 
check_testing_environment()
Check to see if the GIP_TESTING environment variable has been set.		 source code
 
parseOpts(args)
Parse the passed command line options.		 source code
 
config(*args)
Load up the config file.		 source code
 
config_compat(cp)
Currently, all of the configuration information is kept in:		 source code
 
config_compat_osg_attributes(override, cp)
Load up configuration options from osg-attributes.conf into the passed config object; translate between the two formats.		 source code
 
config_compat_gip_attributes(override, cp)
Load up configuration options from gip-attributes.conf into the passed config object; translate between the two formats.		 source code
 
config_compat_gip_attributes_subcluster(gip, override, cp)
Load up the GIP subcluster information from gip-attribtues.conf; as this information wins the award of "the biggest hack on the OSG", we'll separate it out into it's own function.		 source code
 
__write_config(cp, override, dict_object, key, section, option)
Helper function for config_compat; should not be called directly.		 source code
 
getOsgAttributes()
Return a dictionary-like object containing the OSG attributes as configured by the configure-osg.sh script and stored in the $VDT_LOCATION/monitoring/osg-attributes.conf file.		 source code
 
add_giplog_handler()
Add a log file to the default root logger.		 source code
 
getLogger(name)
Returns a logger object corresponding to `name`.		 source code
 
addToPath(new_element)
Add a directory to the path.		 source code
 
HMSToMin(hms)
Helper function to convert something of the form HH:MM:SS to number of minutes.		 source code
 
getTemplate(template, name)
Return a template from a file.		 source code
 
printTemplate(template, info)
Print out the LDIF contained in template using the values from the dictionary `info`.		 source code
 
voList(cp, vo_map=None)
Return the list of valid VOs for this install.		 source code
 
getHostname()
Convenience function for retrieving the local hostname.		 source code
 
fileRead(path)
Return the contents of the file on a given path		 source code
 
fileWrite(path, contents)
Append some contents to a given path		 source code
 
fileOverWrite(path, contents)
Overwrite a file with a contents.		 source code
 
cp_get(cp, section, option, default)
Helper function for ConfigParser objects which allows setting the default.		 source code
 
cp_getBoolean(cp, section, option, default=True)
Helper function for ConfigParser objects which allows setting the default.		 source code
 
cp_getInt(cp, section, option, default)
Helper function for ConfigParser objects which allows setting the default.		 source code
 
pathFormatter(path, slash=False)
Convience function to format a path.		 source code
 
ldap_boolean(val)
Return a LDIF-formatted boolean.		 source code
 
notDefined(val)
Returns TRUE if the input value is possibly not defined (i.e., matches UNAVAILABLE, UNDEFINED, or UNKNOWN).		 source code
 
normalizeFQAN(fqan)
Return fqan in the form of /<VO>[/<VO group>/]Role=<VO Role>		 source code
 
matchFQAN(fqan1, fqan2)
Return True if fqan1 matches with fqan2, False otherwise		 source code
 
parseRvf(name)
Parse the Globus RVF for a specific file.		 source code
 
getURLData(some_url, lines=False)
Return the data from a URL.		 source code
 
getUrlFd(some_url)
Return a file descriptor to a URL.		 source code
 
compare_by(fieldname)
Returns a function which can be used to compare two dictionaries.		 source code
 
ls(directory)
Convenience function for os.listdir; returns a directory listing.		 source code
 
responseTimes(cp, running, waiting, average_job_time=None, max_job_time=None)
Computes the estimated and worst-case response times based on a simple formula.		 source code
 
getFQDNBySiteName(cp, sitename) source code
Variables [hide private]
  py23 = True
True if the current version of Python is 2.3 or higher; enables a few extra capabilities which Python 2.2 does not have.
  loglevel = 'info'
  info_map = {'cores_per_node': '06', 'cpu_model': '03', 'cpu_sp...
  log = getLogger("GIP.common")
  rvf_parse = re.compile(r'(.+?): (?:(.*?)\n)')
Function Details [hide private]

check_gip_location()

source code 

This function checks to make sure that GIP_LOCATION is set and exists. If GIP_LOCATION is not set and $VDT_LOCATION/gip exists, then it adds:

   GIP_LOCATION=$VDT_LOCATION/gip

to the process's environment

It raises ValueErrors if neither situation holds.

This function is automatically run by the *config* function, so it is generally not necessary for provider authors to use this directly.

check_testing_environment()

source code 

Check to see if the GIP_TESTING environment variable has been set. If so, set the gip_testing.replace_command variable to be true. This causes the GIP to read output from static files instead of running the command. This is useful if you want to test, say, PBS on a laptop with no PBS installed.

parseOpts(args)

source code 

Parse the passed command line options.

Does not expect the first element of argv; if you pass this directly, use parseOpts(sys.argv[1:]).

There are three objects returned:

  • keywordOpts: Options of the form --key=val or -key=val or -key val; this is the dictionary of the key: val pairs.
  • passedOpts: Options of the form -name1 or --name2. List of strings.
  • givenOpts: Options which aren't associated with any flags. List of strings.
Parameters:
  • args - A list of strings which are the command line options.
Returns:
keywordOpts, passedOpts, givenOpts. See the docstring.

config(*args)

source code 

Load up the config file. It's taken from the command line, option -c or --config; default is $GIP_LOCATION/etc/gip.conf

If python 2.3 is not available, the command line option is not checked.

If any arguments are supplied to this function, they will be interpreted as filenames for additional config files to read. If the filename considers environmental variables, they will be expanded.

config_compat(cp)

source code 

Currently, all of the configuration information is kept in:

   $VDT_LOCATION/monitoring/osg-attributes.conf

This function will take in the ConfigParser object `cp` and update it with the configurations found from the OSG monitoring.

If gip.override=True, then the config object overrides the OSG settings. If not, then the OSG settings override.

If VDT_LOCATION is not defined, this function does nothing.

config_compat_osg_attributes(override, cp)

source code 

Load up configuration options from osg-attributes.conf into the passed config object; translate between the two formats.

Parameters:
  • override - If true, then options in gip.conf will override the options in osg-attributes.conf
  • cp - Site configuration

config_compat_gip_attributes(override, cp)

source code 

Load up configuration options from gip-attributes.conf into the passed config object; translate between the two formats.

Parameters:
  • override - If true, then options in gip.conf will override the options in osg-attributes.conf
  • cp - Site configuration

add_giplog_handler()

source code 

Add a log file to the default root logger.

Uses a rotating logfile of 10MB, with 5 backups.

getLogger(name)

source code 

Returns a logger object corresponding to `name`.

Parameters:
  • name - Name of the logger object.

getTemplate(template, name)

source code 

Return a template from a file.

Parameters:
  • template - Name of the template file in $GIP_LOCATION/templates.
  • name - Entry in the template file; for now, this is the first entry of the DN.
Returns:
Template string
Raises:
  • e - ValueError if it is unable to find the template in the file.

printTemplate(template, info)

source code 

Print out the LDIF contained in template using the values from the dictionary `info`.

The different entries of the template are matched up to keys in the `info` dictionary; the entries' values are the dictionary values.

To see what keys `info` needs for your template, read the template as found in:

   $GIP_LOCATION/templates
Parameters:
  • info (Dictionary) - Dictionary of information to fill out for the template. The keys correspond to the blank entries in the template string.
  • template - Template string returned from getTemplate.

voList(cp, vo_map=None)

source code 

Return the list of valid VOs for this install. This data is taken from the vo mapper and the blacklist / whitelist in the "vo" section of the config parser `cp` is applied.

Parameters:
  • cp (ConfigParser) - Configuration information for this site.

fileRead(path)

source code 

Return the contents of the file on a given path

Parameters:
  • path - Path to file for reading
Returns:
File contents.
Raises:
  • IOError - When file can't be read (doesn't exist, permission errors)

fileWrite(path, contents)

source code 

Append some contents to a given path

Parameters:
  • path - Path to file we will append.
  • contents - Additional contents for file.
Raises:
  • IOError - When file can't be appended.

fileOverWrite(path, contents)

source code 

Overwrite a file with a contents. If file doesn't exist, create and write as usual.

Parameters:
  • path - Path to file we will [over]write.
  • contents - Contents for file.
Raises:
  • IOError - When file can't be [over]written.

cp_get(cp, section, option, default)

source code 

Helper function for ConfigParser objects which allows setting the default.

ConfigParser objects throw an exception if one tries to access an option which does not exist; this catches the exception and returns the default value instead.

Parameters:
  • cp - ConfigParser object
  • section - Section of config parser to read
  • option - Option in section to retrieve
  • default - Default value if the section/option is not present.
Returns:
Value stored in CP for section/option, or default if it is not present.

cp_getBoolean(cp, section, option, default=True)

source code 

Helper function for ConfigParser objects which allows setting the default.

If the cp object has a section/option of the proper name, and if that value has a 'y' or 't', we assume it's supposed to be true. Otherwise, if it contains a 'n' or 'f', we assume it's supposed to be true.

If neither applies - or the option doesn't exist, return the default

Parameters:
  • cp - ConfigParser object
  • section - Section of config parser to read
  • option - Option in section to retrieve
  • default - Default value if the section/option is not present.
Returns:
Value stored in CP for section/option, or default if it is not present.

cp_getInt(cp, section, option, default)

source code 

Helper function for ConfigParser objects which allows setting the default. Returns an integer, or the default if it can't make one.

Parameters:
  • cp - ConfigParser object
  • section - Section of the config parser to read
  • option - Option in section to retrieve
  • default - Default value if the section/option is not present.
Returns:
Value stored in the CP for section/option, or default if it is not present.

pathFormatter(path, slash=False)

source code 

Convience function to format a path.

Parameters:
  • path - Path to format
  • slash - Add a trailing (on the right) slash to the path if it is not present.
Returns:
right-strip the path of all the '/'; if slash=True, then add a slash to the end.

normalizeFQAN(fqan)

source code 

Return fqan in the form of /<VO>[/<VO group>/]Role=<VO Role>

If VO group is not specified, return /<VO>/Role=<VO Role> If the VO Role is not specified, return /VO/Role=*

matchFQAN(fqan1, fqan2)

source code 

Return True if fqan1 matches with fqan2, False otherwise

fqan1 may actually be more specific than fqan2. So, if fqan1 is /cms/blah and fqan2 is /cms, then there is a match. If the Role=* for fqan2, the value of the Role for fqan1 is ignored.

FQANs may be of the form:

  • VOMS:<FQAN>
  • VO:<VO Name>
  • <FQAN>
  • <VO>
Parameters:
  • fqan1 - The FQAN we are testing for match
  • fqan2 - The FQAN

parseRvf(name)

source code 

Parse the Globus RVF for a specific file.

Retrieves the file from $GLOBUS_LOCATION/share/globus_gram_job_manager/name

See an example RVF file for the patterns this matches. Returns a dictionary of dictionaries; the keys are attributes, and the values are a dictionary of key: value pairs for all the associated information for an attribute.

In the case of an exception, this just returns {}

Parameters:
  • name - Name of RVF file to parse.
Returns:
Dictionary of dictionaries containing information about the attributes in the RVF file; returns an empty dict in case if there is an error.

getURLData(some_url, lines=False)

source code 

Return the data from a URL.

Parameters:
  • some_url - URL to retrieve
  • lines - True to split the content by lines, False otherwise.
Returns:
The data of the URL; a string if lines=False and a list of lines if lines=True

compare_by(fieldname)

source code 

Returns a function which can be used to compare two dictionaries.

Parameters:
  • fieldname - The dictionaries will be compared by applying cmp to this particular keyname.
Returns:
A function which can be used to compare two dictionaries.

responseTimes(cp, running, waiting, average_job_time=None, max_job_time=None)

source code  

Computes the estimated and worst-case response times based on a simple
formula.

We take the
  ERT = average_job_time/(running+1)*waiting 
  WRT = max_job_time/(running+1)*waiting 

If |running| + |waiting| < 10, then ERT=1hr, WRT=24hr unless |running|=0.
If |running|=0 or |waiting|=0, then ERT=1 min.

ERT and WRT must be positive; ERT maxes out at 1 day, WRT maxes out
at 30 days.  WRT must be a minimum of 2*ERT.

@param cp: Site configuration
@param running: Number of jobs running
@param waiting: Number of waiting jobs
@keyword average_job_time: Average runtime (in seconds) for a job
@keyword max_job_time: Maximum runtime (in seconds for a job
@return: ERT, WRT (both are measured in seconds)

Variables Details [hide private]

info_map

Value:
{'cores_per_node': '06',
 'cpu_model': '03',
 'cpu_speed_mhz': '04',
 'cpu_vendor': '02',
 'cpus_per_node': '05',
 'inbound_network': '21',
 'name': '01',
 'node_count': '99',
...

   Trees       Indices       Help   
Generated by Epydoc 3.0.1 on Wed Jan 7 13:01:11 2009 http://epydoc.sourceforge.net