Stuck on Zenoss 3.1.0 for now, lately my collectors have started failing to fetch zencommand and zenjmx config where these commands in data sources depend on a zProperty that is unchanged and inherited from the device type e.g. /Server/Oracle and not instantiated in the device level e.g. /Server/Oracle/devices/device_name. This used to work for ages but presumably some now untraceable change to the system has cased this to start happening every day.
The result when this exception happens is all commands for the device are deleted from the collection schedule of the device with coresponding "Deleting command" logs in the collector's zencommand.log. Usually a restart of the central hub and the collector will restore the commands for a while only to be deleted again when each collector next runs what I presume is a config cycle.
Here is an example of the log lines in zenhub.log when zencommands fail:
2013-05-16 10:55:28,703 INFO zen.ZenHub: Worker reports 2013-05-16 10:55:28,702 ERROR zen.hub: getDeviceCommands() exception for device device_name
Traceback (most recent call last):
File "/usr/local/zenoss/zenoss/Products/ZenHub/services/CommandConfig.py", line 140, in getDataSourceCommands
cmdinfo = getDeviceCommands(dev)
File "/usr/local/zenoss/zenoss/Products/ZenHub/services/CommandConfig.py", line 88, in getDeviceCommands
threshs = getComponentCommands(dev, cache, cmds, dev.getDmd())
File "/usr/local/zenoss/zenoss/Products/ZenHub/services/CommandConfig.py", line 75, in getComponentCommands
cmd.command = ds.getCommand(comp)
File "/usr/local/zenoss/zenoss/Products/ZenModel/RRDDataSource.py", line 206, in getCommand
res = compiled(getEngine().getContext(environ))
File "/usr/local/zenoss/python/lib/python2.6/site-packages/zope.tales-3.4.0-py2.6.egg/zope/tales/expressions.py", line 261, in __call__
v = var(econtext)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/zope.tales-3.4.0-py2.6.egg/zope/tales/expressions.py", line 217, in __call__
return self._eval(econtext)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/Products/PageTemplates/Expressions.py", line 127, in _eval
ob = self._subexprs[-1](econtext)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/zope.tales-3.4.0-py2.6.egg/zope/tales/expressions.py", line 124, in _eval
ob = self._traverser(ob, element, econtext)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/Products/PageTemplates/Expressions.py", line 76, in boboAwareZopeTraverse
object = object.restrictedTraverse(name)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/OFS/Traversable.py", line 310, in restrictedTraverse
return self.unrestrictedTraverse(path, default, restricted=True)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/OFS/Traversable.py", line 283, in unrestrictedTraverse
next = guarded_getattr(obj, name, _marker)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/AccessControl/ImplPython.py", line 729, in guarded_getattr
aq_acquire(inst, name, aq_validate, validate)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/AccessControl/ImplPython.py", line 671, in aq_validate
return validate(inst, object, name, v)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/AccessControl/ImplPython.py", line 565, in validate
self._context)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/AccessControl/ImplPython.py", line 464, in validate
raise Unauthorized(name, value)
Unauthorized: You are not allowed to access 'zOracleUser' in this context
ZenJMX data sources will fail like this also in the central zenhub.log:
2013-05-16 11:00:31,648 INFO zen.ZenHub: Worker reports 2013-05-16 11:00:31,647 ERROR zen.hub: Unhandled exception in zenhub service ZenPacks.zenoss.Z
enJMX.services.ZenJMXConfigService.ZenJMXConfigService: You are not allowed to access 'zJmxAuthenticate' in this context
Traceback (most recent call last):
File "/usr/local/zenoss/zenoss/Products/ZenCollector/services/config.py", line 84, in _wrapFunction
return functor(*args, **kwargs)
File "/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.5.3-py2.6.egg/ZenPacks/zenoss/ZenJMX/services/ZenJMXConfigService.py", line 177, in _createDeviceProxy
device, None, template, ds))
File "/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.5.3-py2.6.egg/ZenPacks/zenoss/ZenJMX/services/ZenJMXConfigService.py", line 95, in __init__
self.copyProperties(device, datasource)
File "/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.5.3-py2.6.egg/ZenPacks/zenoss/ZenJMX/services/ZenJMXConfigService.py", line 115, in copyProperties
value = talesEval('string:%s' % (value,), device)
File "/usr/local/zenoss/zenoss/Products/ZenUtils/ZenTales.py", line 33, in talesEval
res = compiled(getEngine().getContext(contextDict))
File "/usr/local/zenoss/python/lib/python2.6/site-packages/zope.tales-3.4.0-py2.6.egg/zope/tales/expressions.py", line 261, in __call__
v = var(econtext)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/zope.tales-3.4.0-py2.6.egg/zope/tales/expressions.py", line 217, in __call__
return self._eval(econtext)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/Products/PageTemplates/Expressions.py", line 127, in _eval
ob = self._subexprs[-1](econtext)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/zopeNN.tales-3.4.0-py2.6.egg/zope/tales/expressions.py", line 124, in _eval
ob = self._traverser(ob, element, econtext)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/Products/PageTemplates/Expressions.py", line 76, in boboAwareZopeTraverse
object = object.restrictedTraverse(name)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/OFS/Traversable.py", line 310, in restrictedTraverse
return self.unrestrictedTraverse(path, default, restricted=True)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/OFS/Traversable.py", line 283, in unrestrictedTraverse
next = guarded_getattr(obj, name, _marker)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/AccessControl/ImplPython.py", line 729, in guarded_getattr
aq_acquire(inst, name, aq_validate, validate)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/AccessControl/ImplPython.py", line 671, in aq_validate
return validate(inst, object, name, v)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/AccessControl/ImplPython.py", line 565, in validate
self._context)
File "/usr/local/zenoss/python/lib/python2.6/site-packages/Zope2-2.12.1-py2.6-linux-x86_64.egg/AccessControl/ImplPython.py", line 464, in validate
raise Unauthorized(name, value)
Unauthorized: You are not allowed to access 'zJmxAuthenticate' in this context
Why would this happen, or what debugging steps could I take?
Apologies for the size of the tracebacks.