Pyro 3.16: include in cylc distribution + related improvements

README

@@ -33,3 +33,15 @@ Code Contributors (git shortlog -s -n):
   * Annette Osprey
   * Tim Whitcomb
   * Alex Reinecke
+
+Acknowledgement for non-cylc Work:
+  * lib/isodatetime/:
+    Unmodified external software library released under the LGPL license.
+    See [metomi/isodatetime](https://github.com/metomi/isodatetime).
+  * lib/Pyro/:
+    External software library released under the MIT license.
+    Minor modification based on version 3.16.
+    See [Pyro 3.16](https://pypi.python.org/pypi/Pyro).
+
+Licences for non-cylc work included in this distribution can be found in the
+licences/ directory.

bin/cylc-check-software

@@ -77,8 +77,7 @@ fi
 
 # Python packages
 # sqlite3 is part of the standard library since Python 2.5
-PKGS="Pyro-3:Pyro.core \
-Jinja2:jinja2 \
+PKGS="Jinja2:jinja2 \
 pygraphviz:pygraphviz \
 pygtk:pygtk"
 

doc/cug.tex

@@ -386,15 +386,11 @@ \section{Required Software}
     \item {\bf OS: A Linux or Unix variant (including, reportedly, Apple OS X).}
     \item {\bf The Python Language, version 2.6 or later, but not 3.x yet}.
         \newline \url{https://python.org}
-    \item {\bf Pyro-3 (Python Remote Objects), version 3.10$+$, latest
-        tested 3.16}; not Pyro4 as yet. Pyro is used
-        for network communication between server processes (cylc suites)
-        and client programs (running tasks, the gcylc GUI, user commands).
-        \newline \url{https://pypi.python.org/pypi/Pyro/}
     \item {\bf sqlite (a server-less, zero-configuration, SQL database
-        engine}). This is likely included in your Linux distribution
-        already. Cylc generates an sqlite database for each suite as it
-        runs, to record task events and status.
+        engine}). The Python API to sqlite is part of the Python standard
+        library. The command line interface is likely included in your Linux
+        distribution already. Cylc generates an sqlite database for each suite
+        as it runs, to record task events and status.
         \newline \url{http://sqlite.org}
 \end{myitemize}
 
@@ -445,6 +441,12 @@ \section{Required Software}
     \item ImageMagick (for image scaling)
 \end{myitemize}
 
+Note: Older versions of cylc require Pyro-3 (Python Remote Objects). Current
+version of cylc includes a modified version of Pyro 3.16 in its distribution.
+Pyro is used for network communication between server processes (cylc suites)
+and client programs (running tasks, the gcylc GUI, user commands). \newline
+\url{https://pypi.python.org/pypi/Pyro/}
+
 Finally, cylc makes heavy use of Python {\em ordered dictionary}
 data structures.  Significant speedup in parsing large suites can be had
 by installing the
@@ -463,46 +465,6 @@ \subsection{Known Version Compatibility Issues}
 
 Cylc should run ``out of the box'' on recent Linux distributions.
 
-For distributed suites the Pyro versions installed on all suite or task
-hosts must be mutually compatible. Using identical Pyro versions
-guarantees compatibility but may not be strictly necessary because
-cylc uses Pyro rather minimally.
-
-\subsubsection{Pyro 3.9 and Earlier}
-
-Beware of Linux distributions that come packaged with old Pyro versions.
-Pyro 3.9 and earlier is not compatible with the new-style Python classes
-used in cylc. It has been reported that Ubuntu 10.04 (Lucid Lynx),
-released in September 2009, suffers from this problem. Surprisingly,
-so does Ubuntu 11.10 (Oneiric Ocelot), released in October 2011 - and
-therefore, presumably, all earlier Ubuntu releases.  Attempting to run
-a suite with Pyro 3.9 or earlier installed results in the following
-Python traceback:
-
-\lstset{language=transcript}
-\begin{lstlisting}
-Traceback (most recent call last):
- File "/home/hilary/cylc/bin/_run", line 232, in <module>
- server = start()
- File "/home/hilary/cylc/bin/_run", line 92, in __init__
- scheduler.__init__( self )
- File "/home/hilary/cylc/lib/cylc/scheduler.py", line 141, in
-__init__
- self.load_tasks()
- File "/home/hilary/cylc/bin/_run", line 141, in load_tasks_cold
- itask = self.config.get_task_proxy( name, tag, 'waiting',
-stopctime=None, startup=True )
- File "/home/hilary/cylc/lib/cylc/config.py", line 1252, in
-get_task_proxy
- return self.taskdefs[name].get_task_class()( ctime, state,
-stopctime, startup )
- File "/home/hilary/cylc/lib/cylc/taskdef.py", line 453, in
-tclass_init
- print '-', sself.__class__.__name__, sself.__class__.__bases_
-AttributeError: type object 'A' has no attribute '_taskdef__bases_'
-_run --debug testsuite.1322742021 2010010106 failed: 1
-\end{lstlisting}
-
 \subsubsection{Apple Mac OSX}
 
 It has been reported that cylc runs fine on OSX 10.6 SnowLeopard, but on
@@ -542,7 +504,7 @@ \section{Installation}
 
 \subsection{Install The External Dependencies}
 
-First install Pyro, graphviz, Pygraphviz, Jinja2, \TeX, and ImageMagick
+First install graphviz, Pygraphviz, Jinja2, \TeX, and ImageMagick
 using the package manager on your system if possible; otherwise download
 the packages manually and follow their native installation documentation.
 On a modern Linux system, this is very easy. For example, to install
@@ -584,7 +546,6 @@ \subsection{Install The External Dependencies}
  + Graphviz ... ok
  + sqlite ... ok
 Checking for Python packages:
- + Pyro-3 ... ok
  + Jinja2 ... ok
  + pygraphviz ... ok
  + pygtk ... ok
@@ -693,7 +654,6 @@ \subsubsection{Some Guidelines}
  + Graphviz ... ok
  + sqlite ... ok
 Checking for Python packages:
- + Pyro-3 ... ok
  + Jinja2 ... ok
  + pygraphviz ... ok
  + pygtk ... ok
@@ -1370,9 +1330,8 @@ \subsection{Remote Tasks}
 unless the last-resort one way {\em task polling} communication method
 is used.
 
-\item Cylc must be installed on the task host, and for the default task
-    communication method so must Pyro. Other software dependencies like
-    graphviz and Jinja2 are not required there.
+\item Cylc must be installed on the task host. Other software dependencies like
+graphviz and Jinja2 are not required there.
 
 \item The suite passphrase must be installed on the remote account,
 to allow remote tasks to authenticate with the suite.
@@ -4339,7 +4298,7 @@ \subsubsection{Remote Task Hosting}
 \label{RunningTasksOnARemoteHost}
 
 If a task declares an owner other than the suite owner and/or
-a host other than the suite host, cylc will use passwordless ssh to
+a host other than the suite host, cylc will use non-interactive ssh to
 execute the task on the \lstinline=owner@host= account by the configured
 job submission method,
 \lstset{language=suiterc}
@@ -4355,21 +4314,22 @@ \subsubsection{Remote Task Hosting}
 \lstset{language=transcript}
 For this to work,
 \begin{myitemize}
-    \item passwordless ssh must be configured between the suite and task
-    host accounts.
-
-    \item Your shell initialization (.profile, .bashrc, .cshrc, etc) on the
-    remote host must not produce any standard output as it may confuse commands
-    such as scp. See \url{http://www.openssh.com/faq.html#2.9} for more
-    information.
+    \item non-interactive ssh/scp is required from the suite host to the task
+    host accounts. Note: Your shell initialization (.profile, .bashrc, .cshrc,
+    etc) on the remote host must not produce any standard output as it may
+    confuse commands such as scp. See \url{http://www.openssh.com/faq.html#2.9}
+    for more information.
 
-    \item cylc must be installed on task hosts so that remote tasks can
-    use cylc messaging and poll or kill commands.
+    \item cylc must be installed on task hosts.
     \begin{myitemize}
-        \item Pyro and the suite passphrase are needed on tasks hosts,
-            unless the ssh or polling task communication methods are
-            used. Other cylc software dependencies such as graphviz and
+        \item Optional software dependencies such as graphviz and
         Jinja2 are not needed on task hosts.
+        \item If polling task communication is used, there is no other
+        requirement.
+        \item If SSH task communication is configured, non-interactive ssh is
+        required from the task host to the suite host.
+        \item If Pyro (default) task communication is configured, the task host
+        should have access to the Pyro port on the suite host.
     \end{myitemize}
     \item the suite definition directory, or some fraction of its
         content, can be installed on the task host, if needed.
@@ -5290,9 +5250,9 @@ \subsection{Job Polling}
 
 For supported job submission methods, one-way polling can be used to determine
 actual job status: the suite daemon executes a process on the task host, by
-passwordless ssh, to interrogate the batch queueing system there, and to read a
-{\em status file} that is automatically generated by the task job script as it
-runs.
+non-interactive ssh, to interrogate the batch queueing system there, and to
+read a {\em status file} that is automatically generated by the task job script
+as it runs.
 
 Polling may be required to update the suite state correctly after unusual
 events such as a machine being rebooted with tasks running on it, or network
@@ -5328,8 +5288,8 @@ \subsection{Job Polling}
 \subsection{Job Killing}
 
 For supported job submission methods, the suite daemon can execute a process on
-the task host, by passwordless ssh, to kill a submitted or running job according
-to its job submission method.
+the task host, by non-interactive ssh, to kill a submitted or running job
+according to its job submission method.
 
 Tasks can be killed on demand by right-clicking on them in gcylc or using the
 \lstinline=cylc kill= command.
@@ -5511,13 +5471,13 @@ \subsection{How Tasks Interact With Running Suites}
         \item {\bf Pyro:} direct messaging via network sockets using
         Pyro (Python Remote Objects).
         \item {\bf ssh:} for tasks hosts that block access to the
-        network ports required by Pyro, cylc can use passwordless ssh to
+        network ports required by Pyro, cylc can use non-interactive ssh to
         re-invoke task messaging commands on the suite host (where
         ultimately Pyro is still used to connect to the server process).
     \end{myenumerate}
 \item {\bf polling:} for task hosts that do not allow return routing to
 the suite host for Pyro or ssh, cylc can poll tasks at configurable
-intervals, using passwordless ssh.
+intervals, using non-interactive ssh.
 \end{myenumerate}
 
 The Pyro communication method is the default because it is the most
@@ -5636,7 +5596,7 @@ \subsection{How Commands Interact With Running Suites}
 
 User-invoked commands that connect to running suites can also choose
 between direct communication across network sockets (Pyro) and
-re-invocation of commands on the suite host using passwordless ssh
+re-invocation of commands on the suite host using non-interactive ssh
 (there is a \lstinline=--use-ssh= command option for this purpose).
 
 The gcylc GUI requires direct Pyro connections to its target suite. If
@@ -5813,7 +5773,7 @@ \subsection{Remote Control - Passphrases and Network Ports}
 
 So, to connect to a suite running on another account you must install
 the suite passphrase (\ref{passphrases}), and configure
-passwordless ssh so that the port number can be retrieved from the
+non-interactive ssh so that the port number can be retrieved from the
 remote port file. Then use the \lstinline=--user= and
 \lstinline=--host= command options to connect:
 \lstset{language=transcript}

doc/gh-pages/index.html

@@ -209,29 +209,35 @@ <h3><span>Features</span></h3>
                 <h3><span>Requirements</span></h3>
 
                 <ul>
-                    <li> Operating System: Linux or Unix
+                    <li> Operating System: Linux or Unix</li>
 
-                    <li> Python 2.6 or later (but not 3.x as yet).
+                    <li> Python 2.6 or later (but not 3.x as yet).</li>
                     <!-- <sup><a href="#fn2" id="fnr2" title="click to see footnote">2</a></sup> -->
 
                     <li> <a href="http://www.pygtk.org">PyGTK</a>,
                     a Python wrapper for the GTK+ graphical user
                     interface toolkit, included in most Linux Distributions.
-
-                    <li> <a href="https://pypi.python.org/pypi/Pyro/">Pyro 3</a>
-                    (Python Remote Objects) - latest version 3.16.
+                    This requirement is for running the GUIs, and is optional
+                    if you don't use the GUIs.</li>
 
                     <li> The <a href="http://www.graphviz.org">graphviz</a> graph
-                    layout engine (latest version tested: 2.28.0).
-
-                    <li> <a href="http://networkx.lanl.gov/pygraphviz">Pygraphviz</a>,
+                    layout engine (latest version tested: 2.28.0) and
+                    <a href="http://networkx.lanl.gov/pygraphviz">Pygraphviz</a>,
                     a python interface to graphviz (latest version tested: 1.1).
-
-                    <li> <a href="http://jinja.pocoo.org">Jinja2</a>, a template processor for Python.
-
-                    <li> <b>LaTeX</b> (for <code>pdflatex</code>) and <b>tex4ht</b>
-                    (for <code>htlatex</code>) - needed to generate PDF and HTML
-                    copies of the Cylc User Guide from the LaTeX source files.
+                    This requirement is for running <code>cylc gui</code> in
+                    graph view and for commands such as <code>cylc
+                    validate</code> to do a full analysis of the dependency
+                    graph.</li>
+
+                    <li> <a href="http://jinja.pocoo.org">Jinja2</a>,
+                    a template processor for Python. This requirement is for
+                    parsing suite configuration files with Jinja2, and is
+                    optional if you don't have such suites.</li>
+
+                    <li> <b>LaTeX</b> (for <code>pdflatex</code>) and
+                    <b>tex4ht</b> (for <code>htlatex</code>). This requirement
+                    is for generating PDF and HTML copies of the Cylc User Guide
+                    from the LaTeX source files.</li>
                 </ul>
 
                 <p class="p1"><span>

doc/suiterc.tex

@@ -1279,7 +1279,7 @@ \subsection{[runtime]}
 \subparagraph[owner]{[runtime] \textrightarrow [[\_\_NAME\_\_]] \textrightarrow [[[remote]]] \textrightarrow owner}
 
 The username of the task host account. This is (only) used in the
-passwordless ssh command invoked by the suite daemon to submit the remote task
+non-interactive ssh command invoked by the suite daemon to submit the remote task
 (consequently it may be defined using local environment variables
 (i.e.\ the shell in which cylc runs, and [cylc] \textrightarrow [[environment]]).
 

lib/Pyro/EventService/Clients.py

@@ -0,0 +1,91 @@
+#############################################################################
+#  
+#	Event Service client base classes
+#
+#	This is part of "Pyro" - Python Remote Objects
+#	which is (c) Irmen de Jong - irmen@razorvine.net
+#
+#############################################################################
+
+import Pyro.core, Pyro.naming, Pyro.constants
+import Pyro.EventService.Server
+from Pyro.EventService.Event import Event
+from Pyro.errors import *
+
+# SUBSCRIBER: subscribes to certain events.
+class Subscriber(Pyro.core.CallbackObjBase):
+	def __init__(self, ident=None, esURI=None):
+		Pyro.core.CallbackObjBase.__init__(self)
+		Pyro.core.initServer()
+		Pyro.core.initClient()
+		daemon = Pyro.core.Daemon()
+		if esURI:
+			check=Pyro.core.PyroURI(esURI)
+			self.ES_uri=esURI
+		else:		
+			locator = Pyro.naming.NameServerLocator(identification=ident)
+			self.NS = locator.getNS(host=Pyro.config.PYRO_NS_HOSTNAME)
+			daemon.useNameServer(self.NS)
+			self.ES_uri = self.NS.resolve(Pyro.constants.EVENTSERVER_NAME)
+		daemon.connect(self)  #  will also set self.daemon...
+		self.ES_ident=ident
+		self.abortListen=0
+		self.daemon=daemon	# make sure daemon doesn't get garbage collected now
+
+	def getES(self):
+		# we get a fresh proxy to the ES because of threading issues.
+		# (proxies can not be reused across multiple threads)
+		eventservice=Pyro.core.getProxyForURI(self.ES_uri)
+		eventservice._setIdentification(self.ES_ident)
+		return eventservice
+
+	def subscribe(self,subjects):
+		# Subscribe to one or more subjects.
+		# It is safe to call this multiple times.
+		self.getES().subscribe(subjects, self.getProxy())
+	def subscribeMatch(self,subjectPatterns):
+		# Subscribe to one or more subjects (by pattern)
+		# It is safe to call this multiple times.
+		self.getES().subscribeMatch(subjectPatterns, self.getProxy())
+	def unsubscribe(self, subjects):
+		# Unsubscribe the subscriber for the given subject(s).
+		self.getES().unsubscribe(subjects, self.getProxy())
+
+	def abort(self):
+		self.abortListen=1
+
+	def setThreading(self, threaded):
+		self.getDaemon().threaded=threaded
+
+	def listen(self):
+		self.getDaemon().requestLoop(lambda s=self: not s.abortListen)
+
+	def event(self, event):					# callback, override this!
+		print event
+
+# PUBLISHER: publishes events.
+class Publisher(object):
+	def __init__(self, ident=None, esURI=None):
+		Pyro.core.initClient()
+		if esURI:
+			check=Pyro.core.PyroURI(esURI)
+			self.ES_uri=esURI
+		else:
+			locator = Pyro.naming.NameServerLocator(identification=ident)
+			ns = locator.getNS(host=Pyro.config.PYRO_NS_HOSTNAME)
+			self.ES_uri = ns.resolve(Pyro.constants.EVENTSERVER_NAME)
+			ns._release()   # be very sure to release the socket
+		self.ES_ident = ident
+
+	def getES(self):
+		# we get a fresh proxy to the ES because of threading issues.
+		# (proxies can not be reused across multiple threads)
+		eventservice=Pyro.core.getProxyForURI(self.ES_uri)
+		eventservice._setIdentification(self.ES_ident)
+		return eventservice
+
+	def publish(self, subjects, msg):
+		es=self.getES()
+		es.publish(subjects,msg)
+		es._release()   # be very sure to release the socket
+