Skip to content

Commit

Permalink
Responded to KSK review comments. Fixed a few more "JupyterLab Aspect…
Browse files Browse the repository at this point in the history
…" references.

Also consistently capitalized "Aspect" where appropriate.
  • Loading branch information
gpdf committed Apr 5, 2019
1 parent 72b0176 commit 0631deb
Show file tree
Hide file tree
Showing 5 changed files with 29 additions and 26 deletions.
2 changes: 1 addition & 1 deletion backend.tex
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ \section{Backend Services\label{sec:backend}}
\subsection{Database Services}

Key LSST catalogs, both for \textbf{Prompt} and \textbf{Data Release} data products, will be stored in relational databases and made available for querying by users using
% the Structured Query Language (SQL) as well as
% the Structured Query Language (SQL) as well as
the Astronomical Data Query Language (ADQL), with some restrictions.
These products, and expectations surrounding their schemas, are further described in the \DPDD.

Expand Down
8 changes: 6 additions & 2 deletions evolution.tex
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@ \section{Design for Evolution\label{sec:evolution}}

This document captures DM's response to our best estimate of what the expectations of LSST users are likely to be, starting with LSST Commissioning and through the first few years of LSST Operations.

Through a decade of operations, it is likely that both the user expectations will change, as well as the technologies available to respond to them. For example, the emergence of Jupyter as the dominant mode of remote data analysis in the astronomical community has caused the present vision and design of the LSST Science Platform to be markedly different than the original conceptual design of the Science User Interface and Tools (LDM-131). There is no reason to believe that similar shifts will not happen in Operations as well.
Through a decade of operations, it is likely that both the user expectations will change, as well as the technologies available to respond to them.
For example, the emergence of Jupyter as the dominant mode of remote data analysis in the astronomical community has caused the present vision and design of the LSST Science Platform to be markedly different than the original conceptual design of the Science User Interface and Tools (LDM-131).
There is no reason to believe that similar shifts will not happen in Operations as well.

The LSST will therefore proactively design the LSST Science Platform services with such an evolution in mind, to a degree permitted by the available budget and schedule constraints. An example of such \textbf\emph{design for evolution} is the concept of loosely coupled aspects itself (the Portal, JupyterLab, and Web APIs), that all expose different views of the same underlying data model and workspace. Design principles like these will allow for additions of new LSST Science Platform aspects (e.g., a different Jupyter-like technology, should one emerge), replacements of aspects (e.g., migrating to a different Portal technology), as well as retirement of aspects that are not widely used.
The LSST will therefore proactively design the LSST Science Platform services with such an evolution in mind, to a degree permitted by the available budget and schedule constraints.
An example of such \textbf\emph{design for evolution} is the concept of loosely coupled aspects itself (the Portal, Notebook, and Web APIs), that all expose different views of the same underlying data model and workspace.
Design principles like these will allow for additions of new LSST Science Platform Aspects (e.g., a different Jupyter-like technology, should one emerge), replacements of Aspects (e.g., migrating to a different Portal technology), as well as retirement of Aspects that are not widely used.
8 changes: 4 additions & 4 deletions intro.tex
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ \subsection{Goals}
make available the data collected by its telescope and camera,
\footnote{This includes the raw and processed calibration and engineering data, in addition to the data collected by the science sensors.
Because much of LSST science will be systematics limited, access to engineering data will enable a better understanding and correction of subtle instrumental and/or environmental effects.}
as well as enable ``next-to-the-data'' creation of added-value \textit{User Generated} data products (see the \SRD and the \LSR).
as well as enable ``next-to-the-data'' creation of added-value \emph{User Generated} data products (see the \SRD and the \LSR).

This document describes the vision for the services to be put into place to
fulfill the ``\emph{making available}'' and ``\emph{User Generated} data product creation``
Expand Down Expand Up @@ -47,7 +47,7 @@ \subsection{LSST Science Platform Overview}
set of services that will be offered to LSST users.
The platform exposes the LSST data
and services to the user through three primary user-facing ``\textit{Aspects}'' --- the web \textbf{Portal},
and services to the user through three primary user-facing ``\emph{Aspects}'' --- the web \textbf{Portal},
the \textbf{Notebook} analysis environment, and a machine-accessible \textbf{Web API} interface.
These Aspects provide three different ways to access the data sets and analysis services provided in the LSST Data Access Centers (Figure~\ref{fig:layeredLSP}).
Expand Down Expand Up @@ -82,11 +82,11 @@ \subsection{LSST Science Platform Overview}
with other astronomical archives, bringing added value to the LSST dataset.
We describe it in more detail in \secref{sec:apis}.
Enabling these user-facing Aspects is a set of backend services.
The user-facing Aspects all depend on a common set of back-end services, which facilitate the LSP's operation as an integrated whole.
The Data Releases will be organized as catalogs kept in relational database management systems, as well as repositories of files.
The alert distribution system will facilitate the distribution of Alert Streams to community brokers and end-users (see the \DPDD for details).
These services will be complemented by additional User Database, File Storage, and Batch Computing services, as well as a pre-installed Software Tools suite, containing both LSST and community software.
They will provide the computational power, data storage, and analytics capabilities needed to enable LSST data analysis as well as the creation and federation of \textit{User Generated} data products.
They will provide the computational power, data storage, and analytics capabilities needed to enable LSST data analysis as well as the creation and federation of \emph{User Generated} data products.
We further describe these in \secref{sec:backend}.
Finally, the LSST Science Platform is being envisioned to enable and encourage
Expand Down
18 changes: 9 additions & 9 deletions methodology.tex
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@ \subsection{Iterative Development Leveraging Existing Technologies }
The services constructed for the LSST Science Platform will be developed following the iterative Agile methodology. While most of LSST software development follows this approach, adopting it is especially advantageous for user-facing services. There, iterative development and nearly continuous stakeholder feedback can provide guidance as to the details of features to be implemented, the continued validity of the approach taken, and the expected focus of intermediate milestones.

The development of the Portal, Notebook, and Web API Aspects will start from significant existing code bases and prior art.
This is a deliberate approach designed to minimize technological risk and leverage end-user familiarity with these interfaces.
This is a deliberate approach designed to minimize technical risk and leverage end-user familiarity with these interfaces.
The latter also reduces the barrier to user adoption of the products eventually delivered for LSST.

The \textbf{Portal Aspect} is based on the existing, production quality, archive portal interface developed at IRSA/IPAC --- the \emph{Firefly} toolkit.
The primary challenge is integrating the existing Firefly code, and updating the user experience to conform to anticipated user expectations (e.g., supporting all-sky maps and pan/zoom/click-type exploration).
Consistent with the general philosophy, DM should look at achieving the necessary upgrades by re-using existing well-known libraries and tools (e.g., the Plotly library is being adopted to provide 2D and 3D visualizations of tabular data).

The \textbf{Notebook Aspect} environment will be based on the open-source JupyterLab product delivered and maintained by the Jupyter team.
The development of this Aspect of the LSST Science Platform will focus on deployment and integration with the LSST-specific backend services and other aspects of the platform, rather than developing new or radically different features within the JupyterLab product.
The development of this Aspect of the LSST Science Platform will focus on deployment and integration with the LSST-specific backend services and other Aspects of the platform, rather than developing new or radically different features within the JupyterLab product.

Finally, the \textbf{Web API Aspect} is envisioned as implementing existing, widely-adopted, community protocols (in particular, ones from the Virtual Observatory suite of protocols and standards).
Similarly to the other Aspects, it will benefit from leveraging existing codes and libraries wherever appropriate.
Expand All @@ -31,29 +31,29 @@ \subsection{Prioritization Guidance}
\item Deployment of the initial Firefly back-end within the (prototype) LSST Data Access Center at NCSA.
\item Integration of the initial Firefly front- and back-ends with LSST Science Platform backend services. For example, this includes the authentication and authorization mechanisms, relational databases, file stores, etc.
\item User experience improvements, such as addition of all-sky maps with pan/zoom/select navigation metaphors, modernization of the look-and-feel, streamlining of the UI and deprecation of rarely used widgets. \textbf{Once this level of functionality is met (at scale), the Portal Aspect will have achieved the minimum level of viability for deployment to operations}.
\item Improved user workflow integration with other aspects of the LSST Science Platform. For example, it should be possible to begin data exploration in the Portal Aspect (e.g., by interactively selecting data sets) and seamlessly transfer the sub-selected catalogs and images to the Notebook Aspect's JupyterLab environment for further, more complex, analysis using provided Python libraries.
\item Improved user workflow integration with other Aspects of the LSST Science Platform. For example, it should be possible to begin data exploration in the Portal Aspect (e.g., by interactively selecting data sets) and seamlessly transfer the sub-selected catalogs and images to the Notebook Aspect's JupyterLab environment for further, more complex, analysis using provided Python libraries.
\item Addition of new widgets and abilities to the Portal, that address most requested and broadly useful end-user needs.
\item Widget-level integration with JupyterLab.
\end{enumerate}

Notebook Aspect:
\begin{enumerate}
\item Deployment of the initial JupyterLab product within the (prototype) LSST Data Access Center at NCSA.
\item Integration of the JupyterLab product with LSP backend services, most notably authentication and authorization, user management, databases, and file stores. \textbf{Once this level of functionality is met (at scale), the JupyterLab aspect will have achieved the minimum level of viability for deployment to commissioning and operations}.
\item Integration of the JupyterLab product with LSP backend services, most notably authentication and authorization, user management, databases, and file stores. \textbf{Once this level of functionality is met (at scale), the Notebook Aspect will have achieved the minimum level of viability for deployment to commissioning and operations}.
\item Development of libraries and utilities to ease the submission of user-written code from Jupyter notebooks to the batch compute system.
\item Creation and curation of a library of 3rd party code that will be made available to LSP end-users.
\end{enumerate}

Web API Aspect:
\begin{enumerate}
\item Development and deployment of initial data access APIs needed to satisfy the back-end needs of the Portal and JupyterLab aspects.
\item Development and deployment of initial data access APIs needed to satisfy the back-end needs of the Portal and Notebook Aspects.
These may not yet ``speak'' the final, standards-compliant, protocols.
\item Integration of the Web API aspect with LSP backend services, most notably authentication and authorization, user management, databases, and file stores.
\item Integration of the Web API Aspect with LSP backend services, most notably authentication and authorization, user management, databases, and file stores.
\item Deployment of key tabular data in IVOA-compliant data models --- particularly image metadata in the ObsCore format.
\item Deployment of critical protocols (including SCS, TAP, ObsTAP, SIAv2, SODA, VOEvent streaming support, and VO Registry support) at commonly-encountered levels of standards compliance (e.g., the most commonly used ADQL features).
\textbf{Once this level of functionality is met (at scale), the Web API aspect will have achieved the minimum level of viability for deployment to operations}
\item Further improvement of standards-compliance as well as deployment of additional standards-compliant protocols, e.g., DataLink, throughout the Web API aspect, and integration with all other elements of the Platform.
\textbf{Once this level of functionality is met (at scale), the Web API Aspect will have achieved the minimum level of viability for deployment to operations}
\item Further improvement of standards-compliance as well as deployment of additional standards-compliant protocols, e.g., DataLink, throughout the Web API Aspect, and integration with all other elements of the Platform.
\item Further conformance of tabular data to IVOA and community data models, notably CAOM2.
\end{enumerate}

It is assumed that the development of backend services will be driven by the needs of the front-end aspects.
It is assumed that the development of backend services will be driven by the needs of the front-end Aspects.
19 changes: 9 additions & 10 deletions user.tex
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ \subsection{Notebook (JupyterLab) Aspect\label{sec:jupyter}}
\end{figure}

The \textbf{Notebook Aspect}
%, based on the Jupyter family of technologies (such as JupyterHub and JupyterLab),
will be provided to allow for more sophisticated data selection, analysis, and creation of added value \textit{User Generated} data products.
%, based on the Jupyter family of technologies (such as JupyterHub and JupyterLab),
will be provided to allow for more sophisticated data selection, analysis, and creation of added value \emph{User Generated} data products.
A screen capture of a mature prototype of the Notebook Aspect is shown in Figure~\ref{fig:JupyterLab}.

The Notebook Aspect user experience will be nearly identical to working with
Expand All @@ -66,18 +66,17 @@ \subsection{Notebook (JupyterLab) Aspect\label{sec:jupyter}}

We will provide JupyterLab instances to LSST users in an environment carrying a
library of preinstalled commonly used and useful software tools:
Astropy, the LSST science pipelines, Anaconda Scientific Python Distribution, the PyViz visualization toolkit, and others.
Astropy, the LSST science pipelines, Anaconda Scientific Python Distribution, the PyViz visualization toolkit, and others.
The users will be able to upload and install their own tools as well.
Non-trivial shared computing cluster resources will be accessible through this environment as well, enabling the generation of \textit{User Generated} data products.
Non-trivial shared computing cluster resources will be accessible through this environment as well, enabling the generation of \emph{User Generated} data products.

% Add something about writing glue code to make it easy to submit jobs, and how Notebooks will be the primary user interface to the backend batch system...

The Notebook Aspect of the science platform will play a key role in commissioning,
quality assessment, and science validation of the as-built system.
It will be the primary
method of performing interactive analysis of acquired data (e.g., adjusting and executing
prepared notebooks driving commissioning tasks), as well as commanding
the batch resources to execute larger processing tasks.
It will be the primary method of performing interactive analysis of acquired data
(e.g., adjusting and executing prepared notebooks driving commissioning tasks).
Through the terminal functionality of the JupyterLab environment, it will also permit commanding the batch resources to execute larger processing tasks.
Due to this, we expect the
Notebook Aspect to reach maturity earlier than the others, and certainly in time for
commissioning.
Expand Down Expand Up @@ -121,7 +120,7 @@ \subsection{Web API Aspect\label{sec:apis}}

\subsection{Integrated environment}

All the Aspects of the LSST Science Platform are intended to be \textit{well integrated}, enabling a seamless workflow so the users will be able to move back and forth between them as needs dictate.
All the Aspects of the LSST Science Platform are intended to be \emph{well integrated}, enabling a seamless workflow so the users will be able to move back and forth between them as needs dictate.
The aim is to enable a user to find or create data in one Platform Aspect, and view or analyze that data in another.

As an example of how these connections can aid a user in exploring the LSST data, data queries will be shareable across the Portal, Notebook, and Web API Aspects.
Expand All @@ -136,7 +135,7 @@ \subsection{Integrated environment}
\subsection{Next to Data Processing\label{sec:n2d}}
Many LSST science cases will require analysis of the full LSST Object and/or Source tables; subsetting and downloading reduced catalogs will not suffice for science that requires, for example, fitting and computing features of light curves for large numbers of Objects, creating maps of stellar density, or training machine learning models.
The LSST Object table alone will be $\approx$~50TB for Data Release 1 and $\approx$~300TB for Data Release 11 at the close of the 10-year survey.
The LSST Science Platform will enable user-driven large-scale search and analysis capabilities of the LSST data by providing a \textit{Next-to-Data} processing environment, allowing users to bring their analysis to the data.
The LSST Science Platform will enable user-driven large-scale search and analysis capabilities of the LSST data by providing a \emph{Next-to-Data} processing environment, allowing users to bring their analysis to the data.
This environment will be supported by backend services, as described in \ref{sec:backend}

\subsection{Supporting Collaborative Work\label{sec:collab}}
Expand Down

0 comments on commit 0631deb

Please sign in to comment.