Merge branch 'master' into aloftus-update-process-for-request-immutability

Author: andylytical

cpp/style.rst

@@ -807,26 +807,27 @@ Arguments common to all overloads of a function should precede those that differ
 
 .. _style-guide-cpp-3-34:
 
-3-34. Uncertainty values associated with a variable SHOULD be suffixed by one of ``Var``, ``Cov``, ``Sigma``.
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-There is no universal suffix for uncertainties; i.e. no ``Err`` suffix will be used.
-The cases that we have identified, and their appropriate suffixes, are:
+3-34. Uncertainty values associated with a variable SHOULD be suffixed by one of ``Err`` or ``Cov``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-- Standard deviation: ``Sigma`` (not ``Rms``, as ``rms`` doesn't imply that the mean's subtracted)
-- Covariance: ``Cov``
-- Variance: ``Var``
+Following the DPDD, we use ``Err`` (not ``Sigma``) to specify error quantities, as a standard deviation.
+``Sigma`` should be used to specify the inherent width of a distribution or function.
 
 .. code-block:: cpp
 
-   float xAstrom;          // x position computed by a centroiding algorithm
-   float xAstromSigma;     // Uncertainty of xAstrom
-   float yAstrom;
-   float yAstromSigma;
-   float xyAstromCov;
+    // for single measurements
+    float xCentroid;        // x position computed by a centroiding algorithm
+    float xCentroidErr;     // Uncertainty of xCentroid
+    float yCentroid;
+    float yCentroidErr;
+    float xyCentroidCov;    // Covariance of x/y centroid
+
+    // for distributions
+    float fpFluxMean;       // Weighted mean of forced-photometry flux (fpFlux)
+    float fpFluxMeanErr;    // Uncertainty (standard deviation) of fpFluxMean.
+    float fpFluxSigma;      // Standard deviation of the distribution of fpFlux.
 
-The postfix ``Err`` can easily be misinterpreted as error flags.
-Use the full ``Sigma`` since ``Sig`` can easily be misinterpreted as ``Signal``.
+For distribution widths, use the full ``Sigma`` since ``Sig`` can easily be misinterpreted as ``Signal``.
 
 .. _style-guide-cpp-3-35:
 

git/git-lfs.rst

@@ -79,7 +79,7 @@ Trying cloning a small data repository to test your configuration:
 
 .. code-block:: bash
 
-   git clone https://github.com/lsst/testdata_decam
+   git clone https://github.com/lsst/testdata_subaru
 
 *That's it.*
 

legal/copyright.py

@@ -171,7 +171,7 @@ def stringify_year_set(years):
     if len(sys.argv) > 1:
         git_cmd += sys.argv[1:]
 
-    log = subprocess.check_output(git_cmd)
+    log = subprocess.check_output(git_cmd, universal_newlines=True)
 
     # Read in the full hashes of any commits deemed not copyrightable.
     insignificant_list = []

project-docs/technotes.rst

@@ -56,37 +56,27 @@ Create a reStructuredText technote
 ==================================
 
 ReStructuredText-formatted technotes are built with Sphinx_ into websites.
-Create a reStructuredText-formatted technote by messaging the SQuaRE Bot on Slack:
 
-.. code-block:: text
+1. Follow the instructions at `lsst-technote-bootstrap <https://github.com/lsst-sqre/lsst-technote-bootstrap#running-this-cookiecutter-for-development>`__ to manually create a technote repository.
 
-   @sqrbot project create technote series={{<series>}} title={{<title>}} description={{<description>}}
+2. Create a GitHub repository in the appropriate organization with the technote's handle as the name.
+   The organizations are:
 
-Set the ``<title>``, ``<description>`` and ``<series>`` fields (see below) for your technote, but keep the ``{{`` and ``}}`` delimiters.
+   `lsst-dm <https://github.com/lsst-dm>`__
+      DMTN series.
 
-.. note::
-
-   In a Direct Message channel with SQuaRE Bot, don't include the ``@sqrbot`` prefix.
+   `lsst-sqre <https://github.com/lsst-sqre>`__
+      SQR series.
 
-The fields are:
+   `lsst-sims <https://github.com/lsst-sims>`__
+      SMTN series.
 
-series
-   Values can be ``dmtn``, ``sqr`` or ``smtn``.
-   Use ``test`` for testing.
+3. Message the `#dm-docs <https://lsstc.slack.com/archives/dm-docs>`__ Slack channel so that the Travis integration for your technote can be activated.
 
-title
-   Title of the technote.
-   The title doesn't include the handle (DMTN, SQR, or SMTN).
-   You can update the title later by modifying the :ref:`metadata file <technote-rst-metadata>`.
-
-description
-   Short abstract for the technote.
-   The description is used both as an abstract in the technote itself and in the technote's README.
-   You can update description later by editing the technote and the :ref:`metadata file <technote-rst-metadata>`.
+.. note::
 
-SQuaRE Bot prepares technotes in the background after you make your request.
-Go to the GitHub organization for your :ref:`document series <technote-series>` to find your new technote repository.
-Reach out on the `#dm-docs <slack-dm-docs>`_ Slack channel for help.
+   Previously you could use a Slack command, ``@sqrbot project create``, to create a reStructuredText technote.
+   Due to reliability issues with that service, we recommend that you use this manual process for now.
 
 .. _Sphinx: http://www.sphinx-doc.org/en/stable/
 .. _stack-dm-docs: https://lsstc.slack.com/messages/C2B6DQBAL/

python/numpydoc.rst

@@ -1053,6 +1053,13 @@ Class, method, and function docstrings must be placed directly below the declara
 Again, the :ref:`class docstring <py-docstring-class-structure>` takes the place of a docstring for the ``__init__`` method.
 ``__init__`` methods don't have docstrings.
 
+Dunder Methods
+--------------
+
+Special "dunder" methods on classes only need to have docstrings if they are doing anything non-standard.
+For example, if a ``__getslice__`` method cannot take negative indices, that should be noted.
+But if ``__ge__`` returns true if ``self`` is greater than or equal to the argument, that need not be documented.
+
 Examples of Method and Function Docstrings
 ------------------------------------------
 

requirements.txt

@@ -1,2 +1,2 @@
 sphinx_rtd_theme==0.2.0
-documenteer==0.3.0a4
+documenteer==0.3.0

services/ldf-resources.rst

@@ -19,7 +19,7 @@ Machines:  (includes VM and Baremetal)
 - VM server machines - running IBM's Vsphere software.   (@ NCSA 3003, NCSA NPCF, and Chile Base) 
    - virtual machines for numberous development environments 
    - databackbone test beds, container type test beds, demo type machines, monitoring machines 
-- lsst-demo - VM for demos
+- lsst-demo - VM for demos and docker image needs 
 - ATS gateway - VM
 - lsstdb - mysql infrastucture machine for db support 
 - DAQ - camera simulator systems 

services/lsst-dev.rst

@@ -189,18 +189,15 @@ If you are using `Bash`_ — the default shell on ``lsst-dev01`` — try placing
 Load the LSST Environment
 =========================
 
-We provide ready-to-use “shared” versions of the LSST software stack to enable developers to get up and running quickly with no installation step.
-Each shared stack includes a fully fledged Miniconda-based Python environment, a selection of additional development tools, and a selection of builds of the lsst_distrib meta-package.
-The currently maintained stacks are regularly updated to include the latest weekly release, which is tagged as ``current``.
+We provide a ready-to-use “shared” version of the LSST software stack to enable developers to get up and running quickly with no installation step.
+The shared stack includes a fully-fledged Miniconda-based Python environment, a selection of additional development tools, and a selection of builds of the lsst_distrib meta-package.
+The currently stack is regularly updated to include the latest weekly release, which is tagged as ``current``.
 
 The following stacks are currently maintained:
 
 ======================================== ============== ================ =======================================================================================================================================================================================================================
 Path                                     Python Version Toolchain        Description
 ======================================== ============== ================ =======================================================================================================================================================================================================================
-:file:`/ssd/lsstsw/stack2_20171021`      2              ``devtoolset-6`` Located on local, SSD based storage attached to the `lsst-dev01` system: it will support fast interactive use on that machine, but is not accessible across the network.
-:file:`/ssd/lsstsw/stack3_20171021`      3              ``devtoolset-6`` Located on local, SSD based storage attached to the `lsst-dev01` system: it will support fast interactive use on that machine, but is not accessible across the network.
-:file:`/software/lsstsw/stack2_20171022` 2              ``devtoolset-6`` Located on GPFS-based network storage; as such, it is cross-mounted across a variety of LSST systems at NCSA including those configured as part of the `HTCondor pool`_ and :doc:`Verification Cluster <verification>`.
 :file:`/software/lsstsw/stack3_20171023` 3              ``devtoolset-6`` Located on GPFS-based network storage; as such, it is cross-mounted across a variety of LSST systems at NCSA including those configured as part of the `HTCondor pool`_ and :doc:`Verification Cluster <verification>`.
 ======================================== ============== ================ =======================================================================================================================================================================================================================
 
@@ -213,19 +210,14 @@ In addition, the following symbolic links point to particular versions of the st
 =============================== =====================================================================================================
 Path                            Description
 =============================== =====================================================================================================
-:file:`/ssd/lsstsw/stack`       The latest version of the stack on local storage using our standard Python version (currently 3).
-:file:`/ssd/lsstsw/stack2`      The latest version of the stack on local storage and based on Python 2.
-:file:`/ssd/lsstsw/stack3`      The latest version of the stack on local storage and based on Python 3.
 :file:`/software/lsstsw/stack`  The latest version of the stack on networked storage using our standard Python version (currently 3).
-:file:`/software/lsstsw/stack2` The latest version of the stack on networked storage and based on Python 2.
-:file:`/software/lsstsw/stack3` The latest version of the stack on networked storage and based on Python 3.
 =============================== =====================================================================================================
 
 Add a shared stack to your environment and set up the latest build of the LSST applications by running, for example:
 
 .. prompt:: bash
 
-  source /ssd/lsstsw/stack/loadLSST.bash
+  source /software/lsstsw/stack/loadLSST.bash
   setup lsst_apps
 
 (substitute :file:`loadLSST.csh`, :file:`loadLSST.ksh` or :file:`loadLSST.zsh`, depending on your preferred shell).

services/storage.rst

@@ -4,6 +4,12 @@ Storage Resources
 
 There are a few other documents that might have the info you are looking for.
 
+
+- 1.  Look to the - :doc:`data_protection` policy page what the retention policy is, what are immutable files, what is to be placed in each file system area.  
+- 2.  Look to the - :doc:`ldf-resources` LDF resources pages for explaination of each of the file systems, and the type of data and where it's to be located and the policies of each of the file systems.   
+
+This document covers the file systems, and then quotas for currently only the /home file system that is in place.  
+=======
 1.  Look to the :doc:`data_protection` policy page what the retention policy is, what are immutable files, what is to be placed in each file system area.
 2.  Look to the :doc:`ldf-resources` LDF resources pages for explanation of each of the file systems, and the type of data and where it's to be located and the policies of each of the file systems.
 
@@ -15,6 +21,7 @@ Filesystems - in GPFS (4.9PB of storage)
 :file:`/datasets`
     Long term storage of project-approved shared data. Contains immutable data. This is under a disaster recovery policy that every 30 days it is stored and written to nearline tape.
 
+
 :file:`/home`
     Storage of individual-user data. This data is backed up on a daily basis and NCSA retains 30 days of those backups in a snapshot.  It does have quotas on this file system for 1TB for each "directory," and a 1 million INODE quota.
 
@@ -33,29 +40,36 @@ Filesystems - in GPFS (4.9PB of storage)
 Quotas 
 ======
 
+
+Quotas 
+======
+
 Your home directory is the default directory you are placed in when you log on. You should use this space for storing files you want to keep long term such as source code, scripts, etc. Every user has a 1TB home directory quota (total space) and 1 million INODE quota (total number of files).
 
-On **TIME**, quotas were enforced. The soft limit is 1TB and the hard limit is 1.2 TB. The INODE soft quota is 1 million files and the hard limit is 1.2 million files.   If the amount of data in your home directory is over the soft limit  but under the hard limit, there is a grace period of 7 days to get under the soft limit. When the grace period expires, you will not be able to write new files or update any current files until you reduce the amount of data to below the soft limit.
+On 6/17/2018, quotas were enforced. The soft limit is 1TB and the hard limit is 1.2 TB. The INODE soft quota is 1 million files and the hard limit is 1.2 million files.   If the amount of data in your home directory is over the soft limit  but under the hard limit, there is a grace period of 7 days to get under the soft limit. When the grace period expires, you will not be able to write new files or update any current files until you reduce the amount of data to below the soft limit.
+
 
 The command to see your disk usage and limits is quota. Example:
 
 .. code-block:: text
 
-   [jdoe@golubh4 ~]$ quota
+   [jdoe@systemname4 ~]$ quota
    Directories quota usage for user jdoe:
 
    -------------------------------------------------------------------------------------
    |      Fileset       |  Used   |  Soft   |  Hard   |   Used   |   Soft   |   Hard   |
    |                    |  Block  |  Quota  |  Limit  |   File   |   Quota  |   Limit  |
    --------------------------------------------------------------------------------------
-   | home               | 501.1M  | 2G      | 4G      | 14       | 0        | 0        |
-   | cse-shared         | 0       | 1.465T  | 1.953T  | 1        | 0        | 0        |
+
+    | home               | 501.1M  | 2G      | 4G      | 14       | 0        | 0        |
+    | stuff              | 0       | 1.465T  | 1.953T  | 1        | 0        | 0        |
    -------------------------------------------------------------------------------------
 
 Home directories are backed up using snapshots and a separate DR process.
 
-Data Compression
-================
+For space utiziation:  Data Compression
+To reduce space usage in your home directory, an option for files that are not in active use is to compress them. The gzip utility can be used for file compression and decompression. Another alternative is bzip2, which usually yields a better compression ratio than gzip but takes longer to complete. Additionally, files that are typically used together can first be combined into a single file and then compressed using the tar utility.
+
 
 To reduce space usage in your home directory, an option for files that are not in active use is to compress them. The :command:`gzip` utility can be used for file compression and decompression. Another alternative is :command:`bzip2`, which usually yields a better compression ratio than gzip but takes longer to complete. Additionally, files that are typically used together can first be combined into a single file and then compressed using the tar utility.
 
@@ -74,6 +88,7 @@ To decompress the file:
 
 .. code-block:: bash
 
+
    gunzip largefile.dat.gz
 
 Alternatively:
@@ -100,13 +115,9 @@ The convention is to use extension ``.tgz`` in the file name.
 
 To extract the contents of the compressed tar file:
 
-.. code-block:: bash
-
-   tar -xvf largedir.tgz
 
-See the manual pages (``man gzip``, ``man bzip2``, ``man tar``) for more details on these utilities.
+**Notes:**
 
-.. note::
 
    ASCII text and binary files like executables can yield good compression ratios. Image file formats (gif, jpg, png, etc.) are already natively compressed so further compression will not yield much gains.
    Depending on the size of the files, the compression utilities can be compute intensive and take a while to complete. Use the compute nodes via a batch job for compressing large files.

stack/building-pipelines-lsst-io-locally.rst

@@ -34,17 +34,6 @@ Clone the repository:
 
    git clone https://github.com/lsst/pipelines_lsst_io
 
-.. important::
-
-   During this initial phase, ``tickets/DM-11216`` is the integration branch for multi-package builds of `pipelines_lsst_io`_.
-   For now you need to check out that branch:
-
-   .. code-block:: bash
-
-      cd pipelines_lsst_io
-      git checkout tickets/DM-11216
-      cd ..
-
 Then set up the `pipelines_lsst_io`_ package with EUPS:
 
 .. code:: bash