support win dev by moving environment config

.env

@@ -0,0 +1,37 @@
+# this file is read by docker to configure its environment.
+# https://docs.docker.com/compose/environment-variables/
+#
+# on production, these settings are overridden via the './shift' script.
+# ( see also: 'shift.overrides.production' and the 'secrets.example' )
+#
+#### DOCKER ####
+COMPOSE_PROJECT_NAME="shift"   # set the prefix of docker containers
+
+#### DB ####
+MYSQL_ROOT_PASSWORD=da642315cb2d390714590dad93e07c50
+MYSQL_DATABASE=shift   # can force node to use a in-memory db, by changing this to "sqlite"
+MYSQL_USER=shift
+MYSQL_PASSWORD=ok124
+
+### NGINX ###
+NGINX_HTTPS_PORT=4443  # used by nginx and node configuration
+NGINX_HTTP_PORT=4480   # used only by nginx ( for lets encrypt i think )
+LETSENCRYPT_WEBROOT=./services/nginx/letsencrypt
+
+### NODE ####
+SHIFT_DOMAIN="localhost"                   # for generating sharable links
+SHIFT_IMAGE_DIR=/opt/backend/eventimages   # images as seen by the node container
+EVENTIMAGES=./app/eventimages              # source of event images on local machine
+
+#### EMAIL ####
+# with empty values, the default behavior is to log to console.
+# overridden in production via the secrets.example file.
+SMTP_HOST=
+SMTP_USER=
+SMTP_PASS=
+
+# optional configuration for ethereal email.
+# ( see readme.md for more info )
+# export SMTP_HOST=smtp.ethereal.email
+# export SMTP_USER=h42vgcxjpznr6qhx@ethereal.email
+# export SMTP_PASS=E18RgFGSnhwNMzmBdJ

README.md

@@ -27,37 +27,33 @@ The backend uses:
 - If you want to change something about the site configuration or theme, [pull requests](https://help.github.com/articles/creating-a-pull-request/) are welcome.  Once you create a PR, you can immediately check out a link to the [build status, log, and a preview of your changes](https://app.netlify.com/sites/shift-docs/deploys).
 - If you only want to edit CONTENT rather than any code or site styling, (including creating new pages), [this doc shows how to easily do so without writing code](/docs/UPDATING.md).
 
-# Frontend development with Netlify
-
-While creating a pull request does automatically deploy a preview of the frontend to Netlify, you can also create previews manually: this could help you do things like theme development in your own repository before submitting your pr.
-
-1. [fork repo](https://help.github.com/articles/fork-a-repo/)
-2. read the comments in the netlify.toml file around changing the build command in the `[context.production]` section and make changes if needed.
-2. [deploy on Netlify](https://app.netlify.com/start) by linking your forked repo.  Included configuration file `netlify.toml` should mean 0 additional configuration required to get the site running.  If you get a build failure around access denied for ssh, you probably need the advice in step 2 just above this!
-
-If you have trouble with it please [file an issue](https://github.com/shift-org/shift-docs/issues/new) to let us know what you tried and what happened when you did.
-
 # Local development with Docker
 
-The production backend is run in several docker containers; including nginx, mysql, and the node server.
-
-The docker configuration also supports running your own frontend and backend server locally. The following steps assume a Linux, or MacOs development environment. On Windows, you'll need something like the [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install).
+Docker runs the production backend, and Netlify runs the frontend. When developing locally, however, you can use Docker to run both. The following steps assume a Linux, or MacOs development environment.
 
 1. Install Docker: [https://docs.docker.com/get-docker/](https://docs.docker.com/get-docker/)
+1. Install Node: https://nodejs.org/en/download
 2. Download source code: `git clone https://github.com/shift-org/shift-docs.git`
-4. Start shift site: `cd shift-docs ; ./shift up`
-5. If you're standing up the site for the first time, add database tables with the setup script: `./shift mysql-pipe < services/db/seed/setup.sql`.
-6. Visit `https://localhost:4443/` . If this leads to an SSL error in chrome, you may try flipping this flag:  chrome://flags/#allow-insecure-localhost
+3. Start your local server: `cd shift-docs ; ./shift up` ( On Windows, use `npm run up` )
+4. Optionally, watch for content changes: `./shift watch` ( On Windows, use `npm run watch` )
+5. Visit `https://localhost:4443/` . If this leads to an SSL error in chrome, you may try flipping this flag:  chrome://flags/#allow-insecure-localhost
+
+Note that no changes **inside** the Docker containers should ever be needed. The containers read from your **local** filesystem, so updating the files on your machine will automatically update docker too. ( Sometimes after a `./shift down`, `./shift up` to restart the server. )
 
-Note that no changes to the filesystems **inside** the container should ever be needed;  they read from your **local** filesystem so updating the local FS will show up in the container (perhaps after a restart).  Updating, changing branches, etc can be done with git commands **outside** of the container (`git checkout otherbranch` or `git pull`).
+So now you can hopefully access the site.  But for a real end-to-end test of your setup, you'll need to create an event:
 
-So - now you can hopefully access the site.  But a real end-toend test of yoursetup, would be creating an event:
+1. Visit https://localhost:4443/addevent/
+2. Fill out all required fields (ones marked with an asterisk) for a date a day or two in the future.
+3. Save the event (fix any validation errors around missing fields to ensure it saves.)
+4. In production, we send you an email with a link to confirm the ride listing. For local development, we don't actually send the email. Instead, use `./shift emails` ( or `npm run emails` ) to view the email log file.
+5. Publish your event by visiting the url printed to the log file.
+6. See your event on the https://localhost:4443/calendar page!
 
-1. visit https://localhost:4443/addevent/
-2. fill out all required fields (ones marked with an asterisk), for a date a day or two in the future.
-3. save the event (fix any validation errors around missing fields to ensure it saves)
-4. In production, we send you an email with a link to confirm the ride listing; we also write a copy of that email to the file `services/node/shift-mail.log`. For local development, we don't actually send the email, so get the confirmation link from that mail log, visit it, and hit publish event
-5. hopefully see your event on the https://localhost:4443/calendar page!
+You can also create some sample test events using the command:
+
+1. `npm run docker-create-events`
+
+It will print the event urls to the console, but they are automatically published to your test site.
 
 ## Important Project Files
 
@@ -97,7 +93,6 @@ So - now you can hopefully access the site.  But a real end-toend test of yourse
   * This will show the persistent volumes that docker knows about. The shift project volumes are prefixed with `shift_`
   * The "shift_" docker namespace comes from the shift file: `export COMPOSE_PROJECT_NAME="shift"`
 
-
 # Local development with Node.js
 
 You can also do local development with node. These steps will setup a local node server which, by default, acts as both frontend and backend. 
@@ -116,14 +111,12 @@ When you new create events, the link for activating those events will be written
 
 If you are writing javascript code in the node backend, you can test everything is working as expected using `npm test`.
 
-
 ### Local previews using production data
 
 As an alternative to `npm run dev`, you can preview a local frontend with the actual production backend by using: `npm run -w tools preview`.  
 
 **NOTE:** any events you create while previewing this way *will* be seen by the world!
 
-
 ### Ethereal email
 
 Before executing `npm run dev`, you can configure email debugging using `npm run -w tools new-ethereal-cfg`. It will generate a `shift-email.cfg` file in your `bin` directory which will be used when adding new events. Use the username and password listed in that file to check for emails here: https://ethereal.email/login.

docker-compose.yml

@@ -18,6 +18,8 @@ services:
       # NGINX CONF:
       - ./services/nginx/:/opt/nginx/
       - ./services/nginx/conf.d:/etc/nginx/conf.d
+      # maps the hosts letsencrypt onto the ngnix's container
+      # see also services/certbot.sh and ./shift script
       - ${LETSENCRYPT_WEBROOT}:/tmp/letsencrypt-auto
     links:
       # lets shift.conf reference node ( ex. http://node:3080 )
@@ -97,7 +99,7 @@ services:
       # the right-hand side has to be SHIFT_IMAGE_DIR
       # todo: change the host side to a named volume?
       # fix: can the rhs here use the environment var?
-      - ./backend/eventimages/:/opt/backend/eventimages/
+      - "${EVENTIMAGES}:/opt/backend/eventimages"
 
       # DEACTIVATED: access for email log:
       # email logging disabled has been disabled in emailer.js

docs/netlify.md

@@ -0,0 +1,11 @@
+# Frontend development with Netlify
+
+This page is archived from the main readme. Running locally with Netlify is no longer a core supported configuration. Netlify does however still run the production frontend. So this information might be useful.
+
+While creating a pull request does automatically deploy a preview of the frontend to Netlify, you can also create previews manually: this could help you do things like theme development in your own repository before submitting your pr.
+
+1. [fork repo](https://help.github.com/articles/fork-a-repo/)
+2. read the comments in the netlify.toml file around changing the build command in the `[context.production]` section and make changes if needed.
+2. [deploy on Netlify](https://app.netlify.com/start) by linking your forked repo.  Included configuration file `netlify.toml` should mean 0 additional configuration required to get the site running.  If you get a build failure around access denied for ssh, you probably need the advice in step 2 just above this!
+
+If you have trouble with it please [file an issue](https://github.com/shift-org/shift-docs/issues/new) to let us know what you tried and what happened when you did.

package.json

@@ -13,19 +13,30 @@
     "tools"
   ],
   "scripts": {
+    "postinstall": "hugo-installer --version 0.128.0",
+
     "start": "npm -w app start",
     "prestart": "npm i --omit dev --audit false --fund false --ignore-scripts",
+
     "dev": "concurrently -k \"npm:dev-*\"",
     "predev": "env-cmd -f tools/dev.env node tools/setupEventImages.js",
    	"dev-app": "env-cmd -f tools/dev.env npm start -w app",
     "dev-hugo": "exec-bin bin/hugo/hugo -w -s site -d ../bin/dist",
-    "build": "npm run build-hugo",
-    "build-hugo": "exec-bin bin/hugo/hugo -s site -d ../bin/dist",
-    "postinstall": "hugo-installer --version 0.128.0",
-    "test": "npm run test -w app",
+
+    "watch": "exec-bin bin/hugo/hugo -w -s site -d public",
+    "build": "exec-bin bin/hugo/hugo -s site -d public",
+    "prebuild": "npm i",
+
+    "preup": "npm run build",
+    "up": "docker compose up -d",
+    "down": "docker compose down",
+    "emails": "docker compose logs node -f",
+
     "preview": "concurrently -k \"npm:dev-*\"",
     "preview-hugo": "npm run dev-hugo",
-    "preview-app": " npm -w tools run preview"
+    "preview-app": " npm -w tools run preview",
 
+    "docker-create-events": "docker compose exec node node ./tools/makeFakeEvents.js",
+    "test": "npm run test -w app"
   }
 }

secrets.example

@@ -1,8 +1,10 @@
-# production file at api.shift2bikes.org:/opt/shift-docs/secrets
+# ---------------------------------------
+# production environment variables for smtp configuration
+# the real file is on api.shift2bikes.org at `/opt/shift-docs/secrets`
+# ( it contains some outdated unused vars; the ones here are the only ones needed. )
+# see also: shift.overrides.production
+# ---------------------------------------
 
 export SMTP_HOST=
 export SMTP_USER=
 export SMTP_PASS=
-export SMTP_LOGIN=DIGEST-MD5
-export SMTP_DOMAIN=
-export CAL_ADMIN_PASSWORD=

services/nginx/letsencrypt/.gitkeep

@@ -0,0 +1,7 @@
+# tldr: this directory makes nginx happy during development.
+#
+# the nginx container expects the environment variable LETSENCRYPT_WEBROOT 
+# to tell it a location on the host to map its /tmp/letscrypt directory 
+# on production certbot.sh **requires** that the host directory also be called /tmp/letscrypt 
+# but not all development machines have such a location (ie. windows)
+# see also: docs/PRODUCTION_CONFIGURATION.md

shift

@@ -1,53 +1,12 @@
 #!/usr/bin/env bash
 set -e
 
-#### DB ####
-export MYSQL_ROOT_PASSWORD=da642315cb2d390714590dad93e07c50
-export MYSQL_DATABASE=shift
-export MYSQL_USER=shift
-export MYSQL_PASSWORD=ok124
-# can force the node version to use a sqlite in memory database instead of mysql:
-# export MYSQL_DATABASE=sqlite
-
-export NGINX_HTTPS_PORT=4443
-export NGINX_HTTP_PORT=4480
-export LETSENCRYPT_WEBROOT="/tmp/letsencrypt-auto"
-
-export SHIFT_DOMAIN="localhost"
-
-#### DOCKER ####
-export COMPOSE_PROJECT_NAME="shift"
-
-#### SMTP ####
-# with this configuration, emails will log to console, and to a file
-# ./shift emails
-# - or -
-# ./shift attach node
-# $ tail -f /var/log/shift-mail.log
-#
-# it will also try to "send" ethereal emails
-# https://ethereal.email/messages
-#
-# export SMTP_HOST=smtp.ethereal.email
-# export SMTP_USER=h42vgcxjpznr6qhx@ethereal.email
-# export SMTP_PASS=E18RgFGSnhwNMzmBdJ
-# export SMTP_LOGIN=587
-SMTP_HOST=
-SMTP_USER=
-SMTP_PASS=
-SMTP_LOGIN=
-SMTP_DOMAIN=
-
-#####
-# === HERE BE DRAGONS ===
-# These are some convenience pattens and functions that vincent@khougaz.com has found most helpful
-# Feel free to copy into your own projects, or mostly don't worry about it!
-#####
-
-# ENV
+# ====================
+# setup environment overrides
+# ====================
+
+# Get location of this script
 export ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-export EVENTIMAGES="${ROOT}/backend/eventimages"
-export SHIFT_IMAGE_DIR="/opt/backend/eventimages"
 
 # Check for an overrides file
 if [ -f ${ROOT}/shift.overrides ]; then
@@ -59,9 +18,8 @@ if [ -f $ROOT/secrets ]; then
     source $ROOT/secrets
 fi
 
-SCRIPT_NAME=$(basename $0)
-
 # ====================
+# helper to wait for node startup.
 # https://www.marksayson.com/blog/wait-until-docker-containers-initialized/
 
 # Max query attempts before consider setup failed
@@ -94,6 +52,7 @@ function waitUntilServiceIsReady() {
 }
 
 # ====================
+SCRIPT_NAME=$(basename $0)
 
 sub_help() { #                    - Show this text
     echo "Usage: ./${SCRIPT_NAME} <subcommand> [options]"
@@ -104,7 +63,13 @@ sub_help() { #                    - Show this text
 
 sub_up() { #                      - Bring all services up
     if [ ! -d $LETSENCRYPT_WEBROOT ] ; then
-	mkdir $LETSENCRYPT_WEBROOT
+  # tbd: maybe we shouldn't mkdir, and instead let docker/nginx fail?
+  # rationale: certbot.sh creates and fills this directory.
+  # and if certbot hasn't already already run https will fail,
+  # but it won't be obvious until you visit the site.
+  # see docs/PRODUCTION_CONFIGURATION.md
+  echo "*** if this is production: have you run certbot? ***"
+	mkdir -p $LETSENCRYPT_WEBROOT
     fi 
     sub_compose up --force-recreate -d
 

shift.overrides.production

@@ -1,3 +1,11 @@
+# ---------------------------------------
+# production environment variables for nginx and node
+# the './shift' bash script 'sources' "shift.overrides"
+# and on the production server that file is soft-linked to this file ( `/opt/shift-docs/shift.overrides.production` )
+# unlike secrets.example, nothing here is privileged info.
+# ---------------------------------------
 export NGINX_HTTPS_PORT=443
 export NGINX_HTTP_PORT=80
 export SHIFT_DOMAIN=shift2bikes.org
+export LETSENCRYPT_WEBROOT="/tmp/letsencrypt-auto"
+export EVENTIMAGES="/opt/shift-docs/backend/eventimages/"