v0.11.0

.changes/unreleased/.gitkeep

@@ -0,0 +1 @@
+*.yaml

.changes/unreleased/Added-20250111-211546.yaml

@@ -1,3 +0,0 @@
-kind: Added
-body: '[feat]: package `packageOperation` option `checkVersion` implemented'
-time: 2025-01-11T21:15:46.207199643-06:00

.changes/unreleased/Added-20250111-211813.yaml

@@ -1,3 +0,0 @@
-kind: Added
-body: user management added - see docs
-time: 2025-01-11T21:18:13.182822019-06:00

.changes/unreleased/Added-20250113-231248.yaml

@@ -1,3 +0,0 @@
-kind: Added
-body: Support for remote config sources. Only config file and list can be used for now.
-time: 2025-01-13T23:12:48.383700682-06:00

.changes/unreleased/Added-20250128-153524.yaml

@@ -1,3 +0,0 @@
-kind: Added
-body: Cache functionality - still a WIP
-time: 2025-01-28T15:35:24.512485671-06:00

.changes/unreleased/Changed-20250113-231007.yaml

@@ -1,3 +0,0 @@
-kind: Changed
-body: Internal refactoring of config setup
-time: 2025-01-13T23:10:07.215735108-06:00

.changes/unreleased/Changed-20250113-231622.yaml

@@ -1,3 +0,0 @@
-kind: Changed
-body: Formatting and sending for notifications
-time: 2025-01-13T23:16:22.260458782-06:00

.changes/unreleased/Changed-20250128-154204.yaml

@@ -1,3 +0,0 @@
-kind: Changed
-body: "name of `configfetcher` to `remotefetcher`"
-time: 2025-01-28T15:42:04.282668058-06:00

.changes/unreleased/Fixed-20250128-153806.yaml

@@ -1,3 +0,0 @@
-kind: Fixed
-body: Parsing of remote URLs when determining list config file path
-time: 2025-01-28T15:38:06.957506929-06:00

.changes/v0.10.0.md

@@ -0,0 +1,16 @@
+## v0.10.0 - 2025-03-08
+### Added
+* Hooks: improved logging when executing
+* User commands: adding SSH keys using config key `userSshPubKeys`
+* directives: added support for fetching values using directive `%{externalSource:key}%`
+### Changed
+* Commands: if dir is not specified, run in config dir
+* FileDirective: use the config directory if path is not absolute
+* Host: changes to case of some keys
+* Notifications: added external directive to sensitive keys
+### Fixed
+* LocalFetcher: return fetch error
+* Lists: load file key before attempting to load from current file
+* fix: host not in config file, but in ssh config, properly added to hosts struct
+* SSH: password authentication bugs
+* User commands: change user password works

.changes/v0.10.1.md

@@ -0,0 +1,8 @@
+## v0.10.1 - 2025-03-11
+### Added
+* UserCommands: add ssh public keys when running locally
+* UserCommands: add field CreateUserHome
+### Changed
+* UserCommands: create temp file when modifing password over SSH
+* UserCommands: change field name
+* Vault: keys are now referenced by `name`, and the actual data by `data`

.changes/v0.10.2.md

@@ -0,0 +1,6 @@
+## v0.10.2 - 2025-03-19
+### Added
+* Notifications: http service added
+* Variable support. Can be referenced with `%{var:nameOfVar}%` in select string fields.
+### Changed
+* vault: initialize vault before validating config

.changes/v0.11.0.md

@@ -0,0 +1,21 @@
+## v0.11.0 - 2025-11-24
+### Added
+* feat: Package operation `versionCheck` supports regular expressions (see [regexp](https://pkg.go.dev/regexp) package for docs)
+* Command lists: added `cmdLists.[name].notify` object
+* Testing setup with Docker
+* CLI: add global flag --hostsConfig that allows hosts to be dynamic in relation to the main config
+* CLI: Exec subcommand `hosts`. See documentation for more details.
+* CLI: added `exec hosts` subcommand `list`
+* Add support for hosts in parallel
+### Changed
+* Commands: `host` can now be `localhost` or `127.0.0.1` to run commands locally
+* lists loaded from external files only if no list config present in current file
+* `PackageManager.Parse` renamed to `ParseRemotePackageManagerVersionOutput`. This now returns arrays of PackageManagerCommon.Package and errors.
+* Internal: refactoring and renaming functions
+* Commands: moved output-prefixed keys to the `commands.[name].output` object
+* Change internal method name for better understanding
+* Improved error message for remote version package output
+### Fixed
+* Command Lists: hooks now run correctly when commands finish
+* Log file passed using `--log-file` correctly used
+* Cmd Type `script` now correctly appends arguments

.changes/v0.7.0.md

@@ -0,0 +1,16 @@
+## v0.7.0 - 2025-02-11
+### Added
+* [feat]: package `packageOperation` option `checkVersion` implemented
+* user management added - see docs
+* Support for remote config sources. Only config file and list can be used for now.
+* Cache functionality - still a WIP
+* Flag `--s3-endpoint` for config file fetching from S3
+### Changed
+* Internal refactoring of config setup
+* Formatting and sending for notifications
+* name of `configfetcher` to `remotefetcher`
+* Flags that took comma-separated lists now have to be passed multiple times for each argument.
+* Hosts passed to `exec host` now checked against default SSH config files
+### Fixed
+* Parsing of remote URLs when determining list config file path
+* Incorrect error notification template value

.changes/v0.7.1.md

@@ -0,0 +1,3 @@
+## v0.7.1 - 2025-02-14
+### Fixed
+* Incorrect local config file loading logic caused files to not be detected

.changes/v0.7.2.md

@@ -0,0 +1,3 @@
+## v0.7.2 - 2025-02-14
+### Fixed
+* CI configs

.changes/v0.7.3.md

@@ -0,0 +1,3 @@
+## v0.7.3 - 2025-02-14
+### Changed
+* GoReleaser configs

.changes/v0.7.4.md

@@ -0,0 +1,5 @@
+## v0.7.4 - 2025-02-14
+### Changed
+* CI configs
+### Fixed
+* v0.7.1: Incorrect local config file loading logic caused files to not be detected

.changes/v0.7.5.md

@@ -0,0 +1,5 @@
+## v0.7.5 - 2025-02-14
+### Changed
+* CI configs
+### Fixed
+* v0.7.1: Incorrect local config file loading logic caused files to not be detected

.changes/v0.7.6.md

@@ -0,0 +1,4 @@
+## v0.7.6 - 2025-02-14
+### Fixed
+* v0.7.1: Incorrect local config file loading logic caused files to not be detected
+* CI configs

.changes/v0.7.7.md

@@ -0,0 +1,4 @@
+## v0.7.7 - 2025-02-14
+### Fixed
+* v0.7.1: Incorrect local config file loading logic caused files to not be detected
+* CI configs

.changes/v0.7.8.md

@@ -0,0 +1,4 @@
+## v0.7.8 - 2025-02-14
+### Fixed
+* Github CI config
+* v0.7.1: Incorrect local config file loading logic caused files to not be detected

.changes/v0.8.0.md

@@ -0,0 +1,6 @@
+## v0.8.0 - 2025-02-15
+### Changed
+* Breaking: `cmd-lists` key changed to `cmdLists`
+* Properly load list config
+* Config file loading properly errors
+* CI Configs

.changes/v0.9.0.md

@@ -0,0 +1,12 @@
+## v0.9.0 - 2025-02-28
+### Added
+* `list` command with subcommands `cmds` and `lists`
+* Deprecation and unsupported warnings for old config keys
+* CLI flag `--cmdStdOut` to output command's stdout/stderr to stdout
+* Command type `remoteScript`. See docs for more info.
+### Changed
+* change to enums for Command type
+* Cache now stores resources by URL hash for ease-of-lookup
+* Changed PackageOperation to enums
+### Fixed
+* Local command's `dir` full path is now found with home directory

.changes/v0.9.1.md

@@ -0,0 +1,3 @@
+## v0.9.1 - 2025-03-01
+### Changed
+* Use EnvVar AWS_PROFILE to get S3 profile

.github/workflows/release.yml

@@ -15,26 +15,26 @@ jobs:
   goreleaser:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
       - run: git fetch --force --tags
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v5
         with:
-          go-version: '1.20'
+          go-version: '1.23'
           cache: true
       # More assembly might be required: Docker logins, GPG, etc. It all depends
       # on your needs.
-      - name: Get tag
-        id: tag
-        uses: dawidd6/action-get-tag@v1
+      - uses: olegtarasov/get-tag@v2.1.4
+        id: tagName
         with:
-          # Optionally strip `v` prefix
-          strip_v: false
-      - uses: goreleaser/goreleaser-action@v4
+          # tagRegex: "foobar-(.*)"  # Optional. Returns specified group text as tag name. Full tag string is returned if regex is not defined.
+          tagRegexGroup: 1 # Optional. Default is 1.
+      - uses: goreleaser/goreleaser-action@v6
         with:
           distribution: goreleaser
-          version: latest
-          args: release --release-notes=".changes/${{steps.tag.outputs.tag}}.md" -f .goreleaser/github.yml --clean
+          version: 2.7.0
+          args: release --release-notes=".changes/${{ env.GIT_TAG_NAME }}.md" -f .goreleaser/github.yml --clean
         env:
           GITHUB_TOKEN: ${{ secrets.GORELEASER_TOKEN }}
+          GIT_TAG_NAME: ${{ steps.tagName.outputs.tag }}

.gitignore

@@ -1,5 +1,10 @@
+!.changie.yaml
+!.changes/**
 
 dist/
 .codegpt
 
 *.log
+/*.sh
+/*.yaml
+/*.yml

.goreleaser/gitea.yml

@@ -6,7 +6,6 @@ before:
 builds:
   - env:
       - CGO_ENABLED=0
-      - GOPROXY=https://goproxy.io
     goos:
       - freebsd
       - linux
@@ -16,7 +15,7 @@ builds:
       - arm64
 
 archives:
-  - format: tar.gz
+  - formats: tar.gz
     # this name template makes the OS and Arch compatible with the results of uname.
     name_template: >-
       {{ .ProjectName }}_{{ .Version }}_
@@ -28,7 +27,7 @@ archives:
     # use zip for windows archives
     format_overrides:
     - goos: windows
-      format: zip
+      formats: [zip]
 checksum:
   name_template: 'checksums.txt'
 snapshot:

.goreleaser/github.yml

@@ -1,4 +1,3 @@
-# This is an example .goreleaser.yml file with some sensible defaults.
 # Make sure to check the documentation at https://goreleaser.com
 version: 2
 before:
@@ -17,7 +16,7 @@ builds:
       - arm64
 
 archives:
-  - format: tar.gz
+  - formats: tar.gz
     # this name template makes the OS and Arch compatible with the results of uname.
     name_template: >-
       {{ .ProjectName }}_{{ .Version }}_
@@ -29,7 +28,7 @@ archives:
     # use zip for windows archives
     format_overrides:
     - goos: windows
-      format: zip
+      formats: [zip]
 checksum:
   name_template: 'checksums.txt'
 snapshot:

.vscode/settings.json

@@ -9,5 +9,6 @@
         "mautrix",
         "nikoksr",
         "Strs"
-    ]
+    ],
+    "CodeGPT.apiKey": "CodeGPT Plus Beta"
 }

.woodpecker/gitea.yml

@@ -1,11 +1,18 @@
 steps:
-  release:
-    image: goreleaser/goreleaser
+  golang:
+    image: golang:1.23
     commands:
+      - go install github.com/goreleaser/goreleaser/v2@v2.7.0
       - goreleaser release -f .goreleaser/gitea.yml --release-notes=".changes/$(go run backy.go version -V).md"
-    secrets: [ gitea_token ]
+    environment:
+      GITEA_TOKEN:
+        from_secret: gitea_token
+        
     when:
       event: tag
+  # release:
+  #   image: goreleaser/goreleaser
+  #   commands:
 
 when:
   - event: tag

.woodpecker/go-lint.yml

@@ -5,7 +5,7 @@ steps:
       - go build
       - go test
   release:
-    image: golangci/golangci-lint:v1.53.3
+    image: golangci/golangci-lint:v1.64.7
     commands:
       - golangci-lint run -v --timeout 5m
 

.woodpecker/publish-docs.yml

@@ -1,11 +1,11 @@
 steps:
   build:
-    image: hugomods/hugo:ci
+    image: hugomods/hugo:debian-ci-0.147.2
     commands:
       - git submodule foreach 'git fetch origin; git checkout $(git describe --tags `git rev-list --tags --max-count=1`);'
       - cd docs
       - hugo mod get -u github.com/divinerites/plausible-hugo
-      - hugo mod get -u github.com/McShelby/hugo-theme-relearn@7.3.1
+      - hugo mod get -u github.com/McShelby/hugo-theme-relearn@8.2.0
       - hugo
 
   deploy:
@@ -23,7 +23,14 @@ steps:
       - echo "$SSH_DEPLOY_KEY" | tr -d '\r' | DISPLAY=":0.0" SSH_ASKPASS=~/.ssh/.print_ssh_password setsid ssh-add -
       - rsync -atv --delete --progress public/ backy@backy.cybershell.xyz:docs
       - rsync -atv --delete --progress vangen/ backy@backy.cybershell.xyz:vangen-go
-    secrets: [ ssh_host_key, ssh_deploy_key, ssh_passphrase ]
+    environment:
+      SSH_HOST_KEY:
+        from_secret: ssh_host_key
+      SSH_DEPLOY_KEY:
+        from_secret: ssh_deploy_key
+      SSH_PASSPHRASE:
+        from_secret: ssh_passphrase
 
 when:
   - branch: master
+  - path: 'docs/**'

CHANGELOG.md

@@ -6,10 +6,144 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html),
 and is generated by [Changie](https://github.com/miniscruff/changie).
 
 
+## v0.11.0 - 2025-11-24
+### Added
+* feat: Package operation `versionCheck` supports regular expressions (see [regexp](https://pkg.go.dev/regexp) package for docs)
+* Command lists: added `cmdLists.[name].notify` object
+* Testing setup with Docker
+* CLI: add global flag --hostsConfig that allows hosts to be dynamic in relation to the main config
+* CLI: Exec subcommand `hosts`. See documentation for more details.
+* CLI: added `exec hosts` subcommand `list`
+* Add support for hosts in parallel
+### Changed
+* Commands: `host` can now be `localhost` or `127.0.0.1` to run commands locally
+* lists loaded from external files only if no list config present in current file
+* `PackageManager.Parse` renamed to `ParseRemotePackageManagerVersionOutput`. This now returns arrays of PackageManagerCommon.Package and errors.
+* Internal: refactoring and renaming functions
+* Commands: moved output-prefixed keys to the `commands.[name].output` object
+* Change internal method name for better understanding
+* Improved error message for remote version package output
+### Fixed
+* Command Lists: hooks now run correctly when commands finish
+* Log file passed using `--log-file` correctly used
+* Cmd Type `script` now correctly appends arguments
+
+## v0.10.2 - 2025-03-19
+### Added
+* Notifications: http service added
+* Variable support. Can be referenced with `%{var:nameOfVar}%` in select string fields.
+### Changed
+* vault: initialize vault before validating config
+
+## v0.10.1 - 2025-03-11
+### Added
+* UserCommands: add ssh public keys when running locally
+* UserCommands: add field CreateUserHome
+### Changed
+* UserCommands: create temp file when modifing password over SSH
+* UserCommands: change field name
+* Vault: keys are now referenced by `name`, and the actual data by `data`
+
+## v0.10.0 - 2025-03-08
+### Added
+* Hooks: improved logging when executing
+* User commands: adding SSH keys using config key `userSshPubKeys`
+* directives: added support for fetching values using directive `%{externalSource:key}%`
+### Changed
+* Commands: if dir is not specified, run in config dir
+* FileDirective: use the config directory if path is not absolute
+* Host: changes to case of some keys
+* Notifications: added external directive to sensitive keys
+### Fixed
+* LocalFetcher: return fetch error
+* Lists: load file key before attempting to load from current file
+* fix: host not in config file, but in ssh config, properly added to hosts struct
+* SSH: password authentication bugs
+* User commands: change user password works
+
+## v0.9.1 - 2025-03-01
+### Changed
+* Use EnvVar AWS_PROFILE to get S3 profile
+
+## v0.9.0 - 2025-02-28
+### Added
+* `list` command with subcommands `cmds` and `lists`
+* Deprecation and unsupported warnings for old config keys
+* CLI flag `--cmdStdOut` to output command's stdout/stderr to stdout
+* Command type `remoteScript`. See docs for more info.
+### Changed
+* change to enums for Command type
+* Cache now stores resources by URL hash for ease-of-lookup
+* Changed PackageOperation to enums
+### Fixed
+* Local command's `dir` full path is now found with home directory
+
+## v0.8.0 - 2025-02-15
+### Changed
+* Breaking: `cmd-lists` key changed to `cmdLists`
+* Properly load list config
+* Config file loading properly errors
+* CI Configs
+
+## v0.7.8 - 2025-02-14
+### Fixed
+* Github CI config
+* v0.7.1: Incorrect local config file loading logic caused files to not be detected
+
+## v0.7.7 - 2025-02-14
+### Fixed
+* v0.7.1: Incorrect local config file loading logic caused files to not be detected
+* CI configs
+
+## v0.7.6 - 2025-02-14
+### Fixed
+* v0.7.1: Incorrect local config file loading logic caused files to not be detected
+* CI configs
+
+## v0.7.5 - 2025-02-14
+### Changed
+* CI configs
+### Fixed
+* v0.7.1: Incorrect local config file loading logic caused files to not be detected
+
+## v0.7.4 - 2025-02-14
+### Changed
+* CI configs
+### Fixed
+* v0.7.1: Incorrect local config file loading logic caused files to not be detected
+
+## v0.7.3 - 2025-02-14
+### Changed
+* GoReleaser configs
+
+## v0.7.2 - 2025-02-14
+### Fixed
+* CI configs
+
+## v0.7.1 - 2025-02-14
+### Fixed
+* Incorrect local config file loading logic caused files to not be detected
+
+## v0.7.0 - 2025-02-11
+### Added
+* [feat]: package `packageOperation` option `checkVersion` implemented
+* user management added - see docs
+* Support for remote config sources. Only config file and list can be used for now.
+* Cache functionality - still a WIP
+* Flag `--s3-endpoint` for config file fetching from S3
+### Changed
+* Internal refactoring of config setup
+* Formatting and sending for notifications
+* name of `configfetcher` to `remotefetcher`
+* Flags that took comma-separated lists now have to be passed multiple times for each argument.
+* Hosts passed to `exec host` now checked against default SSH config files
+### Fixed
+* Parsing of remote URLs when determining list config file path
+* Incorrect error notification template value
+
 ## v0.6.1 - 2025-01-04
 ### Fixed
 * When running a list, hooks now run explicitly after the command executes. Fixed panic due to improper logic.
-
 ## v0.6.0 - 2025-01-04
 ### Added
 * Command Type Package - allows one to perform package operations [docs](https://backy.cybershell.xyz/config/packages/)

backy.code-workspace

@@ -7,14 +7,20 @@
 	],
 	"settings": {
 		"cSpell.words": [
+			"Autorestic",
+			"changie",
 			"Cmds",
-			"remotefetcher",
+			"CMDSTDOUT",
+			"goreleaser",
 			"knadh",
 			"koanf",
 			"mattn",
 			"maunium",
 			"mautrix",
 			"nikoksr",
+			"packagemanagercommon",
+			"rawbytes",
+			"remotefetcher",
 			"Strs"
 		]
 	}

cmd/backup.go

@@ -12,8 +12,8 @@ import (
 
 var (
 	backupCmd = &cobra.Command{
-		Use:   "backup [--lists=list1,list2,... | -l list1, list2,...]",
-		Short: "Runs commands defined in config file.",
+		Use:   "backup [--lists=list1 --lists list2 ... | -l list1 -l list2 ...]",
+		Short: "Runs commands defined in config file. Use -l flag multiple times to run multiple lists.",
 		Long:  "Backup executes commands defined in config file.\nUse the --lists or -l flag to execute the specified lists. If not flag is not given, all lists will be executed.",
 		Run:   Backup,
 	}
@@ -23,15 +23,21 @@ var (
 var cmdLists []string
 
 func init() {
+	parseS3Config()
 
-	backupCmd.Flags().StringSliceVarP(&cmdLists, "lists", "l", nil, "Accepts comma-separated names of command lists to execute.")
+	backupCmd.Flags().StringArrayVarP(&cmdLists, "lists", "l", nil, "Accepts comma-separated names of command lists to execute.")
 
 }
 
 func Backup(cmd *cobra.Command, args []string) {
-	backyConfOpts := backy.NewOpts(cfgFile, backy.AddCommandLists(cmdLists))
+	backyConfOpts := backy.NewConfigOptions(configFile,
+		backy.AddCommandLists(cmdLists),
+		backy.SetLogFile(logFile),
+		backy.EnableCommandStdOut(cmdStdOut),
+		backy.SetHostsConfigFile(hostsConfigFile))
+
 	backyConfOpts.InitConfig()
-	backyConfOpts.ReadConfig()
+	backyConfOpts.ParseConfigurationFile()
 
 	backyConfOpts.RunListConfig("")
 	for _, host := range backyConfOpts.Hosts {

cmd/backup_test.go

@@ -0,0 +1,54 @@
+package cmd
+
+// import (
+// 	"bufio"
+// 	"encoding/json"
+// 	"os"
+// 	"os/exec"
+// 	"strings"
+// 	"testing"
+
+// 	"github.com/stretchr/testify/assert"
+// )
+
+// // TestConfigOptions tests the configuration options for the backy package.
+// func Test_ErrorHook(t *testing.T) {
+// 	configFile := "-f ../../tests/ErrorHook.yml"
+// 	logFile := "--log-file=ErrorHook.log"
+// 	backyCommand := exec.Command("go", "run", "../../backy.go", configFile, logFile, "backup")
+// 	backyCommand.Stderr = os.Stdout
+// 	backyCommand.Stdout = os.Stdout
+// 	err := backyCommand.Run()
+// 	assert.Nil(t, err)
+// 	os.Remove("ErrorHook.log")
+// 	logFileData, logFileErr := os.ReadFile("ErrorHook.log")
+// 	if logFileErr != nil {
+// 		assert.FailNow(t, logFileErr.Error())
+
+// 	}
+// 	var JsonData []map[string]interface{}
+// 	jsonScanner := bufio.NewScanner(strings.NewReader(string(logFileData)))
+
+// 	for jsonScanner.Scan() {
+// 		var jsonDataLine map[string]interface{}
+// 		err = json.Unmarshal(jsonScanner.Bytes(), &jsonDataLine)
+// 		assert.Nil(t, err)
+// 		JsonData = append(JsonData, jsonDataLine)
+// 	}
+// 	for _, v := range JsonData {
+// 		_, ok := v["error"]
+// 		if !ok {
+// 			assert.FailNow(t, "error does not exist\n")
+// 			// return
+// 		}
+// 	}
+// 	// t.Logf("%s", logFileData)
+// 	// t.Logf("%v", JsonData)
+// }
+
+// func TestBackupErrorHook(t *testing.T) {
+// 	logFile = "ErrorHook.log"
+
+// 	configFile = "../tests/ErrorHook.yml"
+
+// }

cmd/config.go

@@ -20,7 +20,7 @@ package cmd
 
 // func config(cmd *cobra.Command, args []string) {
 
-// 	opts := backy.NewOpts(cfgFile, backy.cronEnabled())
+// 	opts := backy.NewConfigOptions(configFile, backy.cronEnabled())
 // 	opts.InitConfig()
 
 // }

cmd/cron.go

@@ -16,10 +16,16 @@ var (
 )
 
 func cron(cmd *cobra.Command, args []string) {
+	parseS3Config()
+
+	opts := backy.NewConfigOptions(configFile,
+		backy.EnableCron(),
+		backy.SetLogFile(logFile),
+		backy.EnableCommandStdOut(cmdStdOut),
+		backy.SetHostsConfigFile(hostsConfigFile))
 
-	opts := backy.NewOpts(cfgFile, backy.CronEnabled())
 	opts.InitConfig()
-	opts.ReadConfig()
+	opts.ParseConfigurationFile()
 
 	opts.Cron()
 }

cmd/exec.go

@@ -21,20 +21,23 @@ var (
 )
 
 func init() {
-	execCmd.AddCommand(hostExecCommand)
-
-	hostExecCommand.Flags().StringSliceVarP(&hostsList, "hosts", "m", nil, "Accepts comma-separated names of hosts.")
-	hostExecCommand.Flags().StringSliceVarP(&cmdList, "commands", "c", nil, "Accepts comma-separated names of commands.")
+	execCmd.AddCommand(hostExecCommand, hostsExecCommand)
 
 }
 
 func execute(cmd *cobra.Command, args []string) {
+	parseS3Config()
+
 	if len(args) < 1 {
 		logging.ExitWithMSG("Please provide a command to run. Pass --help to see options.", 1, nil)
 	}
 
-	opts := backy.NewOpts(cfgFile, backy.AddCommands(args), backy.SetLogFile(logFile))
+	opts := backy.NewConfigOptions(configFile,
+		backy.AddCommands(args),
+		backy.SetLogFile(logFile),
+		backy.EnableCommandStdOut(cmdStdOut),
+		backy.SetHostsConfigFile(hostsConfigFile))
 	opts.InitConfig()
-	opts.ReadConfig()
+	opts.ParseConfigurationFile()
 	opts.ExecuteCmds()
 }

cmd/host.go

@@ -8,19 +8,25 @@ import (
 
 var (
 	hostExecCommand = &cobra.Command{
-		Use:   "host [--commands=command1,command2, ... | -c command1,command2, ...] [--hosts=host1,hosts2, ... | -m host1,host2, ...] ",
+		Use:   "host [--command=command1 --command=command2 ... | -c command1 -c command2 ...] [--hosts=host1 --hosts=hosts2 ... | -m host1 -m host2 ...] ",
 		Short: "Runs command defined in config file on the hosts in order specified.",
 		Long:  "Host executes specified commands on the hosts defined in config file.\nUse the --commands or -c flag to choose the commands.",
 		Run:   Host,
 	}
 )
 
-// Holds command list to run
+// Holds list of hosts to run commands on
 var hostsList []string
+
+// Holds command list to run
 var cmdList []string
 
 func init() {
 
+	hostExecCommand.Flags().StringArrayVarP(&hostsList, "hosts", "m", nil, "Accepts space-separated names of hosts. Specify multiple times for multiple hosts.")
+	hostExecCommand.Flags().StringArrayVarP(&cmdList, "command", "c", nil, "Accepts space-separated names of commands. Specify multiple times for multiple commands.")
+	parseS3Config()
+
 }
 
 // cli input should be hosts and commands. Hosts are defined in config files.
@@ -29,22 +35,37 @@ func init() {
 //    2. stdin (on command line) (TODO)
 
 func Host(cmd *cobra.Command, args []string) {
-	backyConfOpts := backy.NewOpts(cfgFile, backy.SetLogFile(logFile))
+	backyConfOpts := backy.NewConfigOptions(configFile,
+		backy.SetLogFile(logFile),
+		backy.EnableCommandStdOut(cmdStdOut),
+		backy.SetHostsConfigFile(hostsConfigFile))
 	backyConfOpts.InitConfig()
 
-	backyConfOpts.ReadConfig()
+	backyConfOpts.ParseConfigurationFile()
 
 	// check CLI input
 	if hostsList == nil {
 		logging.ExitWithMSG("error: hosts must be specified", 1, &backyConfOpts.Logger)
 	}
-	// host is only checked when we read the SSH File
-	// so a check may not be needed here
+
 	for _, h := range hostsList {
+		if backy.IsHostLocal(h) {
+			continue
+		}
+		// check if h exists in the config file
 		_, hostFound := backyConfOpts.Hosts[h]
 		if !hostFound {
+			// check if h exists in the SSH config file
+			hostFoundInConfig, s := backy.DoesHostHaveHostName(h)
+			if !hostFoundInConfig {
 				logging.ExitWithMSG("host "+h+" not found", 1, &backyConfOpts.Logger)
 			}
+			if backyConfOpts.Hosts == nil {
+				backyConfOpts.Hosts = make(map[string]*backy.Host)
+			}
+			// create host with hostname and host
+			backyConfOpts.Hosts[h] = &backy.Host{Host: h, HostName: s}
+		}
 	}
 	if cmdList == nil {
 		logging.ExitWithMSG("error: commands must be specified", 1, &backyConfOpts.Logger)
@@ -56,5 +77,5 @@ func Host(cmd *cobra.Command, args []string) {
 		}
 	}
 
-	backyConfOpts.ExecCmdsSSH(cmdList, hostsList)
+	backyConfOpts.ExecCmdsOnHosts(cmdList, hostsList)
 }

cmd/hosts.go

@@ -0,0 +1,93 @@
+package cmd
+
+import (
+	"maps"
+	"slices"
+
+	"git.andrewnw.xyz/CyberShell/backy/pkg/backy"
+	"git.andrewnw.xyz/CyberShell/backy/pkg/logging"
+	"github.com/spf13/cobra"
+)
+
+var (
+	runCommandsInParallel bool
+
+	hostsExecCommand = &cobra.Command{
+		Use:   "hosts [--command=command1 --command=command2 ... | -c command1 -c command2 ...]",
+		Short: "Runs command defined in config file on the hosts in order specified.",
+		Long:  "Hosts executes specified commands on all the hosts defined in config file.\nUse the --commands or -c flag to choose the commands.",
+		Run:   Hosts,
+	}
+
+	hostsListExecCommand = &cobra.Command{
+		Use:   "list list1 list2 ...",
+		Short: "Runs lists in order specified defined in config file on all hosts.",
+		Long:  "Lists executes specified lists on all the hosts defined in hosts config.\nPass the names of lists as arguments after command.",
+		Run:   HostsList,
+	}
+)
+
+func init() {
+	hostsExecCommand.AddCommand(hostsListExecCommand)
+	hostsListExecCommand.Flags().BoolVarP(&runCommandsInParallel, "parallel", "p", false, "Run commands in parallel on hosts")
+	parseS3Config()
+}
+
+// cli input should be hosts and commands. Hosts are defined in config files.
+// commands can be passed by the following mutually exclusive options:
+//    1. as a list of commands defined in the config file
+//    2. stdin (on command line) (TODO)
+
+func Hosts(cmd *cobra.Command, args []string) {
+	backyConfOpts := backy.NewConfigOptions(configFile,
+		backy.SetLogFile(logFile),
+		backy.EnableCommandStdOut(cmdStdOut),
+		backy.SetHostsConfigFile(hostsConfigFile))
+	backyConfOpts.InitConfig()
+
+	backyConfOpts.ParseConfigurationFile()
+
+	for _, h := range backyConfOpts.Hosts {
+
+		hostsList = append(hostsList, h.Host)
+	}
+
+	if cmdList == nil {
+		logging.ExitWithMSG("error: commands must be specified", 1, &backyConfOpts.Logger)
+	}
+	for _, c := range cmdList {
+		_, cmdFound := backyConfOpts.Cmds[c]
+		if !cmdFound {
+			logging.ExitWithMSG("cmd "+c+" not found", 1, &backyConfOpts.Logger)
+		}
+	}
+
+	backyConfOpts.ExecCmdsOnHosts(cmdList, hostsList)
+}
+
+func HostsList(cmd *cobra.Command, args []string) {
+	backyConfOpts := backy.NewConfigOptions(configFile,
+		backy.SetLogFile(logFile),
+		backy.EnableCommandStdOut(cmdStdOut),
+		backy.SetHostsConfigFile(hostsConfigFile))
+	backyConfOpts.InitConfig()
+
+	backyConfOpts.ParseConfigurationFile()
+
+	if len(args) == 0 {
+		logging.ExitWithMSG("error: no lists specified", 1, &backyConfOpts.Logger)
+	}
+
+	for _, l := range args {
+		_, listFound := backyConfOpts.CmdConfigLists[l]
+		if !listFound {
+			logging.ExitWithMSG("list "+l+" not found", 1, &backyConfOpts.Logger)
+		}
+	}
+
+	maps.DeleteFunc(backyConfOpts.CmdConfigLists, func(k string, v *backy.CmdList) bool {
+		return !slices.Contains(args, k)
+	})
+
+	backyConfOpts.ExecuteListOnHosts(args, runCommandsInParallel)
+}

cmd/list.go

@@ -6,16 +6,29 @@ package cmd
 
 import (
 	"git.andrewnw.xyz/CyberShell/backy/pkg/backy"
+	"git.andrewnw.xyz/CyberShell/backy/pkg/logging"
 
 	"github.com/spf13/cobra"
 )
 
 var (
 	listCmd = &cobra.Command{
-		Use:   "list [--list=list1,list2,... | -l list1, list2,...] [ -cmd cmd1 cmd2 cmd3...]",
-		Short: "Lists commands, lists, or hosts defined in config file.",
-		Long:  "Backup lists commands or groups defined in config file.\nUse the --lists or -l flag to list the specified lists. If not flag is not given, all lists will be executed.",
-		Run:   List,
+		Use:   "list [command]",
+		Short: "List commands, lists, or hosts defined in config file.",
+		Long:  "List commands, lists, or hosts defined in config file",
+	}
+
+	listCmds = &cobra.Command{
+		Use:   "cmds [cmd1 cmd2 cmd3...]",
+		Short: "List commands defined in config file.",
+		Long:  "List commands defined in config file",
+		Run:   ListCommands,
+	}
+	listCmdLists = &cobra.Command{
+		Use:   "lists [list1 list2 ...]",
+		Short: "List lists defined in config file.",
+		Long:  "List lists defined in config file",
+		Run:   ListCommandLists,
 	}
 )
 
@@ -23,26 +36,55 @@ var listsToList []string
 var cmdsToList []string
 
 func init() {
-
-	listCmd.Flags().StringSliceVarP(&listsToList, "lists", "l", nil, "Accepts comma-separated names of command lists to list.")
-	listCmd.Flags().StringSliceVarP(&cmdsToList, "cmds", "c", nil, "Accepts comma-separated names of commands to list.")
+	listCmd.AddCommand(listCmds, listCmdLists)
 
 }
 
-func List(cmd *cobra.Command, args []string) {
+func ListCommands(cmd *cobra.Command, args []string) {
 
 	// setup based on whats passed in:
 	//   - cmds
 	//   - lists
 	//   - if none, list all commands
-	if cmdLists != nil {
-
+	if len(args) > 0 {
+		cmdsToList = args
+	} else {
+		logging.ExitWithMSG("Error: list cmds subcommand needs commands to list", 1, nil)
 	}
 
-	opts := backy.NewOpts(cfgFile)
+	parseS3Config()
+
+	opts := backy.NewConfigOptions(configFile,
+		backy.SetLogFile(logFile),
+		backy.SetHostsConfigFile(hostsConfigFile))
 
 	opts.InitConfig()
-	opts.ReadConfig()
+	opts.ParseConfigurationFile()
+
+	for _, v := range cmdsToList {
+		opts.ListCommand(v)
+	}
+}
+
+func ListCommandLists(cmd *cobra.Command, args []string) {
+
+	parseS3Config()
+
+	if len(args) > 0 {
+		listsToList = args
+	} else {
+		logging.ExitWithMSG("Error: lists subcommand needs lists", 1, nil)
+	}
+
+	opts := backy.NewConfigOptions(configFile,
+		backy.SetLogFile(logFile),
+		backy.SetHostsConfigFile(hostsConfigFile))
+
+	opts.InitConfig()
+	opts.ParseConfigurationFile()
+
+	for _, v := range listsToList {
+		opts.ListCommandList(v)
+	}
 
-	opts.ListCommand("rm-sn-db")
 }

cmd/root.go

@@ -13,9 +13,12 @@ import (
 
 var (
 	// Used for flags.
-	cfgFile string
+	configFile      string
+	hostsConfigFile string
 	verbose         bool
+	cmdStdOut       bool
 	logFile         string
+	s3Endpoint      string
 
 	rootCmd = &cobra.Command{
 		Use:   "backy",
@@ -33,11 +36,18 @@ func Execute() {
 }
 
 func init() {
+	rootCmd.PersistentFlags().StringVar(&logFile, "logFile", "", "log file to write to")
+	rootCmd.PersistentFlags().BoolVar(&cmdStdOut, "cmdStdOut", false, "Pass to print command output to stdout")
 
-	rootCmd.PersistentFlags().StringVar(&logFile, "log-file", "", "log file to write to")
-
-	rootCmd.PersistentFlags().StringVarP(&cfgFile, "config", "f", "", "config file to read from")
+	rootCmd.PersistentFlags().StringVarP(&configFile, "config", "f", "", "config file to read from")
+	rootCmd.PersistentFlags().StringVar(&hostsConfigFile, "hostsConfig", "", "yaml hosts file to read from")
 	rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "Sets verbose level")
-
+	rootCmd.PersistentFlags().StringVar(&s3Endpoint, "s3Endpoint", "", "Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.")
 	rootCmd.AddCommand(backupCmd, execCmd, cronCmd, versionCmd, listCmd)
 }
+
+func parseS3Config() {
+	if s3Endpoint != "" {
+		os.Setenv("S3_ENDPOINT", s3Endpoint)
+	}
+}

cmd/version.go

@@ -7,7 +7,7 @@ import (
 	"github.com/spf13/cobra"
 )
 
-const versionStr = "0.6.1"
+const versionStr = "0.11.0"
 
 var (
 	versionCmd = &cobra.Command{

docs/config.yaml

@@ -13,6 +13,7 @@ module:
   imports:
     - path: github.com/divinerites/plausible-hugo
     - path: github.com/McShelby/hugo-theme-relearn
+      version: "v8.2.0"
 params:
   themeVariant:
     - auto: []

docs/content/cli/_index.md

@@ -14,17 +14,21 @@ Usage:
   backy [command]
 
 Available Commands:
-  backup      Runs commands defined in config file.
+  backup      Runs commands defined in config file. Use -l flag multiple times to run multiple lists.
   completion  Generate the autocompletion script for the specified shell
   cron        Starts a scheduler that runs lists defined in config file.
   exec        Runs commands defined in config file in order given.
   help        Help about any command
-  list        Lists commands, lists, or hosts defined in config file.
+  list        List commands, lists, or hosts defined in config file.
   version     Prints the version and exits
 
 Flags:
+      --cmdStdOut            Pass to print command output to stdout
   -f, --config string        config file to read from
   -h, --help                 help for backy
+      --hostsConfig string   yaml hosts file to read from
+      --logFile string       log file to write to
+      --s3Endpoint string    Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
   -v, --verbose              Sets verbose level
 
 Use "backy [command] --help" for more information about a command.
@@ -39,14 +43,18 @@ Backup executes commands defined in config file.
 Use the --lists or -l flag to execute the specified lists. If not flag is not given, all lists will be executed.
 
 Usage:
-  backy backup [--lists=list1,list2,... | -l list1, list2,...] [flags]
+  backy backup [--lists=list1 --lists list2 ... | -l list1 -l list2 ...] [flags]
 
 Flags:
   -h, --help                help for backup
-  -l, --lists strings   Accepts comma-separated names of command lists to execute.
+  -l, --lists stringArray   Accepts comma-separated names of command lists to execute.
 
 Global Flags:
+      --cmdStdOut            Pass to print command output to stdout
   -f, --config string        config file to read from
+      --hostsConfig string   yaml hosts file to read from
+      --logFile string       log file to write to
+      --s3Endpoint string    Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
   -v, --verbose              Sets verbose level
 ```
 
@@ -62,7 +70,11 @@ Flags:
   -h, --help   help for cron
 
 Global Flags:
+      --cmdStdOut            Pass to print command output to stdout
   -f, --config string        config file to read from
+      --hostsConfig string   yaml hosts file to read from
+      --logFile string       log file to write to
+      --s3Endpoint string    Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
   -v, --verbose              Sets verbose level
 ```
 
@@ -77,12 +89,17 @@ Usage:
 
 Available Commands:
   host        Runs command defined in config file on the hosts in order specified.
+  hosts       Runs command defined in config file on the hosts in order specified.
 
 Flags:
   -h, --help   help for exec
 
 Global Flags:
+      --cmdStdOut            Pass to print command output to stdout
   -f, --config string        config file to read from
+      --hostsConfig string   yaml hosts file to read from
+      --logFile string       log file to write to
+      --s3Endpoint string    Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
   -v, --verbose              Sets verbose level
 
 Use "backy exec [command] --help" for more information about a command.
@@ -95,15 +112,19 @@ Host executes specified commands on the hosts defined in config file.
 Use the --commands or -c flag to choose the commands.
 
 Usage:
-  backy exec host [--commands=command1,command2, ... | -c command1,command2, ...] [--hosts=host1,hosts2, ... | -m host1,host2, ...]  [flags]
+  backy exec host [--command=command1 --command=command2 ... | -c command1 -c command2 ...] [--hosts=host1 --hosts=hosts2 ... | -m host1 -m host2 ...]  [flags]
 
 Flags:
-  -c, --commands strings   Accepts comma-separated names of commands.
+  -c, --command stringArray   Accepts space-separated names of commands. Specify multiple times for multiple commands.
   -h, --help                  help for host
-  -m, --hosts strings      Accepts comma-separated names of hosts.
+  -m, --hosts stringArray     Accepts space-separated names of hosts. Specify multiple times for multiple hosts.
 
 Global Flags:
+      --cmdStdOut            Pass to print command output to stdout
   -f, --config string        config file to read from
+      --hostsConfig string   yaml hosts file to read from
+      --logFile string       log file to write to
+      --s3Endpoint string    Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
   -v, --verbose              Sets verbose level
 ```
 
@@ -121,25 +142,74 @@ Flags:
   -V, --vpre   Output the version with v prefixed.
 
 Global Flags:
+      --cmdStdOut            Pass to print command output to stdout
   -f, --config string        config file to read from
+      --hostsConfig string   yaml hosts file to read from
+      --logFile string       log file to write to
+      --s3Endpoint string    Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
   -v, --verbose              Sets verbose level
 ```
 
 ## list
 
 ```
-Backup lists commands or groups defined in config file.
-Use the --lists or -l flag to list the specified lists. If not flag is not given, all lists will be executed.
+List commands, lists, or hosts defined in config file
 
 Usage:
-  backy list [--list=list1,list2,... | -l list1, list2,...] [ -cmd cmd1 cmd2 cmd3...] [flags]
+  backy list [command]
+
+Available Commands:
+  cmds        List commands defined in config file.
+  lists       List lists defined in config file.
 
 Flags:
-  -c, --cmds strings    Accepts comma-separated names of commands to list.
   -h, --help   help for list
-  -l, --lists strings   Accepts comma-separated names of command lists to list.
 
 Global Flags:
+      --cmdStdOut            Pass to print command output to stdout
   -f, --config string        config file to read from
+      --hostsConfig string   yaml hosts file to read from
+      --logFile string       log file to write to
+      --s3Endpoint string    Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
+  -v, --verbose              Sets verbose level
+
+Use "backy list [command] --help" for more information about a command.
+```
+## list cmds
+
+```
+List commands defined in config file
+
+Usage:
+  backy list cmds [cmd1 cmd2 cmd3...] [flags]
+
+Flags:
+  -h, --help   help for cmds
+
+Global Flags:
+      --cmdStdOut            Pass to print command output to stdout
+  -f, --config string        config file to read from
+      --hostsConfig string   yaml hosts file to read from
+      --logFile string       log file to write to
+      --s3Endpoint string    Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
+  -v, --verbose              Sets verbose level
+```
+## list lists
+
+```
+List lists defined in config file
+
+Usage:
+  backy list lists [list1 list2 ...] [flags]
+
+Flags:
+  -h, --help   help for lists
+
+Global Flags:
+      --cmdStdOut            Pass to print command output to stdout
+  -f, --config string        config file to read from
+      --hostsConfig string   yaml hosts file to read from
+      --logFile string       log file to write to
+      --s3Endpoint string    Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
   -v, --verbose              Sets verbose level
 ```

docs/content/cli/exec.md

@@ -7,13 +7,13 @@ The `exec` subcommand can do some things that the configuration file can't do ye
 `exec host` takes the following arguments:
 
 ```sh
-  -c, --commands strings   Accepts comma-separated names of commands.
+  -c, --commands strings   Accepts space-separated names of commands.
   -h, --help               help for host
-  -m, --hosts strings      Accepts comma-separated names of hosts.
+  -m, --hosts strings      Accepts space-separated names of hosts.
 ```
 
 The commands have to be defined in the config file. The hosts need to at least be in the ssh_config(5) file.
 
 ```sh
-backy exec host [--commands=command1,command2, ... | -c command1,command2, ...] [--hosts=host1,hosts2, ... | -m host1,host2, ...]  [flags]
+backy exec host [--commands=command1 -commands=command2 ... | -c command1 -c command2 ...] [--hosts=host1 --hosts=hosts2 ... | -m host1 -m host2 ...]  [flags]
 ```

docs/content/cli/list.md

@@ -0,0 +1,29 @@
+---
+title: List
+---
+
+
+List commands, lists, or hosts defined in config file
+
+Usage:
+```
+  backy list [command]
+```
+
+Available Commands:
+  cmds        List commands defined in config file.
+  lists       List lists defined in config file.
+
+Flags:
+```
+  -h, --help   help for list
+```
+
+Global Flags:
+```
+      --cmdStdOut            Pass to print command output to stdout
+  -f, --config string        config file to read from
+      --log-file string      log file to write to
+      --s3-endpoint string   Sets the S3 endpoint used for config file fetching. Overrides S3_ENDPOINT env variable.
+  -v, --verbose              Sets verbose level
+```

docs/content/config/_index.md

@@ -10,6 +10,13 @@ This is the section on the config file.
 To use a specific file:
 ```backy [command] -f /path/to/file```
 
+You can also use a remote file:
+```
+backy [command] -f `s3/http source`
+```
+
+See remote resources docs for specific info.
+
 If you leave the config path blank, the following paths will be searched in order:
 
 1. `./backy.yml`

docs/content/config/command-lists.md

@@ -2,7 +2,7 @@
 title: "Command Lists"
 weight: 2
 description: >
-  This page tells you how to get use command lists.
+  This page tells you how to use command lists.
 ---
 
 Command lists are for executing commands in sequence and getting notifications from them.
@@ -11,8 +11,16 @@ The top-level object key can be anything you want but not the same as another.
 
 Lists can go in a separate file. Command lists should be in a separate file if:
 
-1. key 'cmd-lists.file' is found
-2. hosts.yml or hosts.yaml is found in the same directory as the backy config file
+1. key 'cmdLists.file' is specified
+2. lists.yml or lists.yaml is found in the same directory as the backy config file (this includes remote config files as of v0.7.0)
+
+{{% notice info %}}
+The lists file is also checked in remote resources. 
+
+The lists file is ignored under the following condition:
+
+If a remote config file is specified (on the command-line using `-f`) and the lists file is not found in the same directory, the lists file is assumed to not exist.
+{{% /notice %}}
 
 ```yaml {lineNos="true" wrap="true" title="yaml"}
   test2:
@@ -27,12 +35,12 @@ Lists can go in a separate file. Command lists should be in a separate file if:
 ```
 
 | key | description | type | required
-| --- | --- | --- | --- |
-| `order` | Defines the sequence of commands to execute | `[]string` | yes |
-| `getOutput` | Command(s) output is in the notification(s) | `bool` | no |
-| `notifications` | The notification service(s) and ID(s) to use on success and failure. Must be *`service.id`*. See the [notifications documentation page](/config/notifications/) for more | `[]string` | no |
-| `name` | Optional name of the list | `string` | no |
-| `cron` | Time at which to schedule the list. Only has affect when cron subcommand is run. | `string` | no |
+| --- | --- | --- | ---
+| `order` | Defines the sequence of commands to execute | `[]string` | yes
+| `sendNotificationOnSuccess` | Whether to send notification on list success with the commands' output | `bool` | no
+| `notifications` | The notification service(s) and ID(s) to use on success and failure. Must be *`service.id`*. See the [notifications documentation page](/config/notifications/) for more | `[]string` | no
+| `name` | Optional name of the list | `string` | no
+| `cron` | Time at which to schedule the list. Only has affect when cron subcommand is run. | `string` | no
 
 ### Order
 
@@ -62,14 +70,14 @@ Name is optional. If name is not defined, name will be the object's map key.
 
 Backy also has a cron mode, so one can run `backy cron` and start a process that schedules jobs to run at times defined in the configuration file.
 
-Adding `cron: 0 0 1 * * *` to a `cmd-lists` object will schedule the list at 1 in the morning. See [https://crontab.guru/](https://crontab.guru/) for reference.
+Adding `cron: 0 0 1 * * *` to a `cmdLists` object will schedule the list at 1 in the morning. See [https://crontab.guru/](https://crontab.guru/) for reference.
 
 {{% notice tip %}}
 Note: Backy uses the second field of cron, so add anything except `*` to the beginning of a regular cron expression.
 {{% /notice %}}
 
 ```yaml {lineNos="true" wrap="true" title="yaml"}
-cmd-lists:
+cmdLists:
   docker-container-backup: # this can be any name you want
     # all commands have to be defined
     order:

docs/content/config/commands.md

@@ -1,153 +0,0 @@
----
-title: "Commands"
-description: Commands are just that, commands
-weight: 1
----
-
-The yaml top-level map can be any string.
-
-The top-level name must be unique.
-
-### Example Config
-
-```yaml
-commands:
-  stop-docker-container:
-    cmd: docker
-    Args:
-      - compose
-      - -f /some/path/to/docker-compose.yaml
-      - down
-    # if host is not defined, command will be run locally
-    # The host has to be defined in either the config file or the SSH Config files
-    host: some-host
-    hooks
-      error:
-        - some-other-command-when-failing
-      success:
-        - success-command
-      final:
-        - final-command
-  backup-docker-container-script:
-    cmd: /path/to/local/script
-    # script file is input as stdin to SSH
-    type: scriptFile # also can be script
-    environment:
-      - FOO=BAR
-      - APP=$VAR
-```
-
-Values available for this section **(case-sensitive)**:
-
-| name | notes | type | required
-| --- | --- | --- | --- |
-| `cmd` | Defines the command to execute | `string` | yes |
-| `Args` | Defines the arguments to the command | `[]string` | no |
-| `environment` | Defines evironment variables for the command | `[]string` | no |
-| `type` | May be `scriptFile`, `script`, or `package`. Runs script from local machine on remote. `Package` is the only one that can be run on local and remote hosts. | `string` | no |
-| `getOutput` | Command(s) output is in the notification(s) | `bool` | no |
-| `host` | If not specified, the command will execute locally. | `string` | no |
-| `scriptEnvFile` | When type is `scriptFile` or `script`, this file is prepended to the input. | `string` | no |
-| `shell` | Run the command in the shell | `string` | no |
-| `hooks` | Hooks are used at the end of the individual command. Must have at least `error`, `success`, or `final`. | `map[string][]string` | no |
-
-#### cmd
-
-cmd must be a valid command or script to execute.
-
-#### Args
-
-args must be arguments to cmd as they would be passed on the command-line:
-
-```sh
-cmd [arg1 arg2 ...]
-```
-
-Define them in an array:
-
-```yaml
-Args:
-  - arg1
-  - arg2
-  - arg3
-```
-
-### getOutput
-
-Get command output when a notification is sent.
-
-Is not required. Can be `true` or `false`.
-
-#### host
-
-{{% notice info %}}
-If any `host` is not defined or left blank, the command will run on the local machine.
-{{% /notice %}}
-
-Host may or may not be defined in the `hosts` section.
-
-{{% notice info %}}
-If any `host` from the commands section does not match any object in the `hosts` section, the `Host` is assumed to be this value. This value will be used to search in the default SSH config files.
-
-For example, say that I have a host defined in my SSH config with the `Host` defined as `web-prod`.
-If I assign a value to host as `host: web-prod` and don't specify this value in the `hosts` object, web-prod will be used as the `Host` in searching the SSH config files.
-{{% /notice %}}
-
-### shell
-
-If shell is defined, the command will run in the specified shell.
-Make sure to escape any shell input.
-
-### scriptEnvFile
-
-Path to a file.
-
-When type is `script` or `scriptFile` , the script is appended to this file.
-
-This is useful for specifying environment variables or other things so they don't have to be included in the script.
-
-### type
-
-May be `scriptFile` or `script`. Runs script from local machine on remote host passed to the SSH session as standard input.
-
-If `type` is `script`, `cmd` is used as the script.
-
-If `type` is `scriptFile`, cmd must be a script file.
-
-If `type` is `package`, there are additional fields that must be specified.
-
-### environment
-
-The environment variables support expansion:
-
-- using escaped values `$VAR` or `${VAR}`
-
-For now, the variables have to be defined in an `.env` file in the same directory as the config file.
-
-If using it with host specified, the SSH server has to be configured to accept those env variables.
-
-If the command is run locally, the OS's environment is added.
-
-### hooks
-
-Hooks are run after the command is run.
-
-Errors are run if the command errors, success if it returns no error. Final hooks are run regardless of error condition.
-
-Values for hooks are as follows:
-
-```yaml
-command:
-  hook:
-    # these commands are defined elsewhere in the file
-    error:
-      - errcommand
-    success:
-      - successcommand
-    final:
-      - donecommand
-```
-
-### packages
-
-See the [dedicated page](/config/packages) for package configuration.

docs/content/config/commands/_index.md

@@ -0,0 +1,164 @@
+---
+title: "Commands"
+description: Commands are just that, commands
+weight: 1
+---
+
+
+
+### Example Config
+
+{{% code file="/examples/example.yml" language="yaml" %}}
+
+Values available for this section **(case-sensitive)**:
+
+| name            | notes                                                                                                   | type                  | required | External directive support |
+| ----------------| ------------------------------------------------------------------------------------------------------- | --------------------- | -------- |----------------------------|
+| `cmd`           | Defines the command to execute                                                                          | `string`              | yes      | No                         |
+| `Args`          | Defines the arguments to the command                                                                    | `[]string`            | no       | No                         |
+| `environment`   | Defines environment variables for the command                                                           | `[]string`            | no       | Partial                    |
+| `type`          | See documentation further down the page. Additional fields may be required.                             | `string`              | no       | No                         |
+| `getOutput`     | Command(s) output is in the notification(s)                                                             | `bool`                | no       | No                         |
+| `host`          | Depricated: use `hosts`. If not specified, the command will execute locally.                                                     | `string`              | no       | No                         |
+| `hosts`         | Must be specified to run commands both locallly and in parrallel.                                                     | `[]string`              | no       | No                         |
+| `scriptEnvFile` | When type is `scriptFile` or `script`, this file is prepended to the input.                             | `string`              | no       | No                         |
+| `shell`         | Run the command in the shell                                                                            | `string`              | no       | No                         |
+| `hooks`         | Hooks are used at the end of the individual command. Must have at least `error`, `success`, or `final`. | `map[string][]string` | no       | No                         |
+
+#### cmd
+
+cmd must be a valid command or script to execute.
+
+#### Args
+
+args must be arguments to cmd as they would be passed on the command-line:
+
+```sh
+cmd [arg1 arg2 ...]
+```
+
+Define them in an array:
+
+```yaml
+Args:
+  - arg1
+  - arg2
+  - arg3
+```
+
+### getOutput
+
+Get command output when a notification is sent.
+
+Is not required. Can be `true` or `false`.
+
+### host
+
+
+{{% notice warning %}}
+Depricated: use `hosts` instead.
+{{% /notice %}}
+
+{{% notice info %}}
+If any `host` is not defined or left blank, the command will run on the local machine.
+{{% /notice %}}
+
+Host may or may not be defined in the `hosts` section.
+
+{{% notice info %}}
+If any `host` from the commands section does not match any object in the `hosts` section, the `Host` is assumed to be this value. This value will be used to search in the default SSH config files.
+
+For example, say that I have a host defined in my SSH config with the `Host` defined as `web-prod`.
+If I assign a value to host as `host: web-prod` and don't specify this value in the `hosts` object, web-prod will be used as the `Host` in searching the SSH config files.
+{{% /notice %}}
+
+### hosts
+
+{{% notice info %}}
+If any `command.[name].hosts` index is `localhost` or `127.0.0.1`, the command will run on the local machine.
+
+You can also remove the field to have the command run locally.
+{{% /notice %}}
+
+Host may or may not be defined in the `hosts` section.
+
+{{% notice info %}}
+If any `host` from the commands section does not match any object in the `hosts` section, the `Host` is assumed to be this value. This value will be used to search in the default SSH config files.
+
+For example, say that I have a host defined in my SSH config with the `Host` defined as `web-prod`.
+If I assign a value to host as `host: web-prod` and don't specify this value in the `hosts` object, web-prod will be used as the `Host` in searching the SSH config files.
+{{% /notice %}}
+
+###### Example:
+
+
+```yaml
+command:
+  start-some-process:
+    cmd: start-server
+    hosts:
+      - prod-1
+      - prod-2
+```
+
+
+### shell
+
+If shell is defined, the command will run in the specified shell.
+Make sure to escape any shell input.
+
+### scriptEnvFile
+
+Path to a file.
+
+When type is `script` or `scriptFile` , the script is appended to this file.
+
+This is useful for specifying environment variables or other things so they don't have to be included in the script.
+
+### type
+
+The following options are available:
+
+| name | description |
+| --- | --- |
+| script | `cmd` is used as the script |
+| scriptFile | Can only be run on a host. `cmd` is read and used as the script, and `scriptEnvFile` can be used to add env variables |
+| package | Run package operations. See [dedicated page](/config/packages) for configuring package commands |
+| user | Run user operations. See [dedicated page](/config/user-commands) for configuring package commands |
+
+### environment
+
+The environment variables support expansion:
+
+- using escaped values `$VAR` or `${VAR}`
+- using any external directive, and if using the env directive, the variable will be read from a `.env` file
+
+<!-- For now, the variables expanded have to be defined in an `.env` file in the same directory that the program is run from. -->
+
+If using it with host specified, the SSH server has to be configured to accept those env variables.
+
+If the command is run locally, the OS's environment is added.
+
+### hooks
+
+Hooks are run after the command is run.
+
+Errors are run if the command errors, success if it returns no error. Final hooks are run regardless of error condition.
+
+Values for hooks are as follows:
+
+```yaml
+command:
+  hook:
+    # these commands are defined elsewhere in the file
+    error:
+      - errcommand
+    success:
+      - successcommand
+    final:
+      - donecommand
+```
+
+### packages
+
+See the [dedicated page](/config/packages) for package configuration.

docs/content/config/commands/packages.md

@@ -8,10 +8,10 @@ This is dedicated to `package` commands. The command `type` field must be `packa
 
 | name | notes | type | required |
 | --- | --- | --- | --- |
-| `packageName` | The name of a package to be modified. | `string` | yes |
+| `packageName` | The name of a package to be modified. | `[]packagemanagercommon.Package` | yes |
 | `packageManager` | The name of the package manger to be used. | `string` | yes |
-| `packageOperation` | The type of operation to be perform. | `string` | yes |
-| `packageVersion` | The version of a package to be modified. | `string` | no |
+| `packageOperation` | The type of operation to perform. | `string` | yes |
+| `packageVersion` | The version of a package. | `string` | no |
 
 
 #### example
@@ -22,7 +22,9 @@ The following is an example of a package command:
  update-docker:
     type: package
     shell: zsh
-    packageName: docker-ce
+    packages:
+	  - name: docker-ce
+	    version: 10
     packageManager: apt
     packageOperation: install
     host: debian-based-host
@@ -35,6 +37,7 @@ The following package operations are supported:
 - `install`
 - `remove`
 - `upgrade`
+- `checkVersion`
 
 #### packageManager
 
@@ -46,11 +49,11 @@ The following package managers are recognized:
 
 #### package command args
 
-You can add additional arguments using the standard `Args` key. This is useful for adding more packages.
+You can add additional arguments using the standard `Args` key. This is useful for adding more packages, yet it does not work with `checkVersion`.
 
 ### Development
 
-The PackageManager interface provides an easy to enforce functions and options. There are two interfaces, `PackageManager` and `ConfigurablePackageManager` in the directory `pkg/pkgman`. Go's import-cycle "feature" caused me to implement functional options using a third interface. `PackageManagerOption`is a function that takes an interface.
+The PackageManager interface provides an easy way to enforce functions and options. There are two interfaces, `PackageManager` and `ConfigurablePackageManager` in the directory `pkg/pkgman`. Go's import-cycle "feature" caused me to implement functional options using a third interface. `PackageManagerOption`is a function that takes an interface.
 
 #### PackageManager
 

docs/content/config/commands/user-commands.md

@@ -0,0 +1,66 @@
+---
+title: "User commands"
+weight: 2
+description: This is dedicated to user commands.
+---
+
+This is dedicated to `user` commands. The command `type` field must be `user`. User is a type that allows one to perform user operations. There are several additional options available when `type` is `user`:
+
+| name            | notes                                                        | type       | required | External directive support
+| ----------------| -------------------------------------------------------------| ---------- | ---------| --------------------------|
+| `userName`      | The name of a user to be configured.                         | `string`   | yes      | no 						 |
+| `userOperation` | The type of operation to perform.                            | `string`   | yes      | no 						 |
+| `userID`        | The user ID to use.                                          | `string`   | no       | no 						 |
+| `userGroups`    | The groups the user should be added to.                      | `[]string` | no       | no 						 |
+| `systemUser`    | Create a system user.                                        | `bool`     | no       | no 						 |
+| `userCreateHome`| Create the home directory.                                   | `bool`     | no       | no 						 |
+| `userSshPubKeys`| The keys to add to the user's authorized keys.               | `[]string` | no       | yes 						 |
+| `userShell`     | The shell for the user.                                      | `string`   | no       | no 						 |
+| `userHome`      | The user's home directory.                                   | `string`   | no       | no 						 |
+| `userPassword`  | The new password value when using the `password` operation.  | `string`   | no       | yes						 |
+
+
+#### example
+
+The following is an example of a package command:
+
+```yaml
+  addUser:
+	name: add user backy with custom home dir
+    type: user
+    userName: backy
+    userHome: /opt/backy
+    userOperation: add
+    host: some-host
+```
+
+#### userOperation
+
+The following package operations are supported:
+
+- `add`
+- `remove`
+- `modify`
+- `password`
+- `checkIfExists`
+
+### Development
+
+The UserManager interface provides an way easy to add new commands. There is one interface `Usermanager` in directory `pkg/usermanager`.
+
+#### UserManager
+
+```go
+// UserManager defines the interface for user management operations.
+// All functions but one return a string for the command and any args.
+type UserManager interface {
+	AddUser(username, homeDir, shell string, isSystem bool, groups, args []string) (string, []string)
+	RemoveUser(username string) (string, []string)
+	ModifyUser(username, homeDir, shell string, groups []string) (string, []string)
+	// Modify password uses chpasswd for Linux systems to build the command to change the password
+	// Should return a password as the last argument
+	// TODO: refactor when adding more systems instead of Linux
+	ModifyPassword(username, password string) (string, *strings.Reader, string)
+	UserExists(username string) (string, []string)
+}
+```

docs/content/config/directives.md

@@ -0,0 +1,15 @@
+---
+title: "External Directives"
+weight: 2
+description: How to set up external directives.
+---
+
+External directives are for including data that should not be in the config file. The following directives are supported:
+
+- `%{file:path/to/file}%`
+- `%{env:ENV_VAR}%`
+- `%{vault:vault-key}%`
+
+See the docs of each command if the field is supported.
+
+If the file path does not begin with a `/`, the config file's directory will be used as the starting point.

docs/content/config/hosts.md

@@ -0,0 +1,24 @@
+---
+title: "Hosts"
+weight: 2
+description: >
+  This page tells you how to use hosts.
+---
+
+| Key                  | Description                                                   | Type     | Required | External directive support |
+|----------------------|---------------------------------------------------------------|----------|----------|----------------------------|
+| `OS`                 | Operating system of the host (used for package commands)      | `string` | no       | No                         |
+| `config`             | Path to the SSH config file                                   | `string` | no       | No                         |
+| `host`               | Specifies the `Host` ssh_config(5) directive                  | `string` | yes      | No                         |
+| `hostname`           | Hostname of the host                                          | `string` | no       | No                         |
+| `knownHostsFile`     | Path to the known hosts file                                  | `string` | no       | No                         |
+| `port`               | Port number to connect to                                     | `uint16` | no       | No                         |
+| `proxyjump`          | Proxy jump hosts, comma-separated                             | `string` | no       | No                         |
+| `password`           | Password for SSH authentication                               | `string` | no       | No                         |
+| `privateKeyPath`     | Path to the private key file                                  | `string` | no       | No                         |
+| `privateKeyPassword` | Password for the private key file                             | `string` | no       | Yes                        |
+| `user`               | Username for SSH authentication                               | `string` | no       | No                         |
+
+## exec host subcommand
+
+Backy has a subcommand `exec host`. This subcommand takes the flags of `-m host1 -m host2`. The commands can also be specified by `-c command1 -c command2`.

docs/content/config/notifications.md

@@ -5,6 +5,7 @@ description: >
   This page tells you how to get set up Backy notifications.
 ---
 
+Notifications are only configurable for command lists, as of right now.
 
 Notifications can be sent on command list completion and failure.
 
@@ -38,23 +39,23 @@ There must be a section with an id (eg. `mail.test-svr`) following one of these
 
 ### mail
 
-| key | description | type
-| --- | --- | ---
-| `host` | Specifies the SMTP host to connect to | `string`
-| `port` | Specifies the SMTP port | `uint16`
-| `senderaddress` | Address from which to send mail | `string`
-| `to` | Recipients to send emails to | `[]string`
-| `username` | SMTP username | `string`
-| `password` | SMTP password | `string`
+| key | description | type | External directive support |
+| --- | --- | --- | --- |
+| `host` | Specifies the SMTP host to connect to | `string` | no
+| `port` | Specifies the SMTP port | `uint16` | no
+| `senderaddress` | Address from which to send mail | `string` | no
+| `to` | Recipients to send emails to | `[]string` | no
+| `username` | SMTP username | `string` | no
+| `password` | SMTP password | `string` | yes
 
 ### matrix
 
-| key | description | type
-| --- | --- | ---
-| `home-server` | Specifies the Matrix server connect to | `string`
-| `room-id` | Specifies the room ID of the room to send messages to | `string`
-| `access-token` | Matrix access token | `string`
-| `user-id` | Matrix user ID | `string`
+| key | description | type | External directive support |
+| --- | --- | ---| ---- |
+| `home-server` | Specifies the Matrix server connect to | `string` | no
+| `room-id` | Specifies the room ID of the room to send messages to | `string` | no
+| `access-token` | Matrix access token  | `string` | yes
+| `user-id` | Matrix user ID | `string` | no
 
 To get your access token (assumes you are using [Element](https://element.io/)) :
 

docs/content/config/remote-resources.md

@@ -0,0 +1,21 @@
+---
+title: "Remote resources"
+weight: 2
+description: This is dedicated to configuring remote resources.
+---
+
+Remote resources can be used for a lot of things, including config files and scripts.
+
+## Config file
+
+For the main config file to be fetched remotely, pass the URL using `-f [url]`.
+
+If using S3, you should use the s3 protocol URI: `s3://bucketName/key/path`. You will also need to set the env variable `S3_ENDPOINT` to the appropriate value. The flag `--s3-endpoint` can be used to override this value or to set this value, if not already set.
+
+## Authentication
+
+Currently, only the AWS authentication credentials file `~/.aws/credentials` is supported. For now, the environment variable `AWS_PROFILE` is used to lookup the profile.
+
+## Scripts
+
+Remote script support is currently limited to http/https endpoints.

docs/content/config/vault.md

@@ -6,7 +6,7 @@ description: Set up and configure vault.
 
 [Vault](https://www.vaultproject.io/) is a tool for storing secrets and other data securely.
 
-Vault config can be used by prefixing `vault:` in front of a password or ENV var.
+A Vault key can be used by prefixing `%{vault:vault.keys.name}%` in a field that supports external directives.
 
 This is the object in the config file:
 
@@ -18,10 +18,12 @@ vault:
   keys:
     - name: mongourl
       mountpath: secret
+      key: data
       path: mongo/url
-      type:  # KVv1 or KVv2
-    - name:
-      path:
-      type:
-      mountpath:
+      type: KVv2  # KVv1 or KVv2
+    - name: someKeyName
+      mountpath: secret
+      key: keyData
+      type: KVv2
+      path: some/path
 ```

docs/content/examples/backy.yaml

@@ -0,0 +1,108 @@
+commands:
+  stop-docker-container:
+    cmd: docker
+    Args:
+      - compose
+      - -f /some/path/to/docker-compose.yaml
+      - down
+    # if host is not defined, cmd will be run locally
+    host: some-host
+    hooks:
+      final:
+        - hostname
+      error:
+        - hostname
+  backup-docker-container-script:
+    cmd: /path/to/script
+    # The host has to be defined in the config file
+    host: some-host
+    environment:
+      - FOO=BAR
+      - APP=$VAR
+  shell-cmd:
+    cmd: rsync
+    shell: bash
+    Args:
+      - -av some-host:/path/to/data ~/Docker/Backups/docker-data
+  hostname:
+    cmd: hostname
+  update-docker:
+    type: package
+    shell: zsh # best to run package commands in a shell
+    packageName: docker-ce
+    Args:
+      - docker-ce-cli
+    packageManager: apt
+    packageOperation: install
+  update-dockerApt:
+    # type: package
+    shell: zsh
+    cmd: apt
+    Args:
+      - update
+      - "&&"
+      - apt install -y docker-ce
+      - docker-ce-cli
+    packageManager: apt
+    packageOperation: install
+
+cmd-lists:
+  cmds-to-run: # this can be any name you want
+    # all commands have to be defined
+    order:
+      - stop-docker-container
+      - backup-docker-container-script
+      - shell-cmd
+      - hostname
+    notifications:
+      - matrix.matrix
+    name: backup-some-server
+    cron: "0 0 1 * * *"
+  hostname:
+    name: hostname
+    order:
+      - hostname
+    notifications:
+      - mail.prod-email
+
+hosts:
+  # any ssh_config(5) keys/values not listed here will be looked up in the config file or the default config file
+  some-host:
+    hostname: some-hostname
+    config: ~/.ssh/config
+    user: user
+    privateKeyPath: /path/to/private/key
+    port: 22
+    # can also be env:VAR
+    password: file:/path/to/file
+    # only one is supported for now
+    proxyjump: some-proxy-host
+
+# optional
+logging:
+  verbose: true
+  file: ./backy.log
+  console: false
+  cmd-std-out: false
+
+
+notifications:
+  mail:
+    prod-email:
+      id: prod-email
+      type: mail
+      host: yourhost.tld
+      port: 587
+      senderAddress: email@domain.tld
+      to:
+        - admin@domain.tld
+      username: smtp-username@domain.tld
+      password: your-password-here
+  matrix:
+    matrix:
+      id: matrix
+      type: matrix
+      home-server: your-home-server.tld
+      room-id: room-id
+      access-token: your-access-token
+      user-id: your-user-id

docs/content/examples/example.yml

@@ -0,0 +1,24 @@
+commands:
+  stop-docker-container:
+    cmd: docker
+    Args:
+      - compose
+      - -f /some/path/to/docker-compose.yaml
+      - down
+    # if host is not defined, command will be run locally
+    # The host has to be defined in either the config file or the SSH Config files
+    host: some-host
+    hooks:
+      error:
+        - some-other-command-when-failing
+      success:
+        - success-command
+      final:
+        - final-command
+  backup-docker-container-script:
+    cmd: /path/to/local/script
+    # script file is input as stdin to SSH
+    type: scriptFile # also can be script
+    environment:
+      - FOO=BAR
+      - APP=$VAR

docs/content/getting-started/config.md

@@ -48,7 +48,7 @@ commands:
 To execute groups of commands in sequence, use a list configuration.
 
 ```yaml
-cmd-lists:
+cmdLists:
   cmds-to-run: # this can be any name you want
     # all commands have to be defined in the commands section
     order:
@@ -97,7 +97,7 @@ hosts:
 
 The notifications object can have two forms.
 
-For more, [see the notification object documentation](/config/notifications). The top-level map key is id that has to be referenced by the `cmd-lists` key `notifications`.
+For more, [see the notification object documentation](/config/notifications). The top-level map key is id that has to be referenced by the `cmdLists` key `notifications`.
 
 ```yaml
 notifications:
@@ -124,13 +124,13 @@ notifications:
 
 ### Logging
 
-cmd-std-out controls whether commands output is echoed to StdOut.
+`cmd-std-out` controls whether commands output is echoed to StdOut.
 
-If logfile is not defined, the log file will be written to the config directory in the file `backy.log`.
+If `logfile` is not defined, the log file will be written to the config directory in the file `backy.log`.
 
 `console-disabled` controls whether the logging messages are echoed to StdOut. Default is false.
 
-`verbose` basically does nothing as all necessary info is already output.
+`verbose` prints out debugging messages.
 
 ```yaml
 logging:
@@ -144,7 +144,7 @@ logging:
 
 [Vault](https://www.vaultproject.io/) can be used to get some configuration values and ENV variables securely.
 
-```
+```yaml
 vault:
   token: hvs.tXqcASvTP8wg92f7riyvGyuf
   address: http://127.0.0.1:8200

docs/go.mod

@@ -3,6 +3,6 @@ module git.andrewnw.xyz/CyberShell/backy/docs
 go 1.19
 
 require (
-	github.com/McShelby/hugo-theme-relearn v0.0.0-20250103114405-80e448e5bdaa // indirect
-	github.com/divinerites/plausible-hugo v1.21.1 // indirect
+	github.com/McShelby/hugo-theme-relearn v0.0.0-20251115105808-d9ca8e8d8f59 // indirect
+	github.com/divinerites/plausible-hugo v1.22.0 // indirect
 )

docs/go.sum

@@ -1,10 +1,4 @@
-github.com/McShelby/hugo-theme-relearn v0.0.0-20230209073138-890d12ea922d h1:weq1mrQ/qNAvGrNgvZVL1K8adbT3bswZf2ABLr/LCIA=
-github.com/McShelby/hugo-theme-relearn v0.0.0-20230209073138-890d12ea922d/go.mod h1:mKQQdxZNIlLvAj8X3tMq+RzntIJSr9z7XdzuMomt0IM=
-github.com/McShelby/hugo-theme-relearn v0.0.0-20241210183303-16d4de84becf h1:bMx4kwM7Q+dAzvSOWs3XWZ25o+n4mI0GPHqzbzeWb3M=
-github.com/McShelby/hugo-theme-relearn v0.0.0-20241210183303-16d4de84becf/go.mod h1:mKQQdxZNIlLvAj8X3tMq+RzntIJSr9z7XdzuMomt0IM=
-github.com/McShelby/hugo-theme-relearn v0.0.0-20250102210630-dd0597ffa4b2 h1:sWaC1/dL65v3iRvblEAaBLpKC5TIT0R9JASk1hZNET8=
-github.com/McShelby/hugo-theme-relearn v0.0.0-20250102210630-dd0597ffa4b2/go.mod h1:mKQQdxZNIlLvAj8X3tMq+RzntIJSr9z7XdzuMomt0IM=
-github.com/McShelby/hugo-theme-relearn v0.0.0-20250103114405-80e448e5bdaa h1:G+OnMEzK4XOzbbcf1SmaGyOYJ0h5idp/IJdguWs8ioU=
-github.com/McShelby/hugo-theme-relearn v0.0.0-20250103114405-80e448e5bdaa/go.mod h1:mKQQdxZNIlLvAj8X3tMq+RzntIJSr9z7XdzuMomt0IM=
-github.com/divinerites/plausible-hugo v1.21.1 h1:ZTWwjhZ0PmLMacCVGlcGiYFEZW7VaYE767tchDskOug=
-github.com/divinerites/plausible-hugo v1.21.1/go.mod h1:cxr+YB3FUwbLon8KCs4pV4Ankbkq6lJxTQUpNb5KqPo=
+github.com/McShelby/hugo-theme-relearn v0.0.0-20251115105808-d9ca8e8d8f59 h1:mnEjz/Wrpv6Hea26KeFJPx94w9g9ZHIurUEWvPdaEvs=
+github.com/McShelby/hugo-theme-relearn v0.0.0-20251115105808-d9ca8e8d8f59/go.mod h1:mKQQdxZNIlLvAj8X3tMq+RzntIJSr9z7XdzuMomt0IM=
+github.com/divinerites/plausible-hugo v1.22.0 h1:2pZheSaIMc+EtwcEeZv0ioU2qBOEZa1Ii7IaR/9II9k=
+github.com/divinerites/plausible-hugo v1.22.0/go.mod h1:cxr+YB3FUwbLon8KCs4pV4Ankbkq6lJxTQUpNb5KqPo=

docs/layouts/_default/baseof.html

@@ -1,7 +1,7 @@
 <!DOCTYPE html>
 {{- block "storeOutputFormat" . }}{{ end }}
 {{- if .IsHome }}
-  {{- $hugoVersion := "0.126.0" }}
+  {{- $hugoVersion := "0.141.0" }}
   {{- if lt hugo.Version $hugoVersion }}
     {{- errorf "The Relearn theme requires Hugo %s or later" $hugoVersion }}
   {{- end }}
@@ -36,12 +36,12 @@
     {{ (printf $link (partial "permalink.gotmpl" (dict "to" .)) .Rel .MediaType.Type ($title | htmlEscape)) | safeHTML }}
       {{- end }}
     {{- end }}
+
     {{- partialCached "favicon.html" . }}
-    {{- partial "stylesheet.html" . }}
-    {{- partial "dependencies.gotmpl" (dict "page" . "location" "header") }}
+    {{- partial "dependencies.html" (dict "page" . "location" "header") }}
     {{- partial "custom-header.html" . }}
   </head>
-  <body class="mobile-support {{ with .Store.Get "relearnOutputFormat" }}{{ . }}{{ else }}html{{ end }}{{- if .Site.Params.disableInlineCopyToClipBoard }} disableInlineCopyToClipboard{{ end }}{{- if .Site.Params.disableHoverBlockCopyToClipBoard }} disableHoverBlockCopyToClipBoard{{ end }}" data-url="{{ partial "permalink.gotmpl" (dict "to" .) }}">
+  <body class="mobile-support {{ with .Store.Get "relearnOutputFormat" }}{{ . }}{{ else }}html{{ end }}{{- if .Site.Params.disableHoverBlockCopyToClipBoard }} disableHoverBlockCopyToClipBoard{{ end }}" data-url="{{ partial "permalink.gotmpl" (dict "to" .) }}">
     <div id="R-body" class="default-animation">
       <div id="R-body-overlay"></div>
       <nav id="R-topbar">
@@ -53,7 +53,7 @@
           {{- $showBreadcrumb := (and (not .Params.disableBreadcrumb) (not .Site.Params.disableBreadcrumb)) }}
           {{- if $showBreadcrumb }}
           <ol class="topbar-breadcrumbs breadcrumbs highlightable" itemscope itemtype="http://schema.org/BreadcrumbList">
-            {{- partial "breadcrumbs.html" (dict "page" .) }}
+            {{- partial "breadcrumbs.html" (dict "page" . "schema" true) }}
           </ol>
           {{- else }}
           <span class="topbar-breadcrumbs highlightable">
@@ -74,11 +74,8 @@
 {{- partial "custom-comments.html" . }}
     </div>
 {{- block "menu" . }}{{ end }}
-{{- $assetBusting := partialCached "assetbusting.gotmpl" . }}
-    <script src="{{"js/clipboard.min.js" | relURL}}{{ $assetBusting }}" defer></script>
-    <script src="{{"js/perfect-scrollbar.min.js" | relURL}}{{ $assetBusting }}" defer></script>
-{{- partial "dependencies.gotmpl" (dict "page" . "location" "footer") }}
-    <script src="{{"js/theme.js" | relURL}}{{ $assetBusting }}" defer></script>
+{{- partial "dependencies.html" (dict "page" . "location" "footer") }}
 {{- partial "custom-footer.html" . }}
+    <div id="toast-container" role="status" aria-live="polite" aria-atomic="false"></div>
   </body>
 </html>

docs/layouts/shortcodes/code.html

@@ -0,0 +1,3 @@
+{{ $file := .Get "file" | readFile }}
+{{ $lang := .Get "language" }}
+{{ (print "```" $lang "\n" $file "\n```") }}

examples/backy.yaml

@@ -29,20 +29,22 @@ commands:
   update-docker:
     type: package
     shell: zsh # best to run package commands in a shell
-    packageName: docker-ce
-    Args:
-      - docker-ce-cli
+    packages:
+      - name: docker-ce
+        version: latest
+      - name: docker-ce-cli
+        version: latest
     packageManager: apt
     packageOperation: install
   update-dockerApt:
     # type: package
     shell: zsh
     cmd: apt
-    Args:
-      - update
-      - "&&"
-      - apt install -y docker-ce
-      - docker-ce-cli
+    packages:
+      - name: docker-ce
+        version: latest
+      - name: docker-ce-cli
+        version: latest
     packageManager: apt
     packageOperation: install
 

examples/example.yml

@@ -0,0 +1,25 @@
+commands:
+  stop-docker-container:
+    cmd: docker
+    Args:
+      - compose
+      - -f /some/path/to/docker-compose.yaml
+      - down
+    # if host is not defined, command will be run locally
+    # The host has to be defined in either the config file or the SSH Config files
+    hosts: 
+     - prod
+    hooks:
+      error:
+        - some-other-command-when-failing
+      success:
+        - success-command
+      final:
+        - final-command
+  backup-docker-container-script:
+    cmd: /path/to/local/script
+    # script file is input as stdin to SSH
+    type: scriptFile # also can be script
+    environment:
+      - FOO=BAR
+      - APP=$VAR

getCommandHelp

@@ -1,67 +1,83 @@
 #!/bin/bash
 CLI_PAGE="docs/content/cli/_index.md"
+
+BACKYCOMMAND="go run backy.go"
+
+{
 echo "---
-title: "CLI"
+title: CLI
 weight: 4
 ---
 
 This page lists documentation for the CLI.
-" > _index.md
+" 
 
-BACKYCOMMAND="go run backy.go"
-
-echo "## Backy " >> _index.md
-echo " " >> _index.md
-echo "\`\`\`" >> _index.md
-eval "${BACKYCOMMAND} -h >>  _index.md"
-echo "\`\`\`" >> _index.md
-echo " " >> _index.md
+echo "## Backy "
+echo " "
+echo "\`\`\`"
+eval "${BACKYCOMMAND} -h"
+echo "\`\`\`"
+echo " "
 
 
-echo "# Subcommands" >> _index.md
-echo "" >> _index.md
 
-echo "## backup" >> _index.md
-echo "" >> _index.md
-echo "\`\`\`" >> _index.md
-eval "${BACKYCOMMAND} backup -h >>  _index.md"
-echo "\`\`\`" >> _index.md
-echo "" >> _index.md
+echo "# Subcommands" 
+echo "" 
 
-echo "## cron" >> _index.md
-echo "" >> _index.md
-echo "\`\`\`" >> _index.md
-eval "${BACKYCOMMAND} cron -h >>  _index.md"
-echo "\`\`\`" >> _index.md
-echo "" >> _index.md
+echo "## backup" 
+echo "" 
+echo "\`\`\`" 
+eval "${BACKYCOMMAND} backup -h"
+echo "\`\`\`" 
+echo "" 
 
-echo "## exec" >> _index.md
-echo "" >> _index.md
-echo "\`\`\`" >> _index.md
-eval "${BACKYCOMMAND} exec -h >>  _index.md"
-echo "\`\`\`" >> _index.md
-echo "" >> _index.md
+echo "## cron" 
+echo "" 
+echo "\`\`\`" 
+eval "${BACKYCOMMAND} cron -h"
+echo "\`\`\`" 
+echo "" 
 
-echo "### exec host" >> _index.md
-echo "" >> _index.md
-echo "\`\`\`" >> _index.md
-eval "${BACKYCOMMAND} exec host -h >>  _index.md"
-echo "\`\`\`" >> _index.md
-echo "" >> _index.md
+echo "## exec" 
+echo "" 
+echo "\`\`\`" 
+eval "${BACKYCOMMAND} exec -h"
+echo "\`\`\`" 
+echo "" 
+
+echo "### exec host" 
+echo "" 
+echo "\`\`\`" 
+eval "${BACKYCOMMAND} exec host -h"
+echo "\`\`\`" 
+echo "" 
 
 
-echo "## version" >> _index.md
-echo "" >> _index.md
-echo "\`\`\`" >> _index.md
-eval "${BACKYCOMMAND} version -h >>  _index.md"
-echo "\`\`\`" >> _index.md
-echo "" >> _index.md
+echo "## version" 
+echo "" 
+echo "\`\`\`" 
+eval "${BACKYCOMMAND} version -h"
+echo "\`\`\`" 
+echo "" 
 
-echo "## list" >> _index.md
-echo "" >> _index.md
-echo "\`\`\`" >> _index.md
-eval "${BACKYCOMMAND} list -h >>  _index.md"
-echo "\`\`\`" >> _index.md
+echo "## list" 
+echo "" 
+echo "\`\`\`" 
+eval "${BACKYCOMMAND} list -h"
+echo "\`\`\`" 
+
+echo "## list cmds" 
+echo "" 
+echo "\`\`\`" 
+eval "${BACKYCOMMAND} list cmds -h"
+echo "\`\`\`" 
+
+echo "## list lists" 
+echo "" 
+echo "\`\`\`" 
+eval "${BACKYCOMMAND} list lists -h"
+echo "\`\`\`" 
+} >> _index.md
 
 
 mv _index.md "$CLI_PAGE"

go.mod

@@ -1,92 +1,99 @@
 module git.andrewnw.xyz/CyberShell/backy
 
-go 1.21
+go 1.23.0
 
-toolchain go1.22.2
-
-replace git.andrewnw.xyz/CyberShell/backy => /home/andrew/Projects/backy
+toolchain go1.23.7
 
 require (
-	github.com/aws/aws-sdk-go-v2/config v1.18.27
-	github.com/aws/aws-sdk-go-v2/service/s3 v1.72.0
-	github.com/go-co-op/gocron v1.33.1
-	github.com/hashicorp/vault/api v1.10.0
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.83.0
+	github.com/dmarkham/enumer v1.5.11
+	github.com/go-co-op/gocron v1.37.0
+	github.com/google/uuid v1.6.0
+	github.com/hashicorp/vault/api v1.20.0
 	github.com/joho/godotenv v1.5.1
 	github.com/kevinburke/ssh_config v1.2.0
-	github.com/knadh/koanf/parsers/yaml v0.1.0
-	github.com/knadh/koanf/providers/rawbytes v0.1.0
-	github.com/knadh/koanf/v2 v2.0.1
-	github.com/mattn/go-isatty v0.0.19
-	github.com/nikoksr/notify v0.41.0
+	github.com/knadh/koanf/parsers/yaml v1.1.0
+	github.com/knadh/koanf/providers/rawbytes v1.0.0
+	github.com/knadh/koanf/v2 v2.2.2
+	github.com/mattn/go-isatty v0.0.20
+	github.com/minio/minio-go/v7 v7.0.94
+	github.com/mitchellh/go-homedir v1.1.0
+	github.com/nikoksr/notify v1.3.0
 	github.com/pkg/errors v0.9.1
-	github.com/rs/zerolog v1.30.0
+	github.com/pkg/sftp v1.13.9
+	github.com/rs/zerolog v1.34.0
 	github.com/sethvargo/go-password v0.3.1
-	github.com/spf13/cobra v1.7.0
-	golang.org/x/crypto v0.31.0
+	github.com/spf13/cobra v1.9.1
+	golang.org/x/crypto v0.40.0
 	gopkg.in/natefinch/lumberjack.v2 v2.2.1
 	gopkg.in/yaml.v3 v3.0.1
-	maunium.net/go/mautrix v0.16.0
-	mvdan.cc/sh/v3 v3.7.0
+	maunium.net/go/mautrix v0.24.1
+	mvdan.cc/sh/v3 v3.12.0
 )
 
 require (
-	github.com/aws/aws-sdk-go-v2 v1.32.7 // indirect
-	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.7 // indirect
-	github.com/aws/aws-sdk-go-v2/credentials v1.13.26 // indirect
-	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.4 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.26 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.26 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/ini v1.3.35 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.26 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.1 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.4.7 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.7 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.7 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sso v1.12.12 // indirect
-	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.12 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sts v1.19.2 // indirect
-	github.com/aws/smithy-go v1.22.1 // indirect
-	github.com/cenkalti/backoff/v3 v3.2.2 // indirect
+	filippo.io/edwards25519 v1.1.0 // indirect
+	github.com/aws/aws-sdk-go-v2 v1.36.5 // indirect
+	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.11 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.36 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.36 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.36 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.4 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.7.4 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.17 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.17 // indirect
+	github.com/aws/smithy-go v1.22.4 // indirect
+	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/go-jose/go-jose/v3 v3.0.0 // indirect
-	github.com/google/uuid v1.3.1 // indirect
+	github.com/dustin/go-humanize v1.0.1 // indirect
+	github.com/go-ini/ini v1.67.0 // indirect
+	github.com/go-jose/go-jose/v4 v4.1.1 // indirect
+	github.com/go-viper/mapstructure/v2 v2.3.0 // indirect
+	github.com/goccy/go-json v0.10.5 // indirect
 	github.com/hashicorp/errwrap v1.1.0 // indirect
 	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
 	github.com/hashicorp/go-multierror v1.1.1 // indirect
-	github.com/hashicorp/go-retryablehttp v0.7.4 // indirect
+	github.com/hashicorp/go-retryablehttp v0.7.8 // indirect
 	github.com/hashicorp/go-rootcerts v1.0.2 // indirect
-	github.com/hashicorp/go-secure-stdlib/parseutil v0.1.7 // indirect
+	github.com/hashicorp/go-secure-stdlib/parseutil v0.2.0 // indirect
 	github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
-	github.com/hashicorp/go-sockaddr v1.0.5 // indirect
-	github.com/hashicorp/hcl v1.0.0 // indirect
+	github.com/hashicorp/go-sockaddr v1.0.7 // indirect
+	github.com/hashicorp/hcl v1.0.1-vault-7 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
-	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/jordan-wright/email v4.0.1-0.20210109023952-943e75fe5223+incompatible // indirect
-	github.com/knadh/koanf/maps v0.1.1 // indirect
-	github.com/mattn/go-colorable v0.1.13 // indirect
+	github.com/klauspost/compress v1.18.0 // indirect
+	github.com/klauspost/cpuid/v2 v2.3.0 // indirect
+	github.com/knadh/koanf/maps v0.1.2 // indirect
+	github.com/kr/fs v0.1.0 // indirect
+	github.com/mattn/go-colorable v0.1.14 // indirect
+	github.com/minio/crc64nvme v1.0.2 // indirect
+	github.com/minio/md5-simd v1.1.2 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
-	github.com/mitchellh/go-homedir v1.1.0 // indirect
 	github.com/mitchellh/mapstructure v1.5.0 // indirect
 	github.com/mitchellh/reflectwalk v1.0.2 // indirect
+	github.com/pascaldekloe/name v1.0.1 // indirect
+	github.com/philhofer/fwd v1.2.0 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/robfig/cron/v3 v3.0.1 // indirect
-	github.com/rogpeppe/go-internal v1.12.0 // indirect
+	github.com/rs/xid v1.6.0 // indirect
 	github.com/ryanuber/go-glob v1.0.0 // indirect
-	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/spf13/pflag v1.0.6 // indirect
 	github.com/stretchr/objx v0.5.2 // indirect
 	github.com/stretchr/testify v1.10.0 // indirect
-	github.com/tidwall/gjson v1.16.0 // indirect
+	github.com/tidwall/gjson v1.18.0 // indirect
 	github.com/tidwall/match v1.1.1 // indirect
 	github.com/tidwall/pretty v1.2.1 // indirect
 	github.com/tidwall/sjson v1.2.5 // indirect
-	go.mau.fi/util v0.0.0-20230906155759-14bad39a8718 // indirect
+	github.com/tinylib/msgp v1.3.0 // indirect
+	go.mau.fi/util v0.8.8 // indirect
 	go.uber.org/atomic v1.11.0 // indirect
-	golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 // indirect
-	golang.org/x/net v0.33.0 // indirect
-	golang.org/x/sync v0.10.0 // indirect
-	golang.org/x/sys v0.28.0 // indirect
-	golang.org/x/text v0.21.0 // indirect
-	golang.org/x/time v0.3.0 // indirect
-	gopkg.in/yaml.v2 v2.4.0 // indirect
-	maunium.net/go/maulogger/v2 v2.4.1 // indirect
+	go.yaml.in/yaml/v3 v3.0.4 // indirect
+	golang.org/x/exp v0.0.0-20250711185948-6ae5c78190dc // indirect
+	golang.org/x/mod v0.26.0 // indirect
+	golang.org/x/net v0.42.0 // indirect
+	golang.org/x/sync v0.16.0 // indirect
+	golang.org/x/sys v0.34.0 // indirect
+	golang.org/x/text v0.27.0 // indirect
+	golang.org/x/time v0.12.0 // indirect
+	golang.org/x/tools v0.35.0 // indirect
 )