* Getting environment variables and passwords from Vault (not tested
yet)
* Vault configuration to config (not tested yet)
* Ability to run scripts from file on local machine on the remote host
* Ability to get ouput in the notification of a list for individual
commands or all commands
* Make SSH connections close after all commands have been run; reuse
previous connections if needed
This commit is contained in:
2023-07-01 21:46:54 -05:00
parent 5e7c52997c
commit 4b382bddd9
831 changed files with 83782 additions and 107 deletions

26
docs/content/_index.md Normal file
View File

@ -0,0 +1,26 @@
+++
archetype = "home"
title = "Backy"
+++
Backy is a tool for automating data backup and remote command execution. It can work over SSH, and provides completion and failure notifications, error reporting, and more.
Why the name Backy? Because I wanted an app for backups.
{{% notice tip %}}
Feel free to open a [PR](https://git.andrewnw.xyz/CyberShell/backy/pulls), raise an [issue](https://git.andrewnw.xyz/CyberShell/backy/issues "Open a Gitea Issue")(s), or request new feature(s).
{{% /notice %}}
## Features
- Allows easy configuration of executable commands
- Allows for commands to be run on many hosts over SSH
- Commands can be grouped in list to run in specific order
- Notifications on completion and failure
- Run in cron mode
- For any command, especially backup commands

101
docs/content/cli/_index.md Normal file
View File

@ -0,0 +1,101 @@
---
title: CLI
weight: 4
---
This page lists documentation for the CLI.
## Backy
```
Backy is a command-line application useful for configuring backups, or any commands run in sequence.
Usage:
backy [command]
Available Commands:
backup Runs commands defined in config file.
completion Generate the autocompletion script for the specified shell
cron Starts a scheduler that runs lists defined in config file.
exec Runs commands defined in config file in order given.
help Help about any command
version Prints the version and exits
Flags:
-f, --config string config file to read from
-h, --help help for backy
-v, --verbose Sets verbose level
Use "backy [command] --help" for more information about a command.
```
# Subcommands
## backup
```
Backup executes commands defined in config file.
Use the --lists or -l flag to execute the specified lists. If not flag is not given, all lists will be executed.
Usage:
backy backup [--lists=list1,list2,... | -l list1, list2,...] [flags]
Flags:
-h, --help help for backup
-l, --lists strings Accepts comma-separated names of command lists to execute.
Global Flags:
-f, --config string config file to read from
-v, --verbose Sets verbose level
```
## cron
```
Cron starts a scheduler that executes command lists at the time defined in config file.
Usage:
backy cron [flags]
Flags:
-h, --help help for cron
Global Flags:
-f, --config string config file to read from
-v, --verbose Sets verbose level
```
## exec
```
Exec executes commands defined in config file in order given.
Usage:
backy exec command ... [flags]
Flags:
-h, --help help for exec
Global Flags:
-f, --config string config file to read from
-v, --verbose Sets verbose level
```
## version
```
Prints the version and exits. No arguments just prints the version number only.
Usage:
backy version [flags]
Flags:
-h, --help help for version
-n, --num Output the version number only. (default true)
-V, --vpre Output the version with v prefixed.
Global Flags:
-f, --config string config file to read from
-v, --verbose Sets verbose level
```

View File

@ -0,0 +1,22 @@
---
title: "Configuring Backy"
weight: 3
description: >
This page tells you how to configure Backy.
---
This is the section on the config file.
To use a specific file:
```backy [command] -f /path/to/file```
If you leave the config path blank, the following paths will be searched in order:
- `./backy.yml`
- `./backy.yaml`
- `~/.config/backy.yml`
- `~/.config/backy.yaml`
Create a file at `~/.config/backy.yml`.
See the rest of the documentation in this section to configure it.

View File

@ -0,0 +1,85 @@
---
title: "Command Lists"
weight: 2
description: >
This page tells you how to get started with Backy.
---
Command lists are for executing commands in sequence and getting notifications from them.
The top-level object key can be anything you want but not the same as another.
```yaml
test2:
name: test2
order:
- test
- test2
notifications:
- prod-email
- matrix
cron: "0 * * * * *"
```
| key | description | type | required
| --- | --- | --- | --- |
| `order` | Defines the sequence of commands to execute | `[]string` | yes |
| `getOutput` | Command(s) output is in the notification(s) | `bool` | no |
| `notifications` | The notification IDs to use on success and failure | `[]string` | no |
| `name` | Optional name of the list | `string` | no |
| `cron` | Time at which to schedule the list. | `string` | no |
### Order
The order is an array of commands to execute in order. Each command must be defined.
```yaml
order:
- cmd-1
- cmd-2
```
### getOutput
Get command output when a notification is sent.
Is not required. Can be `true` or `false`.
### Notifications
An array of notification IDs to use on success and failure. Must match any of the `notifications` object map keys.
### Name
Name is optional for logging. If name is not defined, name will be the object's map key.
### Cron mode
Backy also has a cron mode, so one can run `backy cron` and start a process that schedules jobs to run at times defined in the configuration file.
Adding `cron: 0 0 1 * * *` to a `cmd-configs` object will schedule the list at 1 in the morning. See [https://crontab.guru/](https://crontab.guru/) for reference.
{{% notice tip %}}
Note: Backy uses the second field of cron, so add anything except * to the beginning of a regular cron expression.
{{% /notice %}}
```yaml
cmd-configs:
  cmds-to-run: # this can be any name you want
    # all commands have to be defined
    order:
      - stop-docker-container
      - backup-docker-container-script
      - shell-cmd
      - hostname
    notifications:
      - matrix
    name: backup-some-server
    cron: "0 0 1 * * *"
  hostname:
    name: hostname
    order:
      - hostname
    notifications:
    - prod-email
```

View File

@ -0,0 +1,97 @@
---
title: "Commands"
weight: 1
---
The yaml top-level map can be any string.
The top-level name must be unique.
```yaml
commands:
stop-docker-container:
cmd: docker
Args:
- compose
- -f /some/path/to/docker-compose.yaml
- down
# if host is not defined, command will be run locally
host: some-host
backup-docker-container-script:
cmd: /path/to/script/on/some-host
# The host has to be defined in either the config file or the SSH Config files
host: some-host
environment:
- FOO=BAR
- APP=$VAR
```
Values available for this section:
| name | description | type | required
| --- | --- | --- | --- |
| `cmd` | Defines the command to execute | `string` | yes |
| `args` | Defines the arguments to the command | `[]string` | no |
| `environment` | Defines evironment variables for the command | `[]string` | no |
| `getOutput` | Command(s) output is in the notification(s) | `bool` | no |
| `host` | If not specified, the command will execute locally. | `string` | no |
| `shell` | Only applicable when host is not specified | `string` | no |
#### cmd
cmd must be a valid command or script to execute.
#### args
args must be arguments to cmd as they would be on the command-line:
```sh
cmd [arg1 arg2 ...]
```
Define them in an array:
```yaml
args:
- arg1
- arg2
- arg3
```
### getOutput
Get command output when a notification is sent.
Is not required. Can be `true` or `false`.
#### host
{{% notice info %}}
If any `host` is not defined or left blank, the command will run on the local machine.
{{% /notice %}}
Host may or may not be defined in the `hosts` section.
{{% notice info %}}
If any `host` from the commands section does not match any object in the `hosts` section, the `Host` is assumed to be this value. This value will be used to search in the default SSH config files.
For example, say that I have a host defined in my SSH config with the `Host` defined as `web-prod`.
If I assign a value to host as `host: web-prod` and don't specify this value in the `hosts` object, web-prod will be used as the `Host` in searching the SSH config files.
{{% /notice %}}
### shell
If shell is defined and host is NOT defined, the command will run in the specified shell.
Make sure to escape any shell input.
### environment
The environment variables support expansion:
- using escaped values `$VAR` or `${VAR}`
For now the variables have to be defined in an `.env` file in the same directory as the config file.
If using it with host specified, the SSH server has to be configured to accept those env variables.
If the command is run locally, the OS's environment is added.

View File

@ -0,0 +1,74 @@
---
title: "Notifications"
weight: 3
description: >
This page tells you how to get set up Backy notifications.
---
Notifications can be sent on command list completion and failure.
The supported platforms for notifications are email (SMTP) and [Matrix](https://matrix.org/).
Notifications are defined by type. The top-level object will be the id, and the `type` is required.
```yaml
notifications:
prod-email:
type: mail
host: yourhost.tld
port: 587
senderaddress: email@domain.tld
to:
- admin@domain.tld
username: smtp-username@domain.tld
password: your-password-here
matrix:
type: matrix
home-server: your-home-server.tld
room-id: room-id
access-token: your-access-token
user-id: your-user-id
```
Types recognized are `type: mail` and `type: matrix`
The type's object and its keys are listed below.
### type: mail
| key | description | type
| --- | --- | ---
| `host` | Specifies the SMTP host to connect to | `string`
| `port` | Specifies the SMTP port | `uint16`
| `senderaddress` | Address from which to send mail | `string`
| `to` | Recipients to send emails to | `[]string`
| `username` | SMTP username | `string`
| `password` | SMTP password | `string`
### type: matrix
| key | description | type
| --- | --- | ---
| `home-server` | Specifies the Matrix server connect to | `string`
| `room-id` | Specifies the room ID of the room to send messages to | `string`
| `access-token` | Matrix access token | `string`
| `user-id` | Matrix user ID | `string`
To get your access token (assumes you are using [Element](https://element.io/)) :
1. Log in to the account you want to get the access token for. Click on the name in the top left corner, then "Settings".
2. Click the "Help & About" tab (left side of the dialog).
3. Scroll to the bottom and click on `<click to reveal>` part of Access Token.
4. Copy your access token to a safe place.
To get the room ID:
1. On Element or a similar client, navigate to the room.
2. Navigate to the settings from the top menu.
3. Click on Advanced, the room ID is there.
{{% notice info %}}
Make sure to quote the room ID, as [YAML spec defines tags using `!`](https://yaml.org/spec/1.2.2/#3212-tags).
{{% /notice %}}

View File

@ -0,0 +1,26 @@
---
title: "Vault"
weight: 4
---
[Vault](https://www.vaultproject.io/) is a tool for storing secrets and other data securely.
Vault config can be used by prefixing `vault:` in front of a password or ENV var.
This is the object in the config file:
```yaml
vault:
token: hvs.tXqcASvTP8wg92f7riyvGyuf
address: http://127.0.0.1:8200
enabled: false
keys:
- name: mongourl
mountpath: secret
path: mongo/url
type: # KVv1 or KVv2
- name:
path:
type:
mountpath:
```

View File

@ -0,0 +1,10 @@
---
title: "Getting started"
weight: 2
description: >
This page tells you how to get started with Backy.
---
If you have not installed Backy, [see the install documentation](install).
If you need to configure it, [see the config page](config).

View File

@ -0,0 +1,157 @@
---
title: "Config File Definitions"
description: >
This page tells you how to configure Backy.
---
### Commands
The commands section is for defining commands. These can be run with or without a shell and on a host or locally.
See the [commands documentation](/config/commands) for further information.
```yaml
commands:
  stop-docker-container:
output: true # Optional and only when run in list and notifications are sent
    cmd: docker
    args:
      - compose
      - -f /some/path/to/docker-compose.yaml
      - down
    # if host is not defined, cmd will be run locally
    host: some-host 
  backup-docker-container-script:
    cmd: /path/to/script
    # The host has to be defined in the config file
    host: some-host
    environment:
      - FOO=BAR
    - APP=$VAR # defined in .env file in config directory
  shell-cmd:
    cmd: rsync
    shell: bash
    args:
      - -av
- some-host:/path/to/data
- ~/Docker/Backups/docker-data
script:
type: scriptFile # run a local script on a remote host
cmd: path/to/your/script.sh
host: some-host
  hostname:
    cmd: hostname
```
### Lists
To execute groups of commands in sequence, use a list configuration.
```yaml
cmd-configs:
cmds-to-run: # this can be any name you want
# all commands have to be defined in the commands section
order:
- stop-docker-container
- backup-docker-container-script
- shell-cmd
- hostname
getOutput: true # Optional and only for when notifications are sent
notifications:
- matrix
name: backup-some-server
hostname:
name: hostname
order:
- hostname
notifications:
- prod-email
```
### Hosts
The hosts object may or may not be defined.
{{% notice info %}}
If any `host` from a commands object does not match any `host` object, the needed values will be checked in the default SSH config files.
{{% /notice %}}
```yaml
hosts:
# any needed ssh_config(5) keys/values not listed here will be looked up in the config file or the default config file
some-host:
hostname: some-hostname
config: ~/.ssh/config
user: user
privatekeypath: /path/to/private/key
port: 22
# can also be env:VAR or the password itself
password: file:/path/to/file
# can also be env:VAR or the password itself
privatekeypassword: file:/path/to/file
# only one is supported for now
proxyjump: some-proxy-host
```
### Notifications
The notifications object can have two forms.
For more, [see the notification object documentation](/config/notifications). The top-level map key is id that has to be referenced by the `cmd-configs` key `notifications`.
```yaml
notifications:
prod-email:
type: mail
host: yourhost.tld
port: 587
senderAddress: email@domain.tld
recipients:
- admin@domain.tld
username: smtp-username@domain.tld
password: your-password-here
matrix:
type: matrix
home-server: your-home-server.tld
room-id: room-id
access-token: your-access-token
user-id: your-user-id
```
### Logging
cmd-std-out controls whether commands output is echoed to StdOut.
If logfile is not defined, the log file will be written to the config directory in the file `backy.log`.
`console-disabled` controls whether the logging messages are echoed to StdOut. Default is false.
`verbose` basically does nothing as all necessary info is already output.
```yaml
logging:
verbose: false
file: path/to/log/file.log
console-disabled: false
cmd-std-out: false
```
### Vault
[Vault](https://www.vaultproject.io/) can be used to get some configuration values and ENV variables securely.
```
vault:
token: hvs.tXqcASvTP8wg92f7riyvGyuf
address: http://127.0.0.1:8200
enabled: false
keys:
- name: mongourl
mountpath: secret
path: mongo/url
type: # KVv1 or KVv2
- name:
path:
type:
mountpath:
```

View File

@ -0,0 +1,21 @@
---
title: "Install Backy"
weight: 1
description: >
This page tells you how to install Backy.
---
Binaries are available from the [release page](https://git.andrewnw.xyz/CyberShell/backy/releases). Make sure to get the correct version for your system, which supports x86_64, ARM64, and i386.
### Source Install
You can install from source. You will need [Go installed](https://go.dev/doc/install).
Then run:
```bash
go install git.andrewnw.xyz/CyberShell/backy@master
```
Once set, jump over to the [config docs](/getting-started/config) and start configuring your file.

View File

@ -0,0 +1,10 @@
---
title: "Repositories"
weight: 5
---
The repo mirrors are:
* [https://git.andrewnw.xyz/CyberShell/backy](https://git.andrewnw.xyz/CyberShell/backy)
* [https://git.vern.cc/cybershell/backy](https://git.vern.cc/cybershell/backy)
* [https://github.com/CybersShell/backy](https://github.com/CybersShell/backy)