kwindrem/SetupHelper

The SetupHelper package provides:
    a mechanism to automatically reinstall packages following a Venus OS update
    an automatic update mechanism to keep packages up to date
        from GitHub archives or USB stick
	control of the automatic download and install from the GUI
	add and remove packages from the system
	manual download, install, uninstall from the GUI

    a "blind" install of SetupHelper from SD/USB media

	a blind uninstall mechanism which optionally includes reinstalling Venus OS

	backup and restore SOME settings from com.victronenergy.settings
		this includes custom logos and copying logs to removable media
			SetupHelper
			PackageManager
			gui

SetupHelper is also required for other packages
	SetupHelper includes utilities used by these packages
	and must be installed BEFORE running the other package setup scripts

Blind Install:

By far, the easiest way to install SetupHelper is the "blind install"
	which requires no command-line interaction.

    1) Download venus-data.tgz from the SetupHelper GitHub repo.

	Note: Mac OS and Safari are set by default to unzip packages.
		The Open "safe" files after downloading (bottom of Safair Preferences General)
			must be disabled in order to retain the zip file. 

    2) copy it to the root of a freshly formatted SD card or USB memory stick
    3) place the media in the GX device (Cerbo, CCGX, etc)
    4) reboot the GX device and allow the system to display the GUI

    if you are running Venus OS v2.90 and beyond:
        you should find the Package Manager menu at the bottom of the Settings menu
        you should remove the media at this point
            mechanisms are in place to prevent reinstallation, but removal is still a good idea

    if you are running Venus OS prior to v2.90, perform these additional steps:

    5) reboot the GX device a second time
	6) WHILE the GX device is booting, REMOVE THE MEDIA from the GX device
		VERY IMPORTANT to prevent the next reboot from starting the process all over again
		failure to do so could disable reinstalls following a Venus OS firmware update !!!

    you should find the Package Manager menu at the bottom of the Settings menu


    venus-data.tgz is available here:
        https://github.com/kwindrem/SetupHelper/raw/main/venus-data.tgz


    CAUTION: prior to v2,90, this mechanism overwrites /data/rcS.local !!!!
        If you are using rcS.local to perform boot-time activities,
        /data/rcS.local must be recreated following this "blind" install
        Note that SetupHelper also uses /data/rcS.local for
        reinstallation following a firmware update so use caution in
        recreating rcS.local.
        

Another way to install SetupHelper is to use the following from the command line of the GX device:

	wget -qO - https://github.com/kwindrem/SetupHelper/archive/latest.tar.gz | tar -xzf - -C /data
	mv /data/SetupHelper-latest /data/SetupHelper
	/data/SetupHelper/setup

Once SetupHelper is installed, updates to it and other packages can be performed through the GUI
using the PackageManager menus.

CAUTION:
	Package Manager allows uninstalling SetupHelper.
	This can not be undone since the menus to control Package Manager will go away
	You would need to use the Blind Install or run /data/SetupHelper/setup again to reinstall SetupHelper
	
	Note that removal does not actually remove the package so other setup scripts
	will continue to function.


ssh access:

Setting up ssh access with ssh keys is highly recommended for any system,
but especially when installing third party extensions to Venus OS.
Attaching a serial terminal for direct console access is another option,
especially if you don't have a network setup.

The following document describes ssh access and also serial terminal connections on Cerbo GX:
	https://www.victronenergy.com/live/ccgx:root_access


System Recovery:

It is highly unlikely, but some users have reported a package install leaving their system unresponsive
or with a nonfuncitonal GUI (white screen). In this case, your options depend on the current state of the system.

First, (as always) reboot. This may clear the problem.

Second, if you have a functioning GUI (eitehr locally or via remote console, see if you can access the PackageManager menu.
	If so, you can remove pacakges one at a time from there.
	If you find an offeding package, post an issue to the GitHub repo for that package and include:
		Platform (Cerbo, CCGX, Raspberry PI, etc)
		Venus OS firmware version
		Run a Settings backup and post the logs.zip file on the removble media.
	Remove SetupHelper last since once you do, you loose the PackageManager menus!

Third, if you have terminal or ssh access, try running the package setup scripts to uninstall packages one at a time.

Fourth, try booting to the previous Venus OS version (in Stored backup firmware)
	Then perform a fresh Online firmware update to the latest version or use the .swu update via removable media.
	These procedures are documented: https://www.victronenergy.com/media/pg/Cerbo_GX/en/gx---how-to-update-firmware.html

	Use the Settings / Firmware / Stored backup formware menu if you have GUI access.

	If you don't have GUI access, you can also switch to the backup version from the command line:
		/opt/victronenergy/swupdate-scripts/set-version.sh 2

Fifth, perform the Blind uninstall procedure below.

Finally, if you are running on a Raspberry PI, you can reimage the system SD card.
	Note: this will wipe out all settings and you'll need to reconfigure the GX device from scratch.

The Victron "restore factory default" procedure can be used to will wipe out all settings.
	You'll need to reconfigure the GX device from scratch.
	However, it will NOT replace the operating system and Victron application, nor will it uninstall any packages.
	You will most likely be locked out of ssh access since log-in information and ssh keys
	are stored in the /data partition which is completey erased by this procedure.
	For this reason, I do not recommend using this as part of your attempt to recover a system with no GUI.


Blind UNINSTALL:

A blind uninstall mechanism is provided to recover a system with an unresponsive GUI (white screen)
	or no ssh/terminal access.
This will run all package setup scripts to uninstall that package from system files.

In addition to uninstalling all packages, the blind uninstall can optionally reinstall VenusOS.
To do so, include a .swu file for the platform and desired firmware version on the removable media containing
the blind uninstall venus-data.tar.gz file.

The archive for this is named venus-data.UninstallPackages.tar.gz.

	1) Copy venus-data.UninstallPackages.tar.gz to a USB memory stick or SD card
	2) Rename the copy to venus-data.tar.gz
	3) Plug the removable media into the GX device
	4) Reboot, wait 2 minutes and reboot a second time
	5) when the system automatically reboots after the second manual one, remove the media
		You should eventually see the GUI on the local display if there is one
		or be able to connect via remote console.

	Cauton: Removing media or power cycling the GX device during the uninstall,
		especially if reinstalling firmware could render the system unresponsive!
		Wait to see the GUI before removing media or power cycling.

	Note that a firmware update can take several minutes to complete but will eventually reboot.

	When the blind uninstall finishes, venus-data-tar.gz file on the removable media
	is renamed to venus-data.UninstallPackages.tar.gz so that the blind install will run only once.
	This renaming is necessary to prevent a loop where the system uninstalls and reboots.

Description:

There are two parts to SetupHelper:
1) Package Manager, controls automatic and manual updates to packages

2) Utilities used by other packages' setup scripts.
	These resources simplify the task of writing install/uninstall scripts
	and may be of help to others writing packages of their own.

The latter is of concern only to those writing new Venus modifications
or modifying an existing setup script.
These are described in detail later in the SetupHelperDescription document.

Package Manager:

The Package Manager includes a set of menus on the GX device menu system
	that allows the user to view package versions,
	control automatic package updates
	and manually install, uninstall, add and remove packages.
	
	A python program runs in the background (as a service) to do the actual
		work and to interface with the menus.

	Package Manager menu:

		The first line of this menu provides status for Package Manager,
			indicating what it is currently doing

		Automatic GitHub downloads controls if packages are automatically downloaded
				from GitHub. This occurs if a newer version is available.
			Modes:
				On checks GitHub for a package that is newer than what is stored on the system 
				If multiple downloads are detected, PackageManager waits after a download has occurred before checking another
					The wait time is 10 seconds for the first pass through the active packages.
					After one pass through all packages, the downloads spaced 10 MINUTES apart
				Once checks GitHub for a package, then downloads are turned off
				Off disables GitHub downloads

			Note that when switching on automatic downloads, PackageManager first refreshes the version information from GitHub

		Auto install packages:
			Controls whether new versions of a package are automatically installed.
			Some users may wish to have the system automatically download new updates,
			but install them manually.
			In this case, automatic GitHub downloads may be turned on and Auto install packages turned off
			
			Auto install packages also influences whether packages transferred from SD/USB media 
				are automatically installed or just transferred to local storage

		Active packages:
			displays all active packages
			
			Version information is displayed for each package:
				Git Hub shows the version found on GitHub
				Stored shows the version stored on the GX device
				Installed shows the version actually installed and running

			Tapping on one of the entries leads to the Package editor menu

			Git Hub versions are not refreshed if autodownloads are turned off
				to eliminate internet bandwidth from periodic checks.
			If auto downloads are off:
				Git Hub versions are refreshed once when entering the Package Versions menu - one package every 2 seconds
				The Git Hub version for the package shown in the Package editor menu will be refreshed once
			If auto downloads are on, one Git Hub version is refreshed every 10 minutes
				after an initial pass of one package every 2 seconds.

		Inactive packages:
			displays all INACTIVE packages
				I.e., default packages not yet activated or manually removed
			The first entry is always "new" and allows the operator to enter package name, GitHub user and branch/tag
				from scratch
			Additional lines (if any) are default packages (from the defaultPackageList file)
			If a package is already added to the version list, it will not appear in the Add Package list
			
			Tapping on one of the entries leads to the Add Package menu
		
		Backup & Restore settings:
			saves settings to the settingsBackup file on removable SD/USB media
				or to local media (/data/settingsBackup)
			restores from same 
			/data/SetupHelper/settingsList is a complete list of settings saved to settingsBackup
				GuiMods
				SetupHelper
				ShutdownManager
				SOME Victron stock settings in the following sections
					Alarms
					CGwacs
					DigitalInputs
					Generators
					Gui
					Pump
					Relay
					System
					SystemSetup
					Vrmlogger
			Additionally, backup and restore of the following to/from removable media only
				Any logo files in /data/themes/overlay
				Setup script options in /data/setupOptions
			All logs stored in /data/log are written to logs.zip on the removable media only
				as part of the backup
			The parameters must exist to be saved
			The parameters will be created and set to the backed up value during a restore

			Note: Victron is working on a more comprehensive mechanism but is not working reliably yet
				This part of PackageManager is temporary and will be removed when the Victron functionality is working

		SETTINGS_AUTO_RESTORE:
			An automatic settings restore will be performed when PackageManager if the file named
				SETTINGS_AUTO_RESTORE is detected in the root of removable media

			CAUTION: LEAVING THIS FLASH DRIVE IN THE SYSTEM WILL TRIGGER A SETTINGS RESTORE WITH EVERY BOOT
					YOU MUST REMOVE THE FLASH DRIVE AFTER AUTO RESTORE

		microSD / USB:
			is a duplicate of the menu item in VRM online portal
			it can be used to eject ALL removable media before physically removing them
			NOTE: all media is ejected, so if you are using one for VRM logging,
				you'll need to reboot or unplug, then replug that device.

		AUTO_EJECT:
			If this flag file is found on any removable media, ALL removable media is ejected
				after the media is scanned AND if any transferrers were performed:
					transfer a package from the media (as an archive file) to /data
					restore backup settings
				this will NOT occur for
					manual settings backup or restore

			Removable media can be corrupted if removed while the VRM logger is still writing to it
				so the drive must be ejected to prevent corruption
			A manual eject button is included in the PackageManager menu
			This automatic eject is intended for unattended deployment and will only occur if the AUTO_EJECT file exists
			Unfortunately, the eject mechanism ejects all removable media, not just a specific one.
			The VRM logger automatically uses the first removable media found so there is no control over it,


		AUTO_INSTALL_PACKAGES:
			If the file AUTO_INSTALL_PACKAGES is found on removable media, packages will be installed
			even if the Auto Install menu option is turned off. This is generally used only for system deployment (see below).

		AUTO_UNINSTALL_PACKAGES:
			As above, but will uninstall all packages found in /data
			This is useful if you have not command line access and end up with a GUI that is unresponsive
				or just to clean up a system, returning it (almost) to factory defaults
			This flag file oerrides AUTO_INSTALL_PACKAGES if both are present

			The system is rebooted after the uninstall all just to be sure there's nothing left behind.

			Note: this uses PackageManager, so if SetupHelper isn't installed it will do nothing

		AUTO_INSTALL:
			If the file AUTO_INSTALL is present in a package directory, the package will be installed
				as if the auto install option is set in the PackageManager menu
			Version checks are still performed and DO_NOT_INSTALL is honored

		ONE_TIME_INSTALL
			If the file ONE_TIME_INSTALL is present in a package directory, the package is automatically installed
				even if automatic installs are diabled and the DO_NOT_INSTALL flag is set
			This ONE_TIME_INSTALL is removed when the install is performed
				to prevent repeated installs
			Packages may be deployed with this flag set to insure it is installed
				when a new version is transferred from removable media or downloaded from GitHub

		INITIALZE_PACKAGE_MANAGER and menu item:
			If present, the PackageManager's persistent storage (dbus Settings parameters) are initialized
			and PackageManager restarted
			On restart, PackageManager will rebuild the dbus Settings from packages found in /data
			Only custom Git Hub user and branch information is lost.
		

		Package editor:
			This menu facilitates:
				manual install, uninstall, package add and package remove
				changing GitHub access information for each package

			Normally, you would want to download the latest released version
				but you may also wish to try out a beta version or revert to a previous one.
				Once the GitHub branch is changed, PackageManager will update the GitHub version
					and, if enabled, download this alternate version.

			Remove Package
				Packages that are of no interest to you may be removed from Package Manager.
				Removed packages will no longer appear in the version list or be accessible from the Package Editor menu.
				But you can add the package back in manually.

		Add package menu:
			Allows the package name, GitHub user and GitHub branch or tag to be entered.
			
			Pressing Proceed initiates the package add.
			The package will be added to the package list (and appear in the Package versions menu)
			only IF the name is unique
			
			Pressing Cancel returns to the default package list

			All three fields must be set appropriately or you'll see -- for the GitHub version

		
		Package Manager does not allow removing packages unless they are uninstalled first.
		
		Package Manager DOES permit uninstalling SetupHelper,
			however this will remove the Package Manager itself.
			Once removed, the Blind Install mechanism will be needed again !!

USB/SD updates:

	When the GX device is not connected to the internet, a USB flash drive or microSD card provides an install/upgrade path.

	To use the USB update process
		Navigate to the GitHub, repo, click on tags and select the appropriate branch or specific version tag. 
		Choose the .tar.gz download link.
		(Do not download from the Code tab or download the .zip file. These won't work.)
		Copy the archive file to a USB memory stick or microSD card.
		Do NOT unpack the archive
		Repeat this for all packages you wish to install.
		(These can all be placed on the same media along with the SetupHelper venus-data.tgz file)
		Insert the stick in the GX device. 
		If SetupHelper has not yet been installed, follow the Blind Install process above.
		Once Package Manager is running, it will transfer and unpack the archive files
			and update the package list with the new packages. 
		If Auto install packages is turned on, the packages will then be installed
		
		NOTE: no version checks are made for packages found on SD/USB media!
			Package Manager is quite content to transfer and install an older version!
			So make sure you have the latest version especially if your GX device does not have internet access.



System automatic configuration and package installation:
	It is possible to use SetupHelper to set up a new system based on a template saved from a working system.
	Setup the working system the way you want the new system to behave including custom icons,
	then perform a Settings backup.
	Remove the flash drive from the GX device and plug into a computer that has internet access.
	Copy venus-data.tgz from the SetupHelper GitHub repo to the same flash drive.
	If you wish packages to also be installed, copy the package -latest.tgz file from those repos as well.
	Create SETTINGS_AUTO_RESTORE on the flash drive (contents don't matter - file may be empty).
	Create AUTO_INSTALL_PACKAGES on the flash drive as well.
	Place the flash drive into the GX device to be configured and reboot (once for v2.90 or twice for prior versions).
	REMOVE THE FLASH DRIVE after you have verified that all packages have been installed (check Active packages in PackageManager).


If you are interested in the inner workings of Setup Manager and Package Manager or wish to create a package that can be managed by PackageManager, the document "Package development guidelines"  contains assitional information. Feel free to contact me through the issues part of SetupHelper on GitHub.