More and more automation is moving from salt to python in the accounting project, such as below.

• ./projects.py –setup-projects-for-client creates gitlab projects for the client.
• ./projects.py –template-salt-project-for-client templates similar parts in client salt projects.

And moreover:

• A public project salt-project-template is created with the template for the client’s salt project.
• A public project gitlab-server-job is created – used for scheduling pipelines with salt tasks in gitlab.
• The fake client Xyz is defined – a blank for experiments, but with live salt masters, a project, pipelines, billing, two test virtual machines.
• Salt switched to a multi-master mode with a failover. Now each client has 2 salt masters. Salt minion connects to one of them randomly.
• Pipelines are written in such a way that they themselves find the current active master for the minion and launch the states there.
• The jobs.py scheduler has been written, which is being run every 10 minutes in the accounting pipelines. The scheduler takes job descriptions from accounting.yaml.
• Jobs can be overridden at the client or client server level.
• The rsnapshot_backup job has its own different pipeline with more steps.
• Job rsnapshot_backup can run the backup itself both via SSH (by default) and via salt. Using ssh, we get rid of the layer in the form of a salt with its timeout, and in general it is more reliable and the log in the job is immediately drawn.
• To identify the pipeline, a git tag is created and now in the pipeline errors you can see on which server, when and which command tried to run.
• Now jobs, including backups, are launched per server – that is, now a separate pipeline is created for each job and server in the client’s salt project.
• Now it has become convenient to use backup coverage – so we immediately set it up so that the pipelines are immediately clean and backup with a full set of checks.
• The code of the submodules is cached in the salt code so as not to download the submodule from external sources every time you execute each job in the pipeline.
• When setting up minions, settings are created so that salt masters can connect as root to minions bypassing the salt – needed to perform backups via SSH.
• Since there are 2 salt masters, it is inconvenient to accept keys and the failover scheme provides keys for signing salt master traffic, all keys are pre-generated and placed into client yamls. No keys are generated by minions or masters themselves. Therefore, when reloading, etc., everything remains as it was in terms of pki.
• For servers in the client yaml, we provided the location key.
• With .gitlab-server-job/pipeline_salt_cmd.sh you can create a salt pipeline in the client project, so you don’t have to search for the current active salt master and run commands there.
• Using this approach, we come to the conclusion that new engineers will not need to go to salt servers, and maybe client servers too. You only need the rights to create pipelines in the desired project. That is, access control for clients goes to the gitlab level. And all such actions of admins end up in the job logs.
• For data sharing between clients, for example, when one client backs up another client’s data, we use the git submodules. The serving client includes the serviced client’s turnip.
• In general, client infrastructures become completely independent. Gitlab remains the only point of intersection between clients.

Not implemented yet:

• Generation of a list of servers with the location on the wiki of the client admin project.