Moved all of our active GitHub repositories from the sysadmws organization to the microdevops-com organization (the “microdevops” organization on GitHub and GitLab is squatted 🙂 ). Also, the sysadmws-utils and sysadmws-formula repositories have been renamed to microdevops-utils and microdevops-formula. Accordingly, the links on our site have been changed and now, from the point of view of corporate identity, everything looks more consistent. GitHub made redirects from old links automatically.

A little later, it is planned to rename the deb package sysadmws-utils-v1 to microdevops-utils.

Work has begun on describing PMM dashboards and alerts with code.

A footer and a sidebar with the latest publications have been added to the design of the microdevops.com site.

By the way, News on the site has caught up with the internal Changelog and now the publication of news in both of those places will take place systematically: once a month.

Finally we rewrote rsnapshot_backup/check_backup in python and integrated mysqlsh/xtrabackup dump relevance checks into it.

Converted several clients to use the new version of the rsnapshot_backup utility in python, and mysqlsh backups.

Conducted training on restoring backups created by mysqlsh. The use of such backups has peculiarities.

Added the ability to accounting: a) search and display a list of issues or merge requests in client projects in GitLab directly from the command line; b) quickly add comments and time tracking to them also directly from the console. This will allow you to track time very quickly, which is especially relevant for tasks with a large number of comments that load very slowly in the browser.

We encountered a strange behavior of the ELK/APM library at the client, when the ELK cluster was unavailable, the library led to code hangs. It’s strange when a tool that was created to diagnose code problems ends up causing problems itself.

At the same time, in the process of transferring the same client to tracking Performance Metrics in Sentry, it was verified that Sentry library has no effect of such kind.

We finally switched to using the vaultwarden password instead of sysPass. After activating the Internet in the vaultwarden container for the purpose of sending mail, it unexpectedly turned out that the system itself adds icons to records using url – nice 🙂 . Overall, vaultwarden turned out to be very pleasant to use.

Fixed the problem of errors that occurred when some microdevops-utils were launched in several instances in parallel and tried to rotate the log file at the same time.

Wrote a separate state of the ssh_keys in formula for managing ssh keys on servers.

Added pxc.roles to the pxc formula state: a rewritten version of the grant distribution code, where many old problems are solved.

Several standard disk_alert pillars have been added to the formula to tune the standard (90/95) thresholds for triggering disk usage monitoring.

To rsnapshot_backup (python version, of course) added the possibility to rotate backups more gently, with the help of ionice and nice. Added the possibility of retries to backup items.

Switched to GitLab 15.3 in all instances that we support, including ours.

The utility for checking the state of mysql replication has been rewritten. The new python version of the utility correctly monitors the replication of replicas with several sources at the same time.

Made standard OOM and Zombie monitoring on servers.

Another one has been added to the app states php-fpm, python, static: ruby. Now we can standardly describe the deployment of ruby ​​applications.