Finished sentry.users_and_orgs state, which allows you to describe Sentry users, organizations, projects, membership, teams, tokens, dsn and other small things with code. Management of most objects is not possible through the Sentry API or Sentry CLI, so you have to use a direct access to the Sentry PostgreSQL database, bypassing the code. That is, it’s an ugly hack, but it works well and allows you to “pour” hundreds of automatically created objects into the Sentry without manual operations (which, in principle, is impossible on such volumes).

Tuning capabilities have been added to the mysql_increment_checker utility: trigger level, exclude databases or tables.

The ability to set redirects and aliases has been added to the alerta state, which is useful for migrations to other domains, etc.

We began to insert the description, environment, and location fields from the accounting wherever possible in monitoring. For example, in Alerta and Sentry.

In our salt repository template, links to our own repositories, which are now in the microdevops-com namespace, have been updated.

Added xinetd component check and mysqld process check to the standard pillar of PXC cluster node check.

Changed the severity of raid array errors from major to critical, which is more accurate.

Added missing launch SystemD unit to heartbeat_mesh.receiver installation state.

Also, for the same state, the possibility of setting the length of the queue and the pause between sendings in Alerta has been added.

Completely rewritten state catch_server_mail.sentry. Now the state knows how to find the necessary project and configure the sending of mail from servers to the Sentry in Multi-Tenant setup.

Thanks to these states, we have released for the customers a system for intercepting mail from servers into the Sentry, which is fully automatically configured. Everything is configured in accounting and automatically gets to all servers in service. This way our and customer engineers can use Sentry to analyze cron errors and similar.

We still have servers on Ubuntu Precise in service, it’s sick, we know, but legacy is legacy :). So we noticed that wget stopped working on these systems because it was compiled with an outdated version of OpenSSL, which is incompatible with modern LE/ZeroSSL certificates. The problem can be solved, for example, by installing statically compiled curl.

We started to migrate the default location of our utilities from /opt/sysadmws to /opt/microdevops – so there will be less legacy at least here :).

Added the interception of errors of the Memory cgroup out of memory type to the monitoring, before that OOM errors were not included in the monitoring when the memory was limited through LXC.

Fixed the bug of launching cmd.run states via salt minion. It turned out that, for some reason, we did not notice it before, that the default shell for cmd.run when working through the minion is sh (not bash). Therefore, all commands containing bashism [[ gave an error through the minion.

Added the ability to distribute ssh keys to the gitlab-runner state.