Menu

AdGuard VPN May 8th incident report: Post Mortem

In recent days, a significant portion of AdGuard VPN users have encountered problems with their apps. The problems affected about 25% of AdGuard VPN for Windows users and about 10% of AdGuard VPN users of other platforms. By the time of this publication, all issues had been resolved. Make sure your AdGuard VPN for Windows app is updated to the latest version. If you are a user of a different AdGuard VPN app or an AdGuard VPN browser extension, you don't need to do anything, it should be working normally by now.

We apologize for the inconvenience caused and hope that this incident will not affect your overall satisfaction with AdGuard VPN. This was a one-time occurrence that will not happen again.

Below is the timeline of events that led to the incident, and then we will list the measures we are taking to avoid anything like this in the future.

Events timeline

May 8

It all started with the release of AdGuard VPN v2.3.0 for Windows on May 8th. The mandatory testing before the release went smoothly, but we overlooked one hard-to-notice problem: under certain circumstances the app began endlessly trying to reach the backend server to check the VPN servers' connectivity.

May 9

As a result of this mistake, the load on the backend server had significantly risen. However, despite seeing that something was off, we couldn't reproduce the problem and drew the conclusion that the high load was the consequence of some of the changes we made to the app's operating principles.

May 10

We started receiving first complaints about constant connection drops and "Can't connect to server" errors. According to the user reports, restarting the app helped in the short term. Also, at this time a task was created on GitHub describing this problem.

May 13

The number of user complaints had reached a level high enough to indicate the critical nature of the situation. We still couldn't reproduce the problem on our end, which greatly complicated and delayed the process of finding the root of it. We had several hypotheses, but to test them we needed the help of affected users — huge thanks to everyone who sacrificed their time to help us. But despite all the efforts we weren't able to find a solution.

May 14

According to statistical data and our monitoring system, by 09:00 UTC the amount of requests to the backend server per second had drastically increased compared to the numbers registered on 9th-10th of May.

Between 09:00 UTC and 16:00 UTC we took measures aimed at minimizing the spreading of the "bad" update among the users:

  • Blocked the option to update to v2.3.0 from the app
  • Removed the option to download the AdGuard VPN for Windows app from the official website
  • Everyone who had contacted the tech support was recommended to roll back to the previous version until the problem is fixed

By 15:00 UTC the amount of requests to the backend reached almost 1000 times the norm. To mitigate the negative impact on the AdGuard's infrastructure, at around 13:30 UTC we decided to greatly reduce the server rate limit. Unfortunately, the side effect of this measure was the increase in the number of problems some users encountered when accessing the authenticaion server. This affected not only Windows users, but also users of AdGuard VPN for other platforms.

By 18:00 UTC we managed to identify the cause of the problem, and by 23:30 UTC we were ready with a v2.3.1 hotfix.

By 22:00 UTC the rate limit was returned to the regular value for all apps besides AdGuard VPN for Windows, which helped to fix the problem for the majority of affected users.

May 15

At around 00:45 UTC the v2.3.1 update was released and became available to users.

Conclusions and prevention measures

We are currently in the process of making necessary changes to our quality assurance procedures for product updates, with an emphasis on the apps' network activity monitoring (and their interaction with the authentication server in particular). We are going to optimize the entire internal infrastructure monitoring system as well.

Apart from that, moving forward we will be more quick and transparent in informing our users about any critical issues that could lead to major negative consequences.

We would like to once again apologize to all our users who were affected by this incident and express our gratitude to everyone who provided feedback and helped us resolve this difficult situation.

Liked this post?

AdGuard VPN
for Windows

Use any browser or app and never worry about your anonymity again. The entire world is at your fingertips with AdGuard VPN.
Learn more
Download
By downloading the program you accept the terms of the License agreement

AdGuard VPN
for Mac

In just two clicks, select a city from anywhere in the world — we have 70+ locations — and your data is invisible to the prying eyes of corporations and governments.
Learn more
Download
By downloading the program you accept the terms of the License agreement

AdGuard VPN
for iOS

Boost your online protection by taking it with you wherever you go. Use AdGuard VPN to enjoy your favorite movies and shows!
Learn more
App Store
By downloading the program you accept the terms of the License agreement

AdGuard VPN
for Android

Remain anonymous wherever you go with AdGuard VPN! Dozens of locations, fast and reliable connection — all in your pocket.
Learn more
Google Play
By downloading the program you accept the terms of the License agreement
Download
By downloading the program you accept the terms of the License agreement

AdGuard VPN
for Chrome

Hide your true location and emerge from another place in the world — access any content without speed limits and preserve your web anonymity.
Learn more
Install
By downloading the program you accept the terms of the License agreement

AdGuard VPN
for Edge

Get to a different location in one click, hide your IP, and make your web surfing safe and anonymous.
Learn more
Install
By downloading the program you accept the terms of the License agreement

AdGuard VPN
for Firefox

Protect your privacy, hide your real location, and decide to where you need the VPN and where you don't!
Learn more
Install
By downloading the program you accept the terms of the License agreement

AdGuard VPN
for Opera

Be a ninja in your Opera browser: move quickly to any part of the world and remain unnoticed.
Learn more
Install
By downloading the program you accept the terms of the License agreement
AdGuard VPN
download has started
Click the button indicated by the arrow to start the installation.
Scan to install AdGuard VPN on your mobile device