Mango

joined 2 years ago
sorted by: new top controversial old
13
Opnsense Tailscale Plugin - Firewall (lemmy.dbzer0.com)
 

I feel like this might be a long shot but wanted to see if anyone else was running the tailscale plugin for Opnsense. I have everything set up when my opnsense acts as an exit node. The clients connect up to the exit node just fine but I wanted to redirect traffic to a different gateway instead of my ISP gateway. I tried setting up tailscale net as the source on the tailscale interface to route traffic to a specific gateway but it doesn't seem to work. It seems like tailscale ignores all of opnsense firewall rules which is a lot different than what I am use to working with the wireguard interface on opnsense.

Proton Mail still down as Proton recovers from worldwide outage in c/[email protected]
[–] [email protected] 19 points 3 months ago (1 children)

It was honestly just down for about 2 hours. I hate how you can find out about the status of proton services being down faster on their reddit subreddit than on proton's status page...

Proton's official response on the outage via reddit (wish they would move to lemmy):

Earlier today at around 4PM Zurich, the number of new connections to Proton's database servers increased sharply globally across Proton's infrastructure.

This overloaded Proton's infrastructure, and made it impossible for us to serve all customer connections. While Proton VPN, Proton Pass, Proton Drive/Docs, and Proton Wallet were recovered quickly, issues persisted for longer on Proton Mail and Proton Calendar. For those services, during the incident, approximately 50% of requests failed, leading to intermittent service unavailability for some users (the service would look to be alternating between up and down from minute to minute).

Normally, Proton would have sufficient extra capacity to absorb this load while we debug the problem, but in recent months, we have been migrating our entire infrastructure to a new one based on Kubernetes. This requires us to run two parallel infrastructure at the same time, without having the ability to easily move load between the two very different infrastructures. While all other services have been migrated to the new infrastructure, Proton Mail is still in middle of the migration process.

Because of this, we were not able to automatically scale capacity to handle the massive increase in load. In total, it took us approximately 2 hours to get back to the state where we could service 100% of requests, with users experiencing degraded performance until then. The service was available, but only intermittently, with performance being substantially improved during the second hour of the incident, but requiring an additional hour to fully resolve.

A parallel investigation by our site reliability engineering team identified a software change that we suspected was responsible for the initial load spike. After this change was rolled back, database load returned to normal. This change was not initially suspected because a long period of time had elapsed between when this change was introduced and when the problem manifested itself, and an initial analysis of the code suggested that it should have no impact on the number of database connections. A deeper analysis will be done as part of our post-mortem process to understand this better.

The completion of ongoing infrastructure migrations will make Proton's infrastructure more resilient to unexpected incidents like this by restoring the higher level of redundancy that we typically run, and we are working to complete this work as quickly as possible.