We've been notified that our datacenter in Liberty Lake, Washington will be performing maintenance that is service impacting. Sometime between 14:00 UTC today and 18:00 UTC tomorrow they'll be performing network upgrades which will result in a network outage for all their tenants, us included. No action is required on your behalf.
We apologize for the late notice, all we can do is relay these sort of things as we receive the details and we've just only now received notice.
Be advised that there is currently an ongoing DDoS attack against our Kansas City infrastructure. DDoS mitigation is in place however there may still be degraded network performance until the attacks subside.
No, but seriously. We mass applied some CVE patches in Washington and two hypervisors are being problematic and are still down. Awaiting for the data center to hook us up with a remote access.
Just got off the phone with good friend, Senior Senator Mitch McConnell. He apologizes for the impacted hypervisors in Washington State that are down and was quoted saying, "Shit happens when you apply patches in bulk." We spoke for just shy of 20 minutes.
Everything should be back up in the Netherlands at the moment. The POP was targeted by a decent sized DDoS attack and our upstream null-routed our prefixes that were getting carpet-bombed. So, the bad news is/was service disruption. Sorry, even Sony, Cloudflare, and other big tech gets fucked by DDoS skids. Unlike big tech, we don't have the resources they have to put forth towards mitigation, BUT we ARE doing a major upgrade to this POP's network which includes some great mitigation capacity to hopefully prevent such issues from occurring again.
Hard telling who is behind such attacks. Whether it be a competitor, someone who hates privacy/freedom of speech, someone we've terminated for grave TOS/AUP violations (CSAM, Phishing, etc), or just some regional DDoS mitigation provider wishing to increase demand for their service (mitigation). Will never know. But we'll always try to do what is best for IncogNET and our customers. We've been here for the last five years and will continue to be here for the foreseeable future. 🦾
Probably rustle some jimmies and ruffle some feathers, but unfortunately we're implementing email verification.
Before you complain: Email isn't PII, you can use whatever working email service or privacy forwarding service that you wish. This is to reduce bounced automated messages and decrease administrative headaches of "help I got locked out of my account" when you register with akljdakldj@kljaslkdja.alk
portal.incognet.io/announcem…
Some general updates:
- IncogNET turned 5 years old the other day, wow!
- We're now a happy new sponsor of @ghostbsdproject, check out the project as it's worthy of your attention.
- We also contribute to @FOSSBilling. Although we are not using the software in production, we've been donating a little each month towards the development.
- Big upgrade to our DNS service in the works. 20+ POP anycast update (upgrade from the current 4 POP setup) and toying with the idea of a public DoH resolver... More on that later.
- Seems we're always a little short on help. Going to push hard this weekend to play catch-up on pending tasks and tickets that may be awaiting review. Thanks for everyone's patience while we try to balance development and updates, day to day housekeeping tasks, support and more.
[Notice of Emergency Maintenance] The Liberty Lake, Washington, USA KVM VPS Hypervisor "PedalFast" was scheduled for maintenance at 8:00PM EST today as announced recently via email. This maintenance has been re-scheduled for ASAP to address service degradation due to hardware failure.
If you were unable to process your invoice with crypto, please try again. Appears our @BTCPayServer installation was unable to fetch current market rates via the exchange's APIs due to @Quad9DNS possibly rate-limiting or blocking queries from our network. After pulling hair for a while trying to find out why these API calls were failing, simply changing the resolvers fixed the issue. Will review in more detail later, but for now, the issue should be fixed.
Our web portal will temporarily remain in maintenance mode for the next hour or so.
We are performing updates to our IncogDNS service, and placing the portal in maintenance mode effectively prevents any customers from adding, removing or modifying existing DNS records while we complete this.
(DNS System has gotten a big overhaul.)