CHYP.ai · ORBIT // Cloudy_Day Security Suite

Databases & data

Operational ClickHouse Cloud status

clickhouse.com · Official status page (Statuspage-compatible JSON)

FROM ALL SOURCES

No sign of a widespread ClickHouse Cloud problem right now; every source we read is quiet.

confidence: medium Vendor: operational · users: normal · routing: no data.

Vendor status vs. reality: Vendor status and outside signals roughly agree
The four clocks
ClockStartedRunning
Vendor clock when the vendor's open incident started60 days
Crowd clock when user reports began rising above normalnot seenn/a
Network clock when routing anomaly data first flagged itnot seenn/a
ORBIT clock when our checker first recorded the changenot seenn/a

Evidence

  • Official page: Operational (“All Systems Operational”). evidence confidence: high
  • Vendor stage: resolved on “We are investigating an issue with CH servers in Azure germanywestcentral region”. evidence confidence: high
  • User reports are within normal range (0 in 15 min). evidence confidence: low

What we don't know: what is happening inside ClickHouse Cloud's own systems, how many users are actually affected (reports are self-selected and unverified), and anything from routing / bgp, providers it may run on (inferred). Routing data shows reachability, not whether the app works.

Automated analysis by Cloudy_Day NetWeather; not a personal statement. Generated .

Cloudy_Day's take

Stage vs. health

The vendor's stage is “resolved”. Stage tells you where they are in their process; health is what users experience. Right now vendor: operational · users: normal · routing: no data. Vendor status and outside signals roughly agree.

Who depends on whom

We don't map other services that depend on ClickHouse Cloud. Dependency chains are where single points of failure hide.

Identity & trust lens

Even without an identity system involved, outages push people toward workarounds: password resets, unofficial apps and “status” links in DMs. Each one is a trust decision. Use official channels.

For users

Nothing points to a broad problem. If it's broken for you, try “Is it just me?” and check your own network first.

For security teams

Treat this as an availability event unless an official source says otherwise. Check which of your systems sit on ClickHouse Cloud, watch for outage-themed phishing aimed at your users, and log what you see with timestamps (all four clocks).

What to watch next

Any new vendor incident, a jump in user reports above baseline, or trouble at the providers ClickHouse Cloud may depend on.

Automated analysis by Cloudy_Day NetWeather; not a personal statement. Independent researcher; not affiliated with ClickHouse Cloud. No blame or attack claims without an official source. Generated .

INCIDENT REPLAY

We are investigating an issue with CH servers in Azure germanywestcentral region

minor resolved · started · resolved

8 updates

  1. investigating

    We detected an issue with CHC clusters in some Azure regions. ClickHouse cloud team is investigating. Some operations might be degraded.

  2. investigating

    Only germanywestcentral Azure region is affected.

  3. investigating

    Only one Availability zone is affected in germanywestcentral Azure region. The problem affects internal component (ClickHouse Keepers). Some ClickHouse Services in this region may see increased latency on inserts or DDL queries.

  4. investigating

    The problem is caused by Azure outage in the germanywestcentral region. ClickHouse Cloud team is working with Azure support. Impact: • New ClickHouse services can't be provisioned (appear as stuck in UI). • Some services may experience higher latency due to reduced capacity.

  5. identified

    The Azure team is working on resolution of the issue. The ClickHouse Cloud team will relay any updates related to ClickHouse clusters operation. An excerpt from Azure incident status: > As per our investigation, we believe this stemmed from multiple underlying storage systems becoming unavailable at the same time, which led to attached virtual machines becoming unresponsive, restarting unexpectedly, or being temporarily unavailable.

  6. monitoring

    The issue is still present on Azure infrastructure side and ClickHouse Cloud team is actively monitoring for any impact for existing ClickHouse Clusters. Correction on earlier impact description: Provisioning of new instances is expected to work. Operations like autoscaling of the ClickHouse Keeper replicas can be stuck or slow. Latency for insert and DDL queries are not affected, thanks to cross-AZ redundancy.

  7. monitoring

    Azure support shared they are working on fixing the underlying issue, we started seeing some recovery but the incident is not completely resolved yet. We are working with Azure to get this fixed as soon as possible.

  8. resolved

    The underlying AZ has recovered and we are fully back online. Since ClickHouse cloud architecture uses 3 AZs we were operational during this outage but there might have been some degradations. Azure support shared the details of the issue as following- Between 12:04 UTC and 17:46 UTC on 22 September 2022, a platform issue resulted in an impact to the Virtual Machines (VMs) and Virtual Machine Scale Sets (VMSS) in Germany West Central. Customers may have experienced their virtual machines becoming unresponsive, restarting unexpectedly, or being temporarily unavailable during the impact window.

Official incident page ↗

Community comments (0)

Comments are opinions of visitors, not statements by Cloudy_Day. Every comment is held for review before it appears. Links are removed, and personal data is redacted. Community guidelines.

No approved comments yet.

How to read this: official status comes from each provider's own public status page. User reports are unverified visitor reports. Things that happen at the same time are coincidence, not causation. Cloudy_Day is independent and not an official status source for any company listed.

Where to next