We want to be fully transparent regarding issues that may affect our users and would like to inform you of the causes of the problem and the steps that we are taking to minimize the risk of this happening again.
What happened during the outage?Users could no longer open cloud projects or save changes in cloud projects or offline copies of cloud projects.
What caused the outage?We use Azure, Microsoft’s cloud service, to host Citavi Cloud projects via a web app that runs several instances which process requests from Citavi 6. Web apps are comparable in function to web servers. These web app instances communicate with many other services, such as databases, file storage, and cache services. These services are only accessible via an encrypted message (= token). The token is also included in our web app to allow its running instances to access databases, storage, cache, etc. Decryption is accomplished through what is called a lazy initialization – the token is decrypted once and then stored in memory.
Azure restarts web app instances in regular intervals to install security fixes or to perform maintenance tasks. As several instances run concurrently, interruptions are prevented. On Saturday, December 9, 2017 at 3 p.m. CET one of our web app instances was restarted. During the decryption process of cache login information, the token could not be read. The error was produced within Azure’s framework, had never occurred before, and cannot be reproduced by us. Under normal circumstances it should not have had any effect since everything worked correctly when the same steps were repeated during subsequent testing.
We did not anticipate this error in Citavi’s code. Instead of repeating the decryption process, a faulty lazy object was now stored in the memory. This caused an error each time the cache was accessed, with the result that cloud projects could no longer be opened or edited. As only one instance was affected, Citavi for Windows was falsely notified of fully functioning cloud services. This is why Citavi did not respond properly and did not save the changes made to cloud projects in the local project cache. All changes made during the outage are unfortunately lost.
What steps are we taking?Going forward, all nodes that use lazy connections will be reprogrammed to account for the above described error. Should it occur again, the initialization will simply be repeated and Citavi’s cloud services will function as intended.
Thank you for your understanding!