itslearning users suffered temporarily slowness and timeouts on our learning tools, amongst others assignment and page elements. Some users were also impacted with an overall performance degradation.
Root cause
High amount of traffic targeting our global services caused stacking requests on web servers. We experienced new traffic patterns that caused long running requests that again led to the stacking requests and decreased performance.
Actions to mitigate
Put in place several measures to limit impact across the platform and isolating components
Code fixes
Configuring the customer environments
Releasing code fixes to optimize performance on heavily used pages in the application
Code changes to continuously reduce traffic between components within the application
Put in place fallback solution to restrict certain user functions and reduce customer impact in cooperation with customers
Scaled up number of web servers on global services - as a precaution
Posted Sep 09, 2020 - 14:52 CEST
Resolved
This incident has been resolved.
Posted Sep 08, 2020 - 23:45 CEST
Update
Assignments and pages are now performing well and we don't see any other performance issues. We will keep this issue in monitoring state for a some while to ensure the fixes have solved the problem. Please accept our deepest apologies for the problem caused.
Posted Sep 08, 2020 - 23:40 CEST
Monitoring
Assignment and page is now performing much better. We know some customer still may experience performance issues but we hope to solve this soon. Please accept our deepest apologies for the problem caused.
Posted Sep 08, 2020 - 21:02 CEST
Update
We are continuing to investigate this issue.
Posted Sep 08, 2020 - 20:16 CEST
Investigating
We experience decreased performance on page and assignment elements. Some customers may also experience an overall performance degradation. We are working as hard we can to find a quick resolution for this.