As of 6:38 pm EST we have made and released some additional code optimizing changes. In addition to the computing infrastructure changes we have made and our ongoing monitoring we consider this situation to be resolved. Apologies for any inconvenience caused. If there appear to be any ongoing issues related to page loads or failing LTI launches, please report them to support@yellowdig.com.
Posted Apr 22, 2020 - 18:50 EDT
Update
We are continuing to monitor for any further issues.
Posted Apr 22, 2020 - 17:37 EDT
Monitoring
At 4:12 pm EST we implemented some code changes and added a second host. We are continuing to monitor the situation but since making the changes error volume has decreased and we have not received any user support tickets related to page load or LTI launch problems.
Posted Apr 22, 2020 - 17:26 EDT
Identified
We have tried to scale some network resources to reduce the extent of the performance problems in the short-term. We have identified a number of code changes that should address this issue and we are currently making and testing those. Our plan is to deploy these code changes this afternoon, which should resolve the performance issues that people are continuing to experience. Apologies again for the service problems and thank you for your patience.
Posted Apr 22, 2020 - 14:47 EDT
Investigating
Based on some system alerts over the past few hours and a couple of incoming reports of problems from our support channel, it appears that some users are still experiencing page load problems that are related to the issue that arose on 4/21 around 5:42 EST. Current estimates indicate that since that time about 7 page loads per 1,000 are failing (< 1%) which seems consistent with the level of user reports. Our engineers are continuing to investigate the root cause and will fix it as quickly as possible. We apologize for any inconvenience.
Posted Apr 22, 2020 - 09:01 EDT
This incident affected: Yellowdig (North America) (Frontend, API).