You can read the full details here. The short of it is that we discovered a security vulnerability in RavenDB. This post tells a story. For actionable operations, see the previous link and upgrade your RavenDB instance to a build that includes the fix.
Timeline:
- June 6 – A routine code review inside RavenDB expose a potential flaw in sanitizing external input. It is escalated and confirmed be a security bug. Further investigation classify it as CRTICIAL issue. A lot of sad faces on our slack channels show up. The issue has the trifecta of security problems:
- It is remotely exploitable.
- It is on in the default configuration.
- It provide privilege escalation (and hence, remote code execution).
- June 6 – A fix is implemented. This is somewhat complicated by the fact that we don’t want it to look like a security fix to avoid this issue.
- June 7 – The fix goes through triple code review by independent teams.
- June 7 – An ad hoc team goes through all related functionality to see if similar issues are still present.
- June 8 – Fixed version is deployed to our production environment.
We had to make a choice here, whatever to alert all users immediately, or first provide the fix and urge them to upgrade (while opening them up to attacks in the meanwhile). We also want to avoid the fix, re-fix, for-real-this-time cycle from rushing too often.
As this was discovered internally and there are no indications that this is known and/or exploited in the wild, we chose the more conservative approach and run our full “pre release” cycle, including full 72-96 hours in a production environment serving live traffic.
- June 12 – The fix is now available in a publicly released version (4.0.5).
- June 13 – Begin notification of customers. This was done by:
- Emailing all RavenDB 4.0 users. One of the reasons that we ask for registration even for the free community edition is exactly this. We want to be able to notify users when such an event occur.
- Publishing security notice on our website.
- Pushing a notification to all vulnerable RavenDB nodes warning about this issue. Here is what this looks like:
- Since June 13 – Monitoring of deployed versions and checking for vulnerable builds still in use.
- June 18 – This blog post and public notice in the mailing list to get more awareness of this issue. The website will also contain the following notice for the next couple weeks to make sure that everyone know that they should upgrade:
We are also going to implement a better method to push urgent notices like that in the future, to make sure that we can better alert users. We have also inspected the same areas of the code in earlier versions and verified that this is a new issue and not something that impacts older versions.
I would be happy to hear what more we can do to improve both our security and our security practices.
And yes, I’ll discuss the actual vulnerability in detail in a month or so.