Implementing a file pager in Zig: Reading from the disk
We have finally gotten to the point where we can ask the pager for a page of data (reminder, a page in this case is 8KB of data) and get it back. Here is what this looks like:There are a few things to...
View ArticleImplementing a file pager in Zig: Reclaiming memory
Up to this point, we focused on reading data from the disk, we can do that up to a point. Eventually we’ll run out of memory (assuming that the database is bigger than memory, which is a pretty safe...
View Articlere: Are You Sure You Want to Use MMAP in Your Database Management System?
I was pointed to this paper on twitter: Are You Sure You Want to Use MMAP in Your Database Management System?As you can imagine, this is a topic near and dear to my heart. This is especially the case...
View ArticleImplementing a file pager in Zig: Writing data
At long last, we are now at the point where we can write data back to the disk. Before we can do that, however, we need to figure out what sort of writes we want to allow. The idea that I have in mind...
View ArticleImplementing a file pager in Zig: Write durability and concurrency
In the last blog post I presented the manner in which the Pager can write data to disk. Here is a reminder:We acquire the writer (and a lock on it), call write on the pages we want to write on (passing...
View ArticleImplementing a file pager in Zig: Write behind policies
In the previous post I outlined some ideas about how to implement a more efficient write behind. The idea is that whenever we write pages to the Pager, we’ll not trigger an immediate write to the disk....
View ArticleImplementing a file pager in Zig: Write behind implementation
In the last post, we figured out how we can find what pages we should flush to disk, in this one, I want to talk about how we can actually do the flushing. Once a page is marked as ready for flushing,...
View ArticleClik here to view.
Implementing a file pager in Zig: Pages, buffers and metadata, oh my!
Databases will typically divide their data files into pages, and manage all data in pages. As you can imagine, the notion of pages is pretty important to a database. Up to this point, we worked with...
View ArticleTalking at Carnegie Mellon Database Group: Practical Considerations for...
Next week I’ll be talking at CMU about how to build storage engines and what you need to consider when building them. The talk is open to the public (don’t have to be at CMU to be there).Here are the...
View ArticleRavenDB: Practical Considerations for ACID/MVCC Storage Engines
My talk at the Carnegie Mellon Database Group about the internals of Voron and how we build a transactional storage engine.
View ArticleBadly implementing encryption: Part V–nonce reuse
In the previous post, I moved to using HMAC for the key stream generation. Together with a random nonce, we ensure that each time that we encrypt a value, we’ll get a different encrypted value....
View ArticleBadly implementing encryption: Part X-Additional data
This series has been going on quite a bit longer than I intended it to be. Barring any new developments or questions, I think that this will be the last post I’ll write on the topic.In a previous post,...
View ArticleRavenDB: Domain Modeling and Data Persistency
Dejan Miličić is talking with André Baltieri about data modeling and data persistency.
View ArticleClik here to view.
Performance optimizations in production
Date range queries can be quite expensive for RavenDB. Consider the following query:from index 'Users/Search' wheresearch(DisplayName, "Oren") and CreationDate between "2008-10-13T07:18:01.623" and...
View ArticleWebinar: Database Security in a Hostile World
On March 9, I’m going to be talking about Database Security in a Hostile World. Please join me, I promise it will be interesting.
View ArticleClik here to view.
Negative Space Scheduling
My first project as a professional software developer was to build a scheduling system for a dental clinics chain. That was a huge project (multiple years) and was really quite interesting. Looking...
View ArticleClik here to view.
Defense in depth and the local server
Yesterday I gave a webinar about Database Security in a Hostile World and I got a really interesting question at the end:If your RavenDB and app are on the same server, would you recommend using...
View ArticleRecording: Database Security in a Hostile World
The internet is a hostile place. Publicly accessible machines will be attacked within minutes of being connected, and any unencrypted data in transit is likely to be intercepted and modified. Every day...
View ArticleClik here to view.
Data ownership in a distributed system
When you have a distributed system, one of the key issues that you have to deal with is the notion of data ownership. The problem is that it can be a pretty hard issue to explain properly, given the...
View Article