Using Transactions with Amazon DocumentDB (with MongoDB compatibility) 4.0 - AWS Online Tech Talks

Key Takeaways

all right uh welcome folks uh my name is brian hess i'm a solution architect specialist uh focusing in amazon documentdb uh we're gonna talk a bit today about uh transactions so this is a feature that we added to documentdb back in the fall um so i wanted to talk through uh this this exciting new feature and and uh the the pieces that you can use to uh to in your applications um but what i wanted to start with first in case some folks are less familiar with documentdb is just a really high level a high level overview of documentdb there's there's some really good deeper dives or we have a couple re invent sessions that would be interested uh interesting if you were interested in going a little deeper but just want to cover the highlights so so you sort of uh understand where we're coming from here so amazon documentdb is a fully managed scalable document database and we designed it to cover mongodb workloads and so what do we mean by that so from the fully managed side um this is a cloud architected service that's that's managed for you so we've taken care of all the undifferentiated heavy lifting we've designed it with built-in high availability and automatic backups and point in time recovery the data layer that we have architected is is durable by default so you don't have to do anything to ensure it's configured correctly to make sure that the data survives uh that's just part of the way we've designed it similarly security so we're talking encryption at rest and encryption in motion user privileges and roles and access permissions all those security best practices exist by default in documentdb um and then as i mentioned it's a fully managed service you don't have to take care of any of the patching of the operating system or the database software we take care of all of that similar to other services in aws uh we also have pretty rich environment for monitoring uh the the documentdb cluster so use cloudwatch is is the the way that we uh visualize and give access to those monitoring tools you can also set alerts so you could uh set alerts there if all of a sudden there was an increase in workload that you weren't expecting and that that makes you want to uh you know expand or modify your cluster with response to that so let's sort of get at from fully managed in terms of scalability documentdb was designed in the cloud for the cloud so scalability is a key part of the design principles there uh the ability to scale up and scale out in minutes uh just by going through a couple of either api calls or using our console the storage i mentioned the storage layer in the previous slide as being durable by default it also scales automatically so you don't have to do anything to provision extra space you just start putting more data in and we will under the covers expand that storage as needed and of course the the access i o to that storage uh scales all the way up to 64 terabytes which is quite large actually uh you can configure well documentdb is set up to look a lot like mongodb replica sets if you're familiar uh with mongodb and in that world in that configuration there's uh one writer instance and you can have many reader instances so those reader instances can take over for the writer in the case that the writer has some sort of a failure but they can also be used to increase read traffic so if you have other read read workload you want to add to the cluster you can add extra read replicas and they just take a few minutes to add those instances to the cluster the architecture is set up in such a way that in order for those read replicas to start serving read requests no data needs to be moved around to them because we've separated that that storage layer from the compute layer uh and so we get this sort of scalability built right on in from the start and then lastly uh you know mongodb compatibility it's part of the service name the full name being amazon documentdb with mongodb compatibility and we designed it so that drivers and applications and tools can be used with documentdb just like they could be used with mongodb now we don't have the full coverage of the mongodb api we have really focused on the ones that our customers have told us are the most important and so we're always working backwards from what our customers need and we add to those apis and the operators and the stages uh in response to um working backward from customers and that's exactly sort of how we got to transactions so transactions was something that we had heard from customers as being something really important to their applications and so we were able to add that like i said last fall to the mix so if we talk about what we're gonna we'll talk about today uh i wanted to give a little bit of an understanding of why are transactions even an interesting thing to talk about like why were customers telling us that they would like them added to the api a couple of transaction use cases and then dive a little bit into what do transactions look like in documentdb if you're familiar with transactions in mongodb you'll see a lot of similarities here but they're a little bit different sometimes than some relational systems that you might be familiar with and so it's it's good to call out just exactly what do these transactions look like in documentdb and then we'll have some examples as we go through that so let's start with why transactions and i think that you sort of understand where this comes from we sort of hop in the way back machine and go back to the sort of relational world so relational databases popularized you know several decades ago really and one of the core tenets of a relational data model is to normalize the data and so what you do when you do that is you take redundant or repeated information and you put it in its own table and then you give a link from uh between the tables so you know how to get to that information and understand that relationship so for example if we had a business directory or something like a yelp where we've got businesses and they've got they've got addresses where they have owners you may end up with multiple businesses say at the same address and so we could take that address and put it in its own table and have a link from the profile for that business to the address uh entry in its own table then if we needed to update something about that address i don't know if there's a fire lane out front or if there's sprinklers in the place we can update that address the information about that address once for all the businesses that are in that same place so we get this nice way of operating with things where we can separate out the pieces so we don't need to make many updates and keep things in sync but one of the challenges with that is there still ends up being use cases where you need to update the information in multiple uh tables at the same time and so we might want or just multiple records within the table at the same time and so we now need to sort of keep things in sync we want to make sure that nobody is seeing some weird partial view of the world they're seeing those changes kind of come as an atomic unit you know there's the possibility of of something going wrong in the middle so that if something were to happen while i was making those changes they'd kind of get backed out so nobody is seeing partially committed uh changes to the mix so we end up with with this transactional need based on the transactional model we have this trend this need for transactions and uh and so that's really kind of where the concept got popularized uh because it became a key part of relational models so why are we talking about that if we're talking about document databases um so document databases uh similar to other nosql systems really leverage denormalizing the data uh so we can see here uh kind of this example of that yelp-like use case where we've got a business and and other information and we're keeping that information with the business as an embedded part of it so we see here that the location is a sub field it's kind of its own little uh record within a record uh in inside of the the business record here and the idea is that if you're asking for that business record you're very likely to want the address so why don't we co-locate this information or de-normalize it so that you can get it right away when you go get that record it's probably what you're about to ask for anyway denormalization also used in the relational world usually more in like data warehousing but you definitely see it in the sql world as well and in those cases it's also there to increase performance so we have this denormalized data model so some of these updates that i was saying previously was to update in multiple tables they would now be multiple fields within the same document and we already have that so documentdb's 3.6 engine when we launched documentdb a little over two years ago was this 3.6 compatible engine and and that already supported mult updating multiple fields in the same document as one just atomic unit all or nothing etcetera um so we already have that we have that also in in the 4.0 engine which came out in the fall as well so that's already there but if you go through the document modeling data modeling sort of best practices there are some times when you do want normalization and you do want to sort of talk about how these things uh should be split up into multiple collections and so we're back to wanting to do those updates across multiple collections there's other situations where you want to update multiple documents within the same collection and you want those to be in sync as well and we'll talk about that some of those use cases on the next slide but it's important here to say like we've now entered we've introduced this new tool this transactions tool which can be a bit of a crutch in some situations the document data modeling best practices still apply so don't go crazy with transactions now just because there's a new toy in the toolbox we still want to do the the best practices for data modeling so you can get the best out of your document database we don't want to necessarily just take a relational model and move it over that's not the best way to get all the power of documentdb which is also for what it's worth the same as people say about about mongodb so the similarities uh exist there as well so let's talk about a couple of use cases uh probably the the first like easiest use case for transactions is probably this ledger use case top right it they're kind of financial transactions and the idea is that you've got uh maybe a balanced type data set so this collection is the balance for individual users and if uh we're going to decrement brian and give data over or give money over to phil then we want to decrement brian's account and we want to increment phil's account but we really want that to sort of happen as one thing we don't want to take money away from brian and then something goes wrong and now all of a sudden some money has disappeared from the system we want to to and and similarly the other way around we don't want to give it to phil and then something go wrong and we never finish and all of a sudden money appeared out of nowhere we want these to happen together take money away from brian give it to phil and the world sees um that whole thing as one operation that's just happened uh at the same time the many-to-many relationships one is is one where we start talking about multiple documents um so for instance we could do something like room reservation where we want to know things like uh if a guest has a guest profile we want to see all the reservations that that guest has made and similarly we might have a rooms collection so that we can take a look from the room side uh which reservations were made by which guests so if you go to reserve a room we update both the guest's profile to tell them that he's now reserved a certain room on a certain day and we update the the room profile to indicate that we've got a reservation by a particular guest on a particular day uh so we need to get those two things to happen at the same time we wouldn't want that information to get out of sync uh there are some times where you want to do an operation and log that it happened and so you might be really trying to keep a strict sort of auditing capability that you you manage in your application and the idea is that you don't want to do an operation unless you can also log it you need to make sure that both happen you don't want to have operations that happened but weren't logged and you wouldn't want to have up things in the log that never actually occurred so you sort of want those two things to come together if they happen at all and then last there are some situations where when you do an uh a real world business operation you need to update multiple collections so like for instance if i'm going to create an account i might need to create a document for that new user or that new account in multiple different collections because of the way that we manage our data and so we would want to make sure that we got all of them taken care of and that when the account creation happened we successfully wrote uh that new record or that setup into each of the collections so sort of from a high level a few motivating examples you'll see as we get into uh the characteristics and you know sort of how these you know these use cases and their needs kind of come into what those uh what those characteristics look like so before we get into the details of the actual you know characteristics and and dive into what they look like figured we'd do a really quick example just to see what the api looks like so this is python um the concept here is that it should be pretty simple uh and so we see is we're going to create this session so a session is what we will start our transaction on um at least in the python world it kind of operates this way so the first thing is we create a new session called s and then what we're going to do is start a transaction so we just say s dot start transaction now we're going to try two things so we do this try and catch block just in case there's an error we'll talk a little bit about the types of errors that you can occur but just in case there's an error we want to be able to to to handle that appropriately so the try block is going to try to write doc 1 into the first collection and we'll write doc 2 into the second collection if everything goes well then we jump down to the bottom and we commit the transaction so nothing really has been uh permanently saved in the database in a way that other users can see until we commit the transaction so we just do s dot commit that's going to say anything that was done in this transaction should now be committed to the database and everybody should see it if something does go wrong we're going to catch that exception and we're going to uh we're going to abort so the abort transaction is really a way of saying forget everything i just tried to do please put it back the way it was and pretend i was never here and so none of the partial rights like for instance if the first operation worked and we actually got doc one into the first collection uh if we had an error on doc two writing to the second collection and we choose to abort we're going to pretend like hey just don't do that right to the first collection just back the whole thing out and and make it as if i i had never done anything so that's the the sort of abort transaction piece and then raise we're just going to raise the exception up to the to the calling uh function so that he can handle whatever else he wants to do with this we'll talk a little bit more uh at length about uh how to actually handle uh exceptions and retrying things in a moment but um just wanted to give this idea of pretty simple you've got this session you start the transaction on the session um well i didn't say this but when you do the uh the the write operations and insert and update a delete you want to make sure that you pass on in this session you can actually do a query that's not part of the session uh we'll talk a little bit about how rights that happen or uh that are not part of the session and what their world looks like how things are a little different but if we wanted to be part of the session we have to pass on in that session object so that the insert one method takes an additional optional argument of a session which will tell the database that this is part of the same transaction so that's a really important part um that says session equals s there okay so let's move on to the characteristics so what are these things what do they do what do they feel like what promises are being made here so the first thing uh to understand uh we talked about how single document like multiple fields in the same document those sorts of operations have already been essentially transactional right we can update multiple fields in a single document in an atomic way that just all or nothing and it gets done and people see it as all those updates happen at the same time these transactions these new transactions really are multi-document those documents can be in the same collection um kind of like that ledger use case they can be across different collections they can even be different collections in different databases those update operations typically are also multi-statement so it's multi-everything one of the key features of transactions is really this all or nothing capability and i kind of hinted at that before you're going to do a number of write operations inserts updates or deletes and either all of those writes will occur and be reflected in the database when we're finished or none of them will so all of them are if we commit the transaction and none of them are if we abort the transaction so we really have this sort of no one is able to see uh what's going on while we're doing our transaction they just see as if everything happened at the end snapshot or isolation in general but snapshot isolation is an important part also and this is really in two places or two ways the first is for my transaction when i begin a transaction it's as if i'm freezing the whole world for me right now so everything that i'm going to do with the data is going to be looking just like the data was when i created my transaction so any updates that are happening by other users by other applications by other connect connections i'm not going to see them in my transaction context right so so it's nice and simple that way i can work on data and know that it's not changing underneath me i have a nice little isolated uh view of the world um so only the rights and the updates that happened before i started my transaction are going to be visible um and you know the the other side of the isolation is that with all of those rights and everything no one's going to see any of my partial work until i commit that transaction so i have this little isolated world uh the data's not changing from underneath me and then no one's seeing anything i'm doing while i go until until i commit and the last part is this idea that well no one else can see my rights and my partial work while i'm going but i can and that's an important part so i can do in my transaction a combination of reads and writes so i might uh do do a read figure out a couple of writes to do do some more reading do some more writing and and and back and forth when i go and read from the database i'll be able to see all of the rights that i did even if they haven't been committed yet right so i'm in my little world in my little bubble um and so i get to see the database as if all the things that i was writing had been committed now they're not actually going to be visible to anybody else until i do commit but this allows for my world to stay uh consistent so if i do a write i then immediately do a read i can actually see that that right reflected within my transaction so let's ground this in a couple of examples um so let's imagine we've got uh our our business profile here on the right uh and my transaction is just simply going to be i want to update the review count from 46 to 47. so if i then want to go ahead and read um within my transaction i'll see that the value is 47. if i have a regular read go at the same time and try to read the review count from this record i before my transaction has been committed it will see the old value at 46 so i'm still seeing the old value because my transaction hasn't been committed yet so let's go ahead and commit that transaction and now uh when a regular read comes on in it'll see that value of 47. so we all of a sudden it's revealed okay so pretty straightforward with respect to reading your own rights and the isolation uh the all or nothing kind of capabilities here there's one more aspect that we want to talk about and that has to do with right locking uh there are a couple of different ways to handle consistency or sorry concurrency in general with database systems one of them is to do right locking and another one is what actually documentdb uses or used prior to transactions and still uses for regular operations is something called multi-version concurrency control and this is a way that you can go ahead and have multiple users writing and reading to the database without having to lock records uh when they're being read or when they're being written and it allows for a lot of multi-users multiple users querying at the same time however with transactions because we're trying to do a number of things all at once we do introduce right locking and we're only going to lock the records that we needed to write so if you do a write to a particular document the database will put a little lock on it and that's not really a problem if people read the rest of the documents that are out there no worries everything is fine uh if uh another transaction wants to update the exact same fields uh in sorry the exact same documents as the first transaction then it'll get an error that transaction will get rolled back the whole transaction not just that one little operation and we'll talk a little bit about how we want to handle if and when those errors occur but if you stumble into something in your transaction that somebody else is changing in their transaction uh we're going to to run into an error that we abort if on the other hand somebody does a regular read on that updated record so if we go back to um uh the scenarios uh where we're we're doing a regular a regular read so that's not within a transaction uh that will actually block so it's gonna encounter that a that a write is happening within a transaction and the right will just wait so there's no error it'll just wait until that transaction is completed and then it will be able to read that record just something to note there is still a right timeout in in these operations in document dbn and mongodb we have this query timeout um that happens here and so if your read is going for a very long period of time it will eventually time out so if that transaction blocks for too long some of those reads will get timeouts so those are for writes reads inside of a transaction don't lock the document so if you want to go ahead and do a read that's fine go go for it you're not impacting anybody else there are some times that people will want to do some locking for various reasons even when they do do a read and if you find yourself in that sort of rare case you can do kind of a dummy no-op operation so you can do something that doesn't really matter to the business um so this example here we're incrementing a dummy counter like we're never really going to use that dummy counter field all we've done here is update it so that uh that we do end up creating that right lock like i said it doesn't come up all that often but sometimes uh people do want that um and that will then obviously force the lock and then you kind of have the behavior you were looking for but reads in general don't lock the document i mentioned that regular rights will block while the transaction's happening so will ddl so if there is a ddl operation happening outside the transaction it's going to look at the table and see or the collection and see that some of the documents have been locked and it's going to wait and so that ddl could be something like creating a you know creating an index or something along those lines um so those operations will wait just like just like the the write operations um in uh uh outside of the transaction i do want to call one note here this is the the one big difference between mongodb transactions and documentdb transactions and that's that in mongodb there was an optimization that they took where if you do a right to a document that would result in nothing changing then it doesn't actually lock the document it recognizes this like hey that wasn't actually going to change anything so for example before i was changing that review count from 46 to 47 if instead you actually set it to 46 mongodb will recognize that and say that that actually won't change anything because current value already is 46 i'm not going to lock the record document tb locks the record regardless so it's not going to actually try to go in special case did it did this change really matter um you know that if you do a write it'll always lock and i think that's actually a little bit better from an application development standpoint of understanding what's going to happen in your database um i like the fact that if you do a write it will actually lock that record in the transaction uh you don't have to worry about what was the state beforehand um and so i think from a application development standpoint uh it's it might be a little bit more locking going on uh in documentdb but you kind of understand what your operations do with respect to whether or not things will get get locked so let's take a quick look at an example so we've got our same sort of uh data model here that we've been talking about on transaction number one we're going to go ahead and update that 47. uh what you'll see sort of that that kind of uh circle with the slash is supposed to represent that there's now a lock on this record transaction is still going on so now transaction two comes along it wants to update maybe something different in that field like the price the two dollar signs it will encounter that there was this uh lock on there and it will actually roll back all of transaction number two so it hits a wall and then uh unrolls itself back uh back so that it the transaction is rolled back there you go however if we do a regular write so here we we're back to just doing an update um to maybe this way we're doing it outside the transaction to update two two dollar signs it's just gonna wait that's what i meant by this sort of yield sign here uh once transaction one gets committed then the regular operation will get sent through and you'll see we now update um the price from one dollar sign to two dollar signs okay so regular operations will wait if for whatever reason transaction one was taking a really long time uh then eventually that regular operation would time out all right so we talked about the types of errors that can happen uh what should we do about retrying them so there are two kinds of errors i want to talk about when we do uh these transaction uh operations we can run into an operation failure so it's an error thrown by the drive by the by the server the mongodb api introduced this concept of adding a tag to the error and the tag tells us a little bit more information about what's going on so if we do uh if we encounter a right lock in our transaction so somebody else was already right updating this particular document so we're going to get that rollback scenario we will get an uh operation failure error that will include this transient transaction error tag that's going to be our our notification that um uh that we we encountered that situation now that the transient in there is important it's telling us actually i couldn't do this because somebody else was in the middle of a transaction but that shouldn't stop you from trying again because once that transaction is over i'd be happy to do that operation for you so you can retry uh the whole transaction if you bump into a transient transaction error so that's the first kind of error the second kind of error is one where on the way uh when you try to commit the transaction you might run into an error uh from the server and the problem with that error when you do a commit is you don't know if the error was on the way of communicating that to the server to tell it that it should commit or if it was on the response back from the server telling you that it actually did commit so you don't know whether or not you actually got the commit result you were looking for and that's why the tag is unknown transaction commit result so these can happen when you try to commit unfortunately in documentdb this is slightly different than mongodb we can't retry the commit on its own so uh we wouldn't do that we'd have to figure out how we wanted to approach that from um you know the application standpoint and handle that in its special case because if the operation did commit and we were getting the error we need to detect that situation in a slightly different way um so if you get so the other thing to note is that when the unknown transaction commit result tag is is present so for some reason is the transient transaction error tag so we're going to kind of look for is we want to see that it has the transient transaction error tag but it doesn't have this commit error tag and then we can retry those whole operations so let me ground that in some code so here's a silly little example again very much like the one that we started with uh way back at the beginning and what we see is this first function at the top called run transaction which will start a session start a transaction on that session do the same two inserts and then commit okay so we're not going to do that in a try block we'll do that in a try block in this sort of function two so function two will come on in it'll try to run the transaction if everything works great then we return the result then we're good to go and the function two finishes if there is some kind of an error we're going to catch that error we're catching these operation failure and connection failure and we're going to look to see if the transient transaction error tag is present or label is present we're also going to check that the unknown transaction commit result is not present so we're in that situation where we encountered a right lock so we'll we'll just do a little print and then we'll continue and the continue will send us back around this while true loop so this isn't necessarily the best practice on how to handle all the retries we can get much more fancy logic like we could do some exponential back off so we're not just pounding on the uh server and we keep trying over and over again we might uh add small delays in there we could add a counter that says only retry this two or three times just to keep the brevity of this slide i took that very basic idea of putting it into an infinite loop we're just going to keep trying this transaction until it succeeds there's obviously issues with that like if it doesn't succeed after a while you probably should give up otherwise you're wasting resources so you can feel free to take all sorts of variations on this code i'm just using it here for for uh you know just explanatory uh purposes all right so let's talk a little bit about the similarities with mongodb so if you're familiar with mongodb transactions what's the same and the answer is an awful lot is the same so reads inside of a transaction do not lock the document that's the same as mongodb rights uh so if i do a write and it does a lock writes in another transaction if they encounter a document that i've locked will roll back same as doc mongodb uh regular rights outside of a transaction will just wait so they'll be blocked behind that transaction and once that transaction completes then the regular write will be allowed to finish similarly ddl will happen will also block during while that transaction is happening uh the the the transactions in document db uh are very similar to the way transactions were in mongodb 40. and that covers sort of the next two bullet points later on in in later versions of mongodb some of the some of these two points change but currently document db does not allow for a ddl operation inside of a transaction which is the same as it was in mongodb 4.0 uh and similarly the maximum transaction side is 16 to 16 megabytes again same as mongodb 4.0 and then when you encounter this transient transaction error which sort of indicates that you've hit a right lock contention you should retry those errors or retry those transactions in their entirety there are a couple of differences um just we've talked about them already but i'll reiterate them here the first is if you do an operation a write operation uh in document db regardless of whether or not that update changed anything you will lock that document so mongodb has that no op detection and in in which case it won't lock the document but documentdb will will do a lock uh regardless of whether or not the change made a difference or not and we talked a little also about uh you you can't retry commits if you see commit errors uh from the database so that's just a slightly different best practice in terms of error handling with documentdb did want to call out two things to note it's not so much a difference from mongodb this is just more on the you know you should know about the first one is that there is a time limit on a transaction that's set at a minute and that sort of protects the server from transactions that just stay for a very long time and could potentially keep these locks for a very long time so there is a transaction timeout of uh one minute um and then this it sounds a little silly but you can only do transactions on the primary um the primary is the only one that's allowed to do rights anyway and since transactions really are largely about rights this isn't too much of a surprise but i i did want to sort of just put it out there that if you're going to do this you're going to do transactions you're going to want to set uh your your preferences to the primary and then just kind of summarizing up uh about transactions first thing um i'd be remiss without reiterating that the document data modeling best practices still apply you should still go through and do uh all the best things that we've been talking about even though we have this new tool it shouldn't drastically change uh the way we approach data modeling um and so for instance if you find yourself with transactions being kind of your dominant way of doing reads and writes to the database especially rights to the database uh it's probably a flag that we should take a step back and think about how we're how we're doing this um but transactions are an important uh tool in the toolbox so we should use them appropriately whenever you're doing an operation that you want in a transaction please make sure to pass the session along with the statement uh this is i'm speaking from personal experience of having goofed this up a few times but that extra uh parameter that says this is part of a transaction or part of a session um is important to uh to to remember uh we talked about this last this bullet on handling errors applies really in general but it's worth reiterating with respect to transactions since we've talked about it um handle the errors that we see retry the ones that that you can uh and do all that appropriately and then uh just keeping uh an eye on if you did mean to lock a record like if you wanted to do a read make sure you do a write and that will cause the system to lock that record and with that that that pretty much covers transactions uh in documentdb 4.0 um really exciting new feature added in the fall and uh you know really look forward to seeing what what everybody does with the new tool in the toolbox so thanks