How State Street Uses AI to Transform Millions of Trades Daily

Key Takeaways

hi everyone I'm Casey Ulen hoood I'm a product manager uh at datab bricks and my name is VJ sankaran I'm uh from State Street it's one of the world's largest uh uh Financial Services firm with uh interests in uh custody banking or uh and asset ownership and management we also have uh interests in um the area of uh Investment Management with one of the largest operations um dealing with and it's one of the earliest uh providers of ETFs as you might have uh heard of them uh and in our space uh we have a bunch of um you know interaction with uh generative AI Solutions uh specifically uh because uh we end up with dealing with a lot of documents um in our world um and uh these are financial statements and uh tra transaction documents and we also have a lot of Need for uh a number of different um research databased research uh needs uh across the company so a lot of our customers at datab bricks are trying to figure out hey what should they build with generative Ai and then how do they get Roi out of these use cases and so VJ when we spoke earlier you had mentioned hey there's sort of like three ways to get value from generative AI said like hey you can either get productivity gains you can build a net new product uh or you can have customer improvements that you're launching so kind of like loved that framing so maybe given that can you like walk us through one of the use cases that you have at State Street that's one of your most like strategic generative AI use cases that you all are building there definitely thank you uh so the uh along those three dimensions or axes that we typically measure all our uh products um that we uh try to build out um let me start with the customer Delight or customer experience Improvement kind of things right uh a lot of these um millions of Trades that flow through our system systems over the you know every single day um tend to have a lot of U human touch uh required uh and whenever that happens um we end up with um you know having to it's not just customer support in the traditional sense in many of these cases it is uh simply tracking and actually cancelling a trade that was issued by a an automated system uh so in all those scenarios uh we end up having to conduct research on uh where is it in the overall trade clearance workflow and halt it there so one of the new things that we are innovating on is um the uh instruction comes to us in forms of documents the data uh for our transactions is already in um large uh systems including main frames uh which uh we end up having to uh have copies of and uh data Lake uh oriented structures right so we're uh uh building out a full-fledged uh platform that is going to do smart research uh in terms of locating uh where in the workflow it is and then uh specifying also okay what needs to happen next in terms of uh and the latest um uh Innovation that came out just about last week from um open AI with their strawberry uh model which allows us to think in several steps that's essentially what we were uh building out but that's uh that's definitely upending a little bit of our work or probably accelerating some of our work in that space got it and so what is the business value that you're delivering with this sort of like document Centric automation that you're uh building there are three again it clicks on all three of our axes like first is the customer Delight which is uh what used to take uh approximately uh two to three uh hours in some cases uh is now being uh the time scale is being compressed so the customer is able to uh get what they want uh much faster right uh that also translates into productivity gains at our um at our end which is uh this whole notion of U you know having lesser touch on our uh employees from our employees and then uh of course the uh new product uh introduction we haven't uh yet uh gone in that direction uh fully but there's a lot of thinking going on about how to present this as a offer as an offering to our customers as well what we found at data bricks is many of our customers are really struggling to get to quality like getting quality gen apps into production is hard and so many of our customers are finding ways to sort of embrace data intelligence so by using their Enterprise data to improve uh general intelligence to achieve quality or they're using compound AI systems so rather than shipping like one monolithic model with like maybe like a huge prompt on it you're actually breaking it down into like modularized components where you can like specialize each component in our case even if you keep the prompts uh uh same the variety that we face in the documents that are coming towards us from the on the market um actually the data demographic changes and as a result the response changes and so we start looking at a different part of our operation just because in the document uh structure was different or the words used in the document were different than what the um you know the whole solution was set up to do so what we end up doing is uh a lot of like pre-processing the document categorizing it classifying it uh using both older machine learning based uh and Machine Vision based techniques as well as uh you know uh figuring out how to deal with uh rag inaccuracies by using knowledge graphs uh in addition to the uh original rag itself um and maybe VJ can you also walk us through what's sort of like the are you all building a compound AI system or using data intelligence um at State Street um or how are you all architecting your Solutions today yeah it's a it's a mostly the data intelligence part um in the agentic approach U where you have um each of the agents is uh optimized for uh you know object certain objective functions that they are uh looking at uh right uh when you have a cluster of those they will sort of like uh collaborate to come at the arrive at the solution a compound AI system unfortunately tends to be very engineer Centric in its thinking right it's a sort of a linear uh if this then that kind of a a thing that uh we are all um uh trained on uh trying to translate into software so uh we are sort of like have uh coming somewhere in between that in our compound AI solution right we end up with uh the ra the uh first of all of course rag Centric U use cases then I was referring to the knowledge graph which also uh rolls in some of the um symbolic representation of our uh either it is workflows that uh have to solve a certain problem or uh have the understanding of the domain understanding of certain um types of U you know Securities versus U uh whether it is a uh like a derivative or a foreign Exchange Trade there are different nuances in what happens in those areas so um so that at a high level uh that's one thing but for us uh the bigger challenges were as a very highly regulated uh industry as a player in the industry we end up uh having to demonstrate to our stakeholders uh not only internal but our regulatory stakeholders that we have uh a responsible AI system not just uh you know everybody can have a go at it uh at any llm that's out there so we're going about uh curating those llms and we've come up with a a sort of a a much more deliberate um approach to uh the uh you might call it llm Ops but uh essentially linking the llm them in the appropriate part of the overall solution um and uh you know when we make that call uh to the llm before that or even while we are making the call to the llm we are um not just a single LM multiple llms are uh involved uh so we end up um uh stitching these things together into a quilt work of uh functional calls right uh function calls and then that's uh What uh essentially uh is the overall structure of our um uh solution I see and do you have to track the series of calls for like audit purposes or that design basically correct so there is one is the responsible Ai and I'll probably just quickly show you uh one slide that uh tries to capture what it is that uh we are doing which is uh in this um in this notation right uh we have uh these uh you know of course there are different applications we got we we just um labeled them in the very uh broadest terms but uh we are provisioning this API SDK uh which essentially uh has all of these uh uh you know uh technical capabilities at the back end the prompt catalog rag pipelines and then prompt management prompt security and filtering but essentially what we uh have to uh Pro provision uh is um these kinds of things you got your jail braks input guard rails output guard rails all of these things are built in into the uh into the U uh overall architecture um now we are looking to your compound AI system is applying a bunch of guard rails there correct correct and we also have uh evaluation uh pipelines right so that uh we have to keep track of uh you know whether the response that the llm generated was satisfactory to the uh to the overall Solutions uh context uh and then um so then there's the other interesting thing I'd like to point out I think this goes back to the first uh topic that she brought up we are spending a bunch of time on this area fine-tuning because uh unlike the prompt tuning we are uh learning that uh the um llms have not yet been trained on financial terminology so a leg in uh in the financial services World especially when you're doing a derivatives trade means something else than a body part right so uh we end up having to train those kinds of things so we have uh we're focusing on this and this is where we're getting a lot of uh leverage out of data bricks is uh the Mosaic um uh layer right we uh love that part as well awesome um and VJ one of the things you had mentioned before was uh as you moved into like generative a so a lot like your solution helps sort of like augment a lot of like the manual labor component and sort of like translating documents into like actual trades um but you said like hey back in the day even though it was only maybe like 70% accurate the 30% time it was inaccurate you're able to like very easily figure out what went wrong whereas now it's like 95% accurate but in that 5% of the time it goes wrong you can't really Deb can you kind of walk through like what does it mean now that your problem space has shifted like that yeah thank you for thank you for that prom so the uh no pun intended but uh the idea here is uh prior the pre- llm world right we we always had these uh deep uh CNN uh based uh neural network based uh techniques that uh with Machine Vision where we could uh locate the coordinates on the page where you're extracting data from and TI tie back the information that you extracted back to that uh thing so that if there is something that goes wrong you can at least the humans in the loop uh can uh quickly triangulate to what could no I don't want it from this part of the page or this part of the document I wanted from this other place right and then there was also the feedback that they could provide to improve our models now what has happened with traditional llms they're only linguistic that is they can take in text strings they're not visual yet and uh again that that whole Space is being appended by like the Pix STS and the others that are that have uh multimodal llms right um so uh the text based llms uh lack that um uh that Fidelity to tell us what where you picked up this data from in the document right so uh so that gives a little a lot of trouble to our humans in the loop to repair the findings or the uh the results of the solution and uh that's exactly like you you you said it right we were we were able to increase the uh accuracy of the uh information extracted but we're not able to reduce the effect uh the time to solve in its entirety y makes sense if I if I may put it that way I really appreciate you taking the time to be here and we're excited for what you build next uh on data breaks