Class 28 Video: Feature Extraction and Machine Learning (II)

Key Takeaways

My name is Kik. I'm visiting scholar here at MIT. I'm actually assistant professor at poly techchnical university of Madrid. And my background is actually on engineering. I am aerospace engineer. Uh I did my PhD on numerical methods. But uh after my post dog I decided to start doing things that really passionates me and that's how I end up doing something related with music. No. So I'm relatively new in this field and today I'm going to show you an algorithm that I came up for pattern detection in music which is what I've been working on for the last year and a half more or less. So as running example uh I'm going to use the most listened song of the history which is uh Baby Sark. >> So I don't know if I can play it. Yeah. At least it's the most listened song of the history of YouTube. >> Baby Shark doo doo doo doo doo doo. Baby Shark doo doo doo doo doo doo. Baby Shark doo doo doo doo doo doo. Baby Shark, >> Mommy Shark, doo doo doo doo doo doo. Mommy shark, doo doo doo doo doo doo. Mommy shark, doo doo doo doo doo doo. Mommy shark, >> pretty much like that all the song. This is the melody. So what's the problem of pattern detection in music? Okay, we would like to have a tool, an automatic to tool that help us to identify over the score or over a corpus like the important musical ideas there. No. So I'm highlighting here some fragments. Have a look first at the blue fragment over there. There are three fragments, three occurrences of them. All of them are exactly the same. They have the same notes with the same pitches and same durations. The only difference is that they occur at different moment of the score. But it seems logical. It seems reasonable to group them as the same musical idea and say that they represent the same the same pattern. No, we can call them uh we can put them a label blue pattern. Okay, for the green one is a little trickier because the occurrence number two and the occurrence number three are exactly the same. Okay, probably makes sense to group them together. Occurrence number one has a different durations for the notes and the occurrence number four is even trickier because they just serve with the rest that it's three notes also and the lyrics. No, but it's pretty different. No. So that's one of the main difficulties of the problem of pattern mining in music that even though uh objects that we want to group together because in our mind represent the same musical idea over the table they might look pretty different and there are two mechanisms involved here. The first one is something that we have already studied the transformations. So if we have a a music fragment, if we apply the mathematical transformation over the notes there, we obtain other fragments that under some circumstances our mind will process them as equivalent. And how to deal with this computationally from the point of view of a pattern mining algorithm? My approach was to use the concept of viewpoint. I'm going to show you an example of what is this about but for the moment it's enough to know that it it's it have a double task know on the first hand um it directly takes into account this transformations you will see it in a minute and probably more important it simplifies the representation because one of the constants in this course in this classes is that music is something complex no it has like complex logical structures among among the elements of the score and if we find a way to simplify that representation would be helpful for from the computational point of view. So the viewpoint representation is pretty simple. It's just I'm going to substitute a complex tune like this one by a single sequence of symbols. So actually here I'm showing three viewpoints representation. The pitch viewpoint it's the sequence of these symbols 83 83 83 85 it's the MIDI note then we have the duration and the onset time I'm going to highlight there a a fragment so you can see the different viewpoints representation for that fragment and it's nice that idea of constructing the score by overlapping layers of information no that's the idea formally a viewpoint is a mapping between the current event, the current time slice and all the former ones into a symbol. Usually an integral or float, but could be others. Those there are derived viewpoints and perhaps for your own particular music application, you need to develop your own viewpoint representation. Uh and here is clearly highlighted how the transformations could be directly taken into account by using this technique. You know for instance if you apply uh the octave transformation to that pattern there that fragment you obtain that one over there which one which by the interval uh viewpoint representation uh will have the same symbol. So for sure any pattern mining algorithm that is trying to uh find this sequence uh will see that that one is the same. No. Is that enough to deal with the concept of dissimilarity in music? Unfortunately, not. So have a look at these two occurrences of the green pattern of baby sark. There is no viewpoint representation that can take us from one to the other. So someone said at some point okay it would be nice to have a kind of measure uh a mathematical tool that help us to evaluate how different two fragments are. like uh the aim here is that okay if I use that matrix with this guy and this guy I would expect a lower value than the one that I would obtain when taking this guy and this guy no because they are uh more far no they represent uh farther ideas. So in computer science they use what is called the distance functions. is in a general scenario is that you can give to a function two objects two random objects for instance me myself and this computer and it returns a non- negative real number evaluating the degree of dissimilarity between the two objects in the context of sequences because we are using viewpoint representation so now this fragment will be a sequence for instance let's see let's think that I'm using the duration viewpoint point. So in the literature we have like plenty of distance functions. This was very a very trending topic in music information retrieval like 15 years ago more or less. There were many studies and papers trying to figure out which was the best distance function depending on the style. The lestain distance function perhaps is not the most advanced one but is one of the simplest one and probably the most used one and it accounts for the number of substitutions or insertions to go from one sequence to the other. So uh to go from the sequence 0.5 to the sequence 0.52 we just need a single substitution. So the leven distance function between the fragment two and fragment four is one. Okay. Probably the leven distance function between this guy and this guy it's I don't know two or three at least. No. So cool. Now we have like all the ingredients needed to understand the inputs and outputs of my algorithm. We have a corpus or a piece of music. We have also the viewpoint representation that the user want to choose and some parameters at the input. The minimum support is the minimum number of repetitions that we uh require to a pattern to be considered frequent and to be included in the result list. These two guys the minimum length and maximum length are the parameters to control the lengths of the patterns that we want to mine and some parameters to control the degree of dissimilarity from the former slide. And the result is yes like the list of frequent patterns together with the position. This is an example with the piece of the former slide. Now I'm selecting the interval viewpoint uh and those parameters over there. And this is an example of the output that we might obtain. So we have a yellow pattern that has four occurrences. and have a look that each of these fragments are at most distance one from any of the rest. No, taking the Levenstein distance function and we might have for instance also this guy the purple pattern and this green one which is of length four. So yeah in a real case the pattern structure might be complex as you see because you have nested patterns you have many overlapping I won't go into the details on how I implemented the algorithm just a brief overview uh I divided in four steps the one is the creation of a an initial database a a vertical database it's called sometimes where I just take a sliding window of length minimum length the minimum length selected by the user and I just create a database of do all the patterns together with their positions within the score. Then in the next step I compare every pattern against all the rest trying to uh see if they satisfies the distance constraint. No. And in that case I group them together into uh what I call metric patterns which are like groups of fragments of music. you know at this stage it's safe to deate the entries that doesn't reach the minimum support constraint like they don't have enough occurrences to be considered frequent and finally as here I still have patterns of length equal to the minimum length and we actually want longer patterns so the last stage is overlapping different patterns to form longer ones okay and to finish this I'm going to show you some results this is the last slide Uh I'm actually already using this I started using collaborating with these research groups like seven months ago. This has a a research group from another university of Madrid. They got a good amount of money of the European Union to study the Italian operas. They have a huge database with many features for each piece and in particular they are interested in the emotion that the piece evoke. So here we are trying to find some correlations between the emotions and the pattern structure of the different pieces. These two other projects are pretty similar. The number two it's on a corpus of folk music from a very concrete region of Europe in around the Pyrenees. And the second one is the a database of just solos. So here we are building classifiers like similar to what we did in the last uh class. Here I'm showing like uh the number of patterns per style for the different styles in this uh corpus. uh but perhaps here uh we can see a cool thing of the algorithm that it returns also the position of the patterns not only the fact that a piece contain the pattern no so here I'm plotting it's called coverage this variable but it's actually like the probability function of encountering a pattern within a solo no so I normalize the length of the all the solos by style and the thin line is the average for the solos we You can see for instance that the concentrations of of patterns tends to be at the beginning and not in the end. This is intuitive because like the impro improvisors usually start their solos like in a more organized way taking thematic material from the original melody and then they start to do more crazy stuff. No. Then I'm also trying to build up a pattern dictionary of uh in an Iris corpus. These are the five most frequent patterns in the subset of minor rs in the corpus that I'm working on. So I don't know for a performance that is interested in playing this music perhaps the takeaway of this is okay you might want to start studying these patterns no because they appear quite often in the in the corpus. So better to first master them and for the future with Michael we want to study uh how the algorithm performs in a nice corpus of early music that he has and also we want to analyze some solos of Charlie Parker who was one of the most mathematical improvisers of the history and also for the close future I would like to see if this tool can be helpful for plagiaris detection. No, I suspect that two pieces that uh are very similar share also some similarities in their pattern structure. And that's all. Thank you very much, guys. Any questions? [Applause] >> By the way, just so you get a sense for your final things, that was 12 minutes. So, a tiny bit longer than what you have as a maximum. Cool. >> If any of you are interested in a particular thing of the algorithm, just drop me an email and we can talk about it. Okay. Thank you. >> So, what I want to do is I want to immediately start by putting some of uh thoughts to work. So, uh get with a partner next to you. You can um you can do that if over here we need a group of three or unless you want to participate with a friend or or jump over there. Well, jump over with uh with yeah with uh with Jordan if you do. Um and here are um six melodies. I'm going to play um each one of them. I'm going to play a a bunch of times also. So I'll tell you which one I'm playing and you're going to So and you can talk during this or wait till just after playing. You're going to tell me which melody is the um is most or least similar to A. [Music] Okay. And this is and and this is B. And I want you to know say why [Music] This is C. [Music] And I'm going to play A one more time to get it back into our heads. [Music] And now D. Oh, and by the way, this isn't a right there isn't a right answer one. Great. Now E. And now A one more time before doing F. So this is A again. [Music] And now F. Oops, that's F and E simultaneously, which is very different. Now F. [Music] Great. Talk amongst yourself. I want a ranking from your group. So, somebody write it down. >> And I want to know why. >> Okay, let's bring it all back together. Let's bring it all all back together. So, look at look at your look at your list and we'll we'll count like normal human beings. whichever one is the top in your list is one, whichever one's second to stuff. And we'll we'll vote by fingers first off for the general ranking. So when if you hold up the number of fingers that um you know, if it's closest, it's pulled up one finger. If not to, you know, whatever. And pull palm means I have no idea what that is. Okay. B. Oh, B's doing pretty pretty well. Okay. Not a lot of twos. A couple ones. C. Okay. A little bit more variety. Hold them up so other people can see also. Yeah. Look around the room. It's not just for me. Great. Great. Uh D. Oh, okay. We got a variety. No. No. Whatever. Great. Uh E. Oh, that's still see some palms or things. Great. And F. Okay. Oh, we have one one four. Good. Good. Uh, what did you what you what you you put it down? What do you guys Those who put uh F as four, what' you put as five? >> D. >> B. >> D. >> D. Okay. D. Good. Good. Great. Um, somebody who had who had uh C above B, justify your answer. Who who had C above B? There were a couple people. Yeah. Uh, go ahead, Jake. First. Yeah. Groups. Yeah. >> C was basically just a transposition whereas B like B changed a lot of the rhythms a bit. Like B H B I think fits the exact pitches a little better, but or aside from a couple places where it has some accidentals, but it changes the rhythm quite a bit. Whereas C is just a transposition. So for relative pitch especially, >> yeah, we waited transposition on. Okay, good. Good. I'm already hearing words that I that I'm liking and stuff. Great. Um, somebody who had it the other way around to justify your answer. Who who had it the other way around? Who had a bunch of people had B above C? Yeah, John. >> For like the kind like styling feel that A had. Um, because are similar between A and B is why we put a bit higher relative C. And when you take transposition into account, only part of C's coming transposed. So it doesn't even feel like it's like a full transposition. >> Great. Only part of it transpose. Yeah. Yeah. It kind of gets back on for a bit and then comes back off. Great. Uh okay. Who had who had D um who had D1 or two? Anybody? I can't remember. Yeah. Why' you have a one or two? >> Um because it basically has this. Sorry. It has all of the same notes in the right position or or basically. Yeah. Yeah. All you have to do is remove notes and then you get the same thing and I think with a few exceptions but that never happened in >> Okay, great. Who had uh D very low? >> Yeah. Go ahead. >> Sorry. Sorry. Can you speak a lot? >> Okay. >> Great. Um, and who had um did anybody have E above um above four? >> Okay, we have EX3 and we put it there because the pattern was very similar. I even though the melody wasn't all that close, we figured that count. >> So, so when you say pattern, what the rhythm the rhythmic pattern is about the same? Good. That's an inversion, right? >> Is it an inversion? No. No. That would I would I do that? >> I can't, by the way, I can't remember where melody I I think melody a comes from uh here on uh book and then gives there's uh some some search book and I should I should have my notes better and I'll try to make sure it gets annotated later. Um that had three other melodies. I tried this in the past and it was so obvious everybody had the exact same ranking. I had to agree with them, but I think this is better for making some arguments. Good. Who um who had F anything but F? Actually, somebody who gave F5. Jonathan, why would you did you give F5? >> Yeah. Why'd you give it five? >> I mean, it didn't have any noticeable similarity. Like at first it seemed closer to like E in like it might have been inversion, but then not really. is also completely different or not completely but like it's fairly different. >> Great. So, um I'm just going to point and we'll get some um some uh just what your ranking is. So, so say them from thing. Matthew, what was your your >> alphabetical order? >> BCDE F. Okay, good. BCDEF >> BCE DF. Great. Great. Um, what's it? >> Yeah, CBDF. >> CBD EF. Good. Any anything that feels like you're not being represented on the Hannah, what's your group? >> Um, I think I put CBD. >> CBDF. CBDF. Great. Super. Now, what I want you to do, we're not going to get through all of the exercises today, uh, but I think this is the most important part. What I want you to do is think about what ways, um, start thinking. I'll give you a little bit of things. What are some ways you can make sure that your computer system that is going to classify things by similarity follows your intuition of what is similar and not somebody else's intuition for what is similar. So that's going to be the main theme for the rest of this that we are intelligent people. We are intelligent musicians. We are uh make these choices and yet we are making differences on how far and how similar they are. So in fact I'm going to blake the screen and say the one the one uh takeaway from today's lecture I hope and from Kiki from all these things is that there is no right answer for the similarity between two melodies between the si similarity between two pieces. There may be wrong answers. I will not deny that that if if somebody said that uh that f was uh closer to I don't know than the same thing with one note changed or something I would you know think that that might be wrong then your problem program might be malfunctioning but there isn't a right answer and a lot of it the what different between good answers are what we think of u as important when thinking similarity there is a yearly competition I think it's been suspended since since co so I don't if it's restarted but um for the algorithm that can that can classify songs as the most similar and here is a place where I would say what are your ground truths how do we trust that you have gotten it right and are we just trying having to recreate the views I won't say biases but the views of the people who organize the um the conference and you know what's that going to do for us okay so I want you start thinking that um and I will tell you what f is beforehand. F is one that a lot of computers programs in fact what I like went aha during one of uh Kiki's uh al algorithms that could F is one that a number of algorithms especially older ones will classify as the most similar because what is F unlike any of the other lines F has every single note and every single rhythm if I did it right I was doing it in my head. Every single note and every single rhythm from A just order didn't matter. A is the counter set function, the unordered version, the the P. Well, yeah, the permutation does not matter version of F or F is the permutation does not matter version of A. Did I get it right? >> Right. >> That looks right. >> Looks right. Okay. So, I'm going to go quickly through some things I think you've probably seen before. Some some ways of measuring distance. You all learned this in some point. The um uklidian distance between two points. Take the square root of the x terms. Take square root of the y term add what difference squared plus difference squared square root. Right. Yeah. Square root of x^2 plus difference between x and difference between y. So anyone seen this u thing where the distance between these two points 3 comma 2 and 7 comma 8 is 10 who's soon taxi cab distance Manhattan distance uh we'll go with we'll go with taxi cab since not all of us have been to a Manhattan and had the joys of taking a taxi there and stuff what why this here the distance was 10 here it's approx that's not a negative sign this approximate sign approximately 7.2. Why is the distance greater here? Somebody who's >> triangle inequality. What's talk talk English to me for a second? Talk talk like you're talking to the uh your cab driver and who's who you're explaining to this. No, cab drivers are really smart. Talk but who may not have heard the time inequality. What what is represented by the term Manhattan distance or triang or um taxi cab distance? What's the notion intuition? >> Go along the axis. >> Go along the axis or that go along let's let's get get more literal. Let's one of the things that we science don't do so well is step back into the real world. What is the distance traveled? What constrains the taxi cab from not hitting distance of 7.2 but instead 10? Yeah, >> you go straight up and down to the side. >> You can only go straight up and down the side. You can only go on Let's go even further back. What can What in Manhattan? If you don't want to get arrested, you can only drive on >> streets. And the streets in Manhattan goal >> Yeah. They're orthogonal. There are these little lines. So, you are constrained in where you can go. So if there are constraints on your distance and the most common one is you can go up down left or right um can't always do that in Manhattan but um because of one ways but you know let's assume that we have certain constraints you can be brought down good I wanted to make sure that we all have that and so that we can start thinking about um that first off that what operations are allowed determines the distance metric What operations are around determines how far the distance are? What are some operations we do in music? That's the question. What what operations do we allow and not allow? Adam, >> we can look at MIDI. >> We can look at MIDI difference between notes. So therefore, we can take notes and bring them higher and lower, right? We can we can raise and lower notes. What are other things we can do? What are some some things you've ever done with a piece to make it a little bit different or interesting? >> Yeah, >> you can subdivide or combine. >> You can subdivide or combine notes. Maybe you can, maybe you can't, but yeah, quite often you can. This context where you can um other Yeah, other >> you can change durations. Great. Super. Super super. Um, how about this? What is which of these two chords are closer to the first one? The first one's going to be C major versus [Music] another little similarity problem. Second one was G major. The sorry the the first chord the first one it was G major second one I went from C major to >> C augmented great C augmented triad so those are two things which one who votes that from going from C major to G major is closer who votes that C major and C augmented are closer people. Okay, great. So, a lot of it has to do with your thought um about, you know, well, one one you're the augmented, you're only changing one note and you're only changing by a half step. The minimum distance in our Manhattanized musical world of MIDI and piano keyboards, right? That is their minimum distance is one half step there. Not for all music in the world. Great. C major to G major, you're also just moving one. If you think of something this way, you were moving one distance in what space? >> Oh, what's that word? >> Circle. >> In circle of fifth space, C and G are about as close as you can get without being the identity. C and F probably the other way. Although, I don't know, maybe it's a one-way circle of fifths. You only go around uh one direction or another. Great. based on the time. I don't not going to go through all these other measurements of distance that people can do. Who has heard of Earth Mus distance? That is the amount of work that it takes to move one mound of things over to another place and um sometimes you optimizing depending on how um how much it costs to move distance and how much it costs to move material. Um you can end up with different results. Um, I always, this was one of the charts I think I showed early in the semester, uh, is here's one place where Earth mover distance might be a good use of things, uh, of distances. And then, uh, Levvenstein or edit distance is one that was mentioned in, um, do you use it in your work? >> Yep. So, that's that's what you're talking about. Um so the idea of how to change the word Hondaai into Honda um and uh no no international East Asian uh politics uh please for a second and um we'll you can think of every time okay H and H are the same so it has a cost of zero or we can delete the H and start on O and we have a cost of one but we can kind of find the pattern of as we change we're going to insert a Y after the H we're going to substitute a U for an O. This should be symmetrical the other way around different operations. N is the same. So that's good. D is the same. So it doesn't cost anything. So our cost function goes here. And so we're trying to find the minimum cost from going from one end to another. Uh we don't have time to go through all of the algorithms for this but uh Levvenstein distance edit distance has a lot of good um qualities that makes it useful for a lot of musical similarity tasks uh just so that you know you can say your professor at least uh put the algorithm up on the head for a second. But more importantly, I think a lot of times is thinking about um the particular costs of things in a musical spa in a musical world. So for instance, is deleting what we're trying to think about two pieces, two uh melodies. Um one of them deletes the first note. What would you call the cost on that? Maybe maybe one. But then if it doesn't make up the rhythm, the total rhythm later and everything from here on is going to be off and it's one line within a orchestral piece, that might be a higher um cost. Maybe um some and the classic debate is whether changing a note is that the same or changing a letter uh um in something like this. Is this the same? Does this cost one or does this cost? Well, one way you can change a letter is you delete it and then you add a new letter back with a cost of two. And these are these are things that come up quite a bit in similarity search. Um, and just really want to say that it comes up a lot in music. Don't borrow your distance metric from somebody else. Different ones might be used for different situations. So, the distance between dog and gate. Um, well, we can substitute d for g. Um, I don't know, maybe or maybe we add other things, but I'm going to assert that in different some situations the distance might be two between dog and gut. What we do is we use the substitute uh closely related pet function for cost of one. So dog becomes cat and then translate English to Spanish might cost one. And if you think about large language learning models and things, you might want to have functions like this. And in fact, um this is not a digital humanities text class, but if it were and we were doing computation, we'd definitely be talking about an algorithm called wordtovect, which uh was one of the uh earlier successful algorithms for trying to predict what words are similar to other words, what words are synonyms. So you can create a kind of cost function for uh that is for this word is a synonym for this one that has been substituted that is lower than this then this sentence is different from this one because um it's using a completely different concept. Um I'll skip that. So when we're thinking about um these distances and these weird things like substitute dog for cat on low cost, substitute cat for gateto at low cost. What's the term that we spent a lot of time, maybe even too much time if it felt like at the time talking about earlier in the semester that helps to think about things that are not the same but might be closely related to each other >> equivalence. >> Equivalence or equivalence classes. Yes. So, one of the things you might want to do is define what equivalence classes it could be. I mean I I think last other times I've given the exact same melody up an octave and everybody immediately said oh that is basically the same thing. So everybody was very quickly putting in an O equivalence class things. So I wanted to make sure that we had that and um yeah and so once you have these um these distances we tend to go through and this is if you're in a biology class you'll spend a lot of computational biology a lot of time on this sequence alignment uh a kind of distance metric where you're trying to find the minimum distance between two things that you believe might represent the same type of thing or you might say it's innocent until proven guilty. we'll first try to see if they can be changed into another thing at a low cost and then discard once we realize the cost cannot be u minimized. Um I will say that algorithms that can be shortcircuited that you can prove at a certain point you can't do better than this cost will save speed up a lot of your um run times because once you know you might say um you know that there's no way that this could be more could be better than um 20% or could be yeah there's no way that this could possibly be better than 90%. % similar to this. So, I'm going to stop looking at the rest of the piece or whatever your cut off. Um, one of the classic uh things for sequence alignment is trying to find uh this is Google's uh data set they released at the height of Britney Spears um popularity of all the number of searches that they believed were uh trying to find the top left one, Britney Spears. Um, actually really really impressed that that the number of correct spellings um of a hard name to spell uh outweighs the rest. Anyhow, that's not what I'm talking and um the people who are who are really really good at this and like anytime I'm trying to figure out a similarity sequence alignment or similarity task that I don't know is to look at the people who are trying to align um base pairs in biology or trying to align genes um so that um because they have many many uh many many options Okay. So, I'm just going to keep pounding this term in and as many different ways I can do uh all the things that we're just working like Honda Britney Spears uh jeans those are all strings but we work on notes and and clefts and things like notes and stuff but things like that. So, how do we get them in? Uh so, this is great to get this from two different people. Same thing. We use things called hashes which are very similar to uh the concept of viewpoints um to the rescue. So that try to convert things note hash and note we might say that here our equivalence class is all notes that are names with octave and so we might hash a stream by just joining all the hash notes for all the notes in there. Um, you will find in music 21 if you're working on it, there's a bunch of tools for this already there in music21.sarch, a module that we have not talked about now and we will not talk about again, but if you're doing a lot of searching, it's probably worth reading the um reading the uh module reference for it. I think that there might be a user's guide, but I can't remember if I finished it or if it just trails off after a few words. So we might take a a string a string convert it to a stream that's hard to say very fast and um translate it and we might have some sort of hash function that tries to make everything into um into an asky character. Um though there's no reason that everything needs to be turned into a string like name with octave. um in a lot of ways uh and um strings are just arrays of ints if you know we're talking about that the um that the you know that a is it lower no lowercase a is generally represented internally as anyone remember number remember number remember number remembernumber remember number remember numberremember number remember number remembernumber remember number remember number >> what's that >> 96 97 or 96. I can't remember. 96. Yep. In capital A. Is that one 60? >> 65. Okay. So, some people some people know these. I used to have them all top of the head. So, all the letters you're doing have a particular representation. And mo back in the bad bad days of the 60s and 70s, different computers would have different representations for this. And then we all agreed on the same representation for uh letters. And then we remembered that there are other things in the uh um other characters in the world. That looks too much like an a how do I do a jin or something or an alpha beta things like that. And and then for a while we had a big problem that they weren't all converting the same thing. Anyhow, degression aside, maybe we'll get to the point where we can start converting things besides MIDI numbers and notes into something um more standardized because right now the MIDI numbers is basically the only standardized notes, which is probably why MIDI keeps um uh being used for a lot of computational projects. So the hard part is always finding out what numbers we should use to represent a note. So if we're going to convert name with octave and we want to make a string and then we want to make it a number and then we want to have a whole bunch of numbers. What have we just recently seen that looks like a tool to take a score or a part or something and works like a hash or a viewpoint that tries to convert it to a bunch of numbers. Not asking you to think too far back but farther back than today. Yeah, I'm >> getting a feature representation. >> Get extracting features. Getting a feature representation. Yeah. So feature extraction and this kind of viewpoint searching are very go hand inand with each other. So if you're it's partially why on you know once you finish up a search function you're just going to want to probably try to see if um AI or machine learning can do it better because you have everything ready to go for it. But sometimes what we extract is different from others. Um I want to give a little bit of a caution that um uh great uh back then course six you had to do a little final a final project called the UAP and we thought that making these viewpoints making a hashing system for music 21 for comparisons would be a nice um would would be a nice senior project and then we both realized that no it's uh it's a lot bigger than we thought and a lot more complex than we thought and so it needed to be an MN And Emily Zang was great at creating this and great we did a really great Menge project and then we realized no this this really needs to be a PhD project. We did not continue on. You can work there are so many difficult parts of um hash algorithms because you want to think about things like um yeah we'll not get to it. um going all the way back to the beginning. Um how can we create a viewpoint or something that allows D not to be totally totally different for anybody who didn't put D as the last of all possible results? What kinds of hash what kinds of numbers would we need to represent a piece on to make D not the worst and to really make sure that F isn't the best. So that's going to be our last five six minutes of class. I want you to talk five minutes of y'all talking with each other and five minutes of y'all talk and talking to me. So what kinds of what kinds of feature extractions? What kind of hash function? What kind of viewpoints? These are all slightly different concepts, but they're all in the same area. What kinds of equivalence classes will you need in order to make this happen? Go ahead. Okay. I hear I hear words um continuing but but less frequently. Let's talk about what are some of the ways that um that people thought to create a strategy that doesn't make D and F about the same. >> Yeah. Go ahead. >> Look at the sequence of like local maxima and minima. >> Local maxima and minima. Okay. I think I know what you talk about, but let's give me a little example. What um let's talk about A. What do you what do you >> So you could maybe argue that the C is the local minimum and then the F is higher than both of its neighbors. So it's a maximum and then the D is a minimum. The E is a maximum and then the D and C after that are not really anything until you hit the A on the 16th note. >> Cool. So yeah, we're just looking at every time the direction changes of the pitches. Great. And compare that. So beginning A has D F D E here we have D F D tiny bit different D F but then going down to a little bit different but it's at least giving some numbers we have um yeah does always the question is does your current streak end when you hit a rest or not and maybe it depends on how long the rest is. So good. Other other strategies? >> Adam, >> I would look at where offsets are the same and then check if their notes are the same or not. >> Great. So, we're gonna look at offsets that are the same and see if notes are the same or not. That works really well. And what I'd love to do if if this were um what do you call the the generalized adversarial problem set, the GAN problem set, where one one team has to solve the problem, the other team has to keep giving them things that break that. Uh, and I think it was a great idea and I think it would work in general, but I could generate something like um where all I insert is let's insert a 64th rest at the beginning and then put all random notes and you're going to end up with and then maybe we'll put one note that's the same at the end and you could end up with 100% of the notes on the same offset are the same. Uh I think you I think we we would really work in the real world but but we might want to always think about something like that too. Great great idea John then >> ours kind of builds up on Adam. So >> you kind of start kind of builds up on what Adam was saying. >> Um but first you like take a look at the notes and kind like kind of do like a set kind of like crossover between like a set of D's and A notes and then like A and S notes. So both those would still like show up like relatively high and then like compare the offsets which we want to like take half of it but still keep D relatively high. >> Okay. >> Super. When we're talking about offsets, are we talking about um what kind of offsets >> like um where in the guess like within the two measures? >> Great. Yeah. Where in the two measures where in the measure um we sometimes want to do global offset from the beginning of the measure, but then you can't identify similar phrases or all it takes is put a repeat, put the first four measures, repeat it once and suddenly the whole rest of the piece is different. So yeah, that's great. also similar but a little duration based I guess like if you just start at time= 0 and go all the way through the piece like anytime the two pieces have the same pitch if you score so like the F that they both have on beat two would be like a quarter point because >> the F the F that they both have on beat two would be a quarter point because >> shared duration is only for that 16th note. >> Gotcha. So we look at shared. Great. I like that a lot. Did anybody try to come up with equivalence? Yeah. Yeah. The contour ends up being a kind of a new equivalence class that we hadn't talked about which kind of works out in thinking that everything that doesn't change directions is a kind of passing tone even though not in the proper music theory term and so can kind of be ignored. By the way, I the one I use quite often is I'm just going to look on down beats or strong or on beats and ignore everything else. Um, and that would works pretty well for a lot of a lot of things.