Do ChatGPT Detectors Work? Let's Find Out!

Key Takeaways

hey what's up Matt wolf here I've recently been hearing a lot about these GPT chat detectors you know these tools where you could copy and paste an article or some sort of text blurb into this tool and it will tell you if it was written by chat GPT I know a lot of people are really worried about Google and how they're going to detect this stuff and whether or not they can use AI content inside of their blogged posts and their content marketing is Google going to de-rank it that sort of thing so I got really curious and I recently came across this Reddit thread here which has actually been deleted and the person who posted it is no longer a user on Reddit but they claimed that originality.ai is a huge scam now even though this was deleted and they took it down I don't know if they saw the error in their ways or if they just flat out got removed from Reddit I don't know but it got me thinking how well do these detectors work now I have some content out there on the web that I know was helped with AI and then I have some content out there on the web that I know I personally wrote myself so I thought I would put some of these AI detectors to the test I've never used one of these tools so I'm going to test them out for the first time in this video and you're going to see my process here and I don't know what the results are going to be myself so let's go ahead and dive into this so I went to Future tools and I typed in detector up here and these are the three most popular AI detectors that I've come across we've got originality.ai we've got detect GPT and then we've got the gpt2 output detector which is a free one supposedly detects gpt2 output but also still works fairly well for gpt3 output so I'm going to put these three tools to the test and see what we come up with here so the first one that I want to test out is the gpt2 output detector this is a completely free one that you can find over at open AI dash openai detector.hf dot space the easier way to get to it would be to go to the Future tools website and just you know search GPT to Output detector and you'll come across it but I'm going to test this with some content that I know was written by AI as well as content that I know was not written by AI so funnily enough most of the descriptions on the Future tools website was written by AI so this description for detect GPT no this was actually written by AI let's post it into the gpt2 output detector and it's going to predict real quick and let's see what it says so it says 96.67 real 3.33 percent fake this originality.ai also written here let's go ahead and copy this and paste this one in here and let's see what it says so this one says 97.56 percent real 2.44 fake so this was written with gpt3 and since this is a gpt2 detector it didn't seem to notice that it was fake all right so now I'm going to jump over to mymatwolf.com blog this is all content purely written by me no AI involved on this website I'm gonna go to this one called everything is changing from October of 2022 I know there was absolutely zero AI involved so let me just go ahead and grab a couple paragraphs at random here and copy them over to our GPT to Output detector paste them in let's see what it thinks 99.97 real 0.03 percent fake so it knows that that one's pretty real as well alright so gpt2 output detector probably can't be too relied on since it is designed for gpt2 Content not necessarily gpt3 let's go ahead and try one more thing let's go into chat GPT and prompt something real quick and see what it gives us when we plug it into the GPT to Output detector right a story about a wolf that learns to use a computer and then makes videos about how to use artificial intelligence for productivity and making money let's go ahead and have it write a prompt that we know is 100 AI generated all right let's go ahead and copy this story that it just created and put it in the gpt2 output detector and this will be our final test with this one let's see what it says all right 0.2 percent real 99.98 fake so the stuff that's coming right out of chat GPT the gpt2 output detector still seems to be able to detect it pretty well all right so now let's test the detect GPT tool here let's click on visit to Tech GPT this one is a Chrome extension so I'm going to go ahead and install this extension here add it to Chrome and now we've got it up here let's use this same originality.ai text because I know for sure that this is AI generated copy this go ahead and click this thing up here let's go ahead and paste it in here click check content according to this is 2.44 AI 97.56 percent human so we know that's not true let me pick another tool as well I know this Auto draw this description here is also using GPT let's go ahead and paste this content in and check it and this one is showing 57.78 AI 42.22 percent human but says we are 57.78 confident that this is AI generated so a little more than 50 50. all right so let's go ahead once again let's go ahead and grab this same selection from my blog post which I know is 100 human generated split up into our content Checker here check the content and this is saying 99.93 human zero percent AI so it's getting my blog post right all right and then one last check let's copy this here that we knew we pulled straight in from chat GPT paste this story in here check the content and it knows it's AI so this one seems to work pretty well as well it seems like the shorter text it has a little bit harder time detecting it these one paragraphs here from my future tools site it's kind of 50 50 on whether or not it's detecting it real well the last one I want to play around with is the one that actually charges for you to use it I'm hoping that this one is the best one that's out there because they charge one cent per 100 words per scan now I don't have an account so I'm going to go ahead and sign up for one right now and we'll run the same test through this one so I'm just going to quickly jump through the sign up process here I'll go ahead and purchase some credits I don't have any credits right now so let's go ahead and grab some so let's go ahead and run our first scan here my first scan I am going to grab my auto draw text here come over to originality.ai paste it in go ahead and detect AI sure why not let's do a Jerusalem check as well it'll cost me two credits so go ahead and scan now according to this it's 93 original and seven percent Ai and no plagiarism detected so interesting result because I know for a fact that that was generated with AI and it's saying that it was original all right let's go ahead and do a new check here let me grab the originality.ai this was definitely 100 written by AI let's go ahead and run a check on this one both detect Ai and plagiarism scan this guy this one says a hundred percent original zero percent AI there we go again I know originality AI was 100 written by AI I know for a fact and I didn't run it through any other tools I didn't do any spinning I didn't do any extra stuff this was just purely 100 generated through AI I promise you I wouldn't have used the wording it is available for a low price of one cent per one hundred dollars all right so now I'm gonna go ahead and grab this text here from my blog post which I know for sure is not Ai and let's go ahead and run a new scan let's paste this guy in here get it now for both Ai and plagiarism and see what we get so this one says 98 original two percent AI so I got a better score on the one that I know for sure was a hundred percent AI than the one that I know was a hundred percent not AI interesting so let me let me grab a different Chunk from the same article copy this I'm gonna try one more here this is the one that's actually charging you money to use so I want to be fairly thorough with it 99 original one percent AI so I mean you know it sees it as original but the fact that it thinks it's one percent AI is still interesting to me so let's go ahead and check this story about the wolf who makes money online and see what score it gives me if this thing has wants any credibility it better not show it as original so this one it detected a hundred percent AI so it does seem to be seems to be working for the most part although it didn't seem to do very well with some of these shorter text blurbs here let me see how many credits have I used so I bought twenty dollars worth of credits here and I've used whatever that adds up to be 10 12 14 16 credits out of my two thousand so here's what I'm gonna do one last time let me open up gpt3 here and let's have this open AI write me a gpt3 thing and let's see what it gives me explain to me how gpt3 Works alright so we've got a few paragraphs here written by gpt3 let's go ahead and copy this pull it into originality.ai paste it and scan it now 94 original six percent Ai and even found a little bit of plagiarism so that's kind of interesting it's saying that it plagiarized this blog post here let's go ahead and take the same prompt let's plug it back into this GPT detector plugin let's paste that in here and check the content here 99 AI two percent human those are the comparisons I played around with detect GPT gpt2 output detector originality.ai after playing around with these three my feelings are that gpt2 output detector is probably the least accurate just because it's kind of looking for the older gpt2 technology but it still seems to be able to detect stuff that came out of chat GPT originality.ai seems to be the number two it kind of thought some of my blog content had a little hint of AI to it it also thought some of my AI content had a hint of no AI to it which I found kind of interesting and then the one that I would pick as the winner of these three that seems to be the best at detecting is this detect GPT plugin which is a free Chrome plug-in that you can use so I installed this one if I'm going to be searching for GPT I'm going to be using this detect GPT now I might use originality because I got 19 worth of credits now because I couldn't buy less than that so I'll probably use this from time to time but the one that seems to be the most accurate and find the best from my experimentation has been this Chrome extension which was free there you go those are my thoughts you want to find more tools like this make sure you check out futuretools.io adding new tools to it every single day right now we've got 350 total Tools in here it has an added bonus if you scroll down to the bottom and you put your email in here you'll be able to grab the free AI income database which is all sorts of cool ways to make money using Ai and you'll also be added to our Weekly Newsletter where every Friday I send you 5 five of my favorite tools that came out that week three interesting news articles three cool YouTube videos and one new way to make money with AI you're going to want to be on that mailing list it's growing really fast enjoying almost 2 000 people who have already joined that Weekly Newsletter and I hope you enjoyed this video go do some AI stuff have fun out there kids see in the next one [Music]