How can I optimize for "deep web" crawling?

Key Takeaways

we have a question from Brighton Danny asks what are Google's plans for indexing the Deep Web and are their best practices for Form Construction to optimize for this great question um we recently published a paper in vldb which I believe stands for very large databases um that talks exactly about our criteria all the ways that that we try to do it safely so that if people don't want their forms to be crawled we won't crawl them um and so there are some very simple things you can do so rather than having text that has to be filled out like a zip code if you can make it a drop- down for example that's much more helpful um if you can make it so that it's not a huge form with 20 things to fill out but more like one drop down or two drop downs that's going to be a lot easier as well I definitely encourage you to go read the paper there's nothing super duper confidential in it um and of course if you can make it so you're not part of the Deep Web if you can take those pages that's your database and have an HTML sit map so that people can reach all the different pages on your site by crawling through categories or you know Geographic areas then we don't have to fill out forms and Google's a pretty good company about being able to index the Deep Web through forms uh but not every search engine does that and so if you can expose that database somewhere where people can get to all the pages on your site just by clicking not by submitting a form then you're going to open yourself up to even wider audiences so if you can do that that's what I'd recommend but if you can't do that then I'd say uh check out this paper from the vldb conference where the the team talked about it in more detail