如何用R语言爬取网页表格数据节省一天工作时间

Python018

如何用R语言爬取网页表格数据节省一天工作时间,第1张

方法/步骤fromurllib.requestimporturlopen用于打开网页fromurllib.errorimportHTTPError用于处理链接异常frombs4importBeautifulSoup用于处理html文档importre用正则表达式匹配目标字符串例子用关于抓取百度新闻网页的某些图片链接fromurllib.requestimporturlopenfromurllib.errorimportHTTPErrorfrombs4importBeautifulSoupimportreurl="/"try:html=urlopen(url)exceptHTTPErrorase:print(e)try:bsObj=BeautifulSoup(html.read())images=bsObj.findAll("img",{"src":re.compile(".*")})forimageinimages:print(image["src"])exceptAttributeErrorase:print(e)importjava.io.BufferedReaderimportjava.io.IOExceptionimportjava.io.InputStreamReaderimportjava.net.HttpURLConnectionimportjava.net.MalformedURLExceptionimportjava.net.URLpublicclassCapture{publicstaticvoidmain(String[]args)throwsMalformedURLException,IOException{StringstrUrl="/"URLurl=newURL(strUrl)HttpURLConnectionhttpConnection=(HttpURLConnection)url.openConnection()InputStreamReaderinput=newInputStreamReader(httpConnection.getInputStream(),"utf-8")BufferedReaderbufferedReader=newBufferedReader(input)Stringline=""StringBuilderstringBuilder=newStringBuilder()while((line=bufferedReader.readLine())!=null){stringBuilder.append(line)}Stringstring=stringBuilder.toString()intbegin=string.indexOf("")intend=string.indexOf("")System.out.println("IPaddress:"+string.substring(begin,end))}

如果以后抓取网页碰到动态加载的数据，可以考虑使用 phantomjs 如果想更暴力直接开出一个有界面的浏览器做各式各样的操作，达到ajax无阻碍的，可以用Selenium + Beautifulsoup