β

linux 使用curl命令访问url并模拟cookie

阿辉的博客 2040 阅读

1. 目录

linux下通过命令访问url的方式有多种,主要如下

2. 1. elinks

elinks – lynx-like替代角色模式WWW的浏览器

例如: elinks –dump http://www.baidu.com

2.1. 2.wget

这个会将访问的首页下载到本地

[root@el5-mq2 ~]# wget http://www.baidu.com

3. 3.curl

curl会显示出源码

curl http://www.baidu.com/index.html

4. 4.lynx

lynx http://www.baidu.com

5. 5.curl使用实践

现在有个需求,因为服务器在收集访问数据,抓取cookie中的value,模拟url访问时需要带上cookie参数,curl命令刚好能完成这个功能。
首先查看帮助:
curl -h
-b/–cookie <name=string/file> Cookie string or file to read cookies from (H)
-c/–cookie-jar <file> Write cookies to this file after operation (H)
–create-dirs   Create necessary local directory hierarchy
–crlf          Convert LF to CRLF in upload
–crlfile <file> Get a CRL list in PEM format from the given file
可以使用-b参数来完成,具体使用如下:
curl b “key1=val1;key2=val2;”
或直接使用文件
curl -b ./cookie.txt
编写测试实例:
curl -b “user_trace_token=20150518150621-02994ed9a0fb42d1906a56258e072fc4;LGUID=20150515135257-a33a769c-fac6-11e4-91ce-5254005c3644” http://10.10.26.164:1235/click?v=1&logtype=deliver&position=home_hot-0&orderid=10197777&userid=1942556&positionid=148&url=http%3a%2f%2fwww.lagou.com%2fjobs%2f317000.html%3fsource%3dhome_hot%26i%3dhome_hot-5&fromsite=http%3a%2f%2fwww.lagou.com%2fzhaopin%2fAndroid%3flabelWords%3dlabel%26utm_source%3dAD__baidu_pinzhuan%26utm_medium%3dsem%26utm_campaign%3dSEM&optime=2015-06-15_20:00:00
发现这样还是不可以,url附带的参数取不到。使用-d 参数传递url参数,使用-G 把请求方式配置为GET就OK了,如下:
curl -b “user_trace_token=20150518150621-02994ed9a0fb42d1906a56258e072fc4;LGUID=20150515135257-a33a769c-fac6-11e4-91ce-5254005c3644;LGSID=20150518150621-02994ed9a0fb42d1906a56258e072fc4;LGRID=20150617230732-4ea87972-1580-11e5-9a88-000c29653e90;” -d “v=1&logtype=deliver&position=i_home-1&orderid=10197777&userid=1942556&positionid=148&url=http%3a%2f%2f www.lagou.com%2fjobs%2f317000.html%3fsource%3dhome_hot%26i%3dhome_hot-5&fromsite=http%3a%2f%2fwww.lagou.com%2fzhaopin%2fAndroid%3flabelWords%3dlabel%26utm_source%3dAD__baidu_pinzhuan%26utm_medium%3dsem%26utm_campaign%3dSEM&optime=2015-06-15_20:00:00 ”    -G http://10.10.26.164:1235/click
想要获得response返回的cookie怎么办,使用’-c’参数,同时可以使用-b filename用文件方式表示cookie,配合-c使用更方便
可以先用-c 命令生成一个cookie文件作为模板,再修改这个文件作为-b 参数的文件名。
使用如下:
curl -b c1.txt -c c2.txt -d “v=1&_v=j31&a=406405635&t=pageview&_s=1&dr=http%3a%2f%2fwww.sogou.com%2ftuguang&dl=http%3A%2F%2Fwww.lagou.com%2F%3futm_source%3dad_sougou_pingzhuan&ul=zh-cn&de=UTF-8&dt=%E6%8B%89%E5%8B%BE%E7%BD%91-%E6%9C%80%E4%B8%93%E4%B8%9A%E7%9A%84%E4%BA%92%E8%81%94%E7%BD%91%E6%8B%9B%E8%81%98%E5%B9%B3%E5%8F%B0&sd=24-bit&sr=1600×900&vp=1583×291&je=1&fl=18.0%20r0&_u=MACAAAQBK~&jid=&cid=1312768212.1431333683&tid=UA-41268416-1&z=1204746223”    -G  http://192.168.52.130:1234/collect
生成的c2.txt内容如下:
# Netscape HTTP Cookie File
# http://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.192.168.52.130 FALSE / FALSE 1757574737 user_trace_token 20150914151217-eedd019e-5aaf-11e5-8a69-000c29653e90
192.168.52.130 FALSE / FALSE 1442217595 LGSID 20150914152955-652a13c5-5ab2-11e5-846d-000c29653e90
192.168.52.130 FALSE / FALSE 1442217595 PRE_UTM
192.168.52.130 FALSE / FALSE 1442217595 PRE_HOST www.huxiu.com
192.168.52.130 FALSE / FALSE 1442217595 PRE_SITE http%3A%2F%2Fwww.huxiu.com%2Ftuguang
192.168.52.130 FALSE / FALSE 1442217595 PRE_LAND http%3A%2F%2Fwww.lagou.com%2F%3F
192.168.52.130 FALSE / FALSE 0 LGRID 20150914152955-652a1630-5ab2-11e5-846d-000c29653e90
192.168.52.130 FALSE / FALSE 1757574737 LGUID 20150914151217-eedd0624-5aaf-11e5-8a69-000c29653e90

转载: AI 数据 » linux 使用curl命令访问url并模拟cookie

作者:阿辉的博客
系统 网络 集群 数据库 分布式云计算等 研究
原文地址:linux 使用curl命令访问url并模拟cookie, 感谢原作者分享。

发表评论