• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

ruby爬虫高并发方案

原作者: [db:作者] 来自: [db:来源] 收藏 邀请

em-synchrony

em-http-request

typhoeus

eventmachine

 

 

em-http-request example

Simple client example

  EventMachine.run {
    http = EventMachine::HttpRequest.new('http://127.0.0.1/').get :query => {'keyname' => 'value'}, :timeout => 10

    http.callback {
      p http.response_header.status
      p http.response_header
      p http.response

      EventMachine.stop
    }
  }

Multi-request example

Fire and wait for multiple requests to complete via the MultiRequest interface.

  EventMachine.run {
    multi = EventMachine::MultiRequest.new

    # add multiple requests to the multi-handler
    multi.add(EventMachine::HttpRequest.new('http://www.google.com/').get)
    multi.add(EventMachine::HttpRequest.new('http://www.yahoo.com/').get)

    multi.callback  {
      p multi.responses[:succeeded]
      p multi.responses[:failed]

      EventMachine.stop
    }
  }

Basic-Auth example

Full basic author support. For OAuth, check examples/oauth-tweet.rb file.

  EventMachine.run {
    http = EventMachine::HttpRequest.new('http://www.website.com/').get :head => {'authorization' => ['user', 'pass']}

    http.errback { failed }
    http.callback {
      p http.response_header
      EventMachine.stop
    }
  }

POSTing data example

  EventMachine.run {
    http1 = EventMachine::HttpRequest.new('http://www.website.com/').post :body => {"key1" => 1, "key2" => [2,3]}
    http2 = EventMachine::HttpRequest.new('http://www.website.com/').post :body => "some data"

    # ...
  }

Streaming body processing

Allows you to consume an HTTP stream of content in real-time. Each time a new piece of content is pushed to the client, it is passed to the stream callback for you to operate on.

  EventMachine.run {
    http = EventMachine::HttpRequest.new('http://www.website.com/').get
    http.stream { |chunk| print chunk }
  }

Streaming files from disk

Allows you to efficiently stream a (large) file from disk via EventMachine's FileStream interface.

  EventMachine.run {
    http = EventMachine::HttpRequest.new('http://www.website.com/').post :file => 'largefile.txt'
    http.callback { |chunk| puts "Upload finished!" }
  }

Proxy example

Full transparent proxy support with support for SSL tunneling.

  EventMachine.run {
    http = EventMachine::HttpRequest.new('http://www.website.com/').get :proxy => {
      :host => 'www.myproxy.com',
      :port => 8080,
      :authorization => ['username', 'password'] # authorization is optional
  }

SOCKS5 Proxy example

Tunnel your requests via connect via SOCKS5 proxies (ssh -D port somehost).

EventMachine.run {
  http = EventMachine::HttpRequest.new('http://www.website.com/').get :proxy => {
    :host => 'www.myproxy.com',
    :port => 8080,
    :type => :socks
}

Auto-follow 3xx redirects

Specify the max depth of redirects to follow, default is 0.

  EventMachine.run {
    http = EventMachine::HttpRequest.new('http://www.google.com/').get :redirects => 1
    http.callback { p http.last_effective_url }
  }

WebSocket example

Bi-directional communication with WebSockets: simply pass in a ws:// resource and the client will negotiate the connection upgrade for you. On successful handshake the callback is invoked, and any incoming messages will be passed to the stream callback. The client can also send data to the server at will by calling the "send" method!

  EventMachine.run {
    http = EventMachine::HttpRequest.new("ws://yourservice.com/websocket").get :timeout => 0

    http.errback { puts "oops" }
    http.callback {
      puts "WebSocket connected!"
      http.send("Hello client")
    }

    http.stream { |msg|
      puts "Recieved: #{msg}"
      http.send "Pong: #{msg}"
    }

    http.disconnect { puts "oops, dropped connection?" }
  }


鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap