用twisted来进行批量网址处理

代码来自：http://technicae.cogitat.io/2008/06/async-batching-with-twisted-walkthrough.html

文章由浅入深，最终给出了一种简洁漂亮的代码形式，终于不会twisted（绕）到自己。

 

直接贴最终版代码：

 

from twisted.internet import defer, reactor, task
from twisted.web.client import getPage
 
maxRun = 2
 
urls = [
'http://twistedmatrix.com',
'http://yahoo.com',
'http://www.google.com',
]
 
def pageCallback(result):
   print len(result)
   return result
 
def doWork():
   for url in urls:
      d = getPage(url)
      d.addCallback(pageCallback)
   yield d
 
def finish(ign):
   reactor.stop()
 
def test():
   deferreds = []
   coop = task.Cooperator()
   work = doWork()
   for i in xrange(maxRun):
      d = coop.coiterate(work)
      deferreds.append(d)
   dl = defer.DeferredList(deferreds)
   dl.addCallback(finish)
 
test()
reactor.run()