distributed crawler
Use scrapy_redis
dupefilter to remove duplicates:
request_fingerpint() request fingerprint
Use haslib.sha1 for request.method, request.url, request.headers, request.body is encrypted
40-character hexadecimal string, the disadvantage is too much memory
Optimization:
Change the default dupefilter in scrapy_redis to :
Use BloomFilter to remove duplication, small memory, faster
1. Use the following settings in the project:
#Enable storage request queue scheduling in redis.
SCHEDULER = " scrapy_redis.scheduler.Scheduler "
?
# Make sure that all spiders share the same duplicate filter through redis.
DUPEFILTER_CLASS = " scrapy_redis.dupefilter.RFPDupeFilter "
?
#Store the requested data in redis for post-processing.
ITEM_PIPELINES = {
' scrapy_redis.pipelines.RedisPipeline': 300
}
?
Specify the connection to Redis The host and port used at the time (optional).
REDIS_HOST = 'localhost'
REDIS_PORT = < span class="cm-number">6379 span>
Specific usage reference link: https://github.com/rmax/scrapy-redis
2. Use
import scrapy
from scrapy_redis.spiders import RedisSpider # Use RedisSpider
?
from reading_network.items import ReadingNetworkItem
?
# spider in the basc template Modifications based on:
class GuoxueSpider(RedisSpider): # Change to inherit RedisSpider
name = 'guoxue'
span class="cm-variable">redis_key = 'start_urls' # Set the key to store the crawling address to start_urls, and the value to start Starting url
? span> span>
Note: The running command is different from the previous one. Order:
cd spiders directory
scrapy runspider guoxue.py (crawler file)
3. Data persistence, stored in mysql
import pymysql
from redis import Redis
import json
?
# Take out the data from redis and save it to mysql
conn = pymysql.connect(host='127.0.0.1', port=3306, user=< span class="cm-string">'root', password='450502',
charset='utf8', database='spider1')
cursor = conn.cursor()
redis_conn = Redis()
?
?
def insert_mysql():
while 1 :
# Get data from redis
# print(redis_conn.brpop('guoxue:items) ', timeout=60)) # There will be a 60s delay when it is not available, and an error will be reported after the timeout, (b'guoxue:items', b'{'name':'xxx')')
try:
_, data = redis_conn.brpop('guoxue:items', timeout=60)
except:
cursor.close()
conn.close()
break
?
# Convert to python objects
data = json.loads(data, encoding ='utf-8')
keys = ','.< span class="cm-property">join(data.keys())
values = ','.j oin(['%s'] * len(data)) #'%s, %s, %s'
sql = f'insert into guoxue({keys}) values({values})'
try:
cursor.execute(sql, tuple(data.values())) # Pass a tuple to replace %s
conn.commit()
except Exception as e:
print(e)
conn.rollback()
?
?
if __name__ == '__main__':
insert_mysql() span> span> span> span> span>
p>
Use distributed
1 .Modify configuration
-
The redis configuration file redis.windows.conf needs to be modified under window, restart redis
bind 127.0. 0.1 changed to 0.0.0.0
protected-mode no # Close protected mode
requirepass 123456 # Remote access to other servers requires a password
-
Redis modify the configuration file redis.windows.conf under linux, restart redis
bind 0.0.0.0
requirepass 123456
Other servers connect to local redis: redis-cli -h local ip
A password is required to access: auth 123456
-
-
settings need to change the server address of the shared redis database to:
REDIS_URL = 'redis://:[email protected]local ip:6379' # is not a local ip127.0.0.1
2. Use
< span> is basically the same as above, just using crawlspider; the inherited class is changed to RedisCrawlSpider
import scrapy
from scrapy_redis.< span class="cm-property">spiders import RedisCrawlSpider # Use RedisCrawlSpider
?
from reading_network.items import ReadingNetworkItem
?
# Make changes based on the spider of the crawl template:
class GuoxueSpider(RedisCrawlSpider): # Change to inherit RedisCrawlSpider
name = 'guoxue'
redis_key = 'start_urls' # Set the key to store the start crawling address as start_urls, and the value as the starting url
? span>