I Have This Code For One Movie Is There A Way To Web Crawl More Than One Move At

I have this code for one movie. Is there a way to web crawl more than one move at a time? I have to collect this data for the 25 top and bottom movies. Also the code main_content = urljoin(url,soup.select(“.load-more-data”)[0][‘data-ajaxurl’]) does not seem to be pulling for than the first page of data.

url = ‘https://www.imdb.com/title/tt0111161/reviews?ref_=tt_ql_3’

res = requests.get(url)

soup = BeautifulSoup(res.text,”lxml”)

main_content = urljoin(url,soup.select(“.load-more-data”)[0][‘data-ajaxurl’]) 

response = requests.get(main_content)

broth = BeautifulSoup(response.text,”lxml”)

for item in broth.select(“.review-container”):

  title = item.select(“.title”)[0].text

  review = item.select(“.text”)[0].text

  date = item.select(“.review-date”)[0].text

  user = item.select(“.display-name-link”)[0].text

  #score = item.select(“.rating-other-user-rating”)[0].text

  print(“Title: {}nnDate: {}nnUser: {}nnReview: {}nn”.format(title,date,user,review,))

Prof. Angela


Calculate Price

Price (USD)
Open chat