ruby on rails - nil is not a symbol nor a string

Question

Im trying to use Kimurai to scrape a website. Im running into this error when I want to do /scrape.

def scrape
    url = "https://www.tripadvisor.com/Restaurants-g31892-Rogers_Arkansas.html"
    response = RestaurantsScraper.parse!(response, url, data: {})
    if response[status] == :completed && response[error].nil?
      flash.now[notice] = "Successfully scraped url"
    else
      flash.now[alert] = response[error]
    end
  end

Here is my scraper class

class RestaurantsScraper < Kimurai::Base
    @name = "restaurants_scraper"
    @driver = :selenium_chrome
    @start_urls = ["https://www.tripadvisor.com/Restaurants-g31892-Rogers_Arkansas.html"]

    def parse(response, url:, data: {})
        response.xpath("//div[@class=_1llCuDZj]").each do |a|
            request_to :parse_repo_page, url: absolute_url(a[:href], base: url)
        end
    end
    
    def parse_repo_page(response, url:, data: {})
            item = {}
            item["title"] = t.css('a._15_ydu6b')&.text&.squish&.gsub('[^0-9].', '')
            item["type"] = t.css('span._1p0FLy4t')&.text&.squish
            item["reviews"] = t.css('span.w726Ki5B').text&.squish
            item["top_reviews"] = t.css('a._2uEVo25r _3mPt7dFq').text&.squish

            Restaurant.where(item).first_or_create
    end
end

Here is the error im getting

question from:https://stackoverflow.com/questions/65924182/nil-is-not-a-symbol-nor-a-string

Answer 1

It's because response from RestaurantsScraper.parse!(response, url, data: {}) isn't defined.

From the kimurai docs it says you need to pass a Nokogiri::HTML::Document object.

I haven't used Kimurai and it feels like there is definitely a better way to do this, but something like the following may be enough to get you to the next step:

def scrape
  require 'open-uri'
  url = "https://www.tripadvisor.com/Restaurants-g31892-Rogers_Arkansas.html"  
  html = Nokogiri.parse open(url)
  response = RestaurantsScraper.parse!(html, url, data: {})
  if response[status] == :completed && response[error].nil?
    flash.now[notice] = "Successfully scraped url"
  else
    flash.now[alert] = response[error]
  end
end