I'm trying to use the feeds crawler for this site based on the following URL
http://jstracker.contractortaxation.com/jobs?page=1

I couldn't get "auto detect next link" to work, nor the XPath selector. I have also tried the "URL replacement options" with
URL pattern = http://jstracker.contractortaxation.com/jobs?page=$index.
Initial value of $index = 1
Increment $index by = 1

I get this error message.
Notice: Undefined index: url_pattern in FeedsCrawler->fetch() (line 47 of XXX/sites/all/modules/feeds_crawler/FeedsCrawler.inc).
Notice: Undefined index: FeedsXPathParserHTML in FeedsSource->getConfigFor() (line 577 of XXX/sites/all/modules/feeds/includes/FeedsSource.inc).

What am I missing?

CommentFileSizeAuthor
#11 feeds_crawler-7.x-1.x-dev.tar_.gz8.56 KBdrvdt
#2 Feeds Crawler.PNG36.62 KBsummerofb

Comments

twistor’s picture

twistor commented
Assigned: Unassigned » twistor
Status: Active » Postponed (maintainer needs more info)

I just tested your example and I got it to work fine.

Notice: Undefined index: url_pattern in FeedsCrawler->fetch() (line 47 of XXX/sites/all/modules/feeds_crawler/FeedsCrawler.inc).
Is indeed a bug, but it's minor and shouldn't be affecting your import.

If you were trying different crawling methods, try unchecking "Crawl on first run only."

summerofb’s picture

summerofb commented
StatusFileSize
new Feeds Crawler.PNG36.62 KB

I've just checked it again and it will only create 1 node. I uninstalled and re-install, it made no difference.

I have attached a screen shot of the settings for the feed import. I believe it is all configured correctly.

timb’s picture

timb commented

I have the same issue with the URL replacement options not working when using a setup as depicted in [#2]

natefollmer’s picture

natefollmer commented

Same problem, only I'm trying to get mine to work with a YouTube channel. No errors, but only 1 node imports, but I have about 580 videos I need to import.

dropfen’s picture

dropfen commented

the same Issue.
I think, if you use the the url_pattern for the crawler the origin URL Field schould not be required.

drvdt’s picture

drvdt commented
Component: User interface » Documentation
Assigned: twistor » drvdt

+1

richard.wyke’s picture

richard.wyke commented

Looks as if there is some problem in the data being sent to parseUrl in FeedsCrawler.inc, this fixed it for me but i don't know the rest of the code at all so this change might break other areas of the module.

    private function parseUrl($result, $source_config, $state) {
        if (!isset($state->inc)) {
            $state->inc = $source_config['crawler']['url']['initial'];
        } else {
            $state->inc += $source_config['crawler']['url']['increment'];
        }
        return str_replace('$index', $state->inc, $source_config['crawler']['url']['url_pattern']);
    }
sebastien247’s picture

sebastien247 commented

+1

richard.wyke your method does'nt work for me

bibo’s picture

bibo commented

#7 worked for me after I added a few lines before that! But Im now using dev but the last release, beta-1.

 if(empty($config['url_pattern']) && !empty($config['url']['url_pattern'])){
     // WTF, how can this be missing?
    $config['url_pattern'] = $config['url']['url_pattern'];
  }

    if (
        ($config['auto'] && ($href = $this->parseAuto($result, $source_config)) != FALSE) ||
        ($config['url_pattern'] && ($href = $this->parseUrl($result, $source_config, $state)) != FALSE) ||
        ($config['xpath'] && ($href = $this->parseXPath($result, $source_config)) != FALSE)
        ) {
  // ...

EDIT: nevermind, just use the latest dev-version, it works ok. But this module seems almost abandoned.

sebastien247’s picture

sebastien247 commented

Development version is perfect! Thank you from France!

drvdt’s picture

drvdt commented
Status: Postponed (maintainer needs more info) » Fixed
StatusFileSize
new feeds_crawler-7.x-1.x-dev.tar_.gz8.56 KB

I have tried this version:
7.x-1.x-dev tar.gz (8.56 KB) | zip (9.29 KB) 2011-Feb-25 Notes
It works!
Attachment is that version.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

web226’s picture

web226 commented
Issue summary: View changes

7.x-1.x-dev works well for me too