I'm importing feeds from a news host that uses feedportal as the feeder.
Also I'm fetching some extra data with the Feeds Self Node Processor and I realized that HTTP Fetcher is redirected (HTTP 301) directly to the news host article (thanx Drupal User Agent!).
So, I wanted to store the news host article url as the feed item url, but there was no way.
I modified Feed sources to do so, and I want to share this modifications, but I don't have a development platform, git and all this stuff, ...

Those are the modifications I made:
file libraries/http_request.inc, function http_request_get, line 177:

          $result->code = curl_getinfo($download, CURLINFO_HTTP_CODE);
+         $result->redirect_url = curl_getinfo($download, CURLINFO_EFFECTIVE_URL);

note: redirect_url is also returned by drupal_http_request, but this core patch should be applied.

file plugins/FeedsHTTPFetcher.inc, function FeedsHTTPFetcherResult::getRaw, line 32:

      $result = http_request_get($this->url);
+    if(isset($result->redirect_url)) {
+        $this->redirect_url = $result->redirect_url;
+      }
+      else {
+        $this->redirect_url = $this->url;
+      }

file plugins/FeedsParser.inc, function FeedsParser::getMappingSources:

  public function getMappingSources() {
    self::loadMappers();
    $sources = array();
    feeds_alter('feeds_parser_sources', $sources, feeds_importer($this->id)->config['content_type']);
    if (feeds_importer($this->id)->config['content_type']) {
      $sources['parent:uid'] = array(
        'name' => t('Feed node: User ID'),
        'description' => t('The feed node author uid.'),
      );
    }
    if (is_a(feeds_importer($this->id)->fetcher,'FeedsHTTPFetcher')) { 
      $sources['fetcher:redirect_url'] = array(
        'name' => t('Fetcher: Target url'),
        'description' => t('The fetcher target url. If the feed source url is redirected, this is the final url fetched. Otherwise, this is the original source url.'),
      );
    }
    return $sources;
  }

file plugins/FeedsParser.inc, function FeedsParser::getSourceElement:

  public function getSourceElement(FeedsSource $source, FeedsParserResult $result, $element_key) {
    if ($element_key == 'parent:uid' &&
        $source->feed_nid &&
        ($node = node_load($source->feed_nid))) {
      return $node->uid;
    }
    if ($element_key == 'fetcher:redirect_url' &&
      isset($source->fetcher_result->redirect_url)) {
      return $source->fetcher_result->redirect_url;
    }
    $item = $result->currentItem();
    return isset($item[$element_key]) ? $item[$element_key] : '';
  }

Comments

twistor’s picture

Title: HTTP Fetcher target url as a source mapping » Allow fetchers to declare sources.

I've been thinking about something like this for a while.

bluegeek9’s picture

Status: Active » Closed (outdated)
//www.flaticon.com/free-icons/thank-you Thank you for your contribution!

Unfortunately, Drupal 7 is End of Life and no longer supported. We strongly encourage you to upgrade to a supported version of Drupal.

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.