The way it works!!! Get nutch1.2 run on Ubuntu 10.04

DEAR READER IGNORE THIS INFORMATION ITS NOT CORRECT ONLY conf USING!! :-)

You also have to copy the solrindex-mapping.xml to the conf folder (http://drupal.org/node/811062#comment-3154622) as well as merge the schema.xml and solrconfig.xml (use the ones from the apachesolr module and then check this: http://drupal.org/node/811062#comment-3240566)

PART 2

So after several attempts I got it and here is the second part:

Don't forget I installed the solr and the nutch programm in the /opt folder. Maybe you have different settings then you must adjust the path!!!

1.: Copy from the Drupal ApacheSolr modul (version 2) the schema.xml to the Solr schema.xml. You will find the solr schema.xml here.
/opt/solr/config/scheme.xml

2.: Copy now the ApacheSolr modul solrconfig.xml into the
/opt/solr/config/solrconfig.xml
file

3.: Nutch Version 1.2 open the solrindex-mapping.xml. You will find the file here .
/opt/nutch-1.2/conf/solrindex-mapping.xml
Now change this entry
<field dest="content" source="content"/>
to this
<field dest="body" source="content"/>

4.: Restart Tomcat
/etc/init.d/tomcat6 restart
This nessesary that the new informtions get into the solr modul.

And that's it !!! Quite easy or???

Now let the bot run with a command like this. Remind I use the path of my installation!!!
/home/YOURUSER/public_html/sites/all/modules/nutch/runbot -n '/opt/nutch' -j '/usr/lib/jvm/java-6-sun' -s 'http://localhost:8080/solr' -u 'http://www.example.com!http://www.example1.com' -c '1' -f '100' -d '0'

@karljohann You are right with point of the crawler informations in the nutch-site.xml but it would give errors if you don't fill in the http.agent.name. Maybe the crawler run but it gives an exception without it. So just for the correctness we fill in the infos. :-)
Thx also for the sh / bash info that it is dependent of the distro. I didn't know that. Here I just want to give an running example for Ubuntu 10.04 distros and great thanx for the short tutorial about the permissions of the user who run the bot and which folders need what kind of permissions.

It is not nessesary to merge files or to copy the solrindex-mapping.xml somwhere. This information is just confusing. It's not nesessary for the installation I described here!

So in my case it works without that! I followed the steps in the other posting, but it didn't work. That was the reason for me to make this step by step tutorial. Why is it necessary to get it work and why is it working in my case without this step if it is necessary? Ask yourself, why should so many people post and ask always the same questions, if the tutorial was right and straight? Anyway, maybe it is necessary on a different distro. Which distro did you use and where do you put the solrindex-mapping.xml? How did you integrate that in the solr system or how could solr system know that you copy the file there? You see even me, who got it working has a lot of questions about that. Can you please clarify why, me and the other readers really don't understand it.

Best regards
Frank

Yes, it was an misunderstanding! I agree all your steps. They are the same I described in the step by step tutorial. So the readers can follow this tutorial and get it work

I don't know how to do it at the moment, because I have other problems. But if you found a solution for this, please post it here. Maybe we can collect snippets for the most things user want to change.

I found a nice articel about the german language and how to optimize it for the german language search.

german word compound solr index optimizer

http://www.early-dance.de/news/9189-apachesolr-issues-german-and-other-g...

Update permissions problem:

I ran into the problem with the right permissions so I wrote this small snippet after I get it run.

ad the following to the nutch.admin.inc file to get a result like in the attached picture

Add this code to the nutch_admin_settings() function before the

return system_settings_form($form);

/**
  * Nutch Ordern rechte und besitzer check
  * @ seed/urls    existiert und beschreibbar ?
  * @ crawl/crawldb   existiert und beschreibbar ?
  * @ logs/haadop.log
  */	
  $NUTCH_HOME = variable_get('nutch_nutch_dir', '/usr/local/nutch');
  $check_true = '<span style="color:green;">Ok</span>';
     
  $form['permissions_check'] = array(
     '#type' => 'fieldset',
     '#title' => 'Folder permissions',
     '#description' => t('Folder permissions and owner check. If something is not okay Nutch will not run properly!'),  
     '#collapsible' => true,
     '#collapsed' => true,
  );	

  $apache_uid = posix_getuid();
  $form['permissions_check']['apache_uid'] = array(
    '#title' => 'Apache Uid',
     '#value' => $apache_uid, 
     '#prefix' => '<div>'.t('Apache UID').':',
     '#suffix' => '</div>',
  );	
  
  
  /**
  * Seed
  */
  $form['permissions_check']['seed'] = array(
     '#type' => 'fieldset',
     '#title' => 'Seed check',
     '#description' => t('Overview about your seeds folder and seed/urls file'),  
  );	  
  
  $seed_folder_exist = is_dir($NUTCH_HOME.'/seed');
  $form['permissions_check']['seed']['seed_folder_exist'] = array(
    '#title' => 'seed',
     '#value' => ($seed_folder_exist) ? $check_true : nutch_nutch_check_false(), 
     '#prefix' => '<div>'.t('Seed folder exist check').':',
     '#suffix' => '</div>',
  );	  

  
 $seed_owner_uid = fileowner($NUTCH_HOME.'/seed');
  $form['permissions_check']['seed']['seed_owner'] = array(
    '#title' => 'seed',
     '#value' => ($seed_owner_uid == $apache_uid) ? $check_true : nutch_nutch_check_false($seed_owner_uid), 
     '#prefix' => '<div>'.t('Seed owner check').':',
     '#suffix' => '</div>',
  );	  



  $seed_urls_owner_uid = fileowner($NUTCH_HOME.'/seed/urls');
  $form['permissions_check']['seed']['seed_urls_owner'] = array(
    '#title' => 'seed/urls',
     '#value' => ($seed_owner_uid == $apache_uid) ? $check_true : nutch_nutch_check_false($seed_urls_owner_uid), 
     '#prefix' => '<div>'.t('Seed / urls owner check').':',
     '#suffix' => '</div>',
  );	

  $seed_permission = substr(sprintf('%o', fileperms($NUTCH_HOME.'/seed')), -4); 
  $form['permissions_check']['seed']['seed_permission'] = array(
    '#title' => 'seed',
     '#value' => $seed_permission, 
     '#prefix' => '<div>'.$NUTCH_HOME.t('/seed owner permission').':',
     '#suffix' => '</div>',
  );	

  $seed_url_permission = substr(sprintf('%o', fileperms($NUTCH_HOME.'/seed/urls')), -4); 
  $form['permissions_check']['seed']['seed_url_permission'] = array(
    '#title' => 'seed/urls',
     '#value' => $seed_permission, 
     '#prefix' => '<div>'.$NUTCH_HOME.t('/seed/urls owner permission').':',
     '#suffix' => '</div>',
  );	


  /**
  * crawl
  */
  $form['permissions_check']['crawl'] = array(
     '#type' => 'fieldset',
     '#title' => 'Crawl check',
     '#description' => t('Overview about your crawl folder'),  
  );	  
  
  $crawldb_owner_uid = fileowner($NUTCH_HOME.'/crawl/crawldb');
  $form['permissions_check']['crawl']['crawldb_owner'] = array(
    '#title' => 'crawl/crawldb ',
     '#value' => ($crawldb_owner_uid == $apache_uid) ? $check_true : nutch_nutch_check_false($crawldb_owner_uid), 
     '#prefix' => '<div>'.t('Crawldb owner check').':',
     '#suffix' => '</div>',
  );
   

 $linkdb_owner_uid = fileowner($NUTCH_HOME.'/crawl/linkdb');
  $form['permissions_check']['crawl']['linkdb_owner'] = array(
    '#title' => 'crawl/linkdb ',
     '#value' => ($linkdb_owner_uid == $apache_uid) ? $check_true : nutch_nutch_check_false($linkdb_owner_uid), 
     '#prefix' => '<div>'.t('Linkdb owner check').':',
     '#suffix' => '</div>',
  );  

  $segments_owner_uid = fileowner($NUTCH_HOME.'/crawl/segments');
  $form['permissions_check']['crawl']['segments_owner'] = array(
    '#title' => 'crawl/segments',
     '#value' => ($segments_owner_uid == $apache_uid) ? $check_true : nutch_nutch_check_false($segments_owner_uid), 
     '#prefix' => '<div>'.t('Segments owner check').':',
     '#suffix' => '</div>',
  );
 


  $crawldb_permission = substr(sprintf('%o', fileperms($NUTCH_HOME.'/crawl/crawldb')), -4); 
  $form['permissions_check']['crawl']['crawldb_permission'] = array(
    '#title' => 'crawl/crawldb ',
     '#value' => $crawldb_permission, 
     '#prefix' => '<div>'.$NUTCH_HOME.t('/crawl/crawldb permission check').':',
     '#suffix' => '</div>',
  );
  
  $linkdb_permission = substr(sprintf('%o', fileperms($NUTCH_HOME.'/crawl/linkdb')), -4); 
  $form['permissions_check']['crawl']['lindbdb_permission'] = array(
    '#title' => 'crawl/linkdb ',
     '#value' => $crawldb_permission, 
     '#prefix' => '<div>'.$NUTCH_HOME.t('/crawl/linkdb permission check').':',
     '#suffix' => '</div>',
  );

  $segments_permission = substr(sprintf('%o', fileperms($NUTCH_HOME.'/crawl/segments')), -4); 
  $form['permissions_check']['crawl']['segments_permission'] = array(
    '#title' => 'crawl/segments ',
     '#value' => $segments_permission, 
     '#prefix' => '<div>'.$NUTCH_HOME.t('/crawl/segments permission check').':',
     '#suffix' => '</div>',
  );


  /**
  * Hadoop
  */
  $form['permissions_check']['hadoop'] = array(
     '#type' => 'fieldset',
     '#title' => 'Hadoop check',
     '#description' => t('Overview about your hadoop folder.'),  
  );	  
  
  
  $hadoop_owner_uid = fileowner($NUTCH_HOME.'/logs/hadoop.log');
  $form['permissions_check']['hadoop']['hadoop_owner'] = array(
    '#title' => 'logs/haadop.log',
     '#value' => ($hadoop_owner_uid == $apache_uid) ? $check_true : nutch_nutch_check_false($hadoop_owner_uid), 
     '#prefix' => '<div>'.t('Hadoop owner check').':',
     '#suffix' => '</div>',
  );
   
  
   $hadoop_permission = substr(sprintf('%o', fileperms($NUTCH_HOME.'/logs/hadoop.log')), -4); 
   $form['permissions_check']['hadoop']['hadoop_permission'] = array(
    '#title' => 'logs/haadop.log',
     '#value' => $hadoop_permission,
     '#prefix' => '<div>'.$NUTCH_HOME.t('/logs/haadop.log permission check').':',
     '#suffix' => '</div>',
  );
  

Add this function to the end of the nutch.admin.inc file.

function  nutch_nutch_check_false($permissions = '')
{  
  $output = '<span style="color:red;">Wrong '.$permissions.'</span>';	
  
  return $output;
}

That's it. In my case is the crawler running if your result is equal to my attached picture.

Hope this makes it a litte bit easier to locate the problems and will find the way into the next alpha ??

I added some more checks in the code above. Look at the attached picture if you want toknow how it looks like.

Ah, I never build a patch, but I will try. Give me some days...

@maxmmize It's not so hard to make a patch need round about 1 hour. It looks like we collect really cool infos and functions to make this crawler running. Please don't hesitate. Post your links here, maybe i can help you!

@dstuart Thx for the short hint. This saves me a lot of time.

So here is the patch untested.

So have fun
Frank