I'm working on getting the parallel crawler up & running on a D7.14 site.

I've installed the lua & pcre jit prereqs, the nginx modules, and a custom nginx 1.3.1 build.

My nginx config, based on that provided in /config is,

server {
listen 127.0.0.1:13001;
listen [::1]:13001 ipv6only=on;
server_name cache-warmer.bogus;
limit_conn warmer 192;
access_log /var/log/nginx/cache_warmer.access.log;
error_log /var/log/nginx/cache_warmer.error.log;
root /root/.drush/cache_warmer/lua;
index index.html;
location / { empty_gif; }
location = /cache-warmer { content_by_lua_file '$document_root/cache_warmer_requests.lua'; }
location = /parallel-reqs { content_by_lua_file '$document_root/cache_warmer_client.lua'; }
}

With

drush --version
drush version 6.0-dev

@ exec of the

drush @TEST cache-warmer --updated-last='-2 days' --parallel=20 \
--crawler-service-uri=http://127.0.0.1:13001/cache-warmer --timeout=15 \
--hub-pages-file=hub_pages.txt https://test.loc

fails with,

[{"timestamp":1339361677,"reply":"\r\n500 Internal Server Error<\/title><\/head>\r\n\r\n
500 Internal Server Error<\/h1><\/center>\r\n

nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":500,"time":0.000705}]

logs for this report,

==> /var/log/nginx/cache_warmer.error.log <==
2012/06/10 13:54:37 [error] 31001#0: *5 Failed to load Lua inlined code: cannot open /root/.drush/cache_warmer/lua/cache_warmer_requests.lua: Permission denied, client: 127.0.0.1, server: drupal-cache-warmer.bogus, request: "POST /cache-warmer HTTP/1.1", host: "127.0.0.1:13001"

==> /var/log/nginx/cache_warmer.access.log <==
127.0.0.1 - - [10/Jun/2012:13:54:37 -0700] "POST /cache-warmer HTTP/1.1" 500 186 "-" "-"

nginx execs as wwwrun:www, and the files are owned as same.

Not clear WHAT is denying permission, or if this is a bug.

Ideas?

Comments

perusio’s picture

perusio CreditAttribution: perusio commented

You have a Lua error, hence the 500 status. The problem is that the scripts are not readable by server user. It's a centOS right? Check this thread: http://groups.drupal.org/node/233858

RandyK’s picture

RandyK CreditAttribution: RandyK commented

Hi.

No it's not Centos. It's Opensuse.

But, reading

http://groups.drupal.org/node/233858#comment-765163

even though I'd

chown -R wwwrun:www /root/.drush/cache_warmer

that wasn't enough.

I

mv /root/.drush/cache_warmer/lua /srv/www/lua/cache_warmer
chown -R wwwrun:www  /srv/www/lua/cache_warmer

and

-   root /root/.drush/cache_warmer/lua;
+   root /srv/www/lua/cache_warmer

Now, when I exec the drush command I get no more permission denied.

But now I do get a runtime error,

==> /var/log/nginx/cache_warmer.error.log <==
2012/06/10 17:31:41 [error] 25369#0: *1 lua handler aborted: runtime error: /srv/www/lua/cache_warmer/cache_warmer_client.lua:34: module 'socket.http' not found:
        no field package.preload['socket.http']
        no file './socket/http.lua'
        no file '/usr/local/share/luajit-2.0.0-beta10/socket/http.lua'
        no file '/usr/local/share/lua/5.1/socket/http.lua'
        no file '/usr/local/share/lua/5.1/socket/http/init.lua'
        no file './socket/http.so'
        no file '/usr/local/lib/lua/5.1/socket/http.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
        no file './socket.so'
        no file '/usr/local/lib/lua/5.1/socket.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
        [C]: in function 'require'
        /srv/www/lua/cache_warmer/cache_warmer_client.lua:34: in function </srv/www/lua/cache_warmer/cache_warmer_client.lua:1>, client: 127.0.0.1, server: cache-warmer.bogus, request: "POST /cache-warmer HTTP/1.1", subrequest: "/parallel-reqs", host: "127.0.0.1:13001"

==> /var/log/nginx/cache_warmer.access.log <==
127.0.0.1 - - [10/Jun/2012:17:31:41 -0700] "POST /cache-warmer HTTP/1.1" 200 14 "-" "-"

Which seems to be complaining about not finding the lua 'socket.http' module. But if I check with both lua and luajit

luajit -l socket.http
	LuaJIT 2.0.0-beta10 -- Copyright (C) 2005-2012 Mike Pall. http://luajit.org/
	JIT: ON CMOV SSE2 SSE3 AMD fold cse dce fwd dse narrow loop abc fuse
	>

lua -l socket.http
	Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
	>

it IS there.

Closer! but still not quite there.

RandyK’s picture

RandyK CreditAttribution: RandyK commented

One of our biz consultants just pointed me in the right direction :-/

Looks like it's an nginx 'vs' ENV var issue.

'lua' & 'luajit' @ shell both read LUA_PATH & LUA_CPATH env vars -- and use them.

According to this

http://wiki.nginx.org/HttpLuaModule#lua_package_path
"default: The content of LUA_PATH environ variable or Lua's compiled-in defaults. "

nginx is supposed to too. But for whatever reason, it seems that (my?) nginx &/or cache_warmer do not.

If I explicitly add 

lua_package_path  "<lua_path string>;;";
lua_package_cpath "<lua_cpath string>;;";  

server {...

to the nginx config above, then at exec of 'drush cache-warmer ...', I get:

==> /var/log/nginx/access.log <==
10.42.7.6 - - [10/Jun/2012:19:07:07 -0700] "HEAD / HTTP/1.1" 301 0 "-" "Nginx Cache Warmer"
10.42.7.6 - - [10/Jun/2012:19:07:07 -0700] "GET / HTTP/1.1" 301 178 "-" "Nginx Cache Warmer"
10.42.7.6 - - [10/Jun/2012:19:07:07 -0700] "GET / HTTP/1.1" 301 178 "-" "Nginx Cache Warmer"
10.42.7.6 - - [10/Jun/2012:19:07:07 -0700] "GET / HTTP/1.1" 301 178 "-" "Nginx Cache Warmer"
10.42.7.6 - - [10/Jun/2012:19:07:07 -0700] "GET / HTTP/1.1" 301 178 "-" "Nginx Cache Warmer"
10.42.7.6 - - [10/Jun/2012:19:07:07 -0700] "GET / HTTP/1.1" 301 178 "-" "Nginx Cache Warmer"

==> /var/log/nginx/cache_warmer.access.log <==
127.0.0.1 - - [10/Jun/2012:19:07:07 -0700] "POST /cache-warmer HTTP/1.1" 200 14 "-" "-"

which looks a lot better. Now i need to figure out if it's enough.

perusio’s picture

perusio CreditAttribution: perusio commented

In Debian never had any problem with that. But note that I set in the lua module: 

LUAJIT_INC="/usr/include/luajit-2.0"
LUAJIT_LIB="/usr/lib/$(dpkg-architecture -qDEB_HOST_GNU_TYPE)/libluajit-5.1.so.2.0.0"
RandyK’s picture

RandyK CreditAttribution: RandyK commented

Iiuc, those vars are only relevant to the Luajit *build*, not runtime.

For my DIY-build of luajit,

export LUAJIT_LIB="/usr/local/lib/libluajit-5.1.so"
export LUAJIT_INC="/usr/local/include/luajit-2.0"

It's LUA_PATH & LUA_CPATH that are runtime relevant.