Allow administrator to set logging level for Watchdog

Seems like a decent patch.

A few things, there should be an option to turn off error reporting completely. Also, being able to multiselect what you want would make more sense, but obviously, complicates the patch a bit more.

This patch is great - I've applied it to our multisite install (4.7.3) with over 40k pageviews per day, and it's helping to save us from mysql death.

Kudos, should be part of core.

Why would this make such a big difference? If you want to save on database update, there are better places to focus on IMO. I bet you a beer that this is only a fraction of the total number of write accesses and I bet you another beer that the performance difference won't be noticable.

If it is noticably, I'd like to know more about your setup, and how this is possible. Until then, I'm far from convinced that this is helpful.

Robert, maybe you should support this patch of mine?
http://drupal.org/node/78503

Dries, even if this doesn't improve performance, what about from an admin level?

I administer a lot of sites and sifting through watchdog can be a nuisance. For some sites, I don't even care to see notices like "user John has logged in", where as I really only want to see PHP errors. Maybe even in some cases, I want to turn off watchdog completely--e.g., I have my own mechanism for errors through PHP error reporting logs and so forth. Why do I need Drupal running some extra CPU cycles and queries when I know I don't need it?

What I'd like to see happen is an option for each event (eg. 'search', '404', '403') that lets you define its output channel.

The available output channels would be: (i) the website logs (watchdog table), (ii) syslog and (iii) nothing (/dev/null).

Can't we just extend watchdog() to have a new DESTINATION parameter? "drupal", "syslog", "dev/null" ... of course we'll need a new mechanism for the 2nd. Or do we?

If a PHP error occurs, Drupal will catch it. How does PHP errors know about it? Well if Drupal let's it pass through it's logged... how does that happen? I'm actually not sure just typing out loud :-)

But this seems like a reasonable idea, maybe a bit more discussion on the best way to do this.

Not a bug.

I think it would be best to log everything and filter on display.

(Aside- I think more specialized logging tables would be good. If you were to go 404 hunting today, I don't think the current watchdog UI is well-suited for this. A table with path, last 404 time, and some indication of frequency would be a better tool.)

Neil has a point about the 404s, and the watchdog not being a great tool for that. Maybe we need to think about this some more, and figure out how we'd like to use and filter this information so it can be more useful.

The same is true for the search terms. What you really want is a list of the most popular search terms, ranked by popularity.

@dries

This was extremely helpful for us because we moved from another platform (Scoop) and had 20k nodes and 250k+ comments. Pathauto was helpful for keeping deep links to the story URLs but we chose not to keep deep links to comments. I simply don't need a watchdog INSERT for every 404. I want my 404's in httpd log. And we have a ton of 404's from crawlers, and two+ years worth of links.

Additionally, on a complex site if a single image goes missing (in our case a list item css image), then that's another many thousand 404's every hour until we realize the problem. This patch made the difference between a count(*) watchdog from 500k -> 3k rows. Oh yeah, and I *was* on MyISAM as well and ran into table locking issues.

I've been trolling many of these high performance patches and am in the process of implementing several of them. I see a lot of feedback in the comments to the effect of "so what, no big gain"... Except that too many folks are thinking about things in terms of a new Drupal install, and not a migration like us. $0.02 (or your local currency).

I'll deduct the two beers from the many I owe you for the existence of Drupal ;)

+1 for this patch.

For me this patch gives me the ability to control the extent of logging and makes my life a wee bit easier by not having to wade through events I am not interested in.

Drupal is used in a number of high-traffic sites and in those cases the reduction in the number of queries will also make a difference. Dries is right when he says that there are better areas to focus on for reduction of queries but I argue that this also is one place.

I very much liked Dries approach of reengineering this module to allow admins to configure what event to log as well as the channel to log it to. This will give admins a high degree of control.

However for now I think this patch should be committed.

There are good reasons to keep the information in the database: http://drupal.org/node/79622.

What I think we need is something like this: see attached photoshop mockup. Takers?

Obviously, I lend my support for inclusion into CORE as well. It looks like there is critical mass to maintain this (trivial) patch against 4.8+ in case it's not accepted.

Reduction of queries on *very* trafficked sites is huge. On average each user will generate 1 watchdog entry (just logging on)... if you have a site where users post lots of new content, these watchdog entries can eat up lots of valuable queries and CPU cycles.

I've been working on something like Dries's mock, but using plain old HTML links (which can be AJAXed later) since I don't really see this a place for a form. But, that is not what this issue is about. This is about not logging some things.

Not logging some things honestly scares me a bit. I don't want people put into a situation where they set the logging level to low and then get into a situation where they need the unlogged information. However, other projects do this sort of thing.

I think we certainly need to provide some interface into what is 404ed and what is searched for. Watchdog is probably the wrong thing. This information is relevant to Drupal administrators since it helps them improve their sites. Putting it in the same package makes things much easier for installation, discovery of information, and consistency of experience.

But, all this is beside the point, is it okay to allow users to throw away some logging information? With the number of +1s, I'm thinking this it might be okay.

Dries: I agree that 403/404 errors can be useful. Maybe the fix for that though is to make a level between WATCHDOG_NOTICE and WATCHDOG_ERROR or whatever like WATCHDOG_HTTP for these types of errors. Or, alternateiy, make a much more minor setting like WATCHDOG_STATUS for silly messages like "so and so has logged on" or "so and so completed a search" which can totally clog a busy site and don't need any follow-up afterwards.

Cool. In the interest of splitting things up where possible, I've made a separate issue for adding a WATCHDOG_STATUS message: http://drupal.org/node/80808

I'll try and roll a patch for this later, but the sooner someone does it, the sooner potentially this could get into HEAD, so if you're around over the next few hours, feel free to take a stab. :)

Is the issue:

1. Inserting data in the watchdog table?
2. Storing data and keeping it in memory?
3. Making the data easier to navigate/explore?
4. Any combination of the above?

ma3vck: one insert a second sounds like peanuts to me. What is the ratio of watchdog inserts versus other queries and versus other inserts? It would be great to share some of that data to motivate the need for ignoring certain watchdog events (1).

"This patch is for those administrators who just don't want to log the search terms that people look for, the 404 warnings, new user notifications, new content notifications, new comments, and so forth."

I am part of this group. The website I plan to launch on Drupal will have millions of nodes at first, and I don`t find 'user x has logged on' and 'user x searched for uyaz' useful in my watchdog table (though I understand some might find this info usefull). Also, for high traffic websites, the inserts in watchdog table for just about everything do add up, and if they provide no additional functionality (for me), well, I would love to have the option to set the logging level to what I find suitable.

So its an issue of potential performance gain on high traffic websites, and personal preferences about what should be logged and what shouldnt.

+1 for me - if this doesnt make it into core, any chance of at the very very least, a module?

Also, changed status to 6.x-dev, as I think its a little late for that for 5.0 :)

i haven't read this whole issue, but i was asked to comment here about negative performance impact of watchdog() on big, real sites. see http://drupal.org/node/105223#comment-232094 -- removing the 2 watchdog() calls per release in the packaging script made a *huge* difference in the responsiveness of d.o during the "dreaded packaging hour" (noon/midnight) GMT. it shaved 4 minutes off the total time of the packaing script (from ~23 minutes to ~19, so over 17% of the total time!) and d.o was downright snappy during all but the first minute or so (during the "query of doom")... YMMV.

The patch for hook_watchdog is now in core, which means modules can decide what to do with logging. The module called watchdog.module is now called dblog.module and is no longer mandatory. If it is disabled, then no logging whatsoever will happen. There is also syslog.module in core, which can route messages to flat files or remote hosts, hence relieving the database.

So, do we need this anymore given the above? Perhaps setting a default level is still of value to reduce noise (e.g. if you are using something like the emaillog.module which is part of the logging and alerts project), but the performance aspects can be dealt with using other means (see above).

I rolled a patch against dblog 6.x module to let admin set desired levels for watchdog inserts. This could be useful for very high traffic sites to reduce db inserts.

I adapted the previous patch of kongoji for dblog 7.x module