Follow-on from #2078855: Request::create() is slow, use a helper to duplicate instead which deals with the Request::create() part of that issue.

In Drupal 6/7, router items were loaded alongside their menu links via a join.

While we can't reliably join in 8.x given pluggable storage, we might be able to multiple load routes once route names are available and/or cache using CacheCollector.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

pwolanin’s picture

This query is loading based on an indexed key so should actually be a lot faster than what we would do in Drupal 7:
https://api.drupal.org/api/drupal/includes%21menu.inc/function/menu_get_...

Note that we can already load multiple routes in one call, and in some cases, like Local Tasks, we are pre-loading via a multi-load so that we hit the database only once regardless of how many tabs are being displayed.
see:
#2032311: Load all routes for local tasks via getRoutesByNames()

I think this issue overlaps with Crell's idea that we should have a small set of very commonly used routes potentially dumped to PHP and in memory all the time. A first pass of this could be all the entity routes, and maybe the front page, 403, and 404 pages.

catch’s picture

https://drupal.org/files/Screen%20Shot%202013-09-01%20at%202.26.42%20PM.png is the profiling. The queries are executed via matchRequest(). It looks like I didn't post a screenshot of how much the queries themselves are taking though. iirc it was a high proportion but better to profile again.

The problem isn't that this is a slow query, it's that it's a similar query executed multiple times.

pwolanin’s picture

@catch - yes, we can mitigate that by loading all the needed routes in one call as the linked patch does, but I would much, much rather have a system like tagging some routes to be dumped to PHP for speed than trying to implement yet more caching.

pwolanin’s picture

dawehner’s picture

As there is code all over the place which deals with the "router.route_provider" service directly we could not use the concept of the chained router but would
also have to do something like a chained route provider.

For menu links there is already #2058845: Pre-load non-admin routes which can do that loading.

catch’s picture

I think that's worth looking at here if we think it's viable rather than caching.

Do you have an idea how you'd handle the tagging?

dawehner’s picture

While talking with lukas smith we realized that Drupal should not pass around the url generator but the chained router, which wraps the generate method so you don't have to use a separate fast route provider but all you need is to rely on the fast router itself.

Battle plan: Allow to tag routes. Once they are tagged they are compiled to PHP.
This php is used together with a static route from symfony.

Additional we have to bring our new url generator interface to the chained router (both the interface and the actual implementation)
and never use the url generator directly.

dawehner’s picture

Do you have an idea how you'd handle the tagging?

We could check that tag on our route builder, collect the routes and then use the phpmatcherDumper from symfony.

catch’s picture

Sorry I get that bit - I mean what to name the tag, what the criteria would be for tagging.

A tricky one for tagging is Views routes - some of them are admin/*, another might be the front page or taxonomy/term/*, unless that's an option in the UI.

dawehner’s picture

Sorry I get that bit - I mean what to name the tag, what the criteria would be for tagging.

A tricky one for tagging is Views routes - some of them are admin/*, another might be the front page or taxonomy/term/*, unless that's an option in the UI.

I guess the criteria should be rather less aggressive but people can retag them later.
What about everything which is not accessed read only and is not in the administration interface, so node/{node} would be in there, but node/{node}/edit won't be.

dawehner’s picture

FileSize
8.78 KB

Did some work to compile the node edit route using the php storage system with a lot of help from chx.

Sadly it does not work so let's not torture the testbot.

dawehner’s picture

FileSize
9.17 KB

This works now beside the fact that the symfony php dumper is broken, as it does not export options neither the requirements or anything similar.

dawehner’s picture

Some work done:

  • Add a static router which uses the dumped url matcher as well as a new dumped url generator
  • Problems still: The dumped classes do not contain the full route object, which we need in order to actually solve the problem.
pwolanin’s picture

I think node/{node}/edit should be, since we often render that tab.

I'd tag most entity routes (node, taxonomy, user) as a start.

dawehner’s picture

To be honest we should really keep in mind that adding something here adds a potential overhead for every page.

amateescu’s picture

You mean overhead? :)

dawehner’s picture

Yeah as written ;)

pwolanin’s picture

Status: Active » Needs review

let's see if the tests run.

I'm not sure what the dumped file looks like, but how many routes do we think will be there? maybe 20?

Sees like the overhead of loading one extra file (I assume opcode cached) would be less then the DB queries?

Status: Needs review » Needs work

The last submitted patch, static_routing-2084675-13.patch, failed testing.

dawehner’s picture

The generated code basically does look like the follwing, so even the order of execution potential matters.

  function match($path) {
    if ($path == 'node') {
      return array('route' => 'views.node.page_1');
    }
    elseif (preg_match('...)) {
      return array('route' => 'node_view');
    }
  }
Anonymous’s picture

for the love of $deity - writing out php like this is still caching. lets not continue the BS about compiling just because SF2 advocates pushed that on us.

and we're caching to the filesystem. the hardest thing to scale in HA setups. which drupal has never, ever had to rely on before to get good cache performance.

just saying.

pwolanin’s picture

Status: Needs work » Postponed

I'm a bit with beejeebus. If we think there is a subset of core routes that are often used OOTB, let's just have a hand-built matcher that handles those as efficiently as possible.

I think we should postpone this until we actually have the all the route conversions done, since this is something we can add pretty late with no API change (other than you'd have to e.g. hack core to change the node path patterns).

catch’s picture

Status: Postponed » Needs work

I don't think compiling PHP then committing it to core is better than compiling on the fly. There's obvious problems with compiling on the fly but for scaling we should be looking for general solutions to that (like compiling during build) rather than doing manual bits for core when we know it's not going to work at all for something like Twig. If we don't compile here it'd be good to make sure contrib is able to swap it or other approaches into the router without hacking.

Also this isn't blocked on conversions so no reason to postpone I can see. The issue has been known for more than a year since before the router access system even went in.

I don't have a strong preference at the moment between compiling and a CacheCollector, although concerned CacheCollector could get bloated by things like toolbar if it's not per-role.

Anonymous’s picture

yeah, i didn't mean to have this issue postponed.

i just want to raise the flag, again, that 'compiling' keeps being posed as an alternative to caching, when it is just a form of caching, with some advantages and other drawbacks.

building php cache on build - absolutely, that's what the SF2 projects i've worked on did, and it was awesome, *because there was no UI to change things that required the php caches to be rebuilt*.

catch’s picture

Completely agreed that compiling shouldn't be opposed as an alternative to caching, the 'Symfony doesn't have any caches except for HTTP' line is completely disingenuous. We should probably say 'compared to using the cache API' rather than 'instead of caching' which is apparently not what I did :(

I do think there's going to be Drupal sites that can use it - I've worked on ones where the admin UI was more or less locked down.

Crell’s picture

For some very simple and very common routes (like the 2-3 that are currently called on nearly every page load, like the toolbar), we can also look into just hard coding some logic into a request listener to call the controller, create a response, and set it. That way we bypass routing entirely. It won't work for everything (probably not for any HTML pages), but for maybe a half-dozen really-simple routes (ajax requests and the like) it could be worthwhile.

pwolanin’s picture

@Crell - is there a real advantage vs a set of dumped PHP routes? It feels like that could be broader and more flexibly and at least would fall within the same paradigm as the rest.

Crell’s picture

We'd have to benchmark that to know; I couldn't say off hand.

The other advantage of it is that every time I drop a break point on the cufa() in HttpKernel to call the controller, it always gets picked up at least 3 times since there's 2 extra HTTP requests. So short-circuiting the every-page requests would have a DX benefit, too.

Edit: To be clear, I'm not against PHP-dumping common routes at all. I'm saying that there's another alternative that may work in some specific circumstances.

dawehner’s picture

Issue summary: View changes

With the introduction of the middlewares people could do a way faster routing, if they want, especially if you know that you don't need upcasting.
You could bypass access checking and what not.

catch’s picture

@dawehner how does a middleware help with menu link access checks?

dawehner’s picture

webchick’s picture

Issue tags: +D8 Accelerate Dev Days

Tagging.

catch’s picture

Since this issue was opened we have pre-loading of non-admin routes, which helps the access checking side of things.

For incoming requests (or anywhere where we take a path and convert it to a route since there are more of those now), there is #2406117: potentially cache parts of routing and #2370651: Make (routing by path) cacheable.

dawehner’s picture

Caching the access result could be quite tricky, but we could at least cache the query in RouteProvider itself, that would replace the database query there by default with some cache.

Added a couple of related issues, which at least one tried to do the exact same thhing (potentially cache parts ...)

dawehner’s picture

Caching the access result could be quite tricky, but we could at least cache the query in RouteProvider itself, that would replace the database query there by default with some cache.

Added a couple of related issues, which at least one tried to do the exact same thhing (potentially cache parts ...)

catch’s picture

Status: Needs work » Closed (duplicate)

Between pre-loading and #2480811: Cache incoming path processing and route matching and one or two other issues that are independently major, we can call this fixed now.