right now its 1,000 for URL alias and 10,000 for url column.

CommentFileSizeAuthor
#1 boost-623540.patch4.2 KBmikeytown2

Comments

mikeytown2’s picture

mikeytown2 commented
Status: Active » Needs review
StatusFileSize
new boost-623540.patch4.2 KB
mikeytown2’s picture

mikeytown2 commented
Status: Needs review » Fixed

committed

mikeytown2’s picture

mikeytown2 commented
Status: Fixed » Needs work

The prune function can only run once, otherwise the counter gets messed up. The quick fix is to only prune once; but with the prune function it would be possible for the crawler to be more robust.

Did a quick test and was able to load 1.2M records in about 10 min, a huge improvement compared to the old way.

mikeytown2’s picture

mikeytown2 commented
Status: Needs work » Fixed

committed the fix

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.