Consider option to crawl directories based on a pattern in config #77
Description
I would like to crawl a bunch of directories that are not in the same root folder, but ignore the directories that are out of scope. For instance the US site is,
www.mywebsite.com/beauty/us_en
www.mywebsite.com/fashion/us_en
www.mywebsite.com/watches/us_en
But I don't want
www.mywebsite.com/beauty/uk_en
www.mywebsite.com/fashion/uk_en
www.mywebsite.com/watches/uk_en
This could be done in the config file Pattern command....