Improve directory parsing

Improve the way directories are parsed currently. They are configured to parse 2-level directory structure. Should be multi-level.
Also the parsing is buggy. Does not list files in some cases(needs to be figured out).