/*** HELP START ***//*

## >>> `%dirsAndFiles()` macro: <<< <a name="dirsandfiles-macro"></a> #######################  

The `%dirsAndFiles()` macro allows to extract info about all files 
and subdirectories of a given `root` directory. 

The extracted info may be just a list of files and subdirectories or, if 
the `details=` parameter is set to 1, additional operating system information 
is extracted (information is OS-dependent and gives different results for Linux 
and for Windows)

The extracted info can be narrowed down to files (`keepFiles=1`) or to 
directories (`keepDirs=1`) if need be.

The extracted info can be presented in wide or long format (`longFormat=1`).

The extracted info for files can be narrowed down to only files with particular
extension, for example: `fileExt=sas7bdat`.

The extracted info can be narrowed down maximal path depth 
by setting up the `maxDepth=` parameter.

See examples below for the details.

### REFERENCES: ###################################################################

The macro is based on Kurt Bremser's "*Talking to Your Host*" article 
presented at WUSS 2022 conference.

The article is available [here](https://communities.sas.com/t5/SAS-User-Groups-Library/WUSS-Presentation-Talking-to-Your-Host/ta-p/838344)
and also as an additional content of this package.
The paper was awarded the "Best Paper Award - Programming".

### SYNTAX: ###################################################################

The basic syntax is the following, the `<...>` means optional parameters:
~~~~~~~~~~~~~~~~~~~~~~~sas
%dirsAndFiles(
    root
  <,ODS=>
  <,details=>
  <,keepDirs=>
  <,keepFiles=>
  <,longFormat=>
  <,fileExt=>
  <,maxDepth=>
  <,backslashSens=>
)
~~~~~~~~~~~~~~~~~~~~~~~

**Arguments description**:

1. `root`                     - *Required*, path to be searched
                                for information.

* `ODS=work.dirsAndFilesInfo` - *Optional*, output data set, 
                                name of a dataset to store information.

* `details=0`                 - *Optional*, indicates if detailed info 
                                 will be collected, `1` = yes, `0` = no.

* `keepDirs=1`                - *Optional*, indicates if directories info 
                                will be collected, `1` = yes, `0` = no.

* `keepFiles=1`               - *Optional*, indicates if files info 
                                will be collected, `1` = yes, `0` = no.

* `longFormat=0`              - *Optional*, indicates if output be 
                                in long format, `1` = yes, `0` = no.

* `fileExt=`                  - *Optional*, if not missing then indicates 
                                a list of space-separated file extensions 
                                to filter out results.
                                
* `maxDepth=0`                - *Optional*, if not zero then indicates
                                maximum depth of search in the root path.

* `backslashSens=0`           - *Optional*, if not zero then it indicates 
                                that backslash(`\`) symbol in files and dirs 
                                names is detectable under Linux. Accepted 
                                values: `0` and `1`. Ignored under Windows. 

*//*** HELP END ***/

/* dirsandfiles.sas */

%macro dirsAndFiles(
 root            
,ODS=work.dirsAndFilesInfo
,details=0 /* should detailed info be collected */
,keepDirs=1 /* should directories info be collected */
,keepFiles=1 /* should files info be collected */
,longFormat=0 /* should output be in long format */
,fileExt=     /* file extensions to filter out results (space-separated list) */
,maxDepth=0   /* maximum depth of search in path */
,backslashSens=0 /* indicates if backslash is masked for Linux */
)
/ minoperator des='The dirsAndFiles macro extracts info about all files and subdirectories of a given directory'
;


%if NOT (%superq(details) IN (0 1)) %then 
  %do;
    %put NOTE: The details macro variable can be only: 0 or 1.;
    %put NOTE- The value will be set to 0.;
    %let details=0;
  %end;
%if NOT (%superq(keepDirs) IN (0 1)) %then 
  %do;
    %put NOTE: The keepDirs macro variable can be only: 0 or 1.;
    %put NOTE- The value will be set to 1.;
    %let keepDirs=1;
  %end;
%if NOT (%superq(keepFiles) IN (0 1)) %then 
  %do;
    %put NOTE: The keepFiles macro variable can be only: 0 or 1.;
    %put NOTE- The value will be set to 1.;
    %let keepFiles=1;
  %end;
%if NOT (%superq(longFormat) IN (0 1)) %then 
  %do;
    %put NOTE: The longFormat macro variable can be only: 0 or 1.;
    %put NOTE- The value will be set to 0.;
    %let longFormat=0;
  %end;
%if NOT (%superq(backslashSens) IN (0 1)) %then 
  %do;
    %put NOTE: The backslashSens macro variable can be only: 0 or 1.;
    %put NOTE- The value will be set to 0.;
    %let backslashSens=0;
  %end;

%local backslashDo backslashEnd;
%if 1=&backslashSens. AND NOT(WIN=&SYSSCP.) %then
  %do;
    /* the regexp adds one more backslash to any list of backslashes 
       in a given string, e.g., "A\B\\C" is transformed to "A\\B\\\C"
     */
    %let backslashDo=prxchange('s/(\\+)/\\$1/',-1,;
    %let backslashEnd=);
  %end;

%if %superq(fileExt) NE %then 
  %do; 
    %let keepDirs=0;
  %end;

%local tempOptions;
%let tempOptions = %sysfunc(getOption(NOTES)) %sysfunc(getOption(SOURCE));
options noNOTES noSOURCE;

/* confirm maxDepth is numeric */
/* make sure ODS is "dataset name" */
data _null_;
  call symputX("maxDepth", symgetn("maxDepth"), "L");
  length ods $ 41;
  ods = compress(symget("ODS"),"_.","KAD");
  call symputX("ODS", ods, "L");
run;

/* get full list of files with given extension from &root. (including all subdirectories) */
data &ODS. ; /* just to be sure the file "files" is "empty" */
run ;

data &ODS.( compress=YES ) ;
  length root dname $ 2048 fn $ 256 dir level 8 ;
  root = symget("root") ;
  retain fn dname ' ' level 0 dir 1 ;
  label 
    root  = "Root path"
    fn    = "File Name"
    dname = "Directory Name"
    dir   = "Directory Indicator"
    level = "Directory or File depth in the Root path"
     ;
run ;

data &ODS. ;
  modify &ODS. ;
  fname="_%sysfunc(datetime(),hex7.)" ;
  drop fname;
  rc1=filename(fname, &backslashDo. catx('/',root,dname,fn) &backslashEnd.) ;
  rc2=dopen(fname) ;
  dir = 1 & rc2 ;
  if dir then 
    do ;
      dname=catx('/',dname,fn) ;
      fn=' ' ;
    end ;
  replace ;

  if dir ;

  level=level+1 ;

  do i=1 to dnum(rc2) ;
    fn=dread(rc2,i) ;
    output ;
  end ;
  rc3=dclose(rc2) ; 
run ;

%if %superq(fileExt) NE %then 
  %do;
    %local fileExtNum _fileExt_;
    %let fileExtNum = %sysfunc(countw(%superq(fileExt), %str( ))); 
    %let _fileExt_ = _fileExt_%sysfunc(datetime(),HEX16.)_; /* &_fileExt_. */
    %if 1 < &fileExtNum. %then /* multiple extensions */
      %do;
        filename ____FExt TEMP;
        data _null_;
          file ____FExt lrecl=4096;
          length f $ 4096;
          f = upcase(symget('fileExt'));
          put f;
        run;
        data WORK.&_fileExt_.;
          infile ____FExt lrecl=4096;
          input &_fileExt_. : $ 32. @@;
          if &_fileExt_. ne " " then output;
        run;
        filename ____FExt CLEAR;
      %end;

    data &ODS.(compress=YES) ;
      %if 1 < &fileExtNum. %then /* multiple extensions */
        %do;
          if _N_ = 1 then
            do;
              array ___FExt[&fileExtNum.] $ 32 _temporary_;
              put "INFO: Multiple extensions requested:"; 
              do until (___FExt_EOF);
                set WORK.&_fileExt_. 
                  end=___FExt_EOF
                  curobs=___FExt_curobs
                ;
                ___FExt[___FExt_curobs] = &_fileExt_.;
                put @7 "-" ___FExt[___FExt_curobs];
              end;
            end;
        %end;

      set &ODS. ;

      %if 1 < &fileExtNum. %then /* multiple extensions */
        %do;
          if upcase(scan(fn,-1,".")) IN ___FExt;
        %end;
      %else
        %do;
          where upcase(scan(fn,-1,".")) = upcase(symget('fileExt'));
        %end;

      /* narrow down max depth in search */
      %if %sysevalf(&maxDepth. > 0) %then
        %do;
          if level <= &maxDepth.; 
        %end;
    run ;

    %if 1 < &fileExtNum. %then
      %do;
        proc delete data=WORK.&_fileExt_.;
        run;
      %end;

    /* if filtering by file extension then skip next step */
    %goto skipNextStep;
  %end;

/* this step is executed _only_ if "by extension" filtering was not executed */
data &ODS.(compress=YES) ;
  set &ODS. ;
  where 0
  %if &keepDirs.=1 %then 
    %do; 
      or dir=1 /* keep files in final output */
    %end;
  %if &keepFiles.=1 %then 
    %do; 
      or dir=0 /* keep directories in final output */
    %end;
  ;

  /* narrow down max depth in search */
  %if %sysevalf(&maxDepth. > 0) %then
    %do;
      if level <= &maxDepth.; 
    %end;
run;
%skipNextStep:


%if &details. %then
  %do;
    /* get files info in "Long" format */
    data &ODS.(compress=YES) ;
      set &ODS. ;

      /* get files info */
      fname="_%sysfunc(datetime(),hex7.)" ;
      rc=filename(fname, &backslashDo. catx('/',root,dname,fn) &backslashEnd.) ;

      if dir=0 then
        do ;
        /* files */
          if rc = 0 and fexist(fname) then
             do ;
                fid=fopen(fname,"IS") ;
                length fidTxt $ 1024 ;
                fidTxt=sysmsg();
                if fid then
                  do i=1 to foptnum(fid) ;
                     length infoname infonameShort $ 128 infoval $ 2048 ;
                     infoname=foptname(fid, i) ;
                     infonameShort=compress(infoname,"_","DAK") ;
                     infoval=finfo(fid, infoname) ;
                     output ;
                  end ;
                else
                  do;
                    put fidTxt ;
                    infoname="File Error" ;
                    infonameShort="FError" ;
                    infoval=fidTxt ;
                    output ;
                  end;
                fid=fclose(fid) ;
             end ;
          rc=filename(fname) ;
        end ;
      else 
        do ;
        /* directories */
          if rc = 0 and fexist(fname) then
             do ;
                fid=dopen(fname) ;
                fidTxt=sysmsg();
                if fid then
                  do i=1 to doptnum(fid) ;
                     infoname=doptname(fid, i) ;
                     infonameShort=compress(infoname,"_","DAK") ;
                     infoval=dinfo(fid, infoname) ;
                     output ;
                  end ;
                else
                  do;
                    put fidTxt ;
                    infoname="Directory Error" ;
                    infonameShort="DError" ;
                    infoval=fidTxt ;
                    output ;
                  end;
                fid=dclose(fid) ;
             end ;
        end ;

      rc=filename(fname) ;

      drop fname--i;
    run ;

    /* transforms files info into "Wide" format */
    %if &longFormat.=0 %then
    %do; 
      proc transpose 
        data = &ODS. 
        out=&ODS.(drop =
                    _name_ 
                    /*
                    %if &keepDirs.=0 %then %do; directory %end;
                    %if &keepFiles.=0 %then %do; filename %end;
                    */ 
                  compress=YES
                  ) ;
          by root dname fn level dir notsorted ;
          id infonameShort ;
          idlabel infoname ;
          var infoval ;
      run ;
    %end;
  %end;

options &tempOptions.;
%mend dirsAndFiles;

/*
options nomprint;
options NOTES SOURCE;
*/

/*** HELP START ***//*

### EXAMPLES AND USECASES: ####################################################

**EXAMPLE 1.** Get list of files and directories:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
%dirsAndFiles(C:\SAS_WORK\,ODS=work.result1)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


**EXAMPLE 2.** Get detailed info:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
%dirsAndFiles(C:\SAS_WORK\,ODS=work.result2,details=1)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


**EXAMPLE 3.** Get only files info:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
%dirsAndFiles(C:\SAS_WORK\,ODS=work.result3,keepDirs=0)

%dirsAndFiles(C:\SAS_WORK\,ODS=work.result5,keepDirs=0,details=1)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


**EXAMPLE 4.** Get only directories info:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
%dirsAndFiles(C:\SAS_WORK\,ODS=work.result4,keepFiles=0)

%dirsAndFiles(C:\SAS_WORK\,ODS=work.result6,keepFiles=0,details=1)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


**EXAMPLE 5.** Filter out by `sas` extension:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
%dirsAndFiles(~/,ODS=work.result7,fileExt=sas)

%dirsAndFiles(~/,ODS=work.result8,fileExt=sas,details=1)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


**EXAMPLE 6.** Keep result in the long format:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
%dirsAndFiles(~/,ODS=work.result9,details=1,longFormat=1)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


**EXAMPLE 7.** Get info for maximum depth of 2:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
%dirsAndFiles(C:\SAS_WORK\,ODS=work.result10,details=1,maxDepth=2)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


**EXAMPLE 8.** How locked/unavailable files are handled:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
%dirsAndFiles(%sysfunc(pathname(WORK)),ODS=work.result11,details=1)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


**EXAMPLE 9.** Not existing directory:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
%dirsAndFiles(%sysfunc(pathname(WORK))/noSuchDir,ODS=work.result12,details=1)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

*//*** HELP END ***/

/**###################################################################**/
/*                                                                     */
/*  Copyright Bartosz Jablonski, since 2020.                           */
/*                                                                     */
/*  Code is under the MIT license. If you want - you can use it.       */
/*  But it comes with absolutely no warranty whatsoever.               */
/*  If you cause any damage or something - it will be your own fault.  */
/*  You've been warned! You are using it on your own risk.             */
/*  However, if you decide to use it don't forget to mention author.   */
/*  Bartosz Jablonski (yabwon@gmail.com)                               */
/*                                                                     */
/**###################################################################**/
