/*** HELP START ***//*
 
## >>> `%downloadFilesTo()` macro: <<< <a name="downloadfilesto-macro"></a> #######################  

The downloadFilesTo() macro copy files (in binary mode
using `filename()` function with options `lrecl=1 recfm=n`) 
from list provided by user to a directory indicated 
in the macro call.

Macro can be executed in two possible ways:
1) by providing list of files to download in a `datalines4`(`cards4`) list 
   directly after macro call:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
  %downloadFilesTo(</path/to/target/directory>)
  datalines4;
  <link to file1>
  <link to file2>
   ...
  <link to fileN>
  ;;;;
  run;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

2) by create a dataset with a list of links and use of `DS=`, `DSvar=`, 
   and `DSout=` parameters:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
  %downloadFilesTo(</path/to/target/directory>
  , DS=<dataset with list>
  , DSvar=<variable with list>
  , DSout=<variable with names for downloaded files>
  )
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

See examples below for the details.

The `%downloadFilesTo()` macro **does not** execute as a pure macro code.

Temporary dataset `work.______locationInfoData` is generated during processing.

### SYNTAX: ###################################################################

The basic syntax is the following, the `<...>` means optional parameters:
~~~~~~~~~~~~~~~~~~~~~~~sas
%downloadFilesTo(
    target
  <,DS=>
  <,DSvar=link>
  <,DSout=scan(link,-1,"/\")>
  <,inDev=URL>
  <,outDev=DISK>
  <,inOptions=>
  <,outOptions=>
)
~~~~~~~~~~~~~~~~~~~~~~~

**Arguments description**:

1. `target  `     - *Required*, a path to target directory.
                    If empty the `WORK` location is used.

*. `DS= `         - *Optional*, name of data set with list 
                    of files to download. 

*. `DSvar= `      - *Optional*, name of variable in data set 
                    with list of files to download. 

*. `DSout=`       - *Optional*, name of variable in data set 
                    with list of names for to downloaded files.
                    Default value is: `scan(link,-1,"/\")` it is 
                    an expression to cut last part of the link.

*. `inDev=`       - *Optional*, type of device used by the 
                    `filename()` function to access incoming files.
                    Default value is `URL`.

*. `outDev=`      - *Optional*, type of device used by the 
                    `filename()` function to access outgoing files.
                    Default value is `DISK`.

*. `inOptions=`   - *Optional*, list of additional options for the 
                    `filename()` function to access incoming files.
                    Default value is empty.

*. `outOptions=`  - *Optional*, list of additional options for the 
                    `filename()` function to access outgoing files.
                    Default value is empty.


---

*//*** HELP END ***/

%macro downloadFilesTo(
  target
, DS=
, DSvar=link
, DSout=scan(link,-1,"/\")
, inDev=URL
, outDev=DISK
, inOptions=
, outOptions=
) / 
secure
des='Macro %downloadFilesTo() allows easily download data from web to local directory'
;
  %put &=target.;
  %put &DS. &inDev. &DSvar.;

  data work.______locationInfoData;
    length inOptions outOptions $ 2048 outDev inDev $ 32 path $ 8192;
    inOptions = catx(" ", "lrecl=1 recfm=n", symget('inOptions'));
    outOptions = catx(" ", "lrecl=1 recfm=n", symget('outOptions'));
    inDev = coalescec(symget('inDev'),'URL');
    outDev = coalescec(symget('outDev'),'DISK');
    path = dequote(coalescec(symget('target'),pathname('WORK')));
  run;

  data _null_;
    length in out $ 8;
    if _N_=1 then set work.______locationInfoData;

    %if %superq(DS) = %then
      %do;
        infile cards4;
        input link :$8192.;
      %end;
    %else
      %do;
        set &DS.;
      %end;

    link = &DSvar.;

    length local $ 8192
    rc1txt rc2txt rc3txt $ 1024
    ;


    rc1 = filename(in, strip(link), strip(inDev), inOptions);
      rc1txt = sysmsg();

    local = catx("/",path,&DSout.);
    rc2 = filename(out, strip(local), strip(outDev), outOptions);
      rc2txt = sysmsg();

    rc3 = fcopy(in, out);
      rc3txt = sysmsg();
    if rc3 then
      put (_ALL_) (=/);
    else put "Success: " link;
     
    rc4 = filename(in);
    rc5 = filename(out);

  %if %superq(DS) = %then
    %do;
    %end;
  %else
    %do;
      run;
    %end;
%mend downloadFilesTo;





/*** HELP START ***//*
 
### EXAMPLES AND USECASES: ####################################################

**EXAMPLE 1.** Download data from web with direct list and then copy between directories:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
resetline;
%downloadFilesTo(~/directoryA)
datalines4;
https://www.lexjansen.com/wuss/2023/WUSS-2023-Paper-189.pdf
https://www.lexjansen.com/wuss/2023/WUSS-2023-Paper-189.zip
;;;;
run;

%downloadFilesTo(~/directoryB,inDev=DISK)
datalines4;
~/directoryA/WUSS-2023-Paper-189.pdf
~/directoryA/WUSS-2023-Paper-189.zip
;;;;
run;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


**EXAMPLE 2.** Download data from web using data set with list:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sas
resetline;
data listOfFiles;
infile cards;
input files :$1024.;
cards4;
https://www.lexjansen.com/wuss/2023/WUSS-2023-Paper-201.pdf
https://www.lexjansen.com/wuss/2023/WUSS-2023-Paper-109.pdf
;;;;
run;

%downloadFilesTo(R:\directoryC, DS=listOfFiles, DSvar=files)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---

*//*** HELP END ***/

/**###################################################################**/
/*                                                                     */
/*  Copyright Bartosz Jablonski, since 2023.                           */
/*                                                                     */
/*  Code is under the MIT license. If you want - you can use it.       */
/*  But it comes with absolutely no warranty whatsoever.               */
/*  If you cause any damage or something - it will be your own fault.  */
/*  You've been warned! You are using it on your own risk.             */
/*  However, if you decide to use it don't forget to mention author.   */
/*  Bartosz Jablonski (yabwon@gmail.com)                               */
/*                                                                     */
/**###################################################################**/
