Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
3.7k views
in Technique[技术] by (71.8m points)

Powershell: Get-ChildItem performance to deal with bulk files

The scenario is in a remote server, a folder is shared for ppl to access the log files. The log files will be kept for around 30 days before they got aged, and each day, around 1000 log files will be generated. For problem analysis, I need to copy log files to my own machine, according to the file timestamp.

My previous strategy is:

  1. Use dir /OD command to get the list of the files, to my local PC, into a file
  2. Open the file, find the timestamp, get the list of the files I need to copy
  3. Use copy command to copy the actual log files

It works but needs some manual work, ie step2 I use notepad++ and regular expression to filter the timestamp

I tried to use powershell as:

Get-ChildItem -Path $remotedir | where-object {$_.lastwritetime -gt $starttime -and $_lastwritetime -lt $endtime } |foreach {copy-item $_.fullname -destination .}

However using this approach it took hours and hours and no file has been copied, while compared with the dir solution it took around 7-8 minutes to generate the list of the file than copy itself took sometime but not hours

I guess most of the time spent on the filter file. I'm not quite sure why the get-childitem's performance is so poor.

Can you please advise if there's anything i can change? Thanks


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

For directories with a lot of files, Get-ChildItem is too slow. It looks like most of the time is spent enumerating the directory, then filtering through 'where', then copying each file.

Use .net directly, particularly [io.directoryinfo] with the GetFileSystemInfos() method.

e.g.

$remotedir   = [io.directoryinfo]'\servershare'
$destination = '.'
$filemask    = '*.*'
$starttime   = [datetime]'jan-21-2021 1:23pm'
$endtime     = [datetime]'jan-21-2021 4:56pm'

$remotedir.GetFileSystemInfos($filemask, [System.IO.SearchOption]::TopDirectoryOnly) | % {
    if ($_.lastwritetime -gt $starttime -and $_.lastwritetime -lt $endtime){
        Copy-Item -Path $_.fullname -Destination $destination
    }
}

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...