parallel windows batch-file shell parallel-processing multicore

windows - parallel - Ejecución paralela de procesos de shell



bash parallel execution (5)

¿Hay alguna herramienta disponible para ejecutar varios procesos en paralelo en un archivo por lotes de Windows? He encontrado algunas herramientas interesantes para Linux ( parallel y PPSS ), sin embargo, necesitaría una herramienta para plataformas Windows.

Bonificación: sería genial si la herramienta también permitiera distribuir procesos de forma sencilla entre varias máquinas, ejecutando los procesos remotamente a la PsExec .

Ejemplo: me gustaría eso en el siguiente ciclo

for %F in (*.*) do processFile.exe %F

una cantidad limitada de instancias de processFile.exe se ejecuta en paralelo para aprovechar las CPU de núcleos múltiples.

GNU xargs en Linux tiene un interruptor "-P n" para iniciar procesos "n" en paralelo.

¿Quizás la compilación de xargs de cygwin / mingw también sea compatible con esto?

Entonces puedes usar:

xargs -P 4 processFile < fileList

Sin embargo, no hay un desvanecimiento sofisticado del proceso multinodo.

Intenta start :

start "title of the process" "P:/ath/to.exe"

Abre una nueva ventana con el título dado y ejecuta el archivo BAT, CMD o EXE. También puede establecer la prioridad, establecer el mismo entorno, etc.

Los archivos que no se pueden ejecutar se abren con el programa asociado.

Lectura adicional: Inicio -> Ejecutar

cmd /k start /?

El inicio está disponible al menos desde WinME.

¡Buena suerte!

Suena más como si quisiera usar Powershell 2. Sin embargo, puede generar nuevas ventanas de cmd (u otros procesos) usando start , vea también this respuesta. Aunque probablemente tenga que usar algunas otras herramientas y un pequeño truco para crear algo así como un "grupo de procesos" (para tener un máximo de n instancias ejecutándose a la vez). Puede lograr esto último usando tasklist /im y contando cuántos ya existen ( for loop o wc , si corresponde) y simplemente espere ( ping -n 2 ::1 >nul 2>&1 ) y vuelva a verificar si usted puede engendrar un nuevo proceso.

He improvisado un pequeño lote de prueba para esto:

@echo off for /l %%i in (1,1,20) do call :loop %%i goto :eof :loop call :checkinstances if %INSTANCES% LSS 5 ( rem just a dummy program that waits instead of doing useful stuff rem but suffices for now echo Starting processing instance for %1 start /min wait.exe 5 sec goto :eof ) rem wait a second, can be adjusted with -w (-n 2 because the first ping returns immediately; rem otherwise just use an address that''s unused and -n 1) echo Waiting for instances to close ... ping -n 2 ::1 >nul 2>&1 rem jump back to see whether we can spawn a new process now goto loop goto :eof :checkinstances rem this could probably be done better. But INSTANCES should contain the number of running instances afterwards. for /f "usebackq" %%t in (`tasklist /fo csv /fi "imagename eq wait.exe"^|find /c /v ""`) do set INSTANCES=%%t goto :eof

Genera un máximo de cuatro procesos nuevos que se ejecutan en paralelo y se minimizan. El tiempo de espera debe ajustarse probablemente, dependiendo de cuánto hace cada proceso y cuánto tiempo está funcionando. Probablemente también necesite ajustar el nombre del proceso para el que está buscando la lista de tareas si está haciendo otra cosa.

Sin embargo, no hay manera de contar correctamente los procesos generados por este lote. Una forma sería crear un número aleatorio al comienzo del lote ( %RANDOM% ) y crear un lote auxiliar que haga el procesamiento (o genere el programa de procesamiento) pero que pueda establecer el título de la ventana en un parámetro:

@echo off title %1 "%2" "%3"

Este sería un lote simple que establece su título para el primer parámetro y luego ejecuta el segundo parámetro con el tercero como argumento. A continuación, puede filtrar en la lista de tareas seleccionando solo procesos con el título de ventana especificado ( tasklist /fi "windowtitle eq ..." ). Esto debería funcionar bastante confiable y evita demasiados falsos positivos. La búsqueda de cmd.exe sería una mala idea si todavía tiene algunas instancias ejecutándose, ya que eso limita su grupo de procesos de trabajo.

Puede usar %NUMBER_OF_PROCESSORS% para crear un valor predeterminado razonable de la cantidad de instancias que generará.

También puede adaptar esto fácilmente para utilizar psexec para generar los procesos de forma remota (pero no sería muy viable ya que debe tener privilegios de administrador en la otra máquina, así como proporcionar la contraseña en el lote). Sin embargo, tendría que usar nombres de proceso para filtrar.

Editar - Modifiqué el script para visualizar opcionalmente el resultado de cada proceso

Aquí hay una solución nativa por lotes que ejecuta de manera confiable una lista de comandos en paralelo, nunca iniciando más de n procesos a la vez.

Incluso tiene un mecanismo integrado para distribuir los procesos a CPU específicas o máquinas remotas a través de PSEXEC, pero no he probado esa característica.

El truco para hacer que esto funcione es COMENZAR cada comando a través de un proceso CMD que redirecciona ya sea stdout o un identificador indefinido a un archivo de bloqueo. El proceso mantendrá un bloqueo exclusivo en el archivo hasta que finalice. No importa cómo finaliza el proceso (salida normal, bloqueo, proceso inactivo), el bloqueo se lanzará tan pronto como lo haga.

El script maestro puede probar si el proceso aún está activo al intentar redirigir al mismo archivo de bloqueo. La redirección fallará si el proceso aún está activo, y tendrá éxito si ha finalizado.

Por defecto, el script ignora el resultado de cada proceso. Si se inició con la opción /O como primer parámetro, entonces muestra la salida de cada proceso, sin entrelazado.

Mi demostración establece el límite del proceso en 4, y simplemente ejecuta una serie de comandos PING de longitud variable.

He probado esto en XP, Vista y Windows 7.

@echo off setlocal enableDelayedExpansion :: Display the output of each process if the /O option is used :: else ignore the output of each process if /i "%~1" equ "/O" ( set "lockHandle=1" set "showOutput=1" ) else ( set "lockHandle=1^>nul 9" set "showOutput=" ) :: List of commands goes here. Each command is prefixed with ::: ::: ping /n 05 ::1 ::: ping /n 20 ::1 ::: ping /n 10 ::1 ::: ping /n 15 ::1 ::: ping /n 07 ::1 ::: ping /n 05 ::1 ::: ping /n 20 ::1 ::: ping /n 10 ::1 ::: ping /n 15 ::1 ::: ping /n 07 ::1 :: Define the maximum number of parallel processes to run. :: Each process number can optionally be assigned to a particular server :: and/or cpu via psexec specs (untested). set "maxProc=4" :: Optional - Define CPU targets in terms of PSEXEC specs :: (everything but the command) :: :: If a cpu is not defined for a proc, then it will be run on the local machine. :: I haven''t tested this feature, but it seems like it should work. :: :: set cpu1=psexec //server1 ... :: set cpu2=psexec //server1 ... :: set cpu3=psexec //server2 ... :: etc. :: For this demo force all cpu specs to undefined (local machine) for /l %%N in (1 1 %maxProc%) do set "cpu%%N=" :: Get a unique base lock name for this particular instantiation. :: Incorporate a timestamp from WMIC if possible, but don''t fail if :: WMIC not available. Also incorporate a random number. set "lock=" for /f "skip=1 delims=-+ " %%T in (''2^>nul wmic os get localdatetime'') do ( set "lock=%%T" goto :break ) :break set "lock=%temp%/lock%lock%_%random%_" :: Initialize the counters set /a "startCount=0, endCount=0" :: Clear any existing end flags for /l %%N in (1 1 %maxProc%) do set "endProc%%N=" :: Launch the commands in a loop set launch=1 for /f "tokens=* delims=:" %%A in (''findstr /b ":::" "%~f0"'') do ( if !startCount! lss %maxProc% ( set /a "startCount+=1, nextProc=startCount" ) else ( call :wait ) set cmd!nextProc!=%%A if defined showOutput echo ------------------------------------------------------------------------------- echo !time! - proc!nextProc!: starting %%A 2>nul del %lock%!nextProc! %= Redirect the lock handle to the lock file. The CMD process will =% %= maintain an exclusive lock on the lock file until the process ends. =% start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 !cpu%%N! %%A ) set "launch=" :wait :: Wait for procs to finish in a loop :: If still launching then return as soon as a proc ends :: else wait for all procs to finish :: redirect stderr to null to suppress any error message if redirection :: within the loop fails. for /l %%N in (1 1 %startCount%) do 2>nul ( %= Redirect an unused file handle to the lock file. If the process is =% %= still running then redirection will fail and the IF body will not run =% if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" ( %= Made it inside the IF body so the process must have finished =% if defined showOutput echo =============================================================================== echo !time! - proc%%N: finished !cmd%%N! if defined showOutput type "%lock%%%N" if defined launch ( set nextProc=%%N exit /b ) set /a "endCount+=1, endProc%%N=1" ) ) if %endCount% lss %startCount% ( 1>nul 2>nul ping /n 2 ::1 goto :wait ) 2>nul del %lock%* if defined showOutput echo =============================================================================== echo Thats all folks!

Aquí hay salida de una ejecución de muestra que ignora la salida del proceso

12:24:07.52 - proc1: starting ping /n 05 ::1 12:24:07.52 - proc2: starting ping /n 20 ::1 12:24:07.53 - proc3: starting ping /n 10 ::1 12:24:07.54 - proc4: starting ping /n 15 ::1 12:24:11.60 - proc1: finished ping /n 05 ::1 12:24:11.60 - proc1: starting ping /n 07 ::1 12:24:16.66 - proc3: finished ping /n 10 ::1 12:24:16.66 - proc3: starting ping /n 05 ::1 12:24:17.68 - proc1: finished ping /n 07 ::1 12:24:17.68 - proc1: starting ping /n 20 ::1 12:24:20.72 - proc3: finished ping /n 05 ::1 12:24:20.72 - proc3: starting ping /n 10 ::1 12:24:21.75 - proc4: finished ping /n 15 ::1 12:24:21.75 - proc4: starting ping /n 15 ::1 12:24:26.82 - proc2: finished ping /n 20 ::1 12:24:26.82 - proc2: starting ping /n 07 ::1 12:24:29.86 - proc3: finished ping /n 10 ::1 12:24:32.89 - proc2: finished ping /n 07 ::1 12:24:35.92 - proc4: finished ping /n 15 ::1 12:24:36.93 - proc1: finished ping /n 20 ::1 Thats all folks

Aquí está la salida si se ejecuta con la opción /O muestra la salida del proceso

------------------------------------------------------------------------------- 12:24:51.02 - proc1: starting ping /n 05 ::1 ------------------------------------------------------------------------------- 12:24:51.02 - proc2: starting ping /n 20 ::1 ------------------------------------------------------------------------------- 12:24:51.03 - proc3: starting ping /n 10 ::1 ------------------------------------------------------------------------------- 12:24:51.04 - proc4: starting ping /n 15 ::1 =============================================================================== 12:24:55.10 - proc1: finished ping /n 05 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 5, Received = 5, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:24:55.10 - proc1: starting ping /n 07 ::1 =============================================================================== 12:25:00.17 - proc3: finished ping /n 10 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 10, Received = 10, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:25:00.19 - proc3: starting ping /n 05 ::1 =============================================================================== 12:25:01.22 - proc1: finished ping /n 07 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 7, Received = 7, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:25:01.23 - proc1: starting ping /n 20 ::1 =============================================================================== 12:25:04.27 - proc3: finished ping /n 05 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 5, Received = 5, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:25:04.28 - proc3: starting ping /n 10 ::1 =============================================================================== 12:25:05.30 - proc4: finished ping /n 15 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 15, Received = 15, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:25:05.32 - proc4: starting ping /n 15 ::1 =============================================================================== 12:25:10.38 - proc2: finished ping /n 20 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 20, Received = 20, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms ------------------------------------------------------------------------------- 12:25:10.40 - proc2: starting ping /n 07 ::1 =============================================================================== 12:25:13.44 - proc3: finished ping /n 10 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 10, Received = 10, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms =============================================================================== 12:25:16.48 - proc2: finished ping /n 07 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 7, Received = 7, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms =============================================================================== 12:25:19.52 - proc4: finished ping /n 15 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 15, Received = 15, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms =============================================================================== 12:25:20.54 - proc1: finished ping /n 20 ::1 Pinging ::1 with 32 bytes of data: Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Reply from ::1: time<1ms Ping statistics for ::1: Packets: Sent = 20, Received = 20, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms =============================================================================== Thats all folks