From 05c9c53a9803891314cf5aced7db4b9ce064f304 Mon Sep 17 00:00:00 2001 From: Hilary James Oliver Date: Thu, 24 Mar 2022 12:03:37 +1300 Subject: [PATCH] Tweak reflow and running sections. --- src/glossary.rst | 50 ++++++--- src/img/no-flow-n.png | Bin 32730 -> 29371 bytes .../running-workflows/dynamic-behaviour.rst | 25 ++--- .../external-command-execution.rst | 28 ++--- src/user-guide/running-workflows/index.rst | 12 +- src/user-guide/running-workflows/reflow.rst | 106 +++++++++--------- .../running-workflows/scheduler-log-files.rst | 6 +- .../running-workflows/simulation-modes.rst | 43 ++++--- .../running-workflows/tasks-jobs-ui.rst | 2 +- .../running-workflows/workflow-completion.rst | 7 +- .../writing-workflows/suicide-triggers.rst | 4 +- 11 files changed, 152 insertions(+), 131 deletions(-) diff --git a/src/glossary.rst b/src/glossary.rst index 8c92d69de0..1f28b5630d 100644 --- a/src/glossary.rst +++ b/src/glossary.rst @@ -43,27 +43,36 @@ Glossary retry - Tasks configured to retry on failure will return to the ``waiting`` state - with a :term:`clock trigger` to delay the next try. + task retry + try number + Tasks can be configured to retry automatically on failure, one or more + times. They return to the ``waiting`` state with a :term:`clock trigger` + to delay the retry, and only go to the ``failed`` :term:`state ` once the final try fails. - Any number of retries, with configurable delays between them, are possible. - Task jobs can get their own try number from ``$CYLC_TASK_TRY_NUMBER``. - - If the final try fails, the task goes to the ``failed`` :term:`state - `. + The task try number increments with every automatic retry, and is + passed to the job environment as ``$CYLC_TASK_TRY_NUMBER``. .. seealso:: * :ref:`Cylc User Guide ` + submit number + task submit number + Every time a task re-runs, whether by automatic :term:`retry` or manual + triggering, its submit number increments. Submit number appears in the + job log path so that log files don't get overwritten. + + window n-window active window workflow window + active task pool This is a :term:`graph`-based window or view of the workflow at runtime, - including tasks out to ``n`` graph edges from current active tasks. - The *active window* is ``n=0``. + including tasks out to ``n`` graph edges from current :term:`active + tasks`. The *active window* is ``n=0``. .. seealso:: @@ -143,14 +152,17 @@ Glossary the same as :term:`workflow id`. - active waiting task - An active waiting task is a task in the :term:`scheduler's ` - active window that is "actively waiting" on (i.e. periodically checking) - an :term:`external trigger` or :term:`clock trigger`. + active + active task + An active task is a task in the submitted or running state. + - These are the only waiting tasks that matter to the :term:`scheduler`. - Waiting tasks ahead of the active window in Cylc 8 are entirely - abstract. + active-waiting + active-waiting task + An active-waiting task is a task in the :term:`scheduler's ` + ``n=0`` :term:`active window` that is ready to run according to its task + prerequisites, but is still waiting on a limiting mechanism such as a + :term:`clock trigger`, task :term:`hold`, or :term:`internal queue`. external trigger @@ -172,6 +184,7 @@ Glossary * :ref:`Cylc User Guide ` + queue internal queue Internal queues (so called to distinguish them from external batch queueing systems) allow you to limit how many :term:`tasks ` can be @@ -1424,14 +1437,15 @@ Glossary expected output Task outputs that are not marked as :term:`optional ` in the :term:`graph` are expected to be completed at runtime. If not, the - :term:`scheduler` retains the task as :term:`incomplete ` pending user intervention. + :term:`scheduler` retains the task as :term:`incomplete` pending user + intervention. .. seealso:: * :ref:`Cylc User Guide ` + incomplete incomplete task Incomplete tasks are :term:`tasks ` that finish (succeed or fail) without completing all :term:`expected outputs `. They diff --git a/src/img/no-flow-n.png b/src/img/no-flow-n.png index eb70697bbadfae50d0ab49802cb5e3420f2e0155..efe0cb53eeca8f93fe44b3b81c2fb95fefe1a447 100644 GIT binary patch delta 12933 zcmc(FcQ}@R`~O`^OG%WKSw_i@WLKnYQIeHCGBR?@x|C#7B4kFg`51-F%7_Ls%D9o+ z-nYH)-+8%xzQ^;=?>L^{^ZEXc=laLpb)D~XpRe;g-$z}pxwJ2@2|=8gEPuC7kZZNYN#cm5SR4)f-uON8(C zUc#IzuTE2NUy6*36f~&|b{&e;PA)6cW#usM$krkJLNEIy>ogi|vkI z4QMbGLI{g_h}2uPL_Rp%X(P&4xiVmBWficrO)c=lq0befd4u%)dpm z`QLDhR!K8y2=6=c*X8Bqzr|kQ@5t3>w6(R}O3uB}5d&(})dWzRERMG;c93m| zZ*BGwE3TMvpBi%(AN~=~2%!$S;&PE6L535Wo9Pn57hV9~p_>D*}mzMl_%J^iLndPS^2LvG( zZWo$|Jg&!_cbWX{y7I@*qzy^e0K@0K8r;6V(jG9yV=+%tLEO4hbD1BZ;r1t4y#(~k6l-=~KKgXOsR=*mCe&w&y zqf@VYs~XP$L3&UkS*@HPd6C&|^fjHRO|(Fg@?nPa+$YZ6J{pUN+6I31qda8leJ%Ay zzAIs%WctGdD4hNJH5*hEO_p|^ONpd+FpG?I<>r*Vf~{>195*ZMw6wDeWs-2I`RKE6 zuaI|-D?Y?wBp!QkccTYCc58K{9@O?oHp9?E7D>*1IVI?|;c66dbSQLBdUkg3@UVFm z^GVjTXU`%7_YBwXfE5SG{GUF3nx2&vghv0P2}X>#F^|{nTf5cP*0!H)at!L?%zNAJ z6FE6KyRWbL+~cG+Ty4}IXynPGAwP<*4-0E}!Bs!ro^>kf>w(jM(0UA$oV!NBBCSVv zSB+LhMI|3Q)9VU4?g_u2p>=k;a(%jF=p~~2WLFJ1@3EcQPk;65)k@ukN-AOVZ%4`RD?<2{6Q!jL+^0ksyZ6Li zus4ug>G#pNd6R>JmEDUt@p12=6E&}|_!%!x@_3;Z&0x82Sf8&MvL{;)T#W6=^=J2|=X#SY!w*Q3$ID~8Iq zw6UQfpS8Fe!f+ha4_lo0RZqaz?+q6SI>8&bkJ*)#{4m35^!$yCXeoCWh*}M!eL35+ zCm9(T0afc|RY4b>hWC@L2!DQhIB0ly*mWYO*dqjCeKh<%me!EC_RD0p|KlfajYg&e zy}#{xN?QDhPe5j{AG6z+Vxvc`UBB%*3pk(>r5>G@6^#wy_Z(Al_{v)>cQ*3a{@K^= zvxoRfSg(dLGfn);3-gfXkJu9}WTuD$Naf)}9&f-Iw@dCD+4NQ7EZ!eu2j8Ngg++QF zTD)nsdW2!I22|HDG-R3R$Pqpu!p5l@M~AkeIHw;MKVhV`Zr;3kPOn5@X{vifSogql z#nge9x^@Am5?=kc$A5iDl6rPK7K#bk8bLx+6v7Usu;@P=iX_zXdkG4~1Pge#{%mcf zmf842iC-N_q`fM`aM97!V+y%v7-ZG@_5hiftps*D7}=;t79RhV=s7^0=;u--YFDkb(}`F|2zZC(R8&Y+uH$Cvpj3TZ z>JgzB_LOSUd9uD}sQG@O?__bWTNYJw zwayuugM^Lwrh0ssjGde_OAI3BV0=!e;Pm$N(4T!$+?l&_K7OX=txD{ZmVM9>^_;$o z5K5aof9$vH&#u^*`%17UF4@BjaVA(=z-?jlc7f@2fB=2EYMqNPlz7az_XR<59ZU&Z z>etqDkUwWo{=2Q!v23Z4iLCx5ukJvL0?T$kN{NC>D?!HI#tyviwwtFvk!0!S<~9(( zmEdoY=-J^vYH%K^V4Ho6kUX7i3m7EK?77hLyVyQaz-{970ww?6nGX}W>FMc5_p8%@ zU%ZwI6Kgfqq=x*2D$?J-7oDymZf%#X`uUmic~!}^(G=*GM~oJdp+QKZ2$+s!W*#F~ zr}4eamH_Ft(h}x12!}0KX8#bhkwtAm_az&n2)3cnU2n6pc)ZqH6YGSVB=PGvYOS=`2|JLy6E$O-=2-sD!|*x{V@& zPQ%JZ5$nW69xR0(#L~cnQ0ofWkerjF!fJ8%N8CjRP^~^%(B#Rkg^bPh<@y(D()7H% zyr0C#L_|a!x^0VV+qDg+gzv~Eo&_<+wSlf1gnB$W7+fTyHE=aT9XdWS;S0y1|cu+Om6(M)n{1uw1R^%NZ;pt6fI+GP_Z90#v$406c? z41JASXa7h~civK6&wc70io)E(R8);j9Itb7coRJ48g9S0iDATBSzB8A*9Ot9M;>UN z>B!Ngm09^nIS`NSrK+>~Vk+dY*shJ#+ZN-@H-ww>R1+lWm7@4U3^&KMIR)e2LdDmP z$Pibfg)Kq_yuF|q+1FBf&HDT7=(pZwR=^bPhgp-QMrJlLA#u@IPXzbs$SyWM$~!xj+{LD#m~lkq$!al zadXllB{x@TF84UK;BdUpC zZ=)<|qM~33<&{BZ#+na!gBY*Q4H4!Oi9OLRG7yuNwl*5T%6l;I8GtcNP+Twh^cnIh z*D}NFSn7Lec$W&$Ty~Umwl7h2#XLDn*^WLkyEz38ohw5KdeTRF(H-T5xc9DirTT^DDw3NEt@#foItsoe zWoDj5=i0=rg*26M40!}};yQb;`B+z&NzqBlEau~z>n{di;0lA$jQbWsMrKV}3AyGv zT{>d?{gsGW%q18uU_L1UWHyGe{ls2@yC)@F&Uv8~pXarhB?hCcj=8$FHZ80?Drn17 zWUBTqcsM`20y_!wyYUH=Bhictw$B5@1g_K)NM?)cYALy zGkEmI4u|mX<|e)64^am4CMGF+F%*{F#r9!+TT8`+vc63|adC0Qtr=a=6wO{(VAuf_ zt#@I_Zdq7Zl+QKt5I!M?_-mkJDs9cgHkcZtCcJ~*Z!A3lXaKii>YWDSI?Y+E;698|ih0ioIqG9XQ zbLljLhp#Y?i6vuCO&OU#xVkQ9SSg0 z50iwF-K%<7x~Ne6fFljNE6Kn3xEVPIXHB6*REq@n#qvyLIzsEk~5zG1T0; zx~u$q9)%Y|JOGV!ii(PAp!q3|&xpd7xl+R=z3o^87KEN!az#aLZS8EmQXv#}e-~M4 z4$iNJua==}((tz!YT$RH5g!a@W=u~{-!8V(N7+FUlMnK1z>t7$SFd7YO|~{x1z`xF zJ;8klg%C6bus$1){D>!;n^A|$4O&AF2Xtlo79)Hr_WOswWRZr6v9SOsWH}`zrShe& zdt^&d+=mYzM(Yr!FzNWc5g~xq3h)&b2@4RpX-f+1sOC)xOh6!buj8D_`lEtpO(S?H z?ckR$U*4{8drTqIA%*zj`C*+yG2! zKp&J4wG{?kzH=7xr9Pve*!S2A^&cKg7;mmE=0hgbY!!CET#o`xq51c~7-0*Bn|mTE z&~~!6IAMbO@F3sl^V1qtGsV;!EzQkGh2eHbr_j81VXXDki-rEk-Dma7 zTsW5G5hGY)^<4k>rY-fuMFAhHdIzc^I>V!rAZ!p+^2mx*0lPEBk{D0 zi;HzMyFC0oq*=u2yRN0U!6bm z1MYw{m+#}(c1i2GyRWlMeZ)@pblS_jqKEAPZf6$THr?S$YpEZ@lE1I73L$1lt$;sF zU|EGm-eg{gZZxfFYBCQ_NJICg4KPUQ`Dhpc$Bu)c43ztaG z3%8o`U`dD07q=#C^y!Hubn&${H+%pl@V+`>YtwzEw<2x252!8ldz4ad2m-ikF3&*E zP0z~4zpI#`}bQd%o{MSJU^xIW3Ad4ZtKu{5V%}g09DtQky$Xf41=+F zosx12mKa8GZm8$_QZ2krdG}5P)Y6I===G;MrHZS9NV$)1=>wjPId|>Fh@nM>rX^|v zY(m7g&2^Z=Bqo}6p*al^l(|nS4QTO)EY3GKH}`iL)_$_<9S`6gcAO#=(v7d7pofo5~79)Jjz_V!^mqKt6GNJRxZ{&6T3 z0HbHgy0(&3qW6xF7pa$g3%!VL0p0Qmq zhMcSuN5~d@5)u+9dwfP5OQ9;1Ih``O^>uVsE!^+>!8C@4tsQZuw70h>7D~B-KvSU@ zxT(eHB==bAtv!>*m* z$COLSc7C#cjF9-71+@P~osjEi0EBa4@S+VMRI7 zT;gMK=)cC+doeD|bLPZAwY#rtan4b{?t?Aun&rdSQ%yFIbS-7N z$Uv$qH~4mAFtP!P#}E2Grb8L^<1h9a_)qOSm~V?nyD)RtOkY_YZCLD+X?m8arJSo& z8{se(eIqLHso>Zbb3HDw;YL!=c8^G7GxY!cP=LjL>PLNS*z+CXGguew!(gfr*;LXYmSlDR-(IXN_lk$`i?*wl5*P1>{j3X7uAR&ni*~o; ztfSCg0Ile*kvdMlS`jmEO_u$Tar;Y;lM*^^^%f(+D)o=+ob@DmsUwf)UI1%;p24D) z$jS}QQ{5{bF{zC$b#*tCZv4}nUBEsNZWU89i6kp4r;B`thQ^vf;>>+79!FKx9T7y^ zF*HztA9B|cHPaK%O^6RT)O6kN6(}uRz`e~^{bg4fyTR>W)L;9)$SXui_FnMPI-8pq z8xo@#AqjHBFK;(W?}DG5O!+pRd?@LZA`?rdmotz>g#>Y?`Xb=5=B6FVhkP&BWb&4- z=qT}Zsfr(y&~L3v!wW~5n;7R@WCn(t@+5}g#Sjv&f@H3xla4eloIf-ChwGF*+`KrS zYMprqbPFOwN8k@e@s@&ijc}IRK<(ku`v+Z?$hEKbv_*vMr8T`=3l=BMaAVQ_hfxpj zuxqLhGn@`Ek^XqYdE{3bKJ;s$hJ@BO75|R4H^e`%C#>nKmCu>0(p98`2Rrt5-gdoo z6-}24QXP4R=g;4j-m2BAH=46=C90X47^O57 zflQiMM~DUnL)P)y#CPSFMH#-DNvX+H?0Nmk^2P67vyi^_stXXswM;Q zQ=!G-!{q^ECCgobNIJvmLy)`1q*>=~wN4=~3z5)0J=)N~p1~QcSgW0z=MUxa33!Pq zH8PV}gio8C-GJ9DQHdGF`0;9$5kZ9h>qq1aw@*-FAO;{yb zH2jI1LDRp934Jk;G;s$=b{(6-#yS#Oj|Vc5;Q%!@^3uxi{ZxA?!KN@JQo7W*ZBBVe zR+e+M?NYp=>&~_{6ftH7lwHQ=L%8rozy30fOs4UO7DLu|E3LA+dpmiz`tf-S@ zER-$f`m>ZbQ|~1^+4%F(DPPy|n^RC7-jpe*RR~75YCVs(3}1nN#WXZI6jJJHlBPBX zDtH_vIe4x8p6)zht=nLnny@mLAlCAF5W+kQYXT5qMlW!QA0)ic_6YUqlK{L-%^ThW zY-hRt{w$-@LS5gjV%=Yr(-Rk@;;!rnpH~uhKGPzxOM@nhxA+OGPQ*oHHD|_Ah|2vy zQeu1ZC5GKz%Pd~@_zMdW>3$&H1Yc*RcNFVUVss+>FiJG}^Lf)vMXem$gU;ywq)bi< zi=PZ$8M4-tSh3WXqH4+(sz5<*2a=?hV&6$Q6lfknHq5f#4;m*P;0I%NJQj~<@^!WH zv7(4exXdlpa1@de4V72{@_65QcF3dq5Hy!RMXwd^Vho05Jl46+-z>z4Wq zSarTZQzU|AN6`1!viX)4j(vd^Q%ib-3lM*Aq3tBX$mTu7wH*nZG_|OCSBQ$+a^;fxR22n3l?ue?@EmP4>!p4oF zZFigD>P}d-xv|Z$y+xI%@MKb5w!DK@_}YY%r`bKi58gZe{k~!|ly+tz7nid-I{qA@ zug*-8&HZ6&pYl`^c4&?YDLZnO3XB}OM2cZ@w?nlO)-CZE6_E4okY~P#r^-<%cTQVA zvrW5`M9E#tcG6lBqp|PIQ~#Ak7xwod>{{J(YG1JyqYC(vsaTucH ziAD+f4^5=wKF{kY1@}Hu;Aq;*Yz^MGVvp$BgDdHGEVumN&aPVI-l^KB!HI_Q5h;}N z^LDyy<<@EZOw#7H|0?XFn`C$VrJN45G}jNqk5UOHIRPT2s>Y1Dp9lI7e(`nx!w`n#jazi{Z1 zL$b?)DIlCbhaDJ!$~(Ei^v30+CbLL1@s6eK#NM?*wJKX8Ee_>3{~J`o$hK%fj!+hG z*1aO-a8QycD#1>XYadKzpIBM!7BB&9nPvir!oxY> zg-J@U$tDNY~e4hV0t?t2)(+rq!G%d znTPXNs>#Y-{{AzSI+`c%!I7a_&7#M+7R)^pW)di3glhSkP zdHWG1`As@%Bak>ZFvb8MybXj?`It^Ub)`}x=t&|XOt)x2k{u~Tqaqnva`O*ElemAn z1^U08_Y6nzR3;%Cb^YkgLkz;tMXevPM~kvCnLO211~GrbrgjKjc;Ngv(-nuM6QU$9 zbE&n%Da^u$uY7Yi5({@3*U>r}l02v>^^-B`bOgQSf?Z{t4Iz^GG`5 z^T2zeng1fK()A`oB8c4iw&D zI~FGQ2=Ft891hKC{HPyP&?0arA7uCQ6C$7_FtW*SG^v*LG^wbqD;D_8e(8q?JZx35 z0wcw8EX)AMFqmiOp-7hv4XI99(?}||)VBU~z?EmS!WY$@_8)M*ob$M0WiERrLf()Z zp`7?Ev{w7(lP=JrdI#EA^EpFRYYqFqGid!E|l3v}84cup!>Hjej-Acr$GcLbY z2NXf^pX_7z%pkt3$wk4U_#8Pb)K$bHRh65H=w4-Ujl^k}S^Xmqj}aK$XTs%w#!Ij< z8CL#CyA2dSUXt>c?8nyf_SBBQADfc%=9RDrQ&8UA8E7>Bv)ss9OX&C`n`YQa>j&kL z=Yz~m2axx^o!3^6!3rd+@%Q5LU{P*k05`2NpH<_jqG)Bgx0=rx7 zKJ;Bra@A37&{=ZryFXh7x=cl%uOkudbcfS6xTrAVdTau08zotJ`mp+!$>>n|@R&ik zNIM?b&rM%1ageZ5kN}f66|*N>Y(R;Zov8Fh3F+^(fnR+(3v_ZlpL}(0PR@ec@5kqHSq6O_o7TNKhO~pG2mZpk?psDppNp|*v7$6jK z_+A@BcTp|_g_ASe73EE=s#t%Rfe-vFw(&zxV? zJQsb_-f@S4@q9+r^fyz7vhn0SXZ~bKHzDRH$Uet8x%g3zR99eEMxLehJy=~z;c-4~uVlr(6Ebx+#7>(sMvwM6U6?l`(&0^B4`Sk2)z5-WV9-Tgeb zTs58U+1i~ib2MAmh~<~LH-G=;cgV2&u>mBFpZ@saS1!&xH@7UE7wUI1exbB{YlSFC zf8k*f(q-zD@_qQ~i>6;)-$JrsZLZd5n?_}L6C0il4my9a=eiO~WSCpC`O?rixjH=c zEVpX4n0DkvF1-~9v!O{933Ld%TCTsOO}=} z3r&S*jZNzRqNIOUIH4&WqjuG{py;J!U zpCkd2`S4TL=wFCQ{zfT$Nfx6&50bP%f6{PnTZI1PR^FZ)(Vyzkz!=3Jc>X_d{`VMz z{Qqa2|2@wC9%J+-5{xtH%g{y(NP(rW+! delta 16319 zcmch7cT`i`w=N!!y&xTdfQkZ26OkGqDk{=NKsqR)3P_V0U}HNdEg~Sj2-2JM4k{(m zMS2TOdWoS!0(ol#=ic8N}9_ZP~E23tQ8ZaVkJ@i)g=g^mZl%});K=&~q4>z30ELTR%S6IJbUX*)TfSQIuV_rnA?f=Zw>; z`)oBmWvp`bn^Hbh5eR8l@at~Xs~8rRu_mLPJdW@5JkV$$+WOn8W4VuO{_4#$^)sfA zh>VOZS;u40U%U_a9=Nl0mILW*U9mYC71Urt|LMC@jAYj#e)`;8G_JnNlVrTmU+Al5 za)p<7TyJ>m;Yn6eC87AEnwpw*8^rPP7O`Rfq9V~lm#@)rtN5cfgHgmeFP;ZAKGfI` zA3kUoJ_&`ufzKonF2kNK{_N7aU~gzx7%nX(mc?MWf?~;k+BblEi#xEo{iZWr#u6=d|WY9%I-tl2Fm{S~cjb*wg5TJNN&nB2Nh7 zfHMpU;ZMz)8pi^Wi=S-{(w9Czc%uQBd>GTIQ>Tur z-F}d!;W5E$WN&-u&SJ@UpkFzm3i;&86Z*vOi&;D5Hp*b7=JSryL|)%m6**a1*$NQ2 zTo;1jq04q0{8v;XugeR?-xlybe2RsBd8!SoFOYKX(9>VvgNRwSp1dGv4A4P^kw}}y zkjuHob-ulYR$(vbOEj)96yx2kl%hnHh2sBQSr`zz8_0GRj)%2q#<@Fos;djO(7=bH z`xknwQ{uebGY!f`i);px&)pd*o6Gh)Ay9JTJlMB5gsk^JO9$^;TXO~;IXO8M74Xx3 zLl}HYNVveI8q4O#2=V*Y*WW#Kd4x4&;AuI3tJJbmoj2_@*WV+l85w>cEm}gyd*7Xx z9m8V?0Z)g@E*}nQY*Cd&1aqqfCMkv;*I_DQeOmH0;aW)e&Nc}*^7SS9>x)CTx(~@< zAtDp>SnaIBJslmLXs@lA+$UY>y`_#RFX%rDdJ+BXHz&e)|Aa!`m6b`JpS$oQLn{Y{ zU1C+0p^&e)og7nodb)3i3X-3ZUzh&KX}0_C{-_WtUG3IGgIoC?xHP>nj>RqxmE+!D zQlw9$I(7Ei#l_O8*y;8}ALAQuH+f7dt^2MW!2S`UkTkY6mxJ{atkMSIJjHUIXKQti z=>(Gi{coNd&^Bouiy>5sZLQ2CGT%3rIj)nVU;2Ql@(-sz)3CZOEu*Y6_Y->T0KNh? zm#0p+(#PW~F*wilA*PpyPd)HAF3qN0oYoZ)5o!PN4wpAs)5R&`|sOv*xu+6HVy##i8R5 z&H>O3x{e1BUkiHN6pDX!3L$y@uT!jyCm8uYm+0R(e434Uc(c_dAv;?IpbO?pXPP=1 z5HZ)21oC_?%+TFZAG1(+{JV~9XH&)TU(_7zz~47YT#4t7iFs`NE+JHr z60~v*|K2TX(@(EgWDTjkd!?B4z`jFStd2oT|0DAafOoAgiC-2Ys+MC2x2wIj=@Wai zQ(ht8f6U9fmaH7b$SU^KKTEIp%Dr1%-QD$K)tfEMD6a}*DjygEcd_gULdA06iGe?( zq}&`}CaW13WHH~kO(x|WX%ip!2M{#3?)`l8CT%)E9#plnnu80utWJMWJQLhkMP~Dt z>-vyOVp7tbo3u~J8$@D@t(STUi+E;cCSS5r1nm)OcK@ynE#7-C2%%MLg*koi*)|Y> zT0)jTKWP_Os9hHkxqRzN*nxSDf^A`dPr4Ch;^o0vD8zTn~$%n%LEUw;O zU`9G;8dFuLsK&_%nKp)K=b6Z}iQC<{NoxkcY(Ev_a@W)}2`u0YRj9k15={$oM+7v! zeD7N)2pqI2NV&uXDMx++XUz2Fn}f^nZTGoRWDuSTa3O zY|E#UcQLOoGvVTwH#D4ZDJc}=v%3xa5rS;`3$*ji0&I$C6F`8o%NNT^*7~gyGc%QL zdRKrF&mjBP$+vZWOmnXy&%8WB9SCzZ5Vhl4N41omogHY|kz6;~5@nMgdL~WdN_NSZ zZ^^_5J@S{_-YH3L>37;qVS-qQjD!4PTHa-noQifFn~bszd?uO2{3Ycl9>TL%Z`28n zC50J2@E3SCXpgHCt0D$4i#!Q2ri$w?bDA5&43lDyyfT<~j3ep8y?|(CJEGoXmQVZ@ ztXdzeDl_AA%^+0{%WjWDxb(I%ZoIS}Dif(94H4=ATImx{#5ng`5({%Y`Qg5Ca6`kP zR9}iTpHG}l#BP0rl$hXa4A@pngMysNeCb9B=O_p$-7sZm9o?0#>DzwD=&#}-XFZS} zO(E|#lkQ9%q9A2Su)V)Oof+3F&mBYWf_F1tneS5<@lvaNgDYY9CcJXkCc^_iHRKBJdAwW@ys}F z$(TT^>{2d)vgr#3s?Xa@8s1+b)^O}xWnpDy7-wd^W^&JCoOSrkfrE#6?!G;<$U_yO z#1#weY`V_p;w}U?@O&4#YFzuKWVsc&ThF}{^W2svODH=e^iY|Tul>nS9Wy25rKa&l zh&QM64;Jw&rTPMf?lVcQ8>2pjZ)ENsre0l}D*D+d+_O zfs&CIB|BSlh2Dn)`&n%(Xi(cTsxWJX0-pn2Qjc&Mq)C*+K0G;+IzLb>$nHAw98fIC zm@U!j_gP`H+5j

YQa7)KY0DH>zFgU(p~N4u)P1r=FOxcE#6fi&Q#9Iof9P1edO;ynSO;ng zLKU@LQ1|w9cQr)WXw<5r<&N{nGp33$-riazj=ZF5s0I?RAF4m!SUD z0!7Fs?Y?r~|3M`a-6D??l<_dj_Xvl_kAvS%-&a$k+3j6kn-l#$f5a>%*1a{_p4DPM zk`Wd&qpmtAQ^Q9K4L4&XEd!u%v0Cu?h{QEa5O1@(7O z+sLP26<+{gp}$U^O@42CsVIjMM&m#US3|f zjHkQ4=W17bzPak517%8zii%R(D;-4ga+{o$RmKaC7E=Tedj(? zASMDRIe`F% zmyf<*(IHqv$L~6feDYjR}^SZJIKRBT7wq+hRKs+#Gi_S zbzff|QOrAB^VcaqKY=oyR`KzZU!TOsXMYXj*JBi{+A!wRFZq1Y3f39wDJlU?0_A*Q zuKJ~XfZiX7cpS(etzb3T0Z5>Du*%4YgGt}+Cg^<f{>Vkpi_V*P=b)J^8CU zXmGHm5`ap7(Z)z_XuLuyd;w842CWwmr1|{@g8~?0uY-yOW(wY+!}JvsO<_2zoJz*! z$rj$X4-;W60{0Oe^B4voWbjlszJmVl+hYVto3%#k&{p5NEGUR=2PPxAIaQr3e+07d6UM|H?*ygq;kU|An6;W!5PdE(AQD%Hzp5 z#HEX2O?AEuyt@Ui6-l_QuC7k32vSdDmVg{gqMf>Rgn$f7O-mCRy`d)I`ukCHgfKTA zOdtt*@zdQ|dP=cU63MD@>;U7UeK*=Sf7ejMz!b8@S0v0$+LMVm92^FKTeD{b{E*4P zJ4vOb5&)H1uG_0UaPIG{Dt?2L5AbIUU)FhEoBGe^3He^o_ExxrqgitdQf!SdY@{(I z3P#V3!6_Lg^BAXxE3;iqu(iTM5g_uI^3B@G6R=HrX1(3dTFW??wrq9;7kLc(}sS7S`^d=7{9pknhb!%T;Wz4G6g`LWA1H zPYrIy$^rot4ax#M#nEvoy;)jvpgTcdxwf{ZUG~fpb`S4174u^IQX@PVf$1k%CFaQ- zU%bgYz7h!^2#?vIH#3?1NK8+^q9QR70_LhGO3~tK177hLP4fHTCWu}fEX{rN^@WAC zHQ)DTW^ubm5bPSDwejzfuUw2g$3Ro?15ycYA863H51m~}O-+4sjE<`op78(#fPAnq zmbmf_aT|ldWNPPLU=%cDQ~bd9Hh@)(YfT$yfpMTq1K-kd*S|VOS8(MI1w0s%(LliE zx)R6W%NUnm1Eqjj@|lHF_*LjUmr67#Hzb? ztj-Tj9IMMUYl*Bie5nPV3W3n0$7N^*=BB zmxcPD?*KLePO*xz?d%v9KGB>XEX{9XJP(Fv=M56+6`jgwNbCy@o0RMC-|OozA-18d zRfBGWoa}4?c-DZQG=axZl7F2>TzPloogC=a%2A@b)PYshs1HZ|+QLBjug8z?+_?j^ zDW8h?=&PFH(f-wgzj``5SCBm;p#RV1akRkdRmkDjcMuaW=ee9ZZTHIRH<()4s}36h z$H0hC2)cPw6V!zM{_ebx`^t2U)hN)E(eeo$M_qou&Fiza9Y@OTNIxI+NJyaR^g)0DH%627~i@?!fJQtrGF$53H z3@hCcJ3HAqIam;=-QgaD^_wddBv*PL2+_dd(!-Q#5{7`Ul&gHCrlWHfC_g`-qk+Ga zX=H_7@l!6U5uWQ}VmL6=7=s#}3sBly<%xltvGd|!-}$eVU@irzTO6*Yj1mYyd4Yy3 ze1J;sG3>Pcczth)JsRA56@)%k8WZS_^acY81ja)G&hRC;lmYQT;4Oe(>;8hv-8n`a zQDU|-6@TRc=T0A6T%GS*{w>)ERUGEH`pEk#J1)fTh+I-u7C)Fs#zE!wp#b0DEJ%bvHfu~F01WSE$k0Q@X> znseS@sQ`m#_E-6nqo=9%e!Tze$e`K(eD3Qj(d_hbA2vb}%)|K$3a`o~~{@s0?K#=5GPGyyH8;+<^vpCy?17ie0t zO5eQsZo;WMaN$Y%&i3j|-kZiHjb|CYsaYy_Y*wo&`KnkA4c&H-dmf3WhYrAo4w~fA zKm%)&#!ZXX*7#6A8;nD4h7eOZt7CgV-kuV-D3X>s7vx(OOY?For0$DM8gnN%LuGl2 z(jQb*H-Z!<97?0F!5)3#n7pAb&~4pRH=EU`UChtK>TH%M-Qk4msFY-=nNt@szM)U&W>;xl74#e&w5#iO>SV`0j!%Ka{?WQsRTvPkmROp$8X1FaCmG>vQMffh>Z@!yx zGG;Z@N3RPLM<=aAz9r}YZ!8mk+IKh1Fl(el9RZhraLff>eE+9z{{_5Z?AsnjbK|z> zKMk(13dQraqAnr05k`#0uoDdz2H{xWkbm+dz}3bc)9fxt*OE0$40`VQz`lKE0up-s z>fNr%qkASF0F!0{4YgK&1BvpfoH^$`)$QAVEs48nM;HS`&xH0mt->t*B44FP^p(I?%p$|Gio z^`oo#wIv6`&zst}^i8Ps$kqaEjcA|Q4P$)j_2RoytIn8;O|QE`l4>A?q2?jD0(;9k zU}D;2-&BATG>}isskX$%koesYyea-XepV@3w~Mb7!pr#fmsRNBCYwFxijC=E*U2!z z4cZT>+t0mb5GshX(FN-yU)q5PQc)esO5z`z5*{#apUpIW_HpT1Er(~eRB}L}@(GXu zxxEC(KUx@!^W#7bGY+ijjwwLC!m0|mF7pp^IuWp~XY_kvJN~n5p>KnDV!=MXVd>$| zXz>!hrCr1N$G@W*rRC(Dj8yS2W$vY&C`Y~j81n;@m6I}GjR7zH6_>cb)1!7y{s8MNG|pHeK4N zpuwpdoqL467ac=#B>wPQXpdF%GcI^+F5tE0pl5Oj20SFQk5k&DQjxJZaMr<=STa+- z>0CasS$_fU&NBNUqUB_wLB;hkBG-;NhrG~YGBqBZ&AbH`c>J*!AKLLcmutDM0Y2MF z7no?O!}}QZ){icO1#EBkGV%8F6Gu@)?MG384?2D_5`)9~6QHh60NY=e_AVA?^(0`U z&uSs4SJQxbX<=CUftl?pXwGssQ&q`P)M0=aD4njk%YrXC_1mbzLC{ z_{)-tIhN7lCAL=NC4%6aQydj6)m4;!*Y!{W(+b8I@2kW&oEi*R@CCZK@2Yl{5bKGD z4A7)3**aO;aI+@|H!nv(k_pb%`fOP8LTN6vzJ+#XBROXW$nC6E^cGsqR8#!)1WFc*8>N}!}13bvxdw8`bIeSJY{q2v#t%cWti+Bbx+p2E|HAs4m zF=NRsZn>B_UlmlKLDVo-4%(R<>@7yG5(4<$7vjd!IP?)&Ek};$)t9q45@(GE@<9kZ z_6oT-dU4e>qBpPncM;kW1k7u*cnVi$DmdZU@?4q~J@k_axvkwT;CcS^;=p^zqhXmi zO(A;}Twmb*zW$x}!~nYOc{DX#3HHcq=Z|>u))jhubJS_U!bawdst3WhH`m^NhY7q6 z#s>`hskc$wQ#jgYnDB@xLrQhvM&!aA{w;Tvi)KL~`;`j!VmT=>qjfO%cpu$MaA58* z-xI7X|7)k_1)COPCv|pUmeu5hr?O7j1%c9SBlOtH0JhIQ#$hqWzMX?|vGM1G8)TF} zQ6o0qLwROn(yw=7vGN)y>Y{}1N`T6}Ch3iyyx}NdXr!w>#C$EH$aT_3thRnac)2RM zB2Q0$a}_sU+mRzPi4!Xt-6|oga`-*F zg3o|GY_R&y?7u3b#K)}g?z4jE8k(YT z`GiYW`MK!0T{|Iol8PCcJ$^aR2#1+?yaTa&zh5O%qKdE41t`C6cRL@|Qk#$3 za=0$C_?;9fKecVCwDV1j4s%*va9c$w+zW#z0&3PdqN*-A^nE)v*sK@y_GL>MT=(3tX3quMPq7EwO`aamI7_{cXTUOSBw~% zD+-F_nvEh>AA?4uu-{^0o-AkUTMz0mBbbV^Bbb)s+BgPNXSFVf6 z3Y0B;OwC*jS{JcJk$Q4Hx4*q`6NFmahP(OTkne{|y;AL*0TO=ex3yf3-H1H#bnE7d z_|tn|JFB#YM`2!%=>P1J4NHMhe)yDjSn5O-F3?c}=sZy;ZMj zJWzJX4X3&@sAaq|18LYfUIjR1?y&R36ZtcSeIaW6j?UNUw!sOwc-)ww*-uzS^?BBo ztOiu;hh$G;{Wyi6;F{x=I2Jv(Q}t+Sw1Tow-k#X@xnew@gXCvn^OWz6qfP-A-Puh* z$N1Vkg8#6@^S{eR*vhP;5@x=%R*Va0Z$_o2rt|l@rKZoE?WkUXrVtRS0MY}2rRbcQ za!9>oJi#>sn<&^fak;;r^Fx;Ej_k&eo>px_k5{;h$M5NU^smcdjARdgnQg4`$oDii z9MjGuDc6OR{z5#=1w{|vLBVf78pd^#)&&KZDB{DOo8>`GlZ2}dYhgQO7L&i`IiC@r z?`HkuLs~nz=UUM-Th_lEWYBOJ_EnTR(8ZfW43VejeNAn*t?)!4_3VHHHQY=s(I;6% z_*KML(<{amr^weMsTY*VZfw@Ab$`YD+zsU47U*x4pd03oTvx&^>0$M(}$<5_ZPlNzrj;L3M;aVXlw zesf}O(|5&FQ?ENSb9dtUvx^`<}k4!iB`t#{xXT3a*GLHUg=^~E}Gd%HFhppi3 zrhL)^p`G~wz#?5CWF+;X3fYbR)|;cSkc0LU{qc0+?fKrQd7s6MFsUdPJ_EzE+{)!? z(^E6Q+J~xhKCxdLBp2$gtBkchc1eWMeV@E1@TRZYEYKW@ zdQbW+4PNwQw=^vUQX_;De@$9X_c|^GF7y{%C*rCVVfT3s?8}*Og8(qZ4$e!Dn zxsMg&P6t)5luBfd#;&yXaxEtA$J+##?dkVX z`QwM9T{T4Jo?uRIz_y-DA68!7*ahoik3t=sBfACD$;euMFJG$8- zl=Sk3U3Q5Y1no~nH75BD(nYHMhM)4vicF>!f{Q-MZ*}v@dkw=#|8xJ!N|IkQl=rwj%V*#2*GpE}ZN+4V4b9W=Nm-gE^Q{FQX9IKI0- zWO`hX39gU6(Yy5#qC#*rQ1|hoJ|gev&3y!yUQ-ocZ=79h@EX>c0IJeAdH|lIsU(%| zystrYNyAn=;z43o=~^rElC=`B)ly#(plDJ_zSyg{h-5uY2aY`G!Hthu6OS5N_4Jy! zUm?>m7q>R7GCbE{UhGf2KJrVD?AnrVI*+2(FNCel<)K!D+^p^c!#@7T#-GQa;TafF z|Cyhk9C=#ZzlI}|vR>67;zmd z{u9so>lz`KHgqs?H^%{|`pObOn(j>*irMKWU@KLr;{fCTlO{h+X)2!tM=eexr+-zT zF-4(vCFp_13bfNpRh5<8p4HA$n2cWEpxv4NS?WcUHR{3rkDDWl7LuRye@)JPiB)dz zb~s0H9scBlbl)ux|KW;As_F}*%Sm>qmiS;Waw1srL+2?4=Mwazm`f_K6Q#=_Y*g{G zr#txIcAZ_@*s)k>#bNct_ z1SJrbQ9-L56}n*0&dcFnhz!yz6>bBWu*wH&I*YA(v+Y!iDYJ%54iHkL ztH()%qY6mG*wLm)SR&Xn!B0AFq})roz!k`+_?wtC*123A1fojHsV#CM&<|Bu>j;a9 zW`~EmkULK-Dv%|VSR&RA0K6;s?(r_h?B{nAK~w;8ydFx;UI=#aSVIqAE|ZAZp7wOZ zJ#k<013F7_h1j#IRQJxS$OJ_7KjfU*ZBLz*#n{0k#EDQ5p~qWV%RrmU2zT?jWm`<5 zyUJDTCpAIOQ;jAT^5mo{tVl#l=JDNz1NC74DM5;S(uL&M@cB!r@>4sIrG`e=BqDCR z*XXur)a8v*SP!lmZF9Kccr!4ymxXsO_7PIvSTLg%VQ8%ELnEz~IVv#k&tl!w1(9MpBt7AVtWZiT59WbuB)s$vv6|h(}RCNFK*|BjmLyl>)92?t$!P? zOAgIn4ve##4HdE;+;3!OM*vp zGPeOj8}=ruk#HVWxoFho%^@V0O0{(Tb^S0GkM-^eVl(!slrdzHkYRewhPXm)`^FO~ z(@csCPWF^gky)O6;W`}c;=8R4Ck*tf-;_d^^_veL`XF%7y4#|`!w>Y@8lvYm0*|KU z-C|%lHELk>Q!>1l*;-xwWkVixYmcgYEsc&o4(YhJFf}TzJ+n!J>MIbqpGd~3p8+Jrn-8Xu{G-& zq~c<&%?XpIQ=lA)gZo#jea%UAjO5F`2!LqHR(uNW1{d3 zlgZQV^=?gKlVhu5PA!%8^*2Jl=35(pk&R5PdJi_^?-$X*qEnfQ2{hBL9Gz^|TPoTM zQK@YQsl&1uyXhgh;9RxVw>}w!*)i#a?3V@0Bzd!UEP65>^HuO(^1`Q#!25e|%B8(v z4Q8@u3SX^6dez%%Xm(eUY#qBtasnh5jR=$s>(XUeoQDiGp6JuJm8B<#a9kVX&0jMp z{?tAlt6|*oDBo%kytDGoy|8bqQ*$YBqD8}CNzcom*kQ9?`78uSEzoWIEYrk`u4244u7vs3jEox#x zX_Bz|?W^xRl%g(%seg95MwaBb&d-`RZnUVNN|`x zUmL9V35M%Ulqu?2Zmza1OYdNwBMmjkx=spHJzo>HVCL^9FjNj612Vxs|hz@b8IL-yn-qzqEkWCu)ncD<~i>6 zbyu@cC&frcaPY|0x4V$$`Po@@H(Req!4_K>ey=rF)3nC-p$YsaAJx6VG}jD@>;1kO zR(LJVl%MN`x@z{)z?^Q{f&z?aC8@jcG4~5P%pSU}_dkp*ESxsW-(v0K=>*@4U)ZY& zzGAk{p;kRdrOLYI&jyfN^>ibo8sIlMY;W(Tbj)Z2yiiec(98WdXWDg1(6K{f=?&ICMB`~?nW$VcdN^f)gmdG_tlV{I429%Uf4!~ zvnSiqbWuKV4jd=mU%RW@m5!>)^t7fBSX*N&i^}W!OC8ho^YcO*f-^y)zU1!LyKk;; zM%={^S{y4qaTJ^TBQsc$Ni2VfEP3jPGyMLUO?WSqTp=p&ZEaZ_!q{0_pFf_n&#b7b z4zhY!bS=ZE3q;&5b1$vo?+gb!8m6j6q9LgJGP|6*!6jRnoYo}2D9i3{aDzvDd-M$d zEGfT3d>b^FYbuUe*Tcs3%pBCrF8dVmd`1Pn1hXPI7CxV=qLLJhp;(GjTeHd!++;`P zrv6qmIA+;n0ogUKVry^GnA5+Zp~WzqF}hVE&tj0iqTon!`+h;W*zG_X?dx0izP z)k3hJrg|>oaXaZtyST{Q3c2PcG zMIc(seDzVxjQVXjx=Ckt6Ja-qS@VhUK+oN)B2@+zU~5_vi>DrS<~pdwqy^vEUAWf| zzCajak_vmj%zL?5i>ahaI%r_NAVTv)OHST&v!t|g914hnw7tarzOvw)w=m<(4!Id$ z#G*P{e}Avt97tXh$(#1SqBzxaHheJzzWCF<6nd-8Bi~7_b>3qiYHy{Zw734STi-sV zPGXTD6kPFZ&2-pmzH&+#u=|GAzAukiSX5#?s$#$$^SAq)cb0LcPqRekUgqdWKnkIb zQyLK>vMnH~(2tfqkL+HpRs3vdHHeva{TLHdF6rjwf+YFWd$Gf(nBL_~q1KS_j1Ma1 z*0k)F;Q+3^-9s~#oIH2KVwj}cQy1=Xug(TPjAUQs)JQeTt28)hEr>E)DJ@S4xw{2@ z4BELI5PiKc==n@QbEKN-(nTw3aQ@YxJybpN2UjGUK-?&|OMPOU4ZrU>JlYs3407gn za{9e`nS14;Q>fCS%_rLjKYIr3XyHm`ue8f2kSg$hmK$l|-|$b!9$Vp$_pk2d8hpI* zb?=_=*N5nKGYfx&=>NsYzg_+Re=&^z-Jk#ervFEl_s;*9fc!7}0sVK0`Csw_`gZ{N jhyA#`m!W=vGSz$=ILShkMeg^ZfDj$46fa diff --git a/src/user-guide/running-workflows/dynamic-behaviour.rst b/src/user-guide/running-workflows/dynamic-behaviour.rst index 23e4e7e48b..e2997b41e9 100644 --- a/src/user-guide/running-workflows/dynamic-behaviour.rst +++ b/src/user-guide/running-workflows/dynamic-behaviour.rst @@ -3,19 +3,18 @@ Cylc Broadcast -------------- -The ``cylc broadcast`` command overrides :cylc:conf:`[runtime]` -settings in a running workflow. - -Broadcast can be used to communicate information to downstream tasks by -setting environment variables. Broadcasts can target all tasks, groups of -tasks, or specific tasks, at all cycle points or at specific cycle points. -See broadcast command help for details. - -Broadcast settings targeted at a specific task ID or cycle point expire and -are forgotten as the workflow moves on. Un-targeted variables and those -targeted at a task name persist throughout the workflow run, even across -restarts, unless manually cleared using the broadcast command - and so -should be used sparingly. +The ``cylc broadcast`` command overrides task :cylc:conf:`[runtime]` +settings in a running scheduler. You can think of it as communicating +changed settings (including information via environment variables) to selected +upcoming tasks via the scheduler. + +Broadcasts can target all tasks, groups of tasks, or specific tasks, at all +cycle points or at specific cycle points. See ``cylc broadcast --help`` for +detailed information. + +Broadcast settings targeting a specific task ID or cycle point expire as the +workflow moves on. Non-specific broadcasts persist throughout the run and +across restarts unless manually cleared using the broadcast command. .. _Sub-Workflows: diff --git a/src/user-guide/running-workflows/external-command-execution.rst b/src/user-guide/running-workflows/external-command-execution.rst index fde22d6634..28b2003982 100644 --- a/src/user-guide/running-workflows/external-command-execution.rst +++ b/src/user-guide/running-workflows/external-command-execution.rst @@ -1,19 +1,19 @@ .. _Managing External Command Execution: -External Command Execution --------------------------- +Scheduler Subprocesses +---------------------- Job submission commands, event handlers, and job poll and kill commands, are -executed by the :term:`scheduler` in a "pool" of asynchronous -subprocesses, in order to avoid blocking the workflow process. The process pool -is actively managed to limit it to a configurable size, using +executed by the :term:`scheduler` in a pool of asynchronous subprocesses. + +The process pool is size is configurable via :cylc:conf:`global.cylc[scheduler]process pool size`. -Custom event handlers should be lightweight and quick-running because they -will tie up a process pool member until they complete, and the workflow will -appear to stall if the pool is saturated with long-running processes. -However, to guard against rogue commands that hang indefinitely, processes -are killed after a configurable timeout -(:cylc:conf:`global.cylc[scheduler]process pool timeout`). -All process kills are -logged by the :term:`scheduler`. For killed job submissions, the associated -tasks also go to the *submit-failed* state. + +Event handlers should be lightweight and quick-running because they tie up +a process pool member until they complete, and the workflow will appear to +stall if the pool is saturated with long-running processes. + +To protect the scheduler, processes are killed after a configurable timeout +(:cylc:conf:`global.cylc[scheduler]process pool timeout`). This will be +logged by the :term:`scheduler`. For killed job submissions, associated tasks +go to the *submit-failed* state. diff --git a/src/user-guide/running-workflows/index.rst b/src/user-guide/running-workflows/index.rst index 0f688c03f1..ec68aa391a 100644 --- a/src/user-guide/running-workflows/index.rst +++ b/src/user-guide/running-workflows/index.rst @@ -9,15 +9,15 @@ Running Workflows scheduler-start-up task-job-states tasks-jobs-ui - reflow retrying-tasks - workflow-completion tracking-task-state - authentication-files + workflow-completion + dynamic-behaviour.rst + reflow simulation-modes scheduler-log-files - external-command-execution - handling-job-preemption - dynamic-behaviour.rst + authentication-files workflow-databases.rst workflow-migration.rst + handling-job-preemption + external-command-execution diff --git a/src/user-guide/running-workflows/reflow.rst b/src/user-guide/running-workflows/reflow.rst index a87458c8c8..ca522e820d 100644 --- a/src/user-guide/running-workflows/reflow.rst +++ b/src/user-guide/running-workflows/reflow.rst @@ -1,25 +1,28 @@ .. _user-guide-reflow: -Multiple Flows -============== +Concurrent Flows +================ .. versionadded:: 8.0.0 In Cylc, a *flow* is a self-propagating run through the workflow :term:`graph` -from some initial task or tasks. +from some initial task(s). -Often there is only one flow: the original one started automatically at the -beginning of the graph. But Cylc can manage multiple flows at once. +At start-up the :term:`scheduler` automatically launches a flow from the start +of the graph. But you can use ``cylc trigger --flow=new ID`` to start additional +flows anywhere in the graph, while the original (and any other) flow still +runs. New flows continue on from triggered tasks as dictated by the graph. When a flow advances to a new task in the :term:`graph`, the task will only run if it did not already run in the same flow. -See below for suggested :ref:`use cases` and an -:ref:`example`. +See below for suggested :ref:`use cases` of this +multi-flow capability, and a concrete :ref:`example`. .. note:: - Flows are not entirely independent: they :ref:`merge` - where (and if) tasks collide in the ``n=0`` :term:`active window`. + Flows :ref:`merge` where (and if) tasks collide in the ``n=0`` + :term:`active window`. Downstream of a merge, tasks are considered to belong + to all constituent flows. Flow Numbers @@ -77,8 +80,8 @@ Triggering in Specific Flows This triggers the task with flow numbers ``1`` and ``2``. The result is like the default above, except that tasks in the new front - belong only to the specified flow(s), regardless of which flows are active - at triggering time. + belong only to the specified flow(s), regardless of which flows are + :term:`active` at triggering time. Triggering a New Flow ``cylc trigger --flow=new ID`` @@ -100,41 +103,42 @@ Triggering a Flow-Independent Single Task .. image:: ../../img/no-flow-n.png Special Case: Triggering ``n=0`` Tasks - Tasks in the ``n=0`` window are active, active-waiting, or incomplete. Their - flow membership is already determined - that of the parents that spawned them. + Tasks in the ``n=0`` window are :term:`active`, :term:`active-waiting`, or + :term:`incomplete`. Their flow membership is already determined - that of + the parent tasks that spawned them. - - Triggering an active task has no effect (it is already triggered). - - Triggering an active-waiting task runs it immediately in the same flow. - - Triggering an incomplete task re-runs it immediately in the same flow. + - Triggering an :term:`active task` has no effect (it is already triggered). + - Triggering an :term:`active-waiting task` runs it immediately in the same flow. + - Triggering an :term:`incomplete task` re-runs it immediately in the same flow. .. _flow-merging: -Flow Merging In ``n=0`` +Flow Merging in ``n=0`` ----------------------- If a task spawning into the ``n=0`` :term:`window` finds another instance -of itself (same name and cycle point, different flow) already there, a single -instance of it will carry both (sets of) flow numbers forward from that point. -Downstream tasks belong to both flows. +of itself already there (i.e., same name and cycle point, different flow +number) a single instance will carry both (sets of) flow numbers forward from +that point. Downstream tasks belong to both flows. -Flow merging in ``n=0`` means flows are not entirely independent. One flow -might not be able to overtake another because one or more of its tasks might -merge in ``n=0``. Merging is necessary while task IDs - and associated log -directory paths etc. - do not incorporate flow numbers, because task IDs must -be unique in the active task pool. +Flow merging in ``n=0`` means flows are not completely independent. One flow +might not be able to entirely overtake another because one or more of its tasks +might merge in ``n=0``. Merging is necessary while task IDs - and associated +log directory paths etc. - do not incorporate flow numbers, because task IDs +must be unique in the :term:`active task pool`. -Incomplete tasks -^^^^^^^^^^^^^^^^ +Merging with Incomplete tasks +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Incomplete tasks are retained in the active window in expectation of -retriggering once fixed, to complete expected outputs and continue the flow. +:term:`Incomplete` tasks are retained in the active window in +expectation of retriggering to complete :term:`expected outputs` and continue their flow. -If another flow encounters an incomplete task, one task will carry both flow -numbers forward on successfully completing its expected outputs. - -.. TODO whether or not it automatically reruns in the later flow is still an - open question: https://github.com/cylc/cylc-flow/pull/4737 +If another flow encounters an incomplete task (i.e. if another instance of the +same task collides with it in the ``n=0`` :term:`active window`) the task will +run again and carry both flow numbers forward if it successfully completes its +expected outputs. Stopping Flows @@ -143,22 +147,23 @@ Stopping Flows By default, ``cylc stop`` halts the workflow and shuts the scheduler down. It can also stop specific flows: ``cylc stop --flow=N`` removes the flow number -``N`` from tasks in the active pool. Tasks that have no flow numbers left as a -result do not spawn children at all. If there are no active flows left, the -scheduler shuts down. +``N`` from tasks in the :term:`active task pool`. Tasks that have no flow +numbers left as a result do not spawn children at all. If there are no active +flows left, the scheduler shuts down. .. TODO update this section post https://github.com/cylc/cylc-flow/issues/4741 .. _flow-trigger-use-cases: -Use Cases ---------- +Some Use Cases +-------------- Running Tasks Ahead of Time - To run a task even though its prerequisites are not satisfied, just trigger - it. Use ``--wait`` if you don't want the new front to continue immediately. - Triggered task(s) will not re-run when the main flow front catches up. + To run a task within the existing flow(s) even though its prerequisites are + not satisfied yet, just trigger it. Use ``--wait`` if you don't want the new + flow front to continue immediately. Triggered task(s) will not re-run when + the main front catches up. Regenerating Products Behind a Flow To re-run a sub-graph (e.g. because the original run was affected by a @@ -178,20 +183,21 @@ Rewinding a Workflow Test-running Tasks in a Live Workflow You can trigger individual tasks as many times as you like with - ``--flow=none``, without affecting the workflow. + ``--flow=none``, without affecting the workflow. The task :term:`submit + number` will increment each time. -Processing Flow-specific Data - Flow numbers are passed to task environments, so it is possible to have +Processing Flow-Specific Data + Flow numbers are passed to job environments, so it is possible to have different flows process different datasets through the same graph. However - we do not recommend doing this. Generally, that's what cycling is for; and - besides, each task would have to be capable of processing multiple datasets - at once in case of :ref:`flow-merging`. + **we do not recommend this**. That's what cycling is for; and besides, every + task would have to be capable of processing multiple datasets at once in + case of :ref:`flow-merging`. .. _new-flow-example: -Example: Rerun a Past Sub-graph -------------------------------- +Example: Rerun a Sub-Graph +--------------------------- The following :term:`cycling workflow` runs a :term:`task` called ``model`` in every cycle, followed by a postprocessing task, two product-generating tasks, diff --git a/src/user-guide/running-workflows/scheduler-log-files.rst b/src/user-guide/running-workflows/scheduler-log-files.rst index 11e6145250..3f45671056 100644 --- a/src/user-guide/running-workflows/scheduler-log-files.rst +++ b/src/user-guide/running-workflows/scheduler-log-files.rst @@ -1,7 +1,7 @@ -.. _Workflow Server Logs: +.. _Scheduler Logs: -Workflow Server Logs --------------------- +Scheduler Logs +-------------- Each workflow maintains its own log of time-stamped events in the :term:`workflow log directory` (``$HOME/cylc-run//log/workflow/``). diff --git a/src/user-guide/running-workflows/simulation-modes.rst b/src/user-guide/running-workflows/simulation-modes.rst index 793891f530..b9a2d940d2 100644 --- a/src/user-guide/running-workflows/simulation-modes.rst +++ b/src/user-guide/running-workflows/simulation-modes.rst @@ -1,21 +1,29 @@ .. _SimulationMode: -Simulating Workflow Behaviour ------------------------------ +Simulation Modes +---------------- -Cylc can simulate scheduling without running real task jobs (which may -be long-running and resource-hungry). +Cylc can run a workflow without running the real jobs, which may be +long-running and resource-hungry. **Dummy mode** replaces real task jobs with background ``sleep`` jobs on the scheduler host. This avoids :term:`job runner` directives that request compute resources for real workflow tasks, and it allows any workflow configuration to -run locally in dummy mode. +be run locally in dummy mode. .. code-block:: console $ cylc play --mode=dummy # real dummy jobs -**Simulation mode** does not run real jobs at all. +.. note:: + + All configured task ``script`` items including ``init-script`` are ignored + in dummy mode. If your ``init-script`` is required to run even local dummy + jobs, the relevant environment setup should be done elsewhere. + + +**Simulation mode** does not run real jobs at all, and does not generate job +log files. .. code-block:: console @@ -33,8 +41,8 @@ If :cylc:conf:`[runtime][]execution time limit` and run length is computed by dividing the time limit by the speedup factor. -Limitations of Workflow Simulation -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Limitations +^^^^^^^^^^^ Dummy tasks run locally, so dummy mode does not test communication with remote job platforms. However, it is easy to write a live-mode test workflow with @@ -43,20 +51,11 @@ simple ``sleep 10`` tasks that submit to a remote platform. Alternate path branching is difficult to simulate effectively. You can configure certain tasks to fail via :cylc:conf:`[runtime][][simulation]`, but all branches based -on mutually exclusive custom outputs will run because custom outputs get -artificially completed in dummy and simulation mode. +on mutually exclusive custom outputs will run because all custom outputs get +artificially completed in dummy mode and in simulation mode. .. note:: - All configured task ``script`` items including ``init-script`` are ignored - in dummy mode. If your ``init-script`` is required to run even local dummy - jobs, the relevant environment setup should be done elsewhere. - - -Restarting Workflows With A Different Run Mode? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Run mode is recorded in the workflow run database. Cylc will not let you -*restart* a dummy mode workflow in live mode, or vice versa. To do that, -install a new instance of the workflow and run it from scratch in the new mode. - + Run mode is recorded in the workflow run database. Cylc will not let you + *restart* a dummy mode workflow in live mode, or vice versa. Instead, + install a new instance of the workflow and run it from scratch in the new mode. diff --git a/src/user-guide/running-workflows/tasks-jobs-ui.rst b/src/user-guide/running-workflows/tasks-jobs-ui.rst index 754eb61a52..591dec067e 100644 --- a/src/user-guide/running-workflows/tasks-jobs-ui.rst +++ b/src/user-guide/running-workflows/tasks-jobs-ui.rst @@ -15,7 +15,7 @@ The ``n = 0`` or *active task* window includes: - ``preparing`` tasks - ``submitted`` and ``running`` tasks - i.e. those with active jobs -- ``waiting`` tasks, that are :term:`actively waiting ` on: +- ``waiting`` tasks, that are :term:`actively waiting ` on: - :ref:`clock triggers ` - :ref:`external triggers

` diff --git a/src/user-guide/running-workflows/workflow-completion.rst b/src/user-guide/running-workflows/workflow-completion.rst index e6685f034f..dc7fa230e4 100644 --- a/src/user-guide/running-workflows/workflow-completion.rst +++ b/src/user-guide/running-workflows/workflow-completion.rst @@ -7,8 +7,11 @@ If there is nothing more to run (according to the graph) and there are no :term:`incomplete tasks ` present, the scheduler will report workflow completion and shut down when current active tasks finish. -Workflow Stall -============== + +.. _scheduler stall: + +Scheduler Stall +=============== If there is nothing more to run (according to the graph) but there are :term:`incomplete tasks ` present, the scheduler will diff --git a/src/user-guide/writing-workflows/suicide-triggers.rst b/src/user-guide/writing-workflows/suicide-triggers.rst index 4bd6dec92b..3f2fbc9d38 100644 --- a/src/user-guide/writing-workflows/suicide-triggers.rst +++ b/src/user-guide/writing-workflows/suicide-triggers.rst @@ -56,7 +56,7 @@ To remove a task after any one of several events, use an OR operator: * In Cylc 8 all waiting tasks in front of the active window are virtual and don't need to be "removed" * The only non-virtual waiting tasks in Cylc 8 are those :term:`actively - waiting ` on an external trigger; these might need to + waiting ` on an external trigger; these might need to be removed if they will never run @@ -65,7 +65,7 @@ To remove a task after any one of several events, use an OR operator: Remaining Use Case ------------------ -Suicide triggers may be needed to remove an :term:`active waiting task` when it +Suicide triggers may be needed to remove an :term:`active-waiting task` when it can be inferred from the status of another task that the external trigger will never be satisfied.