PowerShell Client Module

[galvesribeiro]

This initial implementation is part of a series of improvements on tooling.

Many people were asking on Gitter for a way to interact with grains from a console.

This initial push enables user to start and stop GrainClient and call any grain from Powershell.

Currently GrainObservers, Streams, interceptors are not implemented. This first step allow only direct calls to grain methods as in any other client that uses GrainClient does.

A page will be added to docs site on a separated PR explaining how to use it and more work will come to support other client scenarios.

[jthelin]

Nice!

[jdom]

In master we transitioned to using project.json for the nuget packages

[galvesribeiro]

I remember you said that the nuget consumers like samples for example would still be on the packages.config. Ok will change that.

[jdom]

but this is not a nuget consumer, or did I get it wrong? I thought you were adding this to the main Orleans.sln

[galvesribeiro]

Yeah you are right. Will change that.

[jdom]

This looks good to me.
@sergeybykov this will mean we'll probably have to start publishing this module in the Powershell gallery. Do you see any issues with merging this? An alternative is to use OrleansContrib (at least temporarily) although in the long term we probably want to have strong built-in PowerShell support (for cluster management mainly).

[shayhatsor]

LGTM. @galvesribeiro please squash before we can merge. @jdom, I don't see any issues with merging this, only benefits.

[galvesribeiro]

@shayhatsor done. Ah! Remember that while merging PRs GH can squash for you :)

[jdom]

Since this one doesn't require anything from Tester, should we start creating a new test project for the Powershell DLL? I think that's going to be how we do things in the future, and it would help us in the short term since this is not portable to .NET Core.

[galvesribeiro]

It requires a cluster to run. Should we duplicated that test cluster stuff in another project? If yes I can move it right away and update the PR

[jdom]

Either we can move those utilities to another project, or if you don't want to take this work in this PR, you can do the same thing as TesterInternal, and have this new test project reference Tester in order to use BaseTestClusterFixture.
We can refactor BaseTestClusterFixture and some other extra stuff into their own project later. I believe that would be more appropriate instead of conflating that move in this PR.

[jdom]

maybe the change was lost when rebasing? I thought you were creating a new test project

[galvesribeiro]

Not for now. Will keep it there. We need revisit all the tests and make a test kit otherwise we will still keeping the tests duplicated but this time, spread across multiple projects.

I think we should design a new testkit I'm just not sure this is the right time since we have tons of other things to do for the .net core move...

[jdom]

I think this does not suffer from the same as the other provider tests and does not require any test kit. It would be trivial to just move to a new test project in the way I suggested.

[jdom]

We'll then have to define the steps required to publish to the gallery. I assume these can be automated?
Not necessarily need to be addressed in this PR (in case it needs a script), but we'll need that automation in the future. if it's extremely straightforward, we'll just run that command or script when we release new versions of Orleans

[jdom]

you probably don't want to inherit from DefaultTestCluster fixture, and instead use it's base class and start the Silo cluster but not necessarily the grain client (I believe there's an overload to TestCluster's constructor where you just use the ClusterConfiguration but just pass a null ClientConfiguration). You can take the ClusterConfiguration object from the TestClusterOptions.ClusterConfiguration property.

[jdom]

was this a regression? I thought you had an iteration where you were no longer inheriting from Default

[galvesribeiro]

This is outdated @jdom. The current code in the PR already has the change you suggested and inherit from BaseTestClusterFixture

[galvesribeiro]

Check here 06fc7f1#diff-67c19d50200ddd6504cda9466e833332R18

[galvesribeiro]

Yes, the process is the same as publish a nuget. You just call a command line that pass the build output path and the publish key just as nuget. Lets talk offline and I can make another PR for it once this is merged.

[benjaminpetit]

GetGrainTest [](start = 20, length = 12)

It could be nice to test all possible values for key type

[galvesribeiro]

@benjaminpetit you mean test each type of key supported?

[benjaminpetit]

Yes, you could try to get a IGuidGrain, IStringGrain, ect. Just to cover the switch statement in GetGrain.ProcessRecord

[galvesribeiro]

@jdom and @benjaminpetit added the suggested changes.

[jdom]

is there a way to have this populated by the version of the OrleansPSUtils (effectively the same as the rest of Orleans)?

[benjaminpetit]

Sorry I missed that earlier.

It could be nice to have this value synced with what is inside Build\GlobalAssemblyInfo.cs. I believe it could be done by writing a custom target in the csproj.

Same for the version in ReleaseNotes

[benjaminpetit]

At least update to 1.3.0, I think it will be enough for now

[galvesribeiro]

I tried to do this:

  <UsingTask TaskName="UpdateVersion" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
    <ParameterGroup>
      <InputFilename ParameterType="System.String" Required="true" />
      <OutputFilename ParameterType="System.String" Required="true" />
    </ParameterGroup>
    <Task>
      <Reference Include="System.Core" />
      <Using Namespace="System" />
      <Using Namespace="System.IO" />
      <Using Namespace="System.Text.RegularExpressions" />
      <Code Type="Fragment" Language="cs"><![CDATA[
                var version = File.ReadAllText(InputFilename);
                var inputText = File.ReadAllTest(OutputFilename);
                File.WriteAllText(
                    OutputFilename,
                    inputText.Replace("$version$", version)
                    );

              ]]></Code>
    </Task>
  </UsingTask>

and in the end of the .csproj this:

<Target Name="AfterBuild">
    <ReplaceFileText InputFilename="$(SolutionDir)Build\Version.txt" OutputFilename="$(OutputDir)Orleans.psd1" />
  </Target>

I'm not good at MSBuild so if you can put something together for it I can quickly add it here.

Thanks

[benjaminpetit]

This works:

<UsingTask TaskName="UpdateVersion" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
    <ParameterGroup>
      <InputFilename ParameterType="System.String" Required="true" />
      <OutputFilename ParameterType="System.String" Required="true" />
    </ParameterGroup>
    <Task>
      <Reference Include="System.Core" />
      <Using Namespace="System" />
      <Using Namespace="System.IO" />
      <Code Type="Fragment" Language="cs">
                  <![CDATA[
                var version = File.ReadAllText(InputFilename)                                    
                                    .Replace("\r\n", "")
                                    .Replace("\n", "")
                                    .Replace("\r", "");
                var inputText = File.ReadAllText(OutputFilename);
                File.WriteAllText(
                    OutputFilename,
                    inputText.Replace("$version$", version)
                    );
              ]]></Code>
    </Task>
  </UsingTask>

and

  <Target Name="AfterBuild">
        <UpdateVersion InputFilename="$(SolutionDir)Build\Version.txt" OutputFilename="$(OutputPath)Orleans.psd1" />
  </Target>

[galvesribeiro]

Thanks! My mistake was the AfterBuild that was in the wrong place and ofc the endline replaces you add. Now it worked like a charm :)

[galvesribeiro]

• Added the fix for the version update by using the MSBuild code.
• Rebased on latest master.

[galvesribeiro]

Build failure looks like the MSBuild task is breaking it on the CI mahine... @benjaminpetit / @jdom any idea?

[benjaminpetit]

It seems that when you build with MSBuild the OutputPath variable is ignored, and you have to use OutDir instead. But of course, if you use OutDir in your csproj, it will not build in VS anymore. It should be an easy fix, I will post one.

But we have to find a way to copy all output from this project in a dedicated subfolder, right? Because right now with Build.cmd, everything goes to Orleans\Binaries\{Debug|Release}

[galvesribeiro]

Yes, We need have a folder that will end up on the package. So when we call the pack command, it will build the package with this folder so later on, while installing, it will be C:\local_ps_modules\Orleans for instance. That is why I made the Output Dir to be \{config}\Orleans\*

[benjaminpetit]

Adding this in the csproj:

  <PropertyGroup Condition="'$(OutDir)' != ''">
    <OutDir>$(OutDir)\Orleans</OutDir>
    <OutputPath>$(OutDir)</OutputPath>
  </PropertyGroup>

seems to do exactly what we want!

[galvesribeiro]

@benjaminpetit any particular position?

[benjaminpetit]

Just before the first ItemGroup node should be good

[galvesribeiro]

@benjaminpetit done! Thanks for the help!

[shayhatsor]

LGTM

[shayhatsor]

Thanks @galvesribeiro !!

[jdom]

Thanks @galvesribeiro, great contribution. LGTM, I'll merge right after you move this to it's own test project

[jdom]

Doh, didn't realized @shayhatsor had merged already. In the future, @shayhatsor, please give time for other reviewers to validate their comments after the submitter amends the PR. In this case there were still unaddressed comments. Nothing major in this case, but now it puts the pressure on us to fix it right away.

[galvesribeiro]

@jdom it was already merged. Also, related to the tests, until we have fully extracted the testkit, I see no point on have separated projects just to duplicate test code.

[jdom]

@galvesribeiro, in the future please when you address comments with new changes, please do not rebase every time. In some cases it caused regressions, but regardless of that, it's hard to know what are the new changes since the last time I reviewed it, and had to review all over again. We can use the squash and merge button anyway. It would only require rebasing if there's conflicts that do not allow the merge

[shayhatsor]

@jdom, I was under the impression that @galvesribeiro had already addressed all issues. I apologize for jumping the gun.

[jdom]

not sure I understand why you would need a test kit in this case. It's not a provider, you don't need to reuse any of the tests (which is what the test kit would address)

[jdom]

No problem, btw, I'm just mentioning this for future practice. I'll put out a PR on Monday to address the remaining comment, otherwise it provides more friction to the .net core migration.

[veikkoeeva]

@jdom Maybe put something to contributor guidelines? It might be useful to have something like:

1. While in your master git pull upstream master
2. git checkout -b some_branch
3. While working in the topic branch and new code gets pushed git pull --rebase upstream master
4. git push --force-with-lease origin +some_branch

Perhaps with even more detailed instructions (and having rerere on helps). It looks a common tripping point is bleeding commits between branches and/or updating the working branch (some_branch) gets foiled.

Then also how one should open PRs and how in general it is expected it to work. It might make it go smoother. There are probably hesistant people that try to search "how to do it correctly" and we could give instructions. But also tell that it's OK if things go a bit wrong in the process, everything can be fixed.

[jdom]

Hmm, unfortunately this merge broke most of our builds (functionals, load tests, reliability), since some of them still use VSO's way of building that puts everything in a single folder, and I guess the OutDir variable is not being set.

C:\Builds\5879\Orleans\CI-BVT\src\src\Orleans.sln - 1 error(s), 329 warning(s), View Log File
 C:\Builds\5879\Orleans\CI-BVT\src\src\OrleansPSUtils\OrleansPSUtils.csproj (110): The "UpdateVersion" task failed unexpectedly.
System.IO.FileNotFoundException: Could not find file 'C:\Builds\5879\Orleans\CI-BVT\src\src\OrleansPSUtils\$(OutDir)Orleans.psd1'.
File name: 'C:\Builds\5879\Orleans\CI-BVT\src\src\OrleansPSUtils\$(OutDir)Orleans.psd1'
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean checkHost)
   at System.IO.File.InternalReadAllText(String path, Encoding encoding, Boolean checkHost)
   at InlineCode.UpdateVersion.Execute() in c:\Users\xcgbuild\AppData\Local\Temp\tvxtw5sa.0.cs:line 63
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__26.MoveNext()

This is why we were going to run this PR by our build system before merging... Unfortunately it is very common that when we add build tasks, some issues arise. We are trying to get rid of this way of building in some of our tests, but we are not there yet.

Could someone please lend a hand to fix this? Otherwise we are blocked on running tests for other PRs and we'll either have to drop everything and do this emergency fix or revert this for now. I'd prefer not to revert if someone is willing to help asap.

[galvesribeiro]

Without understand and have access to VSO I can't say what variables VSO is using that is making that don't work.

BTW, lets just be clear. The merge doesn't break anything by itself. The common consensus in the past was that we had ALL the build processes working seamless between both Jenkins and VSO and we spent weeks on that migration. If there is something old there, we would never guess since we don't have access to it :(

[jdom]

Let me clarify @galvesribeiro: you did nothing wrong. Nothing at all.
It's just that we take our time to merge, especially when there are dependencies like this, for a reason.
Just as an FYI, what we spent a lot of time making sure it works is for running xUnit tests with both vstest.exe and xunit.console.exe, and the reason is that the build system is still different (we did not get rid of it last time, we made it work across both, remember?). We definitely need to fix this, and we already started porting our battery of tests over in December, and there is a PR coming very soon with some more (@benjaminpetit has been working on it last week).

[shayhatsor]

If this merge is causing any problem with internal builds, I see no harm in reverting for now.
@jdom, what should I watch out for? is it new projects ? changes in the build script(s) ?

[jdom]

I'm troubleshooting now to avoid the revert, I think I'm close, just running the tests.

Yeah, both new projects and new build dependencies (whether it's build scripts, msbuild tasks, test scripts). Also external environmental dependencies.

[jdom]

Nevermind, I wasn't close. @benjaminpetit is taking a look at it now

[galvesribeiro]

I can't actively help on that since I cant simulate the problem here... If there is a way for me to do that let me know

[jdom]

@galvesribeiro Do you know what's required to publish to the gallery? Signed build? We have one. Do we also need to sign any powershell script?

[galvesribeiro]

@jdom sorry for the delay.

Just create an account on powershellgallery.com which is pretty similar to the nuget.org one and grab an APIKey. Invoke the Publish-Module cmdlet and fill all the parameters and you should all be set.

Regarding the signature, I would suggest you to sign the assembly. This project output a .Net Assembly just like any other class library, so the sign process should be the same. There is no real .ps1 script.

Let me know if you have any problems.