add acceleration option to JointPrimaryMarginalizedModel likelihood #4688

WuShichao · 2024-04-07T22:00:02Z

@ahnitz This PR adds an acceleration option to JointPrimaryMarginalizedModel likelihood by assuming all extrinsic parameters can be fixed.

ahnitz · 2024-07-29T16:05:22Z

bin/inference/pycbc_inference

@@ -141,7 +141,12 @@ with ctx:
    if pool.is_main_process():
        for fn in [sampler.checkpoint_file, sampler.backup_file]:
            with loadfile(fn, 'a') as fp:
-                fp.write_config_file(cp)
+                # some models will interally modify original cp for sampling,


This shouldn't be an option as one should always save the original configuration. Why isn't your internal sampler modifying a copy? The version saved here then doesn't have to know (and really shouldn't) that you may have modified internally.

@ahnitz My understanding is cp is saved when it runs logging.info("Loading joint_primary_marginalized model") return super(HierarchicalModel, cls).from_config( cp, submodels=submodels, **kwargs) https://github.com/WuShichao/pycbc/blob/accelerate_multiband/pycbc/inference/models/hierarchical.py#L1002 the cp here is the modified config for sampling, can't be used as the initial config. So how to let PyCBC save the original one here?

@WuShichao I don't see anything related to saving the configfile where you've pointed. The place is in pycbc_inference where I'm comment on. Take a look at my first comment, I say how to do it. Don't modify the configparser you are passed in-place. Make a copy so you aren't editting the original. Then you don't need to save a separate copy and this particular line will just work to begin with.

ahnitz · 2024-07-29T16:06:37Z

bin/inference/pycbc_inference_model_stats

@@ -96,7 +96,10 @@ model.sampling_transforms = None
 def callmodel(arg):
    iteration, paramvals = arg
    # calculate the logposterior to get all stats populated
-    model.update(**{p: paramvals[p] for p in model.variable_params})
+    try:


Again, this is not correct. Your top level model should just have an update method. That method should do whatever is needed to prepare for the log* methods to actually work. You shouldn't be requiring anyone know about a new method 'update_all_models'. It's fine if you want to have an new method for internal use, but not for the top-level api.

So do I need to rename my update_all_models to be update (so overwrite the base one), or just use the original update in callmodel?

@WuShichao A sampler is just using 'update' so why are you doing so differently here? Think through how to do this best, but it should be clear that making this change in this program seems very inconsistent. If it is required, that indicates something is wrong with your model, and if so fix that. Otherwise maybe you made this change in error.

ahnitz · 2024-07-29T16:10:17Z

pycbc/inference/models/hierarchical.py

-        marginalized_params += list(primary_model.static_params.keys())
+        marginalized_params = list(marginalized_params.keys())
+        # add distance or phase if they are marginalized
+        if primary_model.distance_marginalization:


Why do you need to hardcode this? This will be brittle and break easily to changes to marginalization for example, which we don't want. If you need a list of marginalized parameters, why not add this to the class in inference/tools so that it is kept up to date?

OK, I will save it in the tools module and just use it here.

ahnitz · 2024-07-29T16:13:27Z

pycbc/inference/models/relbin.py

@@ -596,10 +596,11 @@ def _loglr(self):
            filt += filter_i
            norm += norm_i

+        loglr = self.marginalize_loglr(filt, norm)
        if self.return_sh_hh:


Why do we need this if statement? Shouldn't the existing flag already used for demarginalization take care of this? E.g. why not use the reconstruct_phase flag?

https://github.com/gwastro/pycbc/blob/master/pycbc/inference/models/tools.py#L241

@ahnitz This is used here https://github.com/WuShichao/pycbc/blob/accelerate_multiband/pycbc/inference/models/hierarchical.py#L752 We need to get LISA's sh and hh.

ahnitz · 2024-07-29T16:16:32Z

pycbc/inference/models/hierarchical.py

+                    if isinstance(value, numpy.ndarray):
+                        nums = len(value)
+                    else:
+                        nums = 1
        # add distance if it has been marginalized,


Again, avoid needed to know explicitly about distance here. Think instead about the format that you require. If the format differs how to generically convert.

ahnitz · 2024-07-29T16:22:31Z

pycbc/inference/models/hierarchical.py

+                    hh_others[i] += hh_other
+                    other_model.return_sh_hh = False
+                    # set logr, otherwise it will store (sh, hh)
+                    setattr(other_model._current_stats, 'loglr',


Do you need to store this? It's not necessarily a problem, but it will slow down the code slightly (maybe not important at the moment). Why not think about why it was being set to a vector (and from where), do you even want this stored in the case of a submodel? Maybe the solution was simply not to store this when it's not actually a marginalized loglr anyway, no?

My understanding is that when pycbc_inference_model_stats check for pi, p in enumerate(model.default_stats): it will try to access submodel's loglr, no? If so, I need to store it. When I not rest it, I found other_model._current_stats also contain (sh, hh) for each point.

@WuShichao OK, good. So now the question is what is the right thing to do in this case?

ahnitz · 2024-07-29T16:23:12Z

pycbc/inference/models/hierarchical.py

                other_model.return_sh_hh = False
+                # set logr, otherwise it will store (sh, hh)
+                setattr(other_model._current_stats, 'loglr',


Same as above. It's not necessarily a problem as it might be useful to have the separate loglrs, but it's not clear that it will always make sense.

ahnitz · 2024-07-30T00:22:48Z

pycbc/inference/models/hierarchical.py

+                    margin_params[p] = \
+                        self.primary_model.current_params[p][i_max_extrinsic]
+                    nums = len(self.primary_model.current_params[p])
+                else:


@WuShichao This logic should already take care of the distance case, except that later on you assume that if any parameter is a scalar they all are. That's the part you should stop assuming. Don't assume they are any particular mix of scalar or vector.

WuShichao · 2024-08-12T14:27:00Z

Replace deprecated ConfigParser readfp with read_file:
python/cpython#92503
https://issues.apache.org/jira/browse/SVN-4899

…wastro#4688) * Update hierarchical.py * Update hierarchical.py * Update hierarchical.py * Update hierarchical.py * Update hierarchical.py * fix cc issues * Update hierarchical.py * Update relbin.py * add complex phase correction for sh_others * Update hierarchical.py * Update relbin.py * fix cc issues * make code more general * update * fix * rename * update * WIP * fix a bug in frame transform * fix overwritten issues * update * update * fix reconstruct * make this PR general * update * update * fix cc issues * rename * rename * add multiband description * fix * add comments * fix hdf's config * fix * fix * fix * fix * remove print * update for Alex's comments * wip * update * fix * update * seems work * fix CC issue * fix * fix demargin

Update hierarchical.py

55e5541

WuShichao added inference LISA labels Apr 7, 2024

WuShichao requested a review from ahnitz April 7, 2024 22:00

WuShichao added the work in progress label Apr 7, 2024

WuShichao and others added 25 commits April 8, 2024 15:11

Update hierarchical.py

a6ac76d

Update hierarchical.py

f4ed98d

Update hierarchical.py

41224cc

Update hierarchical.py

7109db9

fix cc issues

0bbe7a4

Update hierarchical.py

6920a8c

Update relbin.py

55fba36

Merge branch 'gwastro:master' into accelerate_multiband

b993833

add complex phase correction for sh_others

7099515

Update hierarchical.py

94ba798

Update relbin.py

6afbc4e

fix cc issues

e20b6f5

Merge branch 'gwastro:master' into accelerate_multiband

25a4562

make code more general

8fb82a1

update

757a30b

fix

a2d64d1

rename

8a9287e

update

5423d8c

WIP

0b9d44e

fix a bug in frame transform

eeb8890

fix overwritten issues

50e3599

update

dba5292

update

537256e

fix reconstruct

0af3fed

make this PR general

0a09b12

remove print

a5b6d8c

ahnitz reviewed Jul 29, 2024

View reviewed changes

WuShichao added 2 commits July 29, 2024 23:10

update for Alex's comments

ca096ec

wip

e8825be

ahnitz reviewed Jul 30, 2024

View reviewed changes

WuShichao and others added 9 commits July 30, 2024 15:21

update

be2b066

Merge branch 'gwastro:master' into accelerate_multiband

c03652e

fix

02b6937

Merge branch 'gwastro:master' into accelerate_multiband

87f10cb

update

cbcd5a2

seems work

36af111

Merge branch 'gwastro:master' into accelerate_multiband

c084f4f

fix CC issue

709b524

fix

3a17bf5

WuShichao mentioned this pull request Aug 12, 2024

remove readfp (Python 3.12 removes it), use read_file instead #4852

Merged

1 task

WuShichao and others added 3 commits August 12, 2024 16:58

Merge branch 'gwastro:master' into accelerate_multiband

3865d5e

fix demargin

0df23f2

Merge branch 'gwastro:master' into accelerate_multiband

da34461

WuShichao requested a review from ahnitz August 27, 2024 19:26

Merge branch 'gwastro:master' into accelerate_multiband

f2b0798

ahnitz approved these changes Sep 9, 2024

View reviewed changes

ahnitz merged commit 373271e into gwastro:master Sep 9, 2024
29 of 30 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add acceleration option to JointPrimaryMarginalizedModel likelihood #4688

add acceleration option to JointPrimaryMarginalizedModel likelihood #4688

WuShichao commented Apr 7, 2024 •

edited

Loading

ahnitz Jul 29, 2024

WuShichao Jul 29, 2024

ahnitz Jul 29, 2024

ahnitz Jul 29, 2024

WuShichao Jul 29, 2024

ahnitz Jul 29, 2024

ahnitz Jul 29, 2024

WuShichao Jul 29, 2024

ahnitz Jul 29, 2024

WuShichao Jul 29, 2024

ahnitz Jul 29, 2024

ahnitz Jul 29, 2024

WuShichao Jul 29, 2024

ahnitz Jul 29, 2024

ahnitz Jul 29, 2024

ahnitz Jul 30, 2024

WuShichao commented Aug 12, 2024

add acceleration option to JointPrimaryMarginalizedModel likelihood #4688

add acceleration option to JointPrimaryMarginalizedModel likelihood #4688

Conversation

WuShichao commented Apr 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WuShichao commented Aug 12, 2024

WuShichao commented Apr 7, 2024 •

edited

Loading