Planning for expanded community stats system

[jywarren]

We are preparing this for a project proposal with Outreachy.org for this winter. It's a work in progress, but we're happy to answer questions below! 😄

This project builds on the existing community metrics system viewable at http://publiclab.org/stats, which provides statistics and insights into community trends and activity, similarly to GitHub Pulse (https://github.com/publiclab/plots2/pulse/). We’ve collected a preliminary planning issue to organize this here.

The project involves both back-end Ruby on Rails functions to efficiently collect aggregate statistical information about different types of activity from the database, to answer pragmatic questions like “how many people posted this month?” or “how many questions got answered in our Q&A system this week?”. The project also includes front-end development of views/templates in Ruby on Rails to display this information in a rich and useful way on the website, including in graphs.

Our statistics system is already quite powerful, and many of these tasks address consolidating different features which were added at different times over the years, resulting in a messy interface. Others address the concern of performance with basic caching. A new system we propose as part of this project will display the same kind of data that’s currently available, but filtered by topic, using our site’s tagging system.

Finally, this project will include better user interfaces for selecting specific date ranges, as well as better display of data in graphs, especially when viewing specific selected date ranges (currently this data is shown only numerically).

Many of these projects will be solvable in relatively small, stand-alone pieces, ideal for a weekly cycle of proposing new code changes, getting feedback, and publishing the results. It should be fun!

Back-end projects

• downloadable data from stats pages: Raw data from stats page #963 (break-me-up issue)
• statistics on questions answered versus asked (from https://publiclab.org/questions/) Statistics on questions asked vs questions answered #4137
• caching of data weekly (noting this in interface so people know) to reduce load Caching of data #4138
• consolidation/standardization of "range pages" like https://publiclab.org/stats/range/10-05-2015/10-05-2016/ with main stats page at https://publiclab.org/stats?time=September%2026,%202017%2022:43 Consolidation existing "range pages" like with the main stats page #4577
• tags most used in this period (linking into per-tag stats, below) Tags most used in a period #4539
• Add display of Total Contributors "for all time" as default for main /stats - Add the display of Total Contributors "for all time" to the main stats page #3499

Stats per tag

• subscribers for each tag: https://publiclab.org/stats/subscriptions Subscribers for each tag #4541
• closer mirroring between stats pages and tag pages Closer mirroring between stats pages and tag pages #4540
• stats as above, but per-tag? Stats on a per-tag basis #4140
• script to go back through posts and tag first-timer posts with first-time-poster tag, retroactively so they show up at https://publiclab.org/tag/first-time-poster, to improve stats on first-time posting

Interface design projects

• small "bar graph" icons around site which lead to appropriate pages in the stats system (using https://fontawesome.com/v4.7.0/) - see Hide tag graph from tag pages like /tag/ui with a toggle button #4646
• date range selector for stats ranges (calendar or draggable date ranges like https://github.com/publiclab/plots2/graphs/contributors?from=2013-11-07&to=2016-11-18&type=c
• ability to click bar graphs to open that actual collection of items in a view such as https://publiclab.org/tag/mobius

... @ebarry?

Stretch goals

• make statistics system compatible with GrimoireLab using https://github.com/chaoss/grimoirelab-perceval (see below)
• retroactively apply the first-time-poster tag (https://publiclab.org/tag/first-time-poster) to previous posts so that we can retroactively graph newcomer posting rate over time
• Graphs to track subscription rates i.e. overall subscriptions count, and per topic, so we see how interest in a tag grows over time.

We would like to know if a “DevOps for open communities”-style platforms like https://github.com/chaoss/grimoirelab is appropriate for and could be used by a multi-channel community like ours. (via @ebarry )

[jywarren]

Yes, we can definitely be flexible, and this is more of a giant grab-bag list of ideas for what we'd like, that doesn't all have to fit in this winter's project. I think we can work together to organize it according to what makes sense to implement first, second, third, and then also think about how to break down the bigger tasks into smaller parts that'll make it easier to implement.

Finally, I think we can think through some of the ramifications of the changes listed and flesh out any parts we've overlooked or any prerequisites to doing some of the bigger parts. I hope this helps!

[jywarren]

Yes, we can definitely be flexible, and this is more of a giant grab-bag list of ideas for what we'd like, that doesn't all have to fit in this winter's project. I think we can work together to organize it according to what makes sense to implement first, second, third, and then also think about how to break down the bigger tasks into smaller parts that'll make it easier to implement. Finally, I think we can think through some of the ramifications of the changes listed and flesh out any parts we've overlooked or any prerequisites to doing some of the bigger parts. I hope this helps!

…

On Mon, Oct 22, 2018 at 1:58 AM Quinn ***@***.***> wrote: So is the goal to try and get all of this done in the 3 month internship and our proposal is a sort of workflow-order? I don't feel like I have the experience to know how I should expect to set my pacing in my proposal. Is it expected that we may have to adjust our timeline as we progress through the internship? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3498 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ9o3yYBORxt_nbZj79ILwgSra_9Aks5unV59gaJpZM4W7nq7> .

[quinn-codes-synthesis]

Woops, Github was being finicky yesterday and I ended up asking basically the same question here and in the other thread (it wasn't showing me that the post went through).

Thanks for the response! What is the best way to submit a proposal work-in-progress? Email to you and the other mentors, posting it here, something else? Through the Outreachy site directly?

[sagarpreet-chadha]

HI 😄 !
You can post your work-in-progress on publiclab.org !

[jywarren]

Hi, everyone! I'm sorry there was some confusion -- since, unlike in our other Summer of Code programs, we have a single project we've submitted, we weren't sure we needed a call for proposals page. But although we'd love to see group discussion and collaboration around refining the proposals, it does seem helpful to ask each applicant to post a draft proposal (in addition to the one you'll send to Outreachy.org).

So I've posted a call for proposals here, where I'd like to ask each of you to open a draft proposal and get some comments on it! Thank you!!

https://publiclab.org/notes/warren/10-23-2018/call-for-outreachy-2018-19-proposals

[jywarren]

Adding: script to go back through posts and tag first-timer posts with "first-time-poster" tag, retroactively so they show up at https://publiclab.org/tag/first-time-poster, to improve stats on first-time posting !!

[Radhikadua123]

@jywarren I wanted to confirm some things about GrimoireLab. Earlier I didn't research whole GrimoireLab repositories properly, I only checked grimoirelab-perceval. Basically, perceval allows fetching data from different websites in JSON format. I thought since I'll be converting data fetch step to JSON APIs, it would automatically be supported by perceval. But seems like GrimoireLab is much more than that. So before thinking about other ideas, I wanted to know what's the use case of adding support for GrimoireLab. @ebarry May be you can shed some light here ?

[jywarren]

Adding in some screenshots and stuff that could be used to break out the "calendar selector" task in the Interface section:

If you go to this page: https://publiclab.org/questions/new?legacy=true, you can see a calendar selector already in the codebase. There may be other ones that can do range selections too, generally using Bootstrap and jQuery:

[grvsachdeva]

Hi @cesswairimu sorry I was busy with the exams but now available for help. I saw that you created a milestone for your project, that's nice. Could you also keep updating the issue checklist with relevant issue number or PR no. so that we all would be in sync.

I would be available for review and discussion from Sunday, till then I have to go through all the work done so that I can help you better. Also, are you following any schedule like specific work hours for Outreachy project so that I can keep some time for your project accordingly on my schedule.

Thanks!

[grvsachdeva]

okay, in checklist issues are mentioned against some items, any particular order you are following while taking issues?

[cesswairimu]

A redesign for https://publiclab.org/stats. Criticism, inputs and ideas are welcome. Thanks 😃

[jywarren]

Hi @cesswairimu wow this is amazing! I really like it. I'd like to get input from @ebarry @steviepubliclab @publiclab/community-reps but just a note that @ebarry is off today and likely won't be able to respond until Monday.

For my part, a few small things - i know this is a mockup and just wanted to note that we should stick as much as we can to the Bootstrap UI library when it starts getting coded - i think i can see a number of Bootstrap-like elements already there so we should be good. The arrows, for example, could be a standard bootstrap button-bar with Fontawesome icons inside. Rad!

2nd, I wonder if line graphs might make it easier to track each type? with 3 colored lines?

3rd, i think it'll be important to be able to hover over data points to get specific values too. But this looks awesome!!

[ebarry]

Hi Cess! so exciting to see the advances on https://publiclab.org/stats/range!!!! It looks great!

I'll share how i experienced the information on publiclab.org/stats/range and look forward to your guidance :) So, when i opened it i saw:

• a large font displaying a 13 month period
• a drop down menu that says "view stats in the past", to which i'd assume "yes all stats are of the past," so I didn't realize i needed to select something. (Perhaps something like "Choose a date range to view statistics for" might be clearer?) The drop down does offer week | month | year as options, which is good.
• The "Start" and "End" text box range encompasses approximately one week, and then immediately below the stats results are shown, but i'm not sure what period the stats results for, because the page is also displaying the 12 month period at the top.

Looking forward to your input! Thank you so much!!!

[cesswairimu]

Thanks so much @ebarry for the feedback. I will change the text in the select section.
As for the start and end section, by default we have the start as 1 week ago, end as the date today. Maybe you have some suggestions of how that should be or maybe I should have them as blank?
The text as the top changes according to the range selected(If you select a range from the calendar you will see the dates you selected on the top i..basically it shows the start date and the end date you have selected and the stats at the bottom shows the stats between your dates selected. Hope this makes sense and please let me know of any other changes you would like to change so i can work on them as I change the select prompt text. Thanks again

[grvsachdeva]

Hi @cess, you have been doing awesome work till now. I read the theme for this week's blog post and it was really nice. @cess in all the projects whether it's a GSoC project or RGSoC, we try to break some part of our project for the newcomers or other contributors to solve to encourage collaboration among community and flow of new ideas.

The challenge is that you have to guide them to solve those issues to completion and as contributors can take time so it's advisable to choose those tasks where little delay can be compensated.

It's just an idea, it's totally upto you and your plan if it can have some minor changes. Thanks!

[cesswairimu]

@gauravano thanks so much, I love the idea I try and break down some issues and I guess other fixes like the one @ebarry was talking about above could make for good first-timer issues. Thanks again.

[jywarren]

Hi @cesswairimu !! I was thinking of a potential task we could add to the end of the list, if you have time at the end, or if people joining in later may want to carry your work forward;

We're starting to really accelerate getting people to subscribe to different tags. These are with tag_selection and are via the "Subscribe" links everywhere. What if we made a graph of the rate of subscribers 1. overall, i.e. overall subscriptions count, and 2. per topic, so we see how interest in a tag grows over time?

2. could become part of the per-tag stats views.

This need not add to the pile of tasks that we're aiming to complete but I wanted to see what you thought? @SidharthBansal and @ebarry may like this one!

[jywarren]

Hooray!!!

…

On Tue, Jan 15, 2019 at 12:13 PM Cess ***@***.***> wrote: @jywarren <https://github.com/jywarren> I love the idea and with the graphs am working with I already excited about this. I will add it in the stretch goals. Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3498 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ-uvR9M6GRoc_jfsr53SRJW9NoTTks5vDgwggaJpZM4W7nq7> .

[jywarren]

Noting that #4646 fulfills a checklist item above!

[jywarren]

Also noting that a lot of ideas from this year-old planning issue for graphs improvements may either be completed or could be broken up and added to many of the tasks in https://github.com/publiclab/plots2/milestone/20 (i've moved it over into that milestone)

#2068

[jywarren]

@cesswairimu i am also wondering if we could open an issue to display the # of subscribers over time on tag stats pages like this: https://publiclab.org/tag/ui/stats

And finally, add a title to the tag stats pages like "Statistics for the tag ______".

These don't have to be done by you @cesswairimu unless you want to! Just thinking next steps! 🎉

[jywarren]

And, i'll be publishing to production in a moment so you can see this all running 😄

[cesswairimu]

Yeah I would love to Opening an issue. Thanks

[jywarren]

Awesome.

…

On Wed, Mar 6, 2019 at 2:33 AM Cess ***@***.***> wrote: Yeah I would love to Opening an issue. Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3498 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ8e5dtshDhl6QNvBXDYia_nd4m8Dks5vT28WgaJpZM4W7nq7> .

[jywarren]

This is still not 100% online, my apologies, we ran into an issue with optimizing this query: #4924 but are trying to resolve it quickly!

[vaarigupta]

hello @jywarren I am an outreachy applicant and excited to work on this issue. How can I start contributing ?

[grvsachdeva]

@cesswairimu any ideas around-

via @ebarry at #308 (comment): I suppose there is a second part which is that a Stats function can show
how many people have made which types of contributions. It's a community
awareness / eval function, as well as perhaps a support for spam
moderation. Perhaps we could move over to a Stats issue to describe how
that could work.

[jywarren]

If @ebarry can expand on this we can brainstorm a bit - what types how many people made comments, questions, wiki edits? I believe we now have that for overall and per-topic stats, if that helps!

…

On Mon, Mar 25, 2019 at 2:42 PM Gaurav Sachdeva ***@***.***> wrote: @cesswairimu <https://github.com/cesswairimu> any ideas around- via @ebarry <https://github.com/ebarry> at #308 (comment) <#308 (comment)>: I suppose there is a second part which is that a Stats function can show how many people have made which types of contributions. It's a community awareness / eval function, as well as perhaps a support for spam moderation. Perhaps we could move over to a Stats issue to describe how that could work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3498 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJzXe_N92HVrhkCe5iFh0FaH8B1DIks5vaRhqgaJpZM4W7nq7> .

[ebarry]

Thanks! Yes as some may be aware, there are many more registered users on the publiclab.org website than there are active contributors. Defining who is "active" has sometimes led the community development team to identify as "active contributors" the set of user accounts that have taken multiple types of action on publiclab.org, such as posting a note AND editing a wiki AND commenting.

I would like us to brainstorm a little here on how to add up and display tallies of user accounts who have done certain actions in order to understand how many people are active on the website and in what gradations. How does this sound?

[cesswairimu]

Aha! I see @ebarry this would be good..also we should maybe define how an active contributor should be determined...what if someone joined 6 months ago and posted two notes but there is no activity from them since them...maybe we can do ranges and have "active contributors" within a range period be the people who posted something during that period...

[cesswairimu]

With ranges this would be our "active contributors"

Then maybe we can have this as a link to a list of the users

[skilfullycurled]

Hey everyone! First major thank you to everyone who has helped to build this. It's awesome. In particular, I am told that some extra gratitude should be directed to @cesswairimu.

Edit: Whoops, hit some button too early. I'll be right back with what I was going to add to the conversation.

[jywarren]

Indeed this has been largely @cesswairimu's fantastic work -- a big new system and so awesome!!!

…

On Tue, Mar 26, 2019 at 5:33 PM skilfullycurled ***@***.***> wrote: Hey everyone! First major thank you to everyone who has helped to build this. It's awesome. In particular, I am told that some extra gratitude should be directed to @cesswairimu <https://github.com/cesswairimu>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3498 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ8-atM0KQWeGAV_MOfleLoJYy91dks5vapICgaJpZM4W7nq7> .

[skilfullycurled]

I just filed an issue already referenced above regarding the downloading of csv's for tags, specifically community_tags and term_data.

I think it would be great to eventually add it as another option in the drop down. I'm able to merge the files myself but in a download functionality, it would make the data more accessible to a variety of skill levels if term_data and community_tags were merged on tid when it was delivered for download.

Again, thank you for all of this work. It's opening up an entirely new quantitative avenue which will dovetail nicely with the vast amount of qualitative data about our community already on the site. It's add a whole new dimension of research about our community that can be contributed.

[cesswairimu]

Thanks @skilfullycurled glad to hear the raw stats download function is helping in data analysis 😄. Will take a look at the issue and get started on it. Thanks

[skilfullycurled]

Hey all, so I implemented downloads for tags and the accompanying term data, second review in progress!

I'd really like to build on what I've learned, and I was hoping I could do so by implementing some other download options so that I can practice more complex versions of the process of required changes the FTO provided (routes, controllers, views) but this time do the programming.

After working with the data, the experience has given me a few ideas about other downloads we might want to have for people who do not have the background required (either in Excel or computationally) to manipulate the data to answer other questions. For example, it might be nice to download all notes with the "first time posting" (can't remember what it's called) tag. In the future I'll create a list of ideas and post them here.

In the meantime, would people mind I worked on some download by X category function? And if so, what is a good workflow for when you are working on a larger feature. Should I create a new issue to discuss it and add essential updates here, or do you like to keep everything in this thread?

[cesswairimu]

@skilfullycurled just saw your PR looking great 👍 . Yeah it would be good to open another issue for that discussion as we did for the other features and add the link at the top of this issue and some description as a checklist.

[skilfullycurled]

Thanks @cesswairimu! Okay, will do. As an aside, I also posted an issue for a problem I was having with the downloads, but forgot to mention the issue here (or reference it there).

[jywarren]

This is now complete except for the GrimoireLab item!

GrimoireLab can read RSS, so I wonder if this is as simple as fetching the RSS feeds for each tag/topic you're interested in:

https://github.com/chaoss/grimoirelab-perceval#rss

This wouldn't cover wikis or comments, however. But we could make those compatible by producing an RSS feed for each wiki pages comments.

https://publiclab.org/feed/tag/balloon-mapping.rss for example.

Another possibility is to use the PL API, which is JSON based, and to open an issue on GrimoireLab-perceval to create a plots2 backend. Backends don't look too complex; here's one for MediaWiki: https://github.com/chaoss/grimoirelab-perceval/blob/master/perceval/backends/core/mediawiki.py

[jywarren]

I'm going to close this now. But if anyone is interested in the GrimoireLab integration, please open a new issue and cite my last comment! Thank you!!!

[jywarren]

And congrats to @cesswairimu @skilfullycurled @ebarry on a great project!