Contribution system: History heatmap for user

[joubertredrat]

Hi guys,

I think that is a possible feature, a history heatmap similar to github or gitlab.
Actually exists a plugin called Calendar HeatMap. I used this on mine project to heat application log and worked fine here.
Then, is only a idea, what you think? :)

http://cal-heatmap.com/
https://github.com/wa0x6e/cal-heatmap

Reference: gogs/gogs#1640

[bkcsoft]

I would prefer a solution that is in the backend, unless it's required to have it update without reloading. Unfortunately I can't seem to find anything that does that 😒

Also this would require caching, since it will fetch huge amounts of data from disk...

[joubertredrat]

Yes, this plugin build on front-end, with backend I don't know too, but we can consider make component for this.

In my case I use ajax to get data, but build on frontend anyway

[xinity]

following @bkcsoft retention strategy in cache is a must if we don't want gitea to waste ressources.
something like in the latest 15days could be enough don't you think ?

[bkcsoft]

@xinity a simple LRU-cache should suffice in my opinion. Might be useful to be able to set the size of that cache though.
For my own projects I use hashicorp/golang-lru which works just fine and has a configurable size 🙂

[charlesreid1]

d3-calendar illustrates this really nicely and simply with javascript: https://bl.ocks.org/mbostock/4063318

git log is really flexible, so you can use it to output directly to JSON format, filter on users or dates, customize output format, etc (even has its own set of shortcodes). After an aggregation step, that data can be loaded into D3 calendar (basically just json objects with a key of the date and a value of the number of commits).

Here is a single-user activity graph that illustrates embedding the Javascript visualization in the home template file: git.charlesreid1.com and modified home template.

[lunny]

@charlesreid1 could you send a PR to gitea with your heatmap?

[charlesreid1]

I'm happy to put together a visualization in an PR, but a couple of things need to be worked out first:

1. Placement. My example heatmap is commit data scraped for a single user, displayed on the home page for the (single-user) gitea server - which doesn't make sense for multi-user gitea servers. If we wanted to have a per-user heatmap, it could go a couple of places:

• Behind the "Public Activity" tab on the user's profile (most appropriate place, but not very visible)
• Just on top of the three tabs "Repositories"/"Public Activity"/"Starred repositories" on the user's profile (most visible, on user profile so useful for others to see your activity graph)
• On the top of the user's dashboard (less useful since others cannot see your activity graph)

2. Data. The model I'm using for data would need to be changed a bit as well (my visualization points to a JSON file not dynamically generated by gitea). We could modify gitea to cache per-user commit data (updated with a post-commit hook or a daily maintenance task) and provide an endpoint route, where you provide it a username and it returns commit data in a JSON form convenient for visualization.

Thoughts? Potential roadblocks?

[lunny]

I would like put it under the Public Activity tab and top of all the activities on user's profile. Like the below example image:

[joubertredrat]

@lunny I liked this approach, if possible to change color to green can be best.... or not, github is green too.

[charlesreid1]

Thanks @lunny for the suggestion, looks like a great spot for it.

@joubertredrat I'll do something plain and let the hipsters decide what color scheme to use. 🤓

[lafriks]

Green would be more consistent with gitea theme

[charlesreid1]

I will use ColorBrewer's YlGn theme, which looks like the same one Github uses and is nicely consistent with Gitea green.

[charlesreid1]

It looks like gitea/models/graph.go may already implement a model for commit counts. Is that correct?

[charlesreid1]

Maybe not... But it looks like graph.go might be useful for implementing commit counts. There are a couple of files referring to a commit graph. Is this related to the heatmap/calendar idea, or is this referring to the timeline that shows up on a user's/organization's page?

[lunny]

I think we need a new table to store the statistics information.

type UserHeatMap struct {
      ID int64
      UserID int64 `xorm:"index"`
      Day string `xorm:"char(10)"` // 2018-02-26
      Count int
}

And all needed data could read from table action.

[thehowl]

Could we not use a SQL DATE instead of a char(10)?

[lafriks]

Also if this is just activity from action table than probably no new table is needed just select user_id, date, count(1) with group by user_id, date?

[lunny]

@lafriks that maybe
select date, count(1) from action where user_id = ? and date >= ? and date < ? order by date group by date

[lafriks]

@lunny yes that what I meant but I don't know if xorm supports such queries

[lunny]

type Stat struct {
      Date string
      Count int
}
var stats = make([]Stat,0, 366)
x.SQL(`select date, count(1) as count from action where user_id = ? and date >= ? and date < ? order by date group by date`, 
userID, startDate, endDate).Find(&stats)

[thehowl]

would that work in all dialects?

[lafriks]

If we have date column (do we?) than it should work

[lunny]

@lafriks we can use created and remove the time parts.

[lafriks]

@lunny don't think removing time part will be that easy so that it would perform good enaugh for such select

[charlesreid1]

This effort was over my head, as there many decisions where I didn't have a clear sense of how to proceed, so I didn't get very far.

[kolaente]

Any updates on this?

[lunny]

@kolaente no people are working on this.

[kolaente]

Someone created a nice article on how to implement this externally: https://medium.com/@markuman/gitea-contribution-heatmap-af4be1d7a0d5

[kolaente]

I'll try to implement this.

[lafriks]

Problem is with this part:

group by DATE_FORMAT(FROM_UNIXTIME(created_unix), '%Y%m%d')

At least I did not find a way to implement it for sqlite

[kolaente]

@lafriks I did find one:

SELECT created_unix as timestamp, count(user_id) as contributions
FROM `action`
WHERE (user_id = 1)
  AND (created_unix > 1508503562)
GROUP BY strftime('%Y-%m-%d', created_unix, 'unixepoch')
ORDER BY created_unix

I think I'll do it with a hard switch to provide another query if we are dealing with an sqlite db.

[lafriks]

I think you will need different for mssql and Postgres also

[kolaente]

PR is up: #5131

[naiba]

Awesome, can't wait for release this.