Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🤗 What's the main difference in the throughout between v1.* and v2.*? #1139

Closed
fengnex opened this issue Jan 30, 2021 · 21 comments
Closed

Comments

@fengnex
Copy link

fengnex commented Jan 30, 2021

Question description
I had some stress tests on fiber v1.14.6 by accessing a local sqlite (query a few records) or remote redis (get a short string value) and found that the result throughtput can up to around 12000 requests/s.

import "github.com/gofiber/fiber"

But the above sentence actually downloads and uses fiber v1.14.6, rather than v2.*. What I wonder is how to let it use fiber v2 instead and the performance diference between the two major versions.

Code snippet Optional


import (
        "fmt"     

        "qc.com/m/v1/book"                    
        "qc.com/m/v1/database"             
        "github.com/gofiber/fiber"
        "github.com/jinzhu/gorm"
        _ "github.com/jinzhu/gorm/dialects/sqlite"
)


func setupRoutes(app *fiber.App) {  
        app.Get("/api/v1/book", book.GetBooks)
}

func initDatabase() {
        var err error
        database.DBConn, err = gorm.Open("sqlite3", "books.db")
        if err != nil {
                panic("failed to connect database")
        }
        fmt.Println("Connection Opened to Database")
        database.DBConn.AutoMigrate(&book.Book{})
        fmt.Println("Database Migrated")
}

func main() {
        app := fiber.New()
        initDatabase()
        setupRoutes(app)
        app.Listen(3000)
       
        defer database.DBConn.Close()
}

@welcome
Copy link

welcome bot commented Jan 30, 2021

Thanks for opening your first issue here! 🎉 Be sure to follow the issue template! If you need help or want to chat with us, join us on Discord https://gofiber.io/discord

@kiyonlin
Copy link
Contributor

Please see v2 - Changelog

@fengnex
Copy link
Author

fengnex commented Jan 30, 2021

go.mod's contents:
module qc.com/m/v1

go 1.15

require (
github.com/gofiber/fiber v1.14.6
github.com/gomodule/redigo v1.8.3 // indirect
github.com/jinzhu/gorm v1.9.16
)

@Fenny
Copy link
Member

Fenny commented Jan 31, 2021

@fengnex, are you having problems using v2?

See #824 if it helps with initializing your go.mod

Make sure you are using github.com/gofiber/fiber/v2 for your import paths.

@fengnex
Copy link
Author

fengnex commented Feb 1, 2021

@Fenny There are no any problems for me to correctly use v2.

The original go programs need many modificiations before they can successfully run, but what made me surprise is that the result throughput of v2 falls to 11600 requests/s from acround 12800 requests/s of v1 according to several stress tests. The tested url is http://server-ip:3000/api/v1/book/.

What is the root cause for the above test result? What should I do to improve it or how should I understand that?

The new program which can be compiled with fiber2 looks like the following:

package main

import (
        "fmt"
        "time"

        "qc.com/m/v1/book"
        "qc.com/m/v1/database"
        "github.com/gofiber/fiber/v2"
        "github.com/jinzhu/gorm"
        _ "github.com/jinzhu/gorm/dialects/sqlite"

        //"github.com/go-redis/redis"
)

func helloWorld(c *fiber.Ctx) error {
        return c.SendString("Hello, World!")
}

func rtest(c *fiber.Ctx) error {
        result,err := Exec("get","hello")
        if err != nil {
                fmt.Print(err.Error())
        }
        str,_:=red.String(result,err)
        return c.JSON(str)
}

func setupRoutes(app *fiber.App) {
        app.Get("/", helloWorld)

        app.Get("/api/v1/book", book.GetBooks)
        app.Get("/api/v1/book/:id", book.GetBook)
        app.Post("/api/v1/book", book.NewBook)
        app.Post("/api/v1/book_json", book.NewBookJson)
        app.Delete("/api/v1/book/:id", book.DeleteBook)

        app.Get("/api/v1/rtest",rtest)
}

func initDatabase() {
        var err error
        database.DBConn, err = gorm.Open("sqlite3", "books.db")
        if err != nil {
                panic("failed to connect database")
        }
        fmt.Println("Connection Opened to Database")
        database.DBConn.AutoMigrate(&book.Book{})
        fmt.Println("Database Migrated")
}

func main() {
        app := fiber.New()
        initDatabase()

        initRedis()
        Exec("set","hello","world")

        setupRoutes(app)
        app.Listen(":3000")
       
        defer database.DBConn.Close()
}

@fengnex
Copy link
Author

fengnex commented Feb 1, 2021

Source code of book.GetBooks:

func GetBooks(c *fiber.Ctx) error {
        db := database.DBConn
        var books []Book
        db.Find(&books)
        return c.JSON(books)
        //c.Send("All Books")
}

There are only 3 simple records in the local sqlite database.

@fengnex
Copy link
Author

fengnex commented Feb 4, 2021

Is there anyone who would be so kind as to explain the performance differences between fiber v1.* and v2.*?
Can we achieve an increase in fiber v2's performance and how to implement that?

@kiyonlin
Copy link
Contributor

kiyonlin commented Feb 4, 2021

Could you provide an easy and runnable repo to reproduce your issue?

And tell us which benchmark loader you use.

@fengnex
Copy link
Author

fengnex commented Feb 4, 2021

The source code of tested go application using fiber v1 and v2 can be found from the above code.
vegeta (https://github.com/tsenart/vegeta) was used to generate the test results mentioned here, you can obtain similar stress test records using a command like:

echo "GET http://fiber-app-ip:3000/api/v1/rtest/" | ./vegeta attack -rate=13000 -duration=10s > result.bin
./vegeta report -type=json result.bin

One corresponding output is:
{"latencies":{"total":6791532416620,"mean":52242557,"50th":40019999,"90th":62877540,"95th":72578148,"99th":1002333468,"max":1018241338,"min":31253},"bytes_in":{"total":876444,"mean":6.741876923076923},"bytes_out":{"total":0,"mean":0},"earliest":"2021-01-20T14:47:01.953702274+08:00","latest":"2021-01-20T14:47:11.953703133+08:00","end":"2021-01-20T14:47:11.958348869+08:00","duration":10000000859,"wait":4645736,"requests":130000,"rate":12999.998883300095,"throughput":12644.824462187811,"success":0.9731307692307692,"status_codes":{"0":3493,"200":126507},"errors":["Get "http://fiber-app-ip:3000/api/v1/rtest/": dial tcp 0.0.0.0:0-\u003e10.10.39.22:3000: socket: too many open files"]}

I have the both executable files of fiber application and vegeta, but it seems not a good idea to upload them and actually github does not accept such files here.

@kiyonlin
Copy link
Contributor

kiyonlin commented Feb 4, 2021

The source code of tested go application using fiber v1 and v2 can be found from the above code.

Sorry, I don't think these code is complete and runnable. By any chance can you at least make /api/v1/rtest runnable?

@kiyonlin
Copy link
Contributor

kiyonlin commented Feb 4, 2021

And I can see socket: too many open files error in your output, did you set limitation of socket(fd) correctly?

@fengnex
Copy link
Author

fengnex commented Feb 4, 2021

The complete test programs that use fiber v2 are as follows:

  1. main.go
package main

import (
        "fmt"
        "time"

        "qc.com/m/v1/book"
        "qc.com/m/v1/database"
        "github.com/gofiber/fiber/v2"
        "github.com/jinzhu/gorm"
        _ "github.com/jinzhu/gorm/dialects/sqlite"

        //"github.com/go-redis/redis"
        red "github.com/gomodule/redigo/redis"
)

type Redis struct {
        pool     *red.Pool
}

var redis *Redis

func initRedis() {
        redis = new(Redis)
        redis.pool = &red.Pool{
                MaxIdle:     256,
                MaxActive:   0,
                IdleTimeout: time.Duration(120),
                Dial: func() (red.Conn, error) {
                        return red.Dial(
                                "tcp",
                                "ip:6480",
                                red.DialReadTimeout(time.Duration(1000)*time.Millisecond),
                                red.DialWriteTimeout(time.Duration(1000)*time.Millisecond),
                                red.DialConnectTimeout(time.Duration(1000)*time.Millisecond),
                                red.DialDatabase(0),
                                red.DialPassword("pwd"),
                        )
                },
        }
}

func Exec(cmd string, key interface{}, args ...interface{}) (interface{}, error) {
        con := redis.pool.Get()
        if err := con.Err(); err != nil {
                return nil, err
        }
        defer con.Close()
        parmas := make([]interface{}, 0)
        parmas = append(parmas, key)

        if len(args) > 0 {
                for _, v := range args {
                        parmas = append(parmas, v)
                }
        }
        return con.Do(cmd, parmas...)
}

func helloWorld(c *fiber.Ctx) error {
        return c.SendString("Hello, World!")
}

func rtest(c *fiber.Ctx) error {
        result,err := Exec("get","hello")
        if err != nil {
                fmt.Print(err.Error())
        }
        str,_:=red.String(result,err)
        return c.JSON(str)
}

func setupRoutes(app *fiber.App) {
        app.Get("/", helloWorld)

        app.Get("/api/v1/book", book.GetBooks)
        app.Get("/api/v1/book/:id", book.GetBook)
        app.Post("/api/v1/book", book.NewBook)
        app.Post("/api/v1/book_json", book.NewBookJson)
        app.Delete("/api/v1/book/:id", book.DeleteBook)

        app.Get("/api/v1/rtest",rtest)
}

func initDatabase() {
        var err error
        database.DBConn, err = gorm.Open("sqlite3", "books.db")
        if err != nil {
                panic("failed to connect database")
        }
        fmt.Println("Connection Opened to Database")
        database.DBConn.AutoMigrate(&book.Book{})
        fmt.Println("Database Migrated")
}

func main() {
        app := fiber.New()
        initDatabase()

        initRedis()
        Exec("set","hello","world")

        setupRoutes(app)
        app.Listen(":3000")
       
        defer database.DBConn.Close()
}
  1. book/book.go
package book

import (
        "qc.com/m/v1/database"
        "github.com/gofiber/fiber/v2"
        "github.com/jinzhu/gorm"
        _ "github.com/jinzhu/gorm/dialects/sqlite"
)

type Book struct {
        gorm.Model
        Title  string `json:"name"`
        Author string `json:"author"`
        Rating int    `json:"rating"`
}

func GetBooks(c *fiber.Ctx) error {
        db := database.DBConn
        var books []Book
        db.Find(&books)
        return c.JSON(books)
}

func GetBook(c *fiber.Ctx) error {
        id := c.Params("id")
        db := database.DBConn
        var book Book
        db.Find(&book, id)
        return c.JSON(book)
        //c.Send("Single Book")
}

func NewBook(c *fiber.Ctx) error {
        db := database.DBConn
        var book Book
        book.Title = "1984"
        book.Author = "George Orwell"
        book.Rating = 5
        db.Create(&book)
        return c.JSON(book)
}

func NewBookJson(c *fiber.Ctx) error {
        db := database.DBConn
        book := new(Book)
        if err := c.BodyParser(book); err != nil {
                return c.SendStatus(503)
        }
        db.Create(&book)
        return c.JSON(book)
}

func DeleteBook(c *fiber.Ctx) error {
        id := c.Params("id")
        db := database.DBConn

        var book Book
        db.First(&book, id)
        if book.Title == "" {
            return c.SendStatus(500)
        }
        db.Delete(&book)
        return c.SendString("Book Successfully deleted")
}
  1. database/database.go
package database

import (
        "github.com/jinzhu/gorm"
        _ "github.com/jinzhu/gorm/dialects/sqlite"
)

var (
        DBConn *gorm.DB
)
  1. go.mod
module qc.com/m/v1

go 1.15

require (
        github.com/gofiber/fiber/v2 v2.4.0
        github.com/gomodule/redigo v1.8.3 // indirect
        github.com/jinzhu/gorm v1.9.16
)

Related configuration of socket: too many open files:

ulimit -a|grep 'open files'
open files                      (-n) 65535

The url (http://fiber-app-ip:3000/api/v1/rtest/) is to obtain a string value from redis by calling redis's get. The result throughput is similar with that of accessing local sqlite; I guess this is because of capacity limitation of the fiber service.

@fengnex
Copy link
Author

fengnex commented Feb 4, 2021

@kiyonlin
The complete test programs for fiber v1 can be quickly wrote according to the first few posts, and I can also paste them here if you want me to do that.

All the tests are executed on CentOS Linux release 7.8.2003. The test server has 6 cpu cores and 64Gb of memory.

Fiber v2 application has a different banner with that of fiber v1 application and it looks like the following:

┌───────────────────────────────────────────────────┐ 
 │                    Fiber v2.4.0                   │ 
 │               http://127.0.0.1:3000               │ 
 │                                                   │ 
 │ Handlers ............ 11  Processes ........... 1 │ 
 │ Prefork ....... Disabled  PID ............. 14106 │ 
 └───────────────────────────────────────────────────┘ 

@kiyonlin
Copy link
Contributor

kiyonlin commented Feb 5, 2021

I think I found the reason.

This config IdleTimeout: time.Duration(120) in the redis pool means 120ns, so you'll dial redis almost all the time. Please change it to IdleTimeout: time.Second * 120, and set config Wait: true.

The vegeta result on my machine looks like

{"latencies":{"total":30829800748,"mean":237152,"50th":132145,"90th":159838,"95th":201268,"99th":816517,"max":32812825,"min":81229},"bytes_in":{"total":910000,"mean":7},"bytes_out":{"total":0,"mean":0},"earliest":"2021-02-05T08:42:43.680300663+08:00","latest":"2021-02-05T08:42:53.680191261+08:00","end":"2021-02-05T08:42:53.680388308+08:00","duration":9999890598,"wait":197047,"requests":130000,"rate":13000.14222415596,"throughput":12999.886062498605,"success":1,"status_codes":{"200":130000},"errors":[]}

@fengnex
Copy link
Author

fengnex commented Feb 5, 2021

@kiyonlin You are right and I found the maximum throughput can reach 30880 requests/s in my environment where fiber application and Redis are located on different servers, which I think is a great test result.

But there is a throughput decline about 400 requests/s when I attacked http://fiber-app-ip:3000/api/v1/book/ accessing local sqlite in the same environment.

The detailed result of v1:
echo "GET http://fiber-app-ip:3000/api/v1/book/" | ./vt attack -rate=16000 -duration=10s | ./vt report -type=json
{"latencies":{"total":9698308528040,"mean":60614428,"50th":5174020,"90th":238913978,"95th":315546984,"99th":401291065,"max":632856422,"min":30253},"bytes_in":{"total":66534244,"mean":415.839025},"bytes_out":{"total":0,"mean":0},"earliest":"2021-01-30T12:03:16.205139026+08:00","latest":"2021-01-30T12:03:26.204931032+08:00","end":"2021-01-30T12:03:26.328178158+08:00","duration":9999792006,"wait":123247126,"requests":160000,"rate":16000.332797321986,"throughput":12862.14528089804,"success":0.813775,"status_codes":{"0":29796,"200":130204},"errors":["Get "http://fiber-app-ip:3000/api/v1/book/": dial tcp 0.0.0.0:0-\u003e10.10.39.22:3000: socket: too many open files"]}

Corresponding test result of v2:
{"latencies":{"total":9962809811985,"mean":62267561,"50th":4750012,"90th":246529619,"95th":318445771,"99th":412132071,"max":665999169,"min":30332},"bytes_in":{"total":64524481,"mean":403.27800625},"bytes_out":{"total":0,"mean":0},"earliest":"2021-02-05T09:40:39.87155801+08:00","latest":"2021-02-05T09:40:49.871467294+08:00","end":"2021-02-05T09:40:50.047015937+08:00","duration":9999909284,"wait":175548643,"requests":160000,"rate":16000.145146916717,"throughput":12409.367804956184,"success":0.78919375,"status_codes":{"0":33729,"200":126271},"errors":["Get "http://fiber-app-ip:3000/api/v1/book/": dial tcp 0.0.0.0:0-\u003e10.10.39.22:3000: socket: too many open files"]}

I suppose we can conclude that fiber is an outstanding web application from these results. But wheter is it fair if we say there is no noticeable performance improvement and actually there may be certain performance decline sometimes after upgrading to fiber v2?

@kiyonlin
Copy link
Contributor

kiyonlin commented Feb 5, 2021

I think the limitation is on db(sqlite/redis), not the framework. You can test helloWorld to see if there has obvious performance difference.

@fengnex
Copy link
Author

fengnex commented Feb 5, 2021

@kiyonlin I agree with you; but what's your opinion when comparing the test results of fiber v1 and v2?

@kiyonlin
Copy link
Contributor

kiyonlin commented Feb 5, 2021

see #773 (v2 Benchmarks). But the most important thing is the breaking change of api.

@fengnex
Copy link
Author

fengnex commented Feb 5, 2021

I still don't know what we should think about the performance degradation of fiber V2 in accessing the database. I think this is a general problem and I don't think such a problem should occur in the update of a large new version.

I see that fiber V2 supports more and more functions and features, but I still don't know what the main added value is and whether it is worth it. I will be glad to see other opinions.

@kiyonlin
Copy link
Contributor

kiyonlin commented Feb 5, 2021

Maybe it is because fiber is faster and database can hardly to catch up.

@hi019
Copy link
Contributor

hi019 commented Feb 5, 2021

I still don't know what we should think about the performance degradation of fiber V2 in accessing the database. I think this is a general problem and I don't think such a problem should occur in the update of a large new version.

We have not observed any performance regressions from v1 to v2. I think it would be best for you to create a new issue with a small example (preferably one route) that demonstrates the regression, and we can discuss v2 in this issue.

I see that fiber V2 supports more and more functions and features, but I still don't know what the main added value is and whether it is worth it. I will be glad to see other opinions.

The main feature of v2 is the error handling. It is now more elegant and natural. Also, processing of many routes is faster, there is new middleware (monitor, rewritten session, compress improvements and more). All our middleware also uses a standardized storage interface, implemented in https://github.com/gofiber/storage.

In addition, Fiber v1 is considered legacy. While we will still update it with security fixes, eventually it will be unsupported. Like any other major package update, it is better to change your code sooner than later.

@hi019 hi019 closed this as completed Feb 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants