siddontang

siddontang

Member Since 10 years ago

PingCAP, China

Experience Points
2.7k
follower
Lessons Completed
0
follow
Lessons Completed
423
stars
Best Reply Awards
145
repos

42 contributions in the last year

Pinned
⚡ a MySQL proxy powered by Go
⚡ my golang lib
⚡ a fast distributed message queue implemented with go
⚡ libtnet is a tiny high performance c++ network lib, like tornado
⚡ Elasticsearch note
Activity
Jan
17
2 days ago
Activity icon
issue

siddontang issue comment pingcap/tidb

siddontang
siddontang

* : replace fmt.Sprint with strconv

Signed-off-by: Weizhen Wang [email protected]

What problem does this PR solve?

Issue Number: ref #31716

Problem Summary:

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None
siddontang
siddontang

the reference issue doesn't show anything, so why do we do this?

Activity icon
issue

siddontang issue comment tikv/rfcs

siddontang
siddontang

Integer Shard Index

Signed-off-by: TonsnakeLin [email protected] Hot index reduces the write scalability of the TiDB cluster when the written data is monotonically increasing. We find out a proposal to solve the problem that is using an expression index to scatter the hot index, the new expression index is called a shard index.

siddontang
siddontang

don't use Chinese in the benchmark result picture.

Jan
10
1 week ago
started
started time in 1 week ago
Dec
30
2 weeks ago
open pull request

siddontang wants to merge tikv/community

siddontang
siddontang

new PR requirement and commit message proposal

Issue Number: ref #161

@Mini256

siddontang
siddontang
Dec
29
3 weeks ago
push

siddontang push siddontang/tiplay

siddontang
siddontang

add notice in readme

Signed-off-by: siddontang [email protected]

commit sha: fccf3fb2eea49d4c18db80e26612eca79f5fb92d

push time in 2 weeks ago
Activity icon
created branch
createdAt 2 weeks ago
Activity icon
created repository
createdAt 2 weeks ago
Dec
27
3 weeks ago
Activity icon
issue

siddontang issue comment pingcap/tidb

siddontang
siddontang

execution: refine precision of cast as decimal in agg func

What problem does this PR solve?

Problem Summary:

Currently the sum function will cast MySQL integer types TINYINT, SMALLINT, MEDIUMINT, INT, BIGINT to decimals with big precision, which is waste of CPU when pushed down to TiFlash.

What is changed and how it works?

Change the argument decimal precision of sum to be just enough for the integer type.

It speeds up TiFlash in a query for TPCH-100 dataset:

select sum(L_LINENUMBER) from lineitem;
before: cast int to decimal(32, 0)   1.42s
after: cast int to decimal(10, 0)    1.1s

Benchmark tests

We have conducted benchmark tests with this improvement when the cast is executed on TiDB/TiKV/TiFlash.

Results

Here's the summary of results. We expect that TiFlash is what really benefits from this improvement, because of it's implementation of different precision of Decimal. TiDB/TiKV's results are within acceptable experimental errors.

engine sum(int) sum(bigint)
before after % before after %
TiDB 28.40s 28.79s +1.4% 29.60s 28.26s -4.5%
TiFlash 1.52s 1.18s -22.4% 2.40s 1.63s -32.1%
TiKV 29.8s 29.09s -2.4% 26.51s 26.75s +0.9%

plan before PR

These are the plans before this PR respectively:

-- TiFlash
MySQL [tpch_100]> desc select sum(L_LINENUMBER) from lineitem;
+--------------------------------+--------------+-------------------+----------------+-----------------------------------------------------------------------+
| id                             | estRows      | task              | access object  | operator info                                                         |
+--------------------------------+--------------+-------------------+----------------+-----------------------------------------------------------------------+
| HashAgg_23                     | 1.00         | root              |                | funcs:sum(Column#22)->Column#18                                       |
| └─TableReader_25               | 1.00         | root              |                | data:ExchangeSender_24                                                |
|   └─ExchangeSender_24          | 1.00         | batchCop[tiflash] |                | ExchangeType: PassThrough                                             |
|     └─HashAgg_8                | 1.00         | batchCop[tiflash] |                | funcs:sum(Column#26)->Column#22                                       |
|       └─Projection_37          | 600037902.00 | batchCop[tiflash] |                | cast(tpch_100.lineitem.l_linenumber, decimal(32,0) BINARY)->Column#26 |
|         └─TableFullScan_22     | 600037902.00 | batchCop[tiflash] | table:lineitem | keep order:false                                                      |
+--------------------------------+--------------+-------------------+----------------+-----------------------------------------------------------------------+
6 rows in set (0.00 sec)

MySQL [tpch_100]> desc select sum(cast(L_LINENUMBER as signed)) from lineitem;
+--------------------------------+--------------+-------------------+----------------+------------------------------------------------------------------------------------------------+
| id                             | estRows      | task              | access object  | operator info                                                                                  |
+--------------------------------+--------------+-------------------+----------------+------------------------------------------------------------------------------------------------+
| HashAgg_23                     | 1.00         | root              |                | funcs:sum(Column#22)->Column#18                                                                |
| └─TableReader_25               | 1.00         | root              |                | data:ExchangeSender_24                                                                         |
|   └─ExchangeSender_24          | 1.00         | batchCop[tiflash] |                | ExchangeType: PassThrough                                                                      |
|     └─HashAgg_8                | 1.00         | batchCop[tiflash] |                | funcs:sum(Column#26)->Column#22                                                                |
|       └─Projection_37          | 600037902.00 | batchCop[tiflash] |                | cast(cast(tpch_100.lineitem.l_linenumber, bigint(22) BINARY), decimal(43,0) BINARY)->Column#26 |
|         └─TableFullScan_22     | 600037902.00 | batchCop[tiflash] | table:lineitem | keep order:false                                                                               |
+--------------------------------+--------------+-------------------+----------------+------------------------------------------------------------------------------------------------+
6 rows in set (0.00 sec)

-- TiKV

MySQL [tpch_100]> desc select sum(L_LINENUMBER) from lineitem;
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------+
| id                         | estRows      | task      | access object                                           | operator info                                        |
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------+
| StreamAgg_20               | 1.00         | root      |                                                         | funcs:sum(Column#23)->Column#18                      |
| └─IndexReader_21           | 1.00         | root      |                                                         | index:StreamAgg_8                                    |
|   └─StreamAgg_8            | 1.00         | cop[tikv] |                                                         | funcs:sum(tpch_100.lineitem.l_linenumber)->Column#23 |
|     └─IndexFullScan_19     | 600037902.00 | cop[tikv] | table:lineitem, index:PRIMARY(L_ORDERKEY, L_LINENUMBER) | keep order:false                                     |
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------+

MySQL [tpch_100]> desc select sum(cast(L_LINENUMBER as signed)) from lineitem;
+----------------------------+--------------+-----------+---------------------------------------------------------+-------------------------------------------------------------------------------+
| id                         | estRows      | task      | access object                                           | operator info                                                                 |
+----------------------------+--------------+-----------+---------------------------------------------------------+-------------------------------------------------------------------------------+
| StreamAgg_20               | 1.00         | root      |                                                         | funcs:sum(Column#23)->Column#18                                               |
| └─IndexReader_21           | 1.00         | root      |                                                         | index:StreamAgg_8                                                             |
|   └─StreamAgg_8            | 1.00         | cop[tikv] |                                                         | funcs:sum(cast(tpch_100.lineitem.l_linenumber, bigint(22) BINARY))->Column#23 |
|     └─IndexFullScan_19     | 600037902.00 | cop[tikv] | table:lineitem, index:PRIMARY(L_ORDERKEY, L_LINENUMBER) | keep order:false                                                              |
+----------------------------+--------------+-----------+---------------------------------------------------------+-------------------------------------------------------------------------------+
4 rows in set (0.00 sec)

-- TiDB

MySQL [tpch_100]> set tidb_opt_cpu_factor=0; set tidb_opt_copcpu_factor=100000000000000000;
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

MySQL [tpch_100]> desc select sum(L_LINENUMBER) from lineitem;
+----------------------------+--------------+-----------+---------------------------------------------------------+-----------------------------------------------------------------------+
| id                         | estRows      | task      | access object                                           | operator info                                                         |
+----------------------------+--------------+-----------+---------------------------------------------------------+-----------------------------------------------------------------------+
| StreamAgg_9                | 1.00         | root      |                                                         | funcs:sum(Column#25)->Column#18                                       |
| └─Projection_26            | 600037902.00 | root      |                                                         | cast(tpch_100.lineitem.l_linenumber, decimal(32,0) BINARY)->Column#25 |
|   └─IndexReader_25         | 600037902.00 | root      |                                                         | index:IndexFullScan_24                                                |
|     └─IndexFullScan_24     | 600037902.00 | cop[tikv] | table:lineitem, index:PRIMARY(L_ORDERKEY, L_LINENUMBER) | keep order:false                                                      |
+----------------------------+--------------+-----------+---------------------------------------------------------+-----------------------------------------------------------------------+
4 rows in set (0.00 sec)

MySQL [tpch_100]> desc select sum(cast(L_LINENUMBER as signed)) from lineitem;
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------------------------------------------------+
| id                         | estRows      | task      | access object                                           | operator info                                                                                  |
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------------------------------------------------+
| StreamAgg_9                | 1.00         | root      |                                                         | funcs:sum(Column#25)->Column#18                                                                |
| └─Projection_26            | 600037902.00 | root      |                                                         | cast(cast(tpch_100.lineitem.l_linenumber, bigint(22) BINARY), decimal(43,0) BINARY)->Column#25 |
|   └─IndexReader_25         | 600037902.00 | root      |                                                         | index:IndexFullScan_24                                                                         |
|     └─IndexFullScan_24     | 600037902.00 | cop[tikv] | table:lineitem, index:PRIMARY(L_ORDERKEY, L_LINENUMBER) | keep order:false                                                                               |
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------------------------------------------------+
4 rows in set (0.00 sec)

plans after PR

These are the plans after this PR:

-- TiFlash
MySQL [tpch_100]> desc select sum(L_LINENUMBER) from lineitem;
+--------------------------------+--------------+-------------------+----------------+-----------------------------------------------------------------------+
| id                             | estRows      | task              | access object  | operator info                                                         |
+--------------------------------+--------------+-------------------+----------------+-----------------------------------------------------------------------+
| HashAgg_23                     | 1.00         | root              |                | funcs:sum(Column#22)->Column#18                                       |
| └─TableReader_25               | 1.00         | root              |                | data:ExchangeSender_24                                                |
|   └─ExchangeSender_24          | 1.00         | batchCop[tiflash] |                | ExchangeType: PassThrough                                             |
|     └─HashAgg_8                | 1.00         | batchCop[tiflash] |                | funcs:sum(Column#26)->Column#22                                       |
|       └─Projection_37          | 600037902.00 | batchCop[tiflash] |                | cast(tpch_100.lineitem.l_linenumber, decimal(10,0) BINARY)->Column#26 |
|         └─TableFullScan_22     | 600037902.00 | batchCop[tiflash] | table:lineitem | keep order:false                                                      |
+--------------------------------+--------------+-------------------+----------------+-----------------------------------------------------------------------+
6 rows in set (0.01 sec)

MySQL [tpch_100]> desc select sum(cast(L_LINENUMBER as signed)) from lineitem;
+--------------------------------+--------------+-------------------+----------------+------------------------------------------------------------------------------------------------+
| id                             | estRows      | task              | access object  | operator info                                                                                  |
+--------------------------------+--------------+-------------------+----------------+------------------------------------------------------------------------------------------------+
| HashAgg_23                     | 1.00         | root              |                | funcs:sum(Column#22)->Column#18                                                                |
| └─TableReader_25               | 1.00         | root              |                | data:ExchangeSender_24                                                                         |
|   └─ExchangeSender_24          | 1.00         | batchCop[tiflash] |                | ExchangeType: PassThrough                                                                      |
|     └─HashAgg_8                | 1.00         | batchCop[tiflash] |                | funcs:sum(Column#26)->Column#22                                                                |
|       └─Projection_37          | 600037902.00 | batchCop[tiflash] |                | cast(cast(tpch_100.lineitem.l_linenumber, bigint(22) BINARY), decimal(20,0) BINARY)->Column#26 |
|         └─TableFullScan_22     | 600037902.00 | batchCop[tiflash] | table:lineitem | keep order:false                                                                               |
+--------------------------------+--------------+-------------------+----------------+------------------------------------------------------------------------------------------------+
6 rows in set (0.00 sec)

-- TiKV

MySQL [tpch_100]> desc select sum(L_LINENUMBER) from lineitem;
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------+
| id                         | estRows      | task      | access object                                           | operator info                                        |
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------+
| StreamAgg_20               | 1.00         | root      |                                                         | funcs:sum(Column#23)->Column#18                      |
| └─IndexReader_21           | 1.00         | root      |                                                         | index:StreamAgg_8                                    |
|   └─StreamAgg_8            | 1.00         | cop[tikv] |                                                         | funcs:sum(tpch_100.lineitem.l_linenumber)->Column#23 |
|     └─IndexFullScan_19     | 600037902.00 | cop[tikv] | table:lineitem, index:PRIMARY(L_ORDERKEY, L_LINENUMBER) | keep order:false                                     |
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------+
4 rows in set (0.01 sec)

MySQL [tpch_100]> desc select sum(cast(L_LINENUMBER as signed)) from lineitem;
+----------------------------+--------------+-----------+---------------------------------------------------------+-------------------------------------------------------------------------------+
| id                         | estRows      | task      | access object                                           | operator info                                                                 |
+----------------------------+--------------+-----------+---------------------------------------------------------+-------------------------------------------------------------------------------+
| StreamAgg_20               | 1.00         | root      |                                                         | funcs:sum(Column#23)->Column#18                                               |
| └─IndexReader_21           | 1.00         | root      |                                                         | index:StreamAgg_8                                                             |
|   └─StreamAgg_8            | 1.00         | cop[tikv] |                                                         | funcs:sum(cast(tpch_100.lineitem.l_linenumber, bigint(22) BINARY))->Column#23 |
|     └─IndexFullScan_19     | 600037902.00 | cop[tikv] | table:lineitem, index:PRIMARY(L_ORDERKEY, L_LINENUMBER) | keep order:false                                                              |
+----------------------------+--------------+-----------+---------------------------------------------------------+-------------------------------------------------------------------------------+
4 rows in set (0.00 sec)

-- TiDB

MySQL [tpch_100]> desc select sum(L_LINENUMBER) from lineitem;
+----------------------------+--------------+-----------+---------------------------------------------------------+-----------------------------------------------------------------------+
| id                         | estRows      | task      | access object                                           | operator info                                                         |
+----------------------------+--------------+-----------+---------------------------------------------------------+-----------------------------------------------------------------------+
| StreamAgg_9                | 1.00         | root      |                                                         | funcs:sum(Column#25)->Column#18                                       |
| └─Projection_26            | 600037902.00 | root      |                                                         | cast(tpch_100.lineitem.l_linenumber, decimal(10,0) BINARY)->Column#25 |
|   └─IndexReader_25         | 600037902.00 | root      |                                                         | index:IndexFullScan_24                                                |
|     └─IndexFullScan_24     | 600037902.00 | cop[tikv] | table:lineitem, index:PRIMARY(L_ORDERKEY, L_LINENUMBER) | keep order:false                                                      |
+----------------------------+--------------+-----------+---------------------------------------------------------+-----------------------------------------------------------------------+
4 rows in set (0.00 sec)

MySQL [tpch_100]> desc select sum(cast(L_LINENUMBER as signed)) from lineitem;
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------------------------------------------------+
| id                         | estRows      | task      | access object                                           | operator info                                                                                  |
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------------------------------------------------+
| StreamAgg_9                | 1.00         | root      |                                                         | funcs:sum(Column#25)->Column#18                                                                |
| └─Projection_26            | 600037902.00 | root      |                                                         | cast(cast(tpch_100.lineitem.l_linenumber, bigint(22) BINARY), decimal(20,0) BINARY)->Column#25 |
|   └─IndexReader_25         | 600037902.00 | root      |                                                         | index:IndexFullScan_24                                                                         |
|     └─IndexFullScan_24     | 600037902.00 | cop[tikv] | table:lineitem, index:PRIMARY(L_ORDERKEY, L_LINENUMBER) | keep order:false                                                                               |
+----------------------------+--------------+-----------+---------------------------------------------------------+------------------------------------------------------------------------------------------------+
4 rows in set (0.00 sec)

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None
siddontang
siddontang

@zanmato1984

why TiKV can't get too much benefit from this change? another thing is I only see the integration tests, no any unit tests?

Activity icon
issue

siddontang issue comment mattn/go-sixel

siddontang
siddontang

gosr can't show image in Mac terminal

Hi, I tried to use gosr in Mac Terminal, the image couldn't show, like:

image

But using iTerm worked fine:

image

siddontang
siddontang

well, still can't work with longcat. As you can see, the pic doesn't show well below

image

Dec
26
3 weeks ago
Activity icon
issue

siddontang issue comment mattn/go-sixel

siddontang
siddontang

gosr can't show image in Mac terminal

Hi, I tried to use gosr in Mac Terminal, the image couldn't show, like:

image

But using iTerm worked fine:

image

siddontang
siddontang

What I want is to view the xkcd comics in the terminal, so I write https://github.com/siddontang/xkcdsay

I tried different tools to show the image in the terminal like https://github.com/posva/catimg, https://github.com/ichinaski/pxl, etc, All of them can't show the xkcd picture correctly except go-sixel, but seems not all the terminals support Sixel. Bad news :sob:

push

siddontang push siddontang/xkcdsay

siddontang
siddontang

update readme for help

Signed-off-by: siddontang [email protected]

commit sha: 24de87a01971cc404d46a585c358927b0efd17c4

push time in 3 weeks ago
Activity icon
issue

siddontang issue comment siddontang/xkcdsay

siddontang
siddontang

Image(s) not shown under WSL(V2)

The green arrow indicate that this command succeeded, but no image showed: image

siddontang
siddontang

please refer to https://github.com/siddontang/xkcdsay#limitation

I guess your terminal doesn't support Sixel 😭

push

siddontang push siddontang/xkcdsay

siddontang
siddontang

add sixel requirement

Signed-off-by: siddontang [email protected]

commit sha: 814a8c3990ade6c8b3f2258d4396034ec0dc6345

push time in 3 weeks ago
Dec
25
3 weeks ago
Activity icon
issue

siddontang issue mattn/go-sixel

siddontang
siddontang

gosr can't show image in Mac terminal

Hi, I tried to use gosr in Mac Terminal, the image couldn't show, like:

image

But using iTerm worked fine:

image

push

siddontang push siddontang/homebrew-brew

siddontang
siddontang

support 0.2 version

Signed-off-by: siddontang [email protected]

commit sha: 5bd9809a80400dd6441f4e88aa7dd20aa95c9de1

push time in 3 weeks ago
Activity icon
published release v0.2

siddontang in siddontang/xkcdsay create published release v0.2

createdAt 3 weeks ago
Activity icon
created tag
createdAt 3 weeks ago
push

siddontang push siddontang/xkcdsay

siddontang
siddontang

print saying words in xkcdsay

Signed-off-by: siddontang [email protected]

commit sha: 0445e3c1ede804b93614750fb6e33f8642dce66d

push time in 3 weeks ago
push

siddontang push siddontang/xkcdsay

siddontang
siddontang

add brew support

Signed-off-by: siddontang [email protected]

commit sha: 99bed25416e843e64340a17c28944cbc1f716eac

push time in 3 weeks ago
push

siddontang push siddontang/homebrew-brew

siddontang
siddontang

add xkcdsay support

Signed-off-by: siddontang [email protected]

commit sha: 36ed1bc353a2eeaf5af2295e2fdc6e9397d6237c

push time in 3 weeks ago
Activity icon
created branch
createdAt 3 weeks ago
Activity icon
created repository
createdAt 3 weeks ago
Activity icon
created branch
createdAt 3 weeks ago
Activity icon
fork

siddontang forked Homebrew/homebrew-core

⚡ 🍻 Default formulae for the missing package manager for macOS (or Linux)
siddontang BSD 2-Clause "Simplified" License Updated
fork time in 3 weeks ago
Activity icon
fork

siddontang forked Homebrew/brew

⚡ 🍺 The missing package manager for macOS (or Linux)
siddontang BSD 2-Clause "Simplified" License Updated
fork time in 3 weeks ago
Previous