Compare commits

...

299 Commits

Author SHA1 Message Date
Lion 53e6777abd bash comments 2022-06-24 15:50:35 +08:00
Lion 7e41c0c00f xdb java searcher is ready 2022-06-24 15:28:12 +08:00
Lion 9c270d0c39 update the pom for maven release 2022-06-24 15:23:04 +08:00
Lion 0b137336ef optimize the php docs 2022-06-24 12:43:33 +08:00
Lion 9361d9179c add doc for java binding 2022-06-24 12:41:35 +08:00
Lion 20ad3cc722 update the marks for xdb cached search 2022-06-24 12:24:16 +08:00
Lion 05826d8b4a update the version number to 2.6.0 2022-06-24 11:56:28 +08:00
Lion 5d47a27730 impl the bench test 2022-06-24 11:52:33 +08:00
Lion 3e8d616a2a count maybe 0 2022-06-24 11:52:06 +08:00
lion fa4c983710 store the costs first 2022-06-23 17:59:14 +08:00
lion f5d6b490f3 use the raw []byte as vectorIndex to reduce memory alloc 2022-06-23 17:56:28 +08:00
lion 8d03749a37 add java xdb searcher impl 2022-06-23 17:26:17 +08:00
lion 1b6842a2f1 it should be the cols 2022-06-23 17:03:25 +08:00
lion e7f26019bb mark the php xdb search binding as finsihed 2022-06-23 15:04:51 +08:00
lion a2b6eb6b8c reduce the data copy for vectorIndex search with content cached 2022-06-23 15:03:18 +08:00
lion 1f0d242582 optimize the bench average costs stats 2022-06-23 14:57:02 +08:00
lion 4f7813a482 content buffer directly as vectorIndex 2022-06-23 14:52:39 +08:00
lion 8ed03b40e6 remove the vectorIndex autoload for buffer based searcher to reduce memory usage 2022-06-23 11:42:28 +08:00
Lion 05e9a55e02 update docs 2022-06-22 19:23:03 +08:00
Lion b97306af03 add docs for php binding 2022-06-22 19:21:08 +08:00
Lion b1dd15849c update the docs for cache-policy 2022-06-22 19:20:59 +08:00
Lion ba1321fa6b add bench test script 2022-06-22 18:49:02 +08:00
Lion 8797368cda print the cache policy for bench 2022-06-22 18:48:06 +08:00
Lion f5051e3ebc util test script to test all the utils functions 2022-06-22 18:06:50 +08:00
Lion c28d01897f xdb searcher util and tests 2022-06-22 17:36:01 +08:00
Lion 99802f2d28 print the dataLen and dataPtr for debug 2022-06-22 15:59:29 +08:00
Lion 1635de4350 record the start and end segment index ptr 2022-06-22 10:22:40 +08:00
lion e2e530bdf4 add php xdb searcher impl 2022-06-21 18:02:54 +08:00
lion 82ed1e831e break for the undefine option 2022-06-21 18:02:40 +08:00
lion 231f56b291 use the Header struct 2022-06-21 11:34:47 +08:00
lion a3d33152ba add header mgr struct 2022-06-21 10:57:47 +08:00
lion 6bcef64d55 fix the bug of header write and regen the xdb file 2022-06-21 10:47:28 +08:00
lion e3cda45591 add copy right statement and author contact info 2022-06-21 10:21:37 +08:00
lion c3319ff11f add cache-policy option desc 2022-06-20 22:12:21 +08:00
lion 244472d935 add docs for cache searcher create 2022-06-20 21:45:59 +08:00
Lion e9a033b9f5 add vectorIndex/Content cache policy impl 2022-06-20 15:14:33 +08:00
Lion 542753f666 add Makefile and docs for golang xdb maker and binding 2022-06-19 19:07:03 +08:00
Lion fc8dcdd443 add load header func 2022-06-18 19:36:45 +08:00
Lion 1d4d3f9299 add vector index and xdb content load util func 2022-06-18 19:29:24 +08:00
lion 79b3dd2aec update the command docs 2022-06-17 15:48:48 +08:00
lion bae17f007b xdb_maker and usage docs 2022-06-17 15:35:58 +08:00
lion 0873aa7d69 package the source lib to xdb 2022-06-17 15:23:02 +08:00
lion 98e10b9891 Update the docs 2022-06-17 15:06:34 +08:00
lion 4cf2c28d49 rename pacakge to xdb and add SearchByStr impl 2022-06-17 14:45:16 +08:00
lion cb6b06d64a add golang searcher binding 2022-06-17 14:18:44 +08:00
lion 2e4df39af2 Update the doc 2022-06-17 10:10:42 +08:00
lion b1ccd67321 add search and bench doc for golang maker 2022-06-16 16:58:08 +08:00
lion a7374fa440 track the first xdb file 2022-06-16 16:45:44 +08:00
lion ef51d6e271 Add golang xdb maker implemetation 2022-06-16 16:44:02 +08:00
lion 46b25653d7 init the v2.0 framework with v1.0 source kept 2022-06-16 16:27:56 +08:00
lion a8448dd2b3 archive the v1.0 - old db structure 2022-06-16 16:25:31 +08:00
Lion c30f65659f add english README 2022-06-02 12:41:17 +08:00
Lion 87b308c91e
Merge pull request #223 from Morsmalleo/master
English README.md
2022-06-02 12:13:21 +08:00
Morsmalleo 799ed36b26
Update README.md 2022-06-01 10:13:24 +08:00
Morsmalleo 10408713c2
Update README.md 2022-06-01 10:12:06 +08:00
Morsmalleo 73d36ca99d
Rename README-EN.md to README.md 2022-06-01 09:44:50 +08:00
Morsmalleo 47f24e4514
Update and rename README.md to README-EN.md 2022-06-01 09:44:09 +08:00
lion b508896c84 add db design and impl video link 2022-05-21 21:56:27 +08:00
lion 05eaf03ca8 add data upgrade impl video links 2022-05-21 21:52:51 +08:00
lion d0536d9bff add video link 2022-05-21 21:50:19 +08:00
lion e697a80f6e Merge branch 'master' of https://gitee.com/lionsoul/ip2region 2022-05-21 18:18:28 +08:00
lion a09304a36d Add data upgrade based on region detect doc 2022-05-21 18:16:52 +08:00
Lion 75bac23722
Merge pull request #220 from lionsoul2014/dependabot/npm_and_yarn/binding/nodejs/async-2.6.4
Bump async from 2.3.0 to 2.6.4 in /binding/nodejs
2022-04-29 10:32:02 +08:00
dependabot[bot] ee5519b7bf
Bump async from 2.3.0 to 2.6.4 in /binding/nodejs
Bumps [async](https://github.com/caolan/async) from 2.3.0 to 2.6.4.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v2.3.0...v2.6.4)

---
updated-dependencies:
- dependency-name: async
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-04-28 15:55:54 +00:00
Lion 5c947be68b Merge branch 'master' of https://gitee.com/lionsoul/ip2region 2021-10-25 16:38:38 +08:00
Lion 351b914186 Fix the region name and re-generate the binary file 2021-10-25 16:38:23 +08:00
Lion 4e106af3df
Merge pull request #203 from lionsoul2014/dependabot/npm_and_yarn/binding/nodejs/path-parse-1.0.7
Bump path-parse from 1.0.5 to 1.0.7 in /binding/nodejs
2021-10-24 09:48:41 +08:00
Lion b304cd02fc
Merge pull request #206 from lionsoul2014/dependabot/npm_and_yarn/binding/nodejs/tmpl-1.0.5
Bump tmpl from 1.0.4 to 1.0.5 in /binding/nodejs
2021-10-24 09:48:18 +08:00
dependabot[bot] 4de96c8dbe
Bump tmpl from 1.0.4 to 1.0.5 in /binding/nodejs
Bumps [tmpl](https://github.com/daaku/nodejs-tmpl) from 1.0.4 to 1.0.5.
- [Release notes](https://github.com/daaku/nodejs-tmpl/releases)
- [Commits](https://github.com/daaku/nodejs-tmpl/commits/v1.0.5)

---
updated-dependencies:
- dependency-name: tmpl
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-09-20 23:38:34 +00:00
dependabot[bot] e0fed1b805
Bump path-parse from 1.0.5 to 1.0.7 in /binding/nodejs
Bumps [path-parse](https://github.com/jbgutierrez/path-parse) from 1.0.5 to 1.0.7.
- [Release notes](https://github.com/jbgutierrez/path-parse/releases)
- [Commits](https://github.com/jbgutierrez/path-parse/commits/v1.0.7)

---
updated-dependencies:
- dependency-name: path-parse
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2021-08-10 19:27:14 +00:00
Lion f92bcf3170
Merge pull request #193 from lionsoul2014/dependabot/npm_and_yarn/binding/nodejs/handlebars-4.7.7
Bump handlebars from 4.5.3 to 4.7.7 in /binding/nodejs
2021-05-08 09:40:43 +08:00
Lion 5e6335d954
Merge pull request #187 from lionsoul2014/dependabot/npm_and_yarn/binding/nodejs/y18n-3.2.2
Bump y18n from 3.2.1 to 3.2.2 in /binding/nodejs
2021-05-08 09:40:30 +08:00
Lion deb4980dc5
Merge pull request #195 from lionsoul2014/dependabot/npm_and_yarn/binding/nodejs/hosted-git-info-2.8.9
Bump hosted-git-info from 2.4.2 to 2.8.9 in /binding/nodejs
2021-05-08 09:40:14 +08:00
dependabot[bot] dc94fe9320
Bump hosted-git-info from 2.4.2 to 2.8.9 in /binding/nodejs
Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.4.2 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](https://github.com/npm/hosted-git-info/compare/v2.4.2...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>
2021-05-08 01:38:48 +00:00
Lion 4d43d8b581
Merge pull request #194 from lionsoul2014/dependabot/npm_and_yarn/binding/nodejs/lodash-4.17.21
Bump lodash from 4.17.19 to 4.17.21 in /binding/nodejs
2021-05-08 09:36:45 +08:00
dependabot[bot] 64e28305d9
Bump lodash from 4.17.19 to 4.17.21 in /binding/nodejs
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.19 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.19...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>
2021-05-07 16:28:27 +00:00
dependabot[bot] 57da31c171
Bump handlebars from 4.5.3 to 4.7.7 in /binding/nodejs
Bumps [handlebars](https://github.com/wycats/handlebars.js) from 4.5.3 to 4.7.7.
- [Release notes](https://github.com/wycats/handlebars.js/releases)
- [Changelog](https://github.com/handlebars-lang/handlebars.js/blob/master/release-notes.md)
- [Commits](https://github.com/wycats/handlebars.js/compare/v4.5.3...v4.7.7)

Signed-off-by: dependabot[bot] <support@github.com>
2021-05-06 21:46:24 +00:00
dependabot[bot] b7a788791a
Bump y18n from 3.2.1 to 3.2.2 in /binding/nodejs
Bumps [y18n](https://github.com/yargs/y18n) from 3.2.1 to 3.2.2.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
2021-03-30 14:25:51 +00:00
Lion e42d45b61f Upgrade the origin and binary db data to the latest 2020/03/06 version 2021-03-08 23:06:45 +08:00
Lion 57afca223e Merge branch 'master' of https://gitee.com/lionsoul/ip2region 2021-03-08 22:27:16 +08:00
Lion e923b0d7cc Upgrade the origin and binary db data to the latest 2020/03/06 version 2021-03-08 22:26:52 +08:00
狮子的魂 74947ca39e !7 新版 C# 数据库生成器
Merge pull request !7 from Soar360/master
2020-12-12 20:06:10 +08:00
Soar360 e97c34ea8f 新版 C# 数据库生成器 2020-12-03 13:34:39 +08:00
Lion f3f3c9ebfc Merge branch 'master' of https://github.com/lionsoul2014/ip2region 2020-10-31 12:49:02 +08:00
Lion 6b6480e432 Upgrade the origin ip and binary data to the latest 2020/10/31 release version 2020-10-31 12:41:59 +08:00
Lion 813337de9d
Merge pull request #174 from lionsoul2014/dependabot/maven/binding/java/junit-junit-4.13.1
Bump junit from 3.8.1 to 4.13.1 in /binding/java
2020-10-13 17:10:40 +08:00
dependabot[bot] 646873bb17
Bump junit from 3.8.1 to 4.13.1 in /binding/java
Bumps [junit](https://github.com/junit-team/junit4) from 3.8.1 to 4.13.1.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.1.md)
- [Commits](https://github.com/junit-team/junit4/commits/r4.13.1)

Signed-off-by: dependabot[bot] <support@github.com>
2020-10-12 22:56:44 +00:00
Lion e8c9b3916c
Merge pull request #171 from liyang1009/master
fix bug about stack corrupt for example  ip address is spceial  ip 172.16.107.7, Proxy:False, 210.3.213.69
2020-10-11 17:16:28 +08:00
Lion bf8f6a7677
Merge pull request #159 from coderbusy/master
add C# maker
2020-10-10 15:27:42 +08:00
liyang f0be894109 fix bug about stack corrupt for example ip address is spceial "172.16.107.7, Proxy:False, 210.3.213.69" 2020-10-06 21:55:21 +08:00
lionsoul f36b458230
Merge pull request #168 from zzl88824/master
ip2region LabVIEW库
2020-09-18 18:13:48 +08:00
Zhao Zilong db3f5f38b1
Add files via upload 2020-09-18 18:03:30 +08:00
Zhao Zilong 50e931c8be
Update readme.md 2020-09-18 17:59:13 +08:00
Zhao Zilong 7b30c3070a
Update readme.md 2020-09-18 17:58:31 +08:00
Zhao Zilong 9f01d66886
Add files via upload 2020-09-18 17:55:35 +08:00
Zhao Zilong 11da0e97db
Create readme.md 2020-09-18 17:54:17 +08:00
lionsoul 9f01108f45
Merge pull request #163 from Wuxiaodonggit/python_fixbug
Fix multi instance bugs, use instance variable instead of class varia…
2020-09-03 11:08:56 +08:00
wuxiaodong 40bd5a7a2f Fix multi instance bugs, use instance variable instead of class variable. 2020-09-02 17:26:01 +08:00
lionsoul 083768b908
Merge pull request #160 from lionsoul2014/dependabot/npm_and_yarn/binding/nodejs/lodash-4.17.19
Bump lodash from 4.17.13 to 4.17.19 in /binding/nodejs
2020-07-20 20:08:54 +08:00
dependabot[bot] 667cdc8cf6
Bump lodash from 4.17.13 to 4.17.19 in /binding/nodejs
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.13 to 4.17.19.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.13...4.17.19)

Signed-off-by: dependabot[bot] <support@github.com>
2020-07-16 08:58:29 +00:00
Soar360 9101c218d3 add CSharp Macker 2020-07-09 14:01:07 +08:00
lionsoul d2d3a5e15a Merge branch 'master' of https://gitee.com/lionsoul/ip2region 2020-06-18 08:53:41 +08:00
lionsoul d62ede3131 Upgrade the minimist version inspired by the github alerts 2020-06-18 08:53:20 +08:00
lionsoul e8439a6c0e Upgrade the source and binary data to the latest 2020/06/10 version 2020-06-13 22:06:57 +08:00
Soar360 56a9fe834e Util.cs -> ip2long 2020-06-11 23:58:32 +08:00
Soar360 cfa29188e3 add DbMaker.cs 2020-06-11 23:47:31 +08:00
Soar360 06685a3010 add sln file 2020-06-11 22:12:00 +08:00
Soar360 399f4a5ebc rename to Java style. 2020-06-11 22:10:56 +08:00
Killer 1d02185af1 clean code 2020-06-10 18:13:42 +08:00
Killer e13f9b4a56 add DbSearcher 2020-06-10 18:09:57 +08:00
Killer bf07725342 add HeaderBlock.cs
add IndexBlock.cs
add Util.cs
2020-06-10 17:23:01 +08:00
Killer a9571d443f add DataBlock
add DbConfig
2020-06-10 16:59:32 +08:00
lionsoul c6b3e79df6 upgrade the acorn to version 5.7.4 2020-04-02 09:42:44 +08:00
lionsoul 07287ea6ec Upgrade the minimist to 1.2.2 2020-04-02 09:41:14 +08:00
lionsoul d077ab1d3c Merge branch 'master' of https://gitee.com/lionsoul/ip2region 2020-04-02 09:38:27 +08:00
lionsoul e7a666bdf2 Upgrade the kind-of to version 6.0.3 2020-04-02 09:38:10 +08:00
lionsoul 9715647f87
Merge pull request #147 from phpstatic/master
fix may coming memory leak when failed to initialize ip2region and symbol duplicate for php extension binding
2020-03-22 16:13:18 +08:00
PHP Static 44059a4d0e fix php5 symbol duplicate 2020-03-22 11:43:46 +08:00
PHP Static 28f818e91a fix memory leak and symbol duplicate 2020-03-22 11:38:21 +08:00
dongyado e701781239 update db structure intro file link 2020-03-16 14:28:17 +08:00
lionsoul f09285ac17 Update the link for the php ext 2020-02-21 13:56:56 +08:00
lionsoul 6f3d0d5ece Optimize the installation DOC and rename the php ext binding name 2020-02-21 13:56:26 +08:00
lionsoul 710f8065d0 Optimize the installation DOC 2020-02-21 13:50:55 +08:00
lionsoul 33798b5c0b Optimize the README DOC 2020-02-21 13:49:25 +08:00
lionsoul b092eb1549 Optimize the DOC 2020-02-21 13:45:42 +08:00
lionsoul 6c31b18cf9 Merge branch 'master' of https://gitee.com/lionsoul/ip2region 2020-02-21 13:36:35 +08:00
lionsoul b75d215318 Optimize the README DOC 2020-02-21 13:36:21 +08:00
dongyado 25f82ba132 update readme add tested node version 2020-02-20 16:57:50 +08:00
dongyado 2e4d5d583e update nodejs client and use node-ip2region for npm package name 2020-02-20 12:58:56 +08:00
lionsoul 0a2d8fac38
Merge pull request #140 from lionsoul2014/dependabot/npm_and_yarn/binding/nodejs/handlebars-4.5.3
Bump handlebars from 4.1.2 to 4.5.3 in /binding/nodejs
2019-12-28 09:01:06 +08:00
dependabot[bot] 238f11963e
Bump handlebars from 4.1.2 to 4.5.3 in /binding/nodejs
Bumps [handlebars](https://github.com/wycats/handlebars.js) from 4.1.2 to 4.5.3.
- [Release notes](https://github.com/wycats/handlebars.js/releases)
- [Changelog](https://github.com/wycats/handlebars.js/blob/master/release-notes.md)
- [Commits](https://github.com/wycats/handlebars.js/compare/v4.1.2...v4.5.3)

Signed-off-by: dependabot[bot] <support@github.com>
2019-12-27 11:39:13 +00:00
lionsoul 79a1956245
Merge pull request #138 from RocherKong/master
add data result availability checking
2019-12-11 17:20:05 +08:00
rocher 4013387c43 test result data right 2019-12-11 16:59:44 +08:00
lionsoul e57d010ddc Update the data to the latest version 2019-11-23 11:21:45 +08:00
lionsoul 2baa088a7c Upgrade the DOC for upgrade desc 2019-10-28 08:00:25 +08:00
lionsoul 9ef5c4aada Upgrade the data to the version of 2019/10/27 2019-10-28 07:55:13 +08:00
lionsoul 589abb551d Update the README 2019-10-28 07:46:58 +08:00
lionsoul 41ea94f036 Upgrade mixin-deep for version 1.3.2 and set-value to version 2.0.1 2019-09-07 10:36:36 +08:00
lionsoul 5c1394a4f5 add nginx module support 2019-08-22 10:35:26 +08:00
lionsoul f11b8096e3
Merge pull request #125 from Cherisher/master
add support for nginx
2019-08-22 10:16:59 +08:00
Cherisher d071daf34e add support for nginx 2019-08-22 09:02:09 +08:00
lionsoul f483af329e Security Alerts fixing 2019-07-21 17:51:07 +08:00
lionsoul a49b7deaf4 Upgrade some the dependencies version to the one followed by the github alerts 2019-07-07 11:33:31 +08:00
lionsoul bb3740fba1 Upgrade some the dependencies version to the one followed by the github alerts 2019-07-07 11:18:21 +08:00
lionsoul a2e2262bbe Merge branch 'master' of https://gitee.com/lionsoul/ip2region 2019-07-07 11:10:00 +08:00
lionsoul a8f77a9638 fix the bug of raf=null in the close interface of DbSearcher 2019-07-07 11:09:48 +08:00
lionsoul 151de1a13e Optimize the data source list display 2019-05-05 17:25:05 +08:00
lionsoul 4a59fa6a1d Add source data platform or API url 2019-05-05 17:24:09 +08:00
lionsoul e670bc1748 Add source data platform or API url 2019-05-05 17:23:18 +08:00
lionsoul eebf8c6dd1 Optimize the source data list display 2019-05-05 17:21:08 +08:00
lionsoul 354a6c9342 Optimize the source data list display 2019-05-05 17:19:17 +08:00
lionsoul 00f18bfb65 Add source data platform or API url 2019-05-05 17:17:48 +08:00
lionsoul 5767b1e16e Optimize the maker doc 2019-05-02 20:02:51 +08:00
lionsoul abf229270a Add the java implementation for ip2region.db maker 2019-05-02 19:51:29 +08:00
lionsoul 9d23bc53e1 Upgrade the ip source data and db data to the latest version 2019/05/02 2019-05-02 19:30:46 +08:00
lionsoul 4dabf635a8 Upgrade the lodash base version to 4.17.11 2019-02-12 17:39:11 +08:00
lionsoul a7dfe943b6 Update the CHANGES for maven address update and lua comments 2018-12-31 13:32:05 +08:00
lionsoul 41b9c2d763 Track the CHANGES for version 1.8.0 2018-12-31 13:24:03 +08:00
lionsoul 74b2909ea2 ip source text and binary data upgrade to version 2018/12/31 2018-12-31 13:03:50 +08:00
lionsoul ab1c331fbd Update the merge to version 1.2.1 2018-11-03 11:05:48 +08:00
lionsoul a6968d9767
Merge pull request #86 from heweizhen0822/patch-1
Update README.md
2018-10-18 20:59:29 +08:00
lionsoul e4d3772e08
Merge pull request #85 from heweizhen0822/patch-2
Update main.go
2018-10-18 20:59:09 +08:00
heweizhen0822 06e8aa2528
Update main.go
github.com/mohong122/ip2region/binding/golang/ip2region 并没有子包ip2region 需要改成 github.com/lionsoul2014/ip2region/binding/golang/ip2region
2018-10-18 10:34:33 +08:00
heweizhen0822 281237c5c3
Update README.md
github.com/mohong122/ip2region/binding/golang/ip2region  需要改成 github.com/lionsoul2014/ip2region/binding/golang/ip2region
2018-10-18 10:32:32 +08:00
lionsoul 9c37b5be7f Optimize the lua_c module 2018-10-10 11:54:26 +08:00
lionsoul 253dc12917 Add automatically lib dir make 2018-10-07 20:58:54 +08:00
lionsoul 86104eb31f Optimize the ip2long function with trim logic applied 2018-10-07 10:23:17 +08:00
lionsoul bd6dc7be90 Update the README doc for lua_c module 2018-10-06 15:12:24 +08:00
lionsoul 03fd40bb09 Add README doc for lua_c module 2018-10-06 11:18:01 +08:00
lionsoul f9e9dd59a3 lua ip2region c module forward with release version published 2018-10-06 11:05:01 +08:00
lionsoul 93689e09e8 lua binding module interface stdlize with new and ip2long method access by . support 2018-10-05 21:46:21 +08:00
lionsoul e26ed44861 lua_c module forward with basic framework finished(tested) 2018-10-05 21:39:35 +08:00
lionsoul 7bd81290f1 lua_c binding basic framework(Not tested yet...) 2018-10-05 14:03:16 +08:00
lionsoul 7032830d23 Optimize the lua module by add the __index and the __tostring meta-method 2018-10-05 10:55:24 +08:00
lionsoul e9a08e27a3 Optimize the benchmark script 2018-10-05 09:49:32 +08:00
lionsoul 8e278d9c25
Merge pull request #81 from KomaBeyond/master
重构python客户端,提升查询性能,兼容python2.x,3.x
2018-10-05 09:40:15 +08:00
Koma 593aba66d8 重构python客户端,提升查询性能,兼容python2.x,3.x 2018-10-04 11:39:49 +08:00
lionsoul a2d250919d Update the lua interface and add setDbFile interface 2018-10-04 10:22:01 +08:00
lionsoul 5f871ba257 Update the Lua binding 2018-10-03 20:12:30 +08:00
lionsoul e6104dcd1f Update README for lua binding 2018-10-03 20:10:02 +08:00
lionsoul 86ad8057aa Lua binding completed(tested) 2018-10-03 20:05:14 +08:00
lionsoul 5f073d814e lua binding forward with binarySearch and btreeSearch interface implemented 2018-10-03 13:12:05 +08:00
lionsoul 04f093d9bd lua binding basic framework with memory search completed(tested) 2018-10-02 22:36:57 +08:00
lionsoul 1f04c554db Update the maven version number to 1.7.2 2018-10-02 11:24:58 +08:00
lionsoul 679f30acdb Add DbSearcher(dbConfig, dbBinStr) construct method specified by Wendal at https://gitee.com/lionsoul/ip2region/issues/IILFL 2018-10-02 11:14:31 +08:00
lionsoul cd84dff3ac Add some notices 2018-10-02 10:39:54 +08:00
lionsoul d82447c1b7
Merge pull request #76 from a631807682/master
[nodejs] publish package on npm
2018-10-02 10:19:43 +08:00
lionsoul 7322733691 Merge branch 'master' of https://github.com/lionsoul2014/ip2region 2018-09-05 20:45:30 +08:00
lionsoul 1312bdba32 Update the maven version to 1.7 2018-09-05 20:44:37 +08:00
lionsoul 9623e345d9
Merge pull request #72 from stulzq/master
Add use stream method for C#
2018-09-01 20:59:52 +08:00
陈睿 4556afd815 [nodejs] publish package on npm 2018-08-30 18:20:49 +08:00
lionsoul f482591eee
Merge pull request #69 from Maledong/master
[Feature] Add 'memorySearch' and 'memorySearchSync' for Nodejs
2018-08-25 10:59:30 +08:00
stulzq f3ae83afb5 add use stream method 2018-08-19 16:47:01 +08:00
MaleDong 487129d59c [Feature] Add 'memorySearch' and 'memorySearchSync' for Nodejs
1) Add features.
2) Add unit tests.
3) Add benchmark.
2018-07-31 16:12:56 +08:00
lionsoul b7e9755c6b java binding version 1.7 with btree search bug fixed 2018-07-31 16:03:15 +08:00
lionsoul f82d3be466 Fix the bug for btree header fetch 2018-07-31 15:58:11 +08:00
lionsoul f2f80fc5b3 Update to the latest data of version 2018/07/26 2018-07-31 15:49:28 +08:00
lionsoul 4feeb03f85
Merge pull request #68 from RocherKong/master
Use the synchronization method as the first choice
2018-07-31 12:43:11 +08:00
Rocher e1a7052c9d
Update README.md
Use the synchronization method as the first choice.
2018-07-31 11:37:25 +08:00
Rocher 48edca6313
Update README.md
Use the synchronization method as the first choice.
2018-07-31 11:36:14 +08:00
Rocher c4583dc6fd
Merge pull request #3 from lionsoul2014/master
pull newest
2018-07-31 11:33:02 +08:00
lionsoul 6cd49f341d
Merge pull request #67 from Maledong/master
[Fix] Fix csharp formations
2018-07-30 15:26:37 +08:00
MaleDong b177834c7b [Fix] Fix csharp formations
1) In `README.md`, we shouldn't use `c#` but `csharp` to make md
recoginize it's the C# formation of codes.

2) Add 'ConfigureAwait(false)' as for the MSDN magazine usage.

3) Don't need Contribution History in README.md, because we can see the
change logs in GitHub instead.
2018-07-30 14:44:47 +08:00
lionsoul 1cde91a274
Merge pull request #66 from Maledong/NeedDispose
[Fix] Add "Disposible" interface for final release of resouces
2018-07-30 11:31:18 +08:00
MaleDong ee5c05d321 [Fix] Add "Disposible" interface for final release of resouces
`_search` should release resouces because the file state is still open.
2018-07-30 11:21:31 +08:00
lionsoul 3cf1c78ff1
Merge pull request #65 from Maledong/ChangeToTask
[Update] Refactor codes according to RocherHong's changes
2018-07-30 10:56:10 +08:00
MaleDong c6721af1ec [Update] Refactor codes according to RocherHong's changes
1) Considering using xUnit for UnitTest, remove my unit tests.
2) Remove useless `async`, because `async` is just the wrapper of `Task`.
3) Modify unit tests to be sure that the values MUST BE equal to each other when the IP is the same.
2018-07-30 10:48:30 +08:00
lionsoul d40e001802
Merge pull request #61 from Maledong/master
New Feature of async for Nodejs, Refactor of codes in node.js
2018-07-30 10:41:23 +08:00
lionsoul d51db48721 Fix the conflicts 2018-07-30 10:39:15 +08:00
lionsoul a1e433d9c0
Merge pull request #63 from RocherKong/master
Code Refactoring
2018-07-30 10:26:21 +08:00
MaleDong 8a6ac1cdc6 Change from `async Task` to `Task`
Because the async is the wrapper of Task, we don't need to use async
again.
2018-07-30 09:56:25 +08:00
RocherKong 55db35b6f4 1.CodeStandardized 2.Support Netfx4.5 3.TestStandardized 2018-07-30 00:31:49 +08:00
Rocher f0c3b88653
Merge pull request #2 from lionsoul2014/master
merge newest
2018-07-30 00:27:46 +08:00
MaleDong 1331f75997 Refactor of codes in node.js
1) Add async method: `binarySearch` and `btreeSearch`.
2) Refactor of tests by putting them into `tests` folder.
3) Create several new tests in details.
4) Remove useless and reformat codes to be clear.
5) Remove useless snapshots, because they can be recreated when you run `npm run test`.
2018-07-29 20:02:14 +08:00
lionsoul 57d9cd788f
Merge pull request #62 from Maledong/RefactorCSharp
Refactor of CSharp
2018-07-28 15:32:50 +08:00
MaleDong 33570612bc Refactor of CSharp:
1) Add `Async` methods.
2) Remove uesless variables, codes.
3) Remove ConsoleApp and use UnitTest instead.
4) Add README.md instead.
2018-07-20 13:57:32 +08:00
Rocher 7a9e9a6504
Merge pull request #1 from lionsoul2014/master
update to newest
2018-07-17 15:32:37 +08:00
lionsoul 71ec923046
Merge pull request #57 from biluohc/master
Rust: 0.2.0, use std::net::Ipaddr and update Error.
2018-07-03 20:04:22 +08:00
biluohc ea9a3ff7b8 Rust: 0.2.0, use std::net::Ipaddr and update Error. 2018-07-03 15:53:16 +08:00
dongyado 94ed13cdd0 update php_extension client intro file 2018-07-03 15:27:33 +08:00
dongyado b63a566641 add new author and email for nodejs binding client 2018-07-03 15:17:47 +08:00
Monster Slayer 1d3c083dc8
Merge pull request #50 from leeching/refactor/nodejs
refactor(nodejs): write nodejs script in javascript way
2018-07-03 15:14:41 +08:00
lionsoul 4afaf4513c
Merge pull request #55 from biluohc/master
add rust implement.
2018-07-03 10:55:40 +08:00
leeching 1ffe2b985e feat: add destroy method for backward compatibility 2018-07-03 10:07:25 +08:00
leeching 834e5939d8 feat(nodejs): add create static method 2018-07-03 09:48:41 +08:00
leeching b89ed1c397 refactor(nodejs): write nodejs script in javascript way 2018-07-03 09:48:41 +08:00
biluohc 1d09fee3c5 add a example 2018-07-02 23:04:46 +08:00
biluohc 02db5c5c56 use workspace 2018-07-02 22:17:10 +08:00
biluohc 4120c651eb add rust implement. 2018-06-30 16:16:00 +08:00
lionsoul 751c095ca0
Merge pull request #54 from Insua/master
fix issuse golang执行测试报错 #29
2018-06-30 10:40:37 +08:00
insua d8a7e34a84 fix(move golang package in directory and typo in readme): 2018-06-29 16:49:38 +08:00
lionsoul b6da4602e5
Merge pull request #52 from felixonmars/patch-1
Fix some typos in Ip2Region.class.php
2018-06-28 10:51:19 +08:00
Felix Yan 34be04bc4a
Fix some typos in Ip2Region.class.php 2018-06-28 02:00:05 +08:00
dongyado 550425a2b5 keep php class and php_extention return data key same, we change php_extension return data key : cityId to city_id 2018-06-27 10:44:29 +08:00
lionsoul 35cc0f37c7
Merge pull request #48 from leeching/fix/nodejs-var
remove unnecessary var
2018-06-26 17:34:03 +08:00
lionsoul 250b99de29
Merge pull request #49 from leeching/fix/nodejs-snapshot
test(nodejs): update snapshot
2018-06-26 17:33:08 +08:00
leeching b961c1f0ac test(nodejs): update snapshot 2018-06-26 17:10:56 +08:00
leeching 3d6db4931a remove unnecessary var 2018-06-26 16:26:00 +08:00
lionsoul 1634e7ae5e
Merge pull request #47 from jwcn/master
重命名 php 测试脚本
2018-06-26 12:37:00 +08:00
Wei.Jiang 470b29db3c
Merge pull request #1 from jwcn/jwcn-patch-1
重命名 php 测试脚本
2018-06-26 11:06:46 +08:00
Wei.Jiang 98616d0df8
重命名 php 测试脚本
重命名 php 测试脚本,与 README 和其他语言脚本名保持一致
2018-06-26 11:05:48 +08:00
lionsoul 1b275b49da
Merge pull request #46 from WoShiSunWuKong/patch-1
add nodejs sample
2018-06-26 10:09:09 +08:00
我是孙悟空 fb732f0535
add nodejs sampale 2018-06-25 12:59:52 +08:00
lionsoul 02506277ad fix the region and re-generate the ip2region.db file 2018-06-19 17:38:33 +08:00
dongyado 7e6cbdc652 update php_extension install guides 2018-05-17 12:11:13 +08:00
lionsoul 0cab5514eb
Merge pull request #44 from Chunlin-Li/fix-binding-nodejs
fix: ReferenceError: m is not defined
2018-05-17 09:20:26 +08:00
Chunlin Li fdac1c0101
fix: ReferenceError: m is not defined 2018-05-16 19:01:17 +08:00
lionsoul 42764c495b Ignore the testSearch generated by Makefile 2018-04-09 09:31:48 +08:00
lionsoul 09c2ba55fc
Merge pull request #41 from helloKaiDing/master
Makefile for binding/c with ip2region.db auto generate.
2018-04-09 09:28:45 +08:00
helloKaiDing d4dab91796
Makefile for binding/c
This is an Makefile for the binding/c program, can use to generate the db file ip2config.db  and the test program testSearcher, can be very convient for the new starter.
2018-04-08 17:23:03 +08:00
lionsoul d22d96cae2
Merge pull request #39 from panjunjie/patch-1
Update README.md
2018-03-30 00:39:55 +08:00
Jack a1ae4c8299
Update README.md
更改 markdow h 标签错误格式!
2018-03-29 19:04:48 +08:00
lionsoul 74b3045339 Update the ip data to version 2018/03/26 2018-03-27 14:38:33 +08:00
lionsoul 7cb8bc4b9d
Merge pull request #31 from RocherKong/master
add C# support ( from RocherKong)
2018-02-11 11:56:51 +08:00
RocherKong b65d8206ba add support for C# 2018-02-09 22:04:01 +08:00
lionsoul 39555fc266 Update the data to version 2018/01/12 2018-01-16 15:08:30 +08:00
lionsoul 68540b4db4 Fix the doc error 2017-11-23 21:25:32 +08:00
lionsoul 217a7dab9e Untrack the binding/java/target/ directory 2017-10-28 15:38:09 +08:00
lionsoul 07ea5994b8 Update the REAMD 2017-10-23 15:31:46 +08:00
lionsoul 02391e2057 Add the maven repository address 2017-10-23 13:50:57 +08:00
lionsoul 5f29715ec2 java binding maven upload & track the latest dbMaker.jar 2017-10-23 13:48:49 +08:00
lionsoul 4d0ce55008 bold the upgrade notice 2017-10-21 13:10:16 +08:00
lionsoul 29bacb0c8a Update the README 2017-10-21 13:08:26 +08:00
lionsoul a49fae2ce8 Update the README 2017-10-21 13:07:29 +08:00
lionsoul 1f0b07d42b Update the notice 2017-10-21 13:06:08 +08:00
lionsoul b0aecd7046 active the upgrade notice 2017-10-21 13:05:14 +08:00
lionsoul b48dbe5248 Bug fixed: Adjust the super header length to 8192 for all client implementation 2017-10-21 12:47:24 +08:00
Wilson Wang 4ffb1334a1 !4 python 3正常运行
Merge pull request !4 from Wilson Wang/master
2017-10-21 12:13:29 +08:00
wfifi 0f39924b08 add python3 support 2017-10-20 18:09:07 +08:00
lionsoul 7f1216eb6f Merge branch 'master' of http://git.oschina.net/lionsoul/ip2region 2017-10-20 12:13:30 +08:00
lionsoul 6eec7a2c42 Update the to data 2017/10/09 version 2017-10-20 12:13:19 +08:00
lionsoul 499bc1887e Merge pull request #19 from chengpiaopiao/master
fix incorrect python test helper.
2017-09-16 09:39:27 +08:00
chengpiaopiao 1ad6274118 fix incorrect python test helper. 2017-09-15 15:40:04 +08:00
dongyado 769ab315b1 correct post path 2017-09-08 14:10:31 +08:00
lionsoul 979e9bdda7 Merge pull request #15 from mohong122/master
Golang binding 增加控制台实例
2017-08-01 12:18:56 +08:00
mohong e4f54054c4 Merge pull request #1 from lionsoul2014/master
合并远程
2017-05-25 14:51:46 +08:00
lionsoul af9c8794ff Merge branch 'master' of https://github.com/lionsoul2014/ip2region 2017-04-19 10:49:13 +08:00
lionsoul f93b8d2dfb Fix the demo doc error 2017-04-19 10:48:10 +08:00
lionsoul 546be60b0a Merge pull request #11 from zhuangya/jest
chore: test in jest
2017-04-14 13:11:52 +08:00
Ya Zhuang eae269f37d
chore: test in jest 2017-04-14 12:47:55 +08:00
lionsoul 90682e06eb track the latest binary db data 2017-04-10 11:04:02 +08:00
lionsoul 7d4a36d9bc Update the 1.2.4 version desc 2017-04-07 19:04:12 +08:00
lionsoul fe4057d8aa Add the changes for version 1.2.4 2017-04-07 18:53:55 +08:00
lionsoul bd3ffadabf Update the source and the binary ip data to version 2017/03/15 2017-04-07 18:53:29 +08:00
lionsoul b7376d5868 Add the author info for binding/c_mmap module 2017-01-12 18:21:36 +08:00
lionsoul 387f5a2097 Merge branch 'master' of http://git.oschina.net/lionsoul/ip2region 2017-01-12 18:14:16 +08:00
lionsoul 9de920ef2b stdlize the code style and add the author info 2017-01-12 18:14:09 +08:00
我的上铺叫路遥 639c852534 Merge pull request !3 from 我的上铺叫路遥/master 2017-01-12 18:13:16 +08:00
Leo Ma 3c8cfa1282 Fix memory leak
Signed-off-by: Leo Ma <begeekmyfriend@gmail.com>
2017-01-12 18:04:07 +08:00
我的上铺叫路遥 a67936d835 Merge pull request !2 from 我的上铺叫路遥/master 2017-01-12 17:51:43 +08:00
Leo Ma 9bdf030fe0 Add database file memory map as c_mmap binding
File I/O optimization.

- No more `fread` call when the file is mapped.

- Close database file as soon as possible after `mmap`

Signed-off-by: Leo Ma <begeekmyfriend@gmail.com>
2017-01-12 17:33:58 +08:00
lionsoul 7a2b9943b5 stdlize the license add free statement 2016-12-28 09:53:13 +08:00
lionsoul c0955fedff update the merge&binary db file to version/2016/12/14 2016-12-21 16:53:14 +08:00
lionsoul 21bb060839 fix 10+ rows failed ip range records 2016-11-07 14:07:03 +08:00
lionsoul b0f7bff2de text and binary data version 2016/11/02 2016-11-05 21:13:36 +08:00
lionsoul 643e6a2463 update the db make doc and jar file 2016-11-03 17:32:16 +08:00
lionsoul ec395be524 add db make doc & update the CHANGES 2016-11-03 16:40:28 +08:00
lionsoul 7dcec80080 track the db make jar tools 2016-11-03 15:53:20 +08:00
lionsoul bedaca5b70 Merge pull request #4 from mohong122/master
golang实现
2016-10-20 17:08:17 +08:00
208 changed files with 1365749 additions and 98357 deletions

59
.gitignore vendored
View File

@ -3,12 +3,71 @@
*.o *.o
*.pyc *.pyc
*~ *~
*.log
*.la
*.so
*.xdb
*.iml
META-INF/ META-INF/
# Binary Files # # Binary Files #
*.jar *.jar
!dbMaker-*.jar
# ignore all xdb except the one in ./data/
*.xdb
!/data/*.xdb
# Package Files # # Package Files #
.settings/ .settings/
.classpath .classpath
.project .project
# vim swp file # # vim swp file #
*.swp *.swp
.idea
# binding
/v1.0//binding/java/classes/
/v1.0//binding/java/doc/
/v1.0//binding/java/target/
/v1.0//binding/java/*.jar
/binding/java/classes/
/binding/java/doc/
/binding/java/target/
/binding/java/*.jar
/binding/c/testSearcher
# golang
/binding/golang/searcher
/binding/golang/xdb_searcher
/binding/golang/golang
# rust
Cargo.lock
target
# VS ignore cases
/**/*.sln
/v1.0/binding/c#/**/.vs/
/v1.0/binding/c#/**/packages
/v1.0/binding/c#/**/bin
/v1.0/binding/c#/**/obj
/binding/c#/**/.vs/
/binding/c#/**/packages
/binding/c#/**/bin
/binding/c#/**/obj
# Nodejs
/v1.0/binding/nodejs/tests/unitTests/__snapshots__
/v1.0/binding/nodejs/coverage
/binding/nodejs/tests/unitTests/__snapshots__
/binding/nodejs/coverage
# maker
## golang
/maker/golang/dbmaker
/maker/golang/xdb_maker
/maker/golang/golang

View File

@ -1,7 +0,0 @@
### 1.2
1. 新增分列式升级算法大大的增加了数据的准确率基本避免之前各大网友反馈的些许ip定位错误
2. 数据升级是2016/06/30版
3. 优化数据文件生成算法ip2region.db文件由原来的3.5M降为1.5M42亿个IP地址皆大欢喜啊
4. C/PHP/JAVA客户端增加纯内存搜索模式python, php扩展nodejs会在后续版本加上
说明因为数据文件只有1.5M对于PHP,java这类IO优化类语言提升速度不大C有一个数量级的提升

View File

@ -201,9 +201,9 @@
limitations under the License. limitations under the License.
========================================================================== ==========================================================================
The following license applies to the JQuery JavaScript library The following license applies to the ip2region library
-------------------------------------------------------------------------- --------------------------------------------------------------------------
Copyright (c) 2010 John Resig, http://jquery.com/ Copyright (c) 2015 Lionsoul<chenxin619315@gmail.com>
Permission is hereby granted, free of charge, to any person obtaining Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the a copy of this software and associated documentation files (the
@ -223,4 +223,3 @@ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@ -1,66 +0,0 @@
ip2region - 最自由的ip地址查询库ip到地区的映射库提供Binary,B树和纯内存三种查询算法妈妈再也不用担心我的ip地址定位。
**1. 99.9%准确率,定时更新:**
数据聚合了一些知名ip到地名查询提供商的数据这些是他们官方的的准确率经测试着实比纯真啥的准确多了。<br />
每次聚合一下数据需要1-2天会不定时更新。
**2. 标准化的数据格式:**
每条ip数据段都固定了格式_城市Id|国家|区域|省份|城市|ISP_
只有中国的数据精确到了城市其他国家只能定位到国家后前的选项全部是0已经包含了全部你能查到的大大小小的国家。
请忽略前面的城市Id个人项目需求
**3. 体积小:**
生成的数据库文件ip2region.db只有1.5M1.2版本前是3.5M
**4. 多查询客户端的支持0.0x毫秒级别的查询**
已经集成的客户端有java, php, c, pythonnodejsphp扩展(支持linux, php5, php7版本已支持)。
提供了两种查询算法,响应时间如下:
客户端/binary算法/b-tree算法/Memory算法
java/0.x毫秒/0.x毫秒/0.1x毫秒 (使用RandomAccessFile)
php/0.x毫秒/0.1x毫秒/0.1x毫秒
c/0.0x毫秒/0.0x毫秒/0.00x毫秒(b-tree算法基本稳定在0.02x毫秒级别)
python/0.x毫秒/0.1x毫秒/未知
任何客户端b-tree都比binary算法快当然Memory算法固然是最快的
**5. 测试程序:**
java:
cd binding/java
ant all
java -jar ip2region-{version}.jar ./data/ip2region.db
php:
php binding/php/testSearcher ./data/ip2region.db
c:
cd binding/c/
gcc -g -O2 testSearcher.c ip2region.c
./a.out ../../data/ip2region.db
python:
python binding/python/testSearcher ./data/ip2region.db
均会看到如下界面:
initializing B-tree ...
+----------------------------------+
| ip2region test script |
| Author: chenxin619315@gmail.com |
| Type 'quit' to exit program |
+----------------------------------+
p2region>> 101.105.35.57
2163|中国|华南|广东省|深圳市|鹏博士 in 0.02295 millseconds
输入ip地址开始测试第一次会稍微有点慢在运行命令后面接入binary,memory来尝试其他算法建议使用b-tree算法速度和并发需求的可以使用memory算法。
具体集成请参考不同客户端的测试源码。

83
ReadMe.md Normal file
View File

@ -0,0 +1,83 @@
# Ip2region 是什么
ip2region v2.0 - 是一个离线IP地址定位库和IP定位数据管理框架10微秒级别的查询效率提供了众多主流编程语言的 `xdb` 数据生成和查询客户端实现。v1.0 旧版本: [v1.0版本入口](v1.0)
# Ip2region 特性
### 1、标准化的数据格式
每个 ip 数据段的 region 信息都固定了格式:`国家|区域|省份|城市|ISP`只有中国的数据巨绝大部分精确到了城市其他国家部分数据只能定位到国家后前的选项全部是0。
### 2、数据去重和压缩
`xdb` 格式生成程序会自动去重和压缩部分数据,默认的全部 IP 数据,生成的 ip2region.xdb 数据库是 11MiB随着数据的详细度增加数据库的大小也慢慢增大。
### 3、极速查询响应
即使是完全基于 `xdb` 文件的查询,单次查询响应时间在十微秒级别,可通过如下两种方式开启内存加速查询:
1. `vIndex` 索引缓存 :使用固定的 `512KiB` 的内存空间缓存 vector index 数据,减少一次 IO 磁盘操作保持平均查询效率稳定在10-20微秒之间。
2. `xdb` 整个文件缓存:将整个 `xdb` 文件全部加载到内存,内存占用等同于 `xdb` 文件大小,无磁盘 IO 操作,保持微秒级别的查询效率。
### 4、IP 数据管理框架
v2.0 格式的 `xdb` 支持亿级别的 IP 数据段行数region 信息也可以完全自定义,例如:你可以在 region 中追加特定业务需求的数据例如GPS信息/国际统一地域信息编码/邮编等。也就是你完全可以使用 ip2region 来管理你自己的 IP 定位数据。
# 查询客户端
使用文档和测试程序请参考各个查询客户端下的 ReadMe 介绍,全部查询 binding 实现情况如下:
| 编程语言 | 描述 | 开发状态 | 贡献者 |
| :--- | :--- |:-------| :--- |
| [golang](binding/golang) | golang 查询客户端实现 | 已完成 | [Lion](https://github.com/lionsoul2014) |
| [php](binding/php) | php 查询客户端实现 | 已完成 | [Lion](https://github.com/lionsoul2014) |
| [java](binding/java) | java 查询客户端实现 | 已完成 | [Lion](https://github.com/lionsoul2014) |
| [lua](binding/lua) | lua 查询客户端实现 | 开发中... | [Lion](https://github.com/lionsoul2014) |
| [c](binding/c) | ANSC c 查询客户端实现 | 开发中... | [Lion](https://github.com/lionsoul2014) |
| [lua_c](binding/lua_c) | lua 查询客户端实现 | 开发中... | [Lion](https://github.com/lionsoul2014) |
| [rust](binding/rust) | rust 查询客户端实现 | 开发中... | [Lion](https://github.com/lionsoul2014) |
| [python](binding/python) | python 查询客户端实现 | 待开始 | 待确定 |
| [nodejs](binding/nodejs) | nodejs 查询客户端实现 | 待开始 | 待确定 |
| [php5_ext](binding/php5_ext) | php5 c 扩展查询客户端实现 | 待开始 | 待确定 |
| [php7_ext](binding/php7_ext) | php7 c 扩展查询客户端实现 | 待开始 | 待确定 |
| [nginx](binding/nginx) | nginx的扩展查询客户端实现 | 待开始 | 待确定 |
| [csharp](binding/csharp) | c# 查询客户端实现 | 待开始 | 待确定 |
# xdb数据生成
使用文档和测试程序请参考不同生成程序下的 ReadMe 介绍,全部生成 maker 实现情况如下:
| 编程语言 | 描述 | 开发状态 | 贡献者 |
| :--- | :--- | :--- | :--- |
| [golang](maker/golang) | golang xdb 生成程序实现 | 已完成 | [Lion](https://github.com/lionsoul2014) |
| [java](maker/java) | java xdb 生成程序实现 | 开发中... | [Lion](https://github.com/lionsoul2014) |
| [c](maker/c) | ANSC c xdb 生成客户端实现 | 开发中... | [Lion](https://github.com/lionsoul2014) |
# 并发查询必读
全部查询客户端的 search 接口都 <b>不是</b> 并发安全的实现,不同进程/线程/携程需要通过创建不同的查询对象来安全使用,并发量很大的情况下,基于文件查询的方式可能会导致打开文件数过多的错误,请修改内核的最大允许打开文件数(fs.file-max=一个更高的值)或者将整个xdb加载到内存进行安全并发使用。
# 相关备注
### 1、使用声明
ip2region 重点在于<b>研究 IP 定位数据的存储设计和各种语言的查询实现</b>,并没有原始 IP 数据的支撑,本项目不保证及时的数据更新,没有也不会有商用版本,你可以使用自定义的数据导入 ip2region 进行管理。
### 2、技术交流
ip2region 微信交流群请先加微信lionsoul2014 (请备注 ip2region)
### 3、数据更新
基于检测算法的数据更新方式视频分享:[数据更新实现视频分享 - part1](https://www.bilibili.com/video/BV1934y1E7Q5/)[数据更新实现视频分享 - part2](https://www.bilibili.com/video/BV1pF411j7Aw/)
### 4、数据结构
1. v1.0数据库文件的结构和原理 blog[“ip2region 数据库文件的结构和原理”](https://github.com/dongyado/dongyado.github.io/blob/master/_posts/2016-08-18-structure-of-ip2region-database-file.md)
2. v1.0素据库文件的结构和原理 视频:[ip2region 数据结构设计和实现视频分享](https://www.bilibili.com/video/BV1wv4y1N7SD)

7
binding/c/ReadMe.md Normal file
View File

@ -0,0 +1,7 @@
# ip2region c 查询客户端实现
# 使用方式
# 查询测试
# bench 测试

7
binding/csharp/ReadMe.md Normal file
View File

@ -0,0 +1,7 @@
# ip2region csharp 查询客户端实现
# 使用方式
# 查询测试
# bench 测试

10
binding/golang/Makefile Normal file
View File

@ -0,0 +1,10 @@
# ip2region golang binding makefile
all: build
.PHONY: all
build:
go build -o xdb_searcher
test:
go test -v ./...
clean:
find ./ -name xdb_searcher | xargs rm -f

139
binding/golang/ReadMe.md Normal file
View File

@ -0,0 +1,139 @@
# ip2region xdb golang 查询客户端实现
# 使用方式
### package 获取
```bash
go get github.com/lionsoul2014/ip2region/binding/golang
```
### 完全基于文件的查询
```golang
import (
"fmt"
"github.com/lionsoul2014/ip2region/binding/golang/xdb"
"time"
)
func main() {
var dbPath = "ip2region.xdb file path"
searcher, err := xdb.NewWithFileOnly(dbPath)
if err != nil {
fmt.Printf("failed to create searcher: %s\n", err.Error())
return
}
defer searcher.Close()
// do the search
var ip = "1.2.3.4"
var tStart = time.Now()
region, err := searcher.SearchByStr(ip)
if err != nil {
fmt.Printf("failed to SearchIP(%s): %s\n", ip, err)
return
}
fmt.Printf("{region: %s, took: %s}\n", region, time.Since(tStart))
// 备注:并发使用,每个 goroutine 需要创建一个独立的 searcher 对象。
}
```
### 缓存 `VetorIndex` 索引
可以预先加载 vecotorIndex 缓存,然后做成全局变量,每次创建 searcher 的时候使用全局的 vectorIndex可以减少一次固定的 IO 操作从而加速查询,减少系统 io 压力。
```golang
// 1、从 dbPath 加载 VectorIndex 缓存,把下述 vIndex 变量全局到内存里面。
vIndex, err := LoadVectorIndexFromFile(dbPath)
if err != nil {
fmt.Printf("failed to load vector index from `%s`: %s\n", dbPath, err)
return
}
// 2、用全局的 vIndex 创建带 VectorIndex 缓存的查询对象。
searcher, err := xdb.NewWithVectorIndex(dbPath, vIndex)
if err != nil {
fmt.Printf("failed to create searcher with vector index: %s\n", err)
return
}
// 备注:并发使用,全部 goroutine 共享全局的只读 vIndex 缓存,每个 goroutine 创建一个独立的 searcher 对象
```
### 缓存整个 xdb 数据
可以预先加载整个 ip2region.xdb 到内存,完全基于内存查询,类似于之前的 memory search 查询。
```golang
// 1、从 dbPath 加载整个 xdb 到内存
cBuff, err := LoadContentFromFile(dbPath)
if err != nil {
fmt.Printf("failed to load content from `%s`: %s\n", dbPath, err)
return
}
// 2、用全局的 cBuff 创建完全基于内存的查询对象。
searcher, err := xdb.NewWithBuffer(cBuff)
if err != nil {
fmt.Printf("failed to create searcher with content: %s\n", err)
return
}
// 备注:并发使用,用整个 xdb 缓存创建的 searcher 对象可以安全用于并发。
```
# 编译测试程序
通过如下方式编译得到 xdb_searcher 可执行程序:
```bash
# 切换到 golang binding 根目录
make
```
# 查询测试
通过 `xdb_searcher search` 命令来测试 ip2region.xdb 的查询:
```
➜ golang git:(v2.0_xdb) ./xdb_searcher search
./xdb_searcher search [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如:使用默认的 data/ip2region.xdb 进行查询测试
```bash
➜ golang git:(v2.0_xdb) ✗ ./xdb_searcher search --db=../../data/ip2region.xdb
ip2region xdb searcher test program, type `quit` to exit
ip2region>> 1.2.3.4
{region:美国|0|华盛顿|0|谷歌, took:101.57µs}
```
输入 ip 地址进行查询即可,输入 quit 退出测试程序。可以设置 `cache-policy` 为 file/vectorIndex/content 来测试不同的查询缓存机制。
# bench 测试
通过 `xdb_searcher bench` 命令来进行自动 bench 测试一方面确保程序和xdb文件都没有错误一方面通过大量的查询得到平均查询性能
```bash
➜ golang git:(v2.0_xdb) ./xdb_searcher bench
./xdb_searcher bench [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如:通过 data/ip2region.xdb 和 data/ip.merge.txt 进行 bench 测试:
```bash
➜ golang git:(v2.0_xdb) ✗ ./xdb_searcher bench --db=../../data/ip2region.xdb --src=../../data/ip.merge.txt
Bench finished, {total: 3417955, took: 28.211578339s, cost: 8253 ns/op}
```
可以设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 不同缓存机制的实现。
*请注意 bench 使用的 src 文件需要是生成对应的 xdb 文件的相同的源文件*。bench 程序会逐行读取 `src` 指定的源IP文件然后每个 IP 段选取 5 个固定位置的 IP 进行测试,以确保查询的 region 信息和原始的 region 信息是相同。测试途中没有调试信息的输出,有错误会打印错误信息并且终止运行,所以看到 `Bench finished` 就表示 bench 成功了cost 是表示每次查询操作的平均时间(ns)。

5
binding/golang/go.mod Normal file
View File

@ -0,0 +1,5 @@
module github.com/lionsoul2014/ip2region/binding/golang
go 1.17
require github.com/mitchellh/go-homedir v1.1.0

2
binding/golang/go.sum Normal file
View File

@ -0,0 +1,2 @@
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=

View File

@ -1,70 +1,270 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package main package main
import ( import (
"os"
"github.com/mohong122/ip2region/binding/golang"
"bufio" "bufio"
"fmt" "fmt"
"github.com/lionsoul2014/ip2region/binding/golang/xdb"
"github.com/mitchellh/go-homedir"
"log"
"os"
"strings" "strings"
"errors"
"time" "time"
) )
func main() { func printHelp() {
fmt.Printf("ip2region xdb searcher\n")
fmt.Printf("%s [command] [command options]\n", os.Args[0])
fmt.Printf("Command: \n")
fmt.Printf(" search search input test\n")
fmt.Printf(" bench search bench test\n")
}
db := os.Args[1] func testSearch() {
var err error
_,err:= os.Stat(db) var dbFile, cachePolicy = "", "vectorIndex"
if os.IsNotExist(err){ for i := 2; i < len(os.Args); i++ {
panic("not found db " + db) r := os.Args[i]
} if len(r) < 5 {
region, err := ip2region.New(db)
defer region.Close()
fmt.Println(`initializing
+-------------------------------------------------------+
| ip2region test script |
| format 'ip type' |
| type option 'b-tree','binary','memory' default b-tree |
| Type 'quit' to exit program |
+-------------------------------------------------------+`)
reader := bufio.NewReader(os.Stdin)
for {
fmt.Print("ip2reginon >> ")
data, _, _ := reader.ReadLine()
begin:= time.Now()
commands := strings.Fields(string(data))
ip := ip2region.IpInfo{}
len := len(commands)
if len == 0{
continue continue
} }
if commands[0] == "quit"{ if strings.Index(r, "--") != 0 {
continue
}
var sIdx = strings.Index(r, "=")
if sIdx < 0 {
fmt.Printf("missing = for args pair '%s'\n", r)
return
}
switch r[2:sIdx] {
case "db":
dbFile = r[sIdx+1:]
case "cache-policy":
cachePolicy = r[sIdx+1:]
default:
fmt.Printf("undefined option `%s`\n", r)
return
}
}
if dbFile == "" {
fmt.Printf("%s search [command options]\n", os.Args[0])
fmt.Printf("options:\n")
fmt.Printf(" --db string ip2region binary xdb file path\n")
fmt.Printf(" --cache-policy string cache policy: file/vectorIndex/content\n")
return
}
dbPath, err := homedir.Expand(dbFile)
if err != nil {
fmt.Printf("invalid xdb file path `%s`: %s", dbFile, err)
return
}
// create the searcher with the cache policy setting
searcher, err := createSearcher(dbPath, cachePolicy)
if err != nil {
fmt.Printf("failed to create searcher: %s\n", err.Error())
return
}
defer func() {
searcher.Close()
fmt.Printf("searcher test program exited, thanks for trying\n")
}()
fmt.Printf(`ip2region xdb searcher test program, cachePolicy: %s
type 'quit' to exit
`, cachePolicy)
reader := bufio.NewReader(os.Stdin)
for {
fmt.Print("ip2region>> ")
str, err := reader.ReadString('\n')
if err != nil {
log.Fatalf("failed to read string: %s", err)
}
line := strings.TrimSpace(strings.TrimSuffix(str, "\n"))
if len(line) == 0 {
continue
}
if line == "quit" {
break break
} }
if !(len > 1) { tStart := time.Now()
commands = append(commands, "b-tree") region, err := searcher.SearchByStr(line)
}
switch commands[1] {
case "b-tree":
ip, err = region.BtreeSearch(commands[0])
case "binary":
ip, err = region.BinarySearch(commands[0])
case "memory":
ip, err = region.MemorySearch(commands[0])
default:
err = errors.New("parameter error")
}
if err != nil { if err != nil {
fmt.Printf("\x1b[0;31m{err: %s, ioCount: %d}\x1b[0m\n", err.Error(), searcher.GetIOCount())
fmt.Println( fmt.Sprintf("\x1b[0;31m%s\x1b[0m",err.Error())) } else {
}else{ fmt.Printf("\x1b[0;32m{region: %s, ioCount: %d, took: %s}\x1b[0m\n", region, searcher.GetIOCount(), time.Since(tStart))
fmt.Println( fmt.Sprintf("\x1b[0;32m%s %s\x1b[0m",ip.String(),time.Since(begin).String()))
} }
} }
} }
func testBench() {
var err error
var dbFile, srcFile, cachePolicy = "", "", "vectorIndex"
for i := 2; i < len(os.Args); i++ {
r := os.Args[i]
if len(r) < 5 {
continue
}
if strings.Index(r, "--") != 0 {
continue
}
var sIdx = strings.Index(r, "=")
if sIdx < 0 {
fmt.Printf("missing = for args pair '%s'\n", r)
return
}
switch r[2:sIdx] {
case "db":
dbFile = r[sIdx+1:]
case "src":
srcFile = r[sIdx+1:]
case "cache-policy":
cachePolicy = r[sIdx+1:]
default:
fmt.Printf("undefined option `%s`\n", r)
return
}
}
if dbFile == "" || srcFile == "" {
fmt.Printf("%s bench [command options]\n", os.Args[0])
fmt.Printf("options:\n")
fmt.Printf(" --db string ip2region binary xdb file path\n")
fmt.Printf(" --src string source ip text file path\n")
fmt.Printf(" --cache-policy string cache policy: file/vectorIndex/content\n")
return
}
dbPath, err := homedir.Expand(dbFile)
if err != nil {
fmt.Printf("invalid xdb file path `%s`: %s", dbFile, err)
return
}
searcher, err := createSearcher(dbPath, cachePolicy)
if err != nil {
fmt.Printf("failed to create searcher: %s\n", err.Error())
return
}
defer func() {
searcher.Close()
}()
handle, err := os.OpenFile(srcFile, os.O_RDONLY, 0600)
if err != nil {
fmt.Printf("failed to open source text file: %s\n", err)
return
}
var count, tStart, costs = int64(0), time.Now(), int64(0)
var scanner = bufio.NewScanner(handle)
scanner.Split(bufio.ScanLines)
for scanner.Scan() {
var l = strings.TrimSpace(strings.TrimSuffix(scanner.Text(), "\n"))
var ps = strings.SplitN(l, "|", 3)
if len(ps) != 3 {
fmt.Printf("invalid ip segment line `%s`\n", l)
return
}
sip, err := xdb.CheckIP(ps[0])
if err != nil {
fmt.Printf("check start ip `%s`: %s\n", ps[0], err)
return
}
eip, err := xdb.CheckIP(ps[1])
if err != nil {
fmt.Printf("check end ip `%s`: %s\n", ps[1], err)
return
}
if sip > eip {
fmt.Printf("start ip(%s) should not be greater than end ip(%s)\n", ps[0], ps[1])
return
}
mip := xdb.MidIP(sip, eip)
for _, ip := range []uint32{sip, xdb.MidIP(sip, mip), mip, xdb.MidIP(mip, eip), eip} {
sTime := time.Now()
region, err := searcher.Search(ip)
if err != nil {
fmt.Printf("failed to search ip '%s': %s\n", xdb.Long2IP(ip), err)
return
}
costs += time.Since(sTime).Nanoseconds()
// check the region info
if region != ps[2] {
fmt.Printf("failed Search(%s) with (%s != %s)\n", xdb.Long2IP(ip), region, ps[2])
return
}
count++
}
}
cost := time.Since(tStart)
fmt.Printf("Bench finished, {cachePolicy: %s, total: %d, took: %s, cost: %d μs/op}\n",
cachePolicy, count, cost, costs/count/1000)
}
func createSearcher(dbPath string, cachePolicy string) (*xdb.Searcher, error) {
switch cachePolicy {
case "nil", "file":
return xdb.NewWithFileOnly(dbPath)
case "vectorIndex":
vIndex, err := xdb.LoadVectorIndexFromFile(dbPath)
if err != nil {
return nil, fmt.Errorf("failed to load vector index from `%s`: %w", dbPath, err)
}
return xdb.NewWithVectorIndex(dbPath, vIndex)
case "content":
cBuff, err := xdb.LoadContentFromFile(dbPath)
if err != nil {
return nil, fmt.Errorf("failed to load content from '%s': %w", dbPath, err)
}
return xdb.NewWithBuffer(cBuff)
default:
return nil, fmt.Errorf("invalid cache policy `%s`, options: file/vectorIndex/content", cachePolicy)
}
}
func main() {
if len(os.Args) < 2 {
printHelp()
return
}
// set the log flag
log.SetFlags(log.Ldate | log.Ltime | log.Lshortfile)
switch strings.ToLower(os.Args[1]) {
case "search":
testSearch()
case "bench":
testBench()
default:
printHelp()
}
}

View File

@ -0,0 +1,36 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package xdb
import (
"encoding/binary"
"fmt"
)
type Header struct {
// data []byte
Version uint16
IndexPolicy IndexPolicy
CreatedAt uint32
StartIndexPtr uint32
EndIndexPtr uint32
}
func NewHeader(input []byte) (*Header, error) {
if len(input) < 16 {
return nil, fmt.Errorf("invalid input buffer")
}
return &Header{
Version: binary.LittleEndian.Uint16(input),
IndexPolicy: IndexPolicy(binary.LittleEndian.Uint16(input[2:])),
CreatedAt: binary.LittleEndian.Uint32(input[4:]),
StartIndexPtr: binary.LittleEndian.Uint32(input[8:]),
EndIndexPtr: binary.LittleEndian.Uint32(input[12:]),
}, nil
}

View File

@ -0,0 +1,222 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// ip2region database v2.0 searcher.
// @Note this is a Not thread safe implementation.
//
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package xdb
import (
"encoding/binary"
"fmt"
"os"
)
const (
HeaderInfoLength = 256
VectorIndexRows = 256
VectorIndexCols = 256
VectorIndexSize = 8
SegmentIndexBlockSize = 14
)
type IndexPolicy int
const (
VectorIndexPolicy IndexPolicy = 1
BTreeIndexPolicy IndexPolicy = 2
)
func (i IndexPolicy) String() string {
switch i {
case VectorIndexPolicy:
return "VectorIndex"
case BTreeIndexPolicy:
return "BtreeIndex"
default:
return "unknown"
}
}
type Searcher struct {
handle *os.File
// header info
header *Header
ioCount int
// use it only when this feature enabled.
// Preload the vector index will reduce the number of IO operations
// thus speedup the search process
vectorIndex []byte
// content buffer.
// running with the whole xdb file cached
contentBuff []byte
}
func baseNew(dbFile string, vIndex []byte, cBuff []byte) (*Searcher, error) {
var err error
// content buff first
if cBuff != nil {
return &Searcher{
vectorIndex: nil,
contentBuff: cBuff,
}, nil
}
// open the xdb binary file
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, err
}
return &Searcher{
handle: handle,
vectorIndex: vIndex,
}, nil
}
func NewWithFileOnly(dbFile string) (*Searcher, error) {
return baseNew(dbFile, nil, nil)
}
func NewWithVectorIndex(dbFile string, vIndex []byte) (*Searcher, error) {
return baseNew(dbFile, vIndex, nil)
}
func NewWithBuffer(cBuff []byte) (*Searcher, error) {
return baseNew("", nil, cBuff)
}
func (s *Searcher) Close() {
if s.handle != nil {
err := s.handle.Close()
if err != nil {
return
}
}
}
// GetIOCount return the global io count for the last search
func (s *Searcher) GetIOCount() int {
return s.ioCount
}
// SearchByStr find the region for the specified ip string
func (s *Searcher) SearchByStr(str string) (string, error) {
ip, err := CheckIP(str)
if err != nil {
return "", err
}
return s.Search(ip)
}
// Search find the region for the specified long ip
func (s *Searcher) Search(ip uint32) (string, error) {
// reset the global ioCount
s.ioCount = 0
// locate the segment index block based on the vector index
var il0 = (ip >> 24) & 0xFF
var il1 = (ip >> 16) & 0xFF
var idx = il0*VectorIndexCols*VectorIndexSize + il1*VectorIndexSize
var sPtr, ePtr = uint32(0), uint32(0)
if s.vectorIndex != nil {
sPtr = binary.LittleEndian.Uint32(s.vectorIndex[idx:])
ePtr = binary.LittleEndian.Uint32(s.vectorIndex[idx+4:])
} else if s.contentBuff != nil {
sPtr = binary.LittleEndian.Uint32(s.contentBuff[HeaderInfoLength+idx:])
ePtr = binary.LittleEndian.Uint32(s.contentBuff[HeaderInfoLength+idx+4:])
} else {
// read the vector index block
var buff = make([]byte, 8)
err := s.read(int64(HeaderInfoLength+idx), buff)
if err != nil {
return "", fmt.Errorf("read vector index block at %d: %w", HeaderInfoLength+idx, err)
}
sPtr = binary.LittleEndian.Uint32(buff)
ePtr = binary.LittleEndian.Uint32(buff[4:])
}
// fmt.Printf("sPtr=%d, ePtr=%d", sPtr, ePtr)
// binary search the segment index to get the region
var dataLen, dataPtr = 0, uint32(0)
var buff = make([]byte, SegmentIndexBlockSize)
var l, h = 0, int((ePtr - sPtr) / SegmentIndexBlockSize)
for l <= h {
m := (l + h) >> 1
p := sPtr + uint32(m*SegmentIndexBlockSize)
err := s.read(int64(p), buff)
if err != nil {
return "", fmt.Errorf("read segment index at %d: %w", p, err)
}
// decode the data step by step to reduce the unnecessary operations
sip := binary.LittleEndian.Uint32(buff)
if ip < sip {
h = m - 1
} else {
eip := binary.LittleEndian.Uint32(buff[4:])
if ip > eip {
l = m + 1
} else {
dataLen = int(binary.LittleEndian.Uint16(buff[8:]))
dataPtr = binary.LittleEndian.Uint32(buff[10:])
break
}
}
}
//fmt.Printf("dataLen: %d, dataPtr: %d", dataLen, dataPtr)
if dataLen == 0 {
return "", nil
}
// load and return the region data
var regionBuff = make([]byte, dataLen)
err := s.read(int64(dataPtr), regionBuff)
if err != nil {
return "", fmt.Errorf("read region at %d: %w", dataPtr, err)
}
return string(regionBuff), nil
}
// do the data read operation based on the setting.
// content buffer first or will read from the file.
// this operation will invoke the Seek for file based read.
func (s *Searcher) read(offset int64, buff []byte) error {
if s.contentBuff != nil {
cLen := copy(buff, s.contentBuff[offset:])
if cLen != len(buff) {
return fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
} else {
_, err := s.handle.Seek(offset, 0)
if err != nil {
return fmt.Errorf("seek to %d: %w", offset, err)
}
s.ioCount++
rLen, err := s.handle.Read(buff)
if err != nil {
return fmt.Errorf("handle read: %w", err)
}
if rLen != len(buff) {
return fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
}
return nil
}

159
binding/golang/xdb/util.go Normal file
View File

@ -0,0 +1,159 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package xdb
import (
"encoding/binary"
"fmt"
"os"
"strconv"
"strings"
)
func CheckIP(ip string) (uint32, error) {
var ps = strings.Split(ip, ".")
if len(ps) != 4 {
return 0, fmt.Errorf("invalid ip address `%s`", ip)
}
var buff = make([]byte, 4)
for i, s := range ps {
d, err := strconv.Atoi(s)
if err != nil {
return 0, fmt.Errorf("the %dth part `%s` is not an integer", i, s)
}
if d < 0 || d > 255 {
return 0, fmt.Errorf("the %dth part `%s` should be an integer bettween 0 and 255", i, s)
}
buff[i] = byte(d)
}
// convert the ip to integer
return binary.BigEndian.Uint32(buff), nil
}
func Long2IP(ip uint32) string {
var buff = make([]string, 4)
buff[0] = fmt.Sprintf("%d", (ip>>24)&0xFF)
buff[1] = fmt.Sprintf("%d", (ip>>16)&0xFF)
buff[2] = fmt.Sprintf("%d", (ip>>8)&0xFF)
buff[3] = fmt.Sprintf("%d", (ip>>0)&0xFF)
return strings.Join(buff, ".")
}
func MidIP(sip uint32, eip uint32) uint32 {
return uint32((uint64(sip) + uint64(eip)) >> 1)
}
// LoadHeader load the header info from the specified handle
func LoadHeader(handle *os.File) (*Header, error) {
_, err := handle.Seek(0, 0)
if err != nil {
return nil, fmt.Errorf("seek to the header: %w", err)
}
var buff = make([]byte, HeaderInfoLength)
rLen, err := handle.Read(buff)
if err != nil {
return nil, err
}
if rLen != len(buff) {
return nil, fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
return NewHeader(buff)
}
// LoadHeaderFromFile load header info from the specified db file path
func LoadHeaderFromFile(dbFile string) (*Header, error) {
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("open xdb file `%s`: %w", dbFile, err)
}
return LoadHeader(handle)
}
// LoadHeaderFromBuff wrap the header info from the content buffer
func LoadHeaderFromBuff(cBuff []byte) (*Header, error) {
return NewHeader(cBuff[0:256])
}
// LoadVectorIndex util function to load the vector index from the specified file handle
func LoadVectorIndex(handle *os.File) ([]byte, error) {
// load all the vector index block
_, err := handle.Seek(HeaderInfoLength, 0)
if err != nil {
return nil, fmt.Errorf("seek to vector index: %w", err)
}
var buff = make([]byte, VectorIndexRows*VectorIndexCols*VectorIndexSize)
rLen, err := handle.Read(buff)
if err != nil {
return nil, err
}
if rLen != len(buff) {
return nil, fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
return buff, nil
}
// LoadVectorIndexFromFile load vector index from a specified file path
func LoadVectorIndexFromFile(dbFile string) ([]byte, error) {
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("open xdb file `%s`: %w", dbFile, err)
}
return LoadVectorIndex(handle)
}
// LoadContent load the whole xdb content from the specified file handle
func LoadContent(handle *os.File) ([]byte, error) {
// get file size
fi, err := handle.Stat()
if err != nil {
return nil, fmt.Errorf("stat: %w", err)
}
size := fi.Size()
// seek to the head of the file
_, err = handle.Seek(0, 0)
if err != nil {
return nil, fmt.Errorf("seek to get xdb file length: %w", err)
}
var buff = make([]byte, size)
rLen, err := handle.Read(buff)
if err != nil {
return nil, err
}
if rLen != len(buff) {
return nil, fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
return buff, nil
}
// LoadContentFromFile load the whole xdb content from the specified db file path
func LoadContentFromFile(dbFile string) ([]byte, error) {
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("open xdb file `%s`: %w", dbFile, err)
}
return LoadContent(handle)
}

View File

@ -0,0 +1,49 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package xdb
import (
"fmt"
"testing"
"time"
)
func TestLoadVectorIndex(t *testing.T) {
vIndex, err := LoadVectorIndexFromFile("../../../data/ip2region.xdb")
if err != nil {
fmt.Printf("failed to load vector index: %s\n", err)
return
}
fmt.Printf("vIndex length: %d\n", len(vIndex))
}
func TestLoadContent(t *testing.T) {
buff, err := LoadContentFromFile("../../../data/ip2region.xdb")
if err != nil {
fmt.Printf("failed to load xdb content: %s\n", err)
return
}
fmt.Printf("buff length: %d\n", len(buff))
}
func TestLoadHeader(t *testing.T) {
header, err := LoadHeaderFromFile("../../../data/ip2region.xdb")
if err != nil {
fmt.Printf("failed to load xdb header info: %s\n", err)
return
}
fmt.Printf("Version : %d\n", header.Version)
fmt.Printf("IndexPolicy : %s\n", header.IndexPolicy.String())
fmt.Printf("CreatedAt : %d(%s)\n", header.CreatedAt, time.Unix(int64(header.CreatedAt), 0).Format(time.RFC3339))
fmt.Printf("StartIndexPtr : %d\n", header.StartIndexPtr)
fmt.Printf("EndIndexPtr : %d\n", header.EndIndexPtr)
}

201
binding/java/ReadMe.md Normal file
View File

@ -0,0 +1,201 @@
# ip2region xdb java 查询客户端实现
# 使用方式
### maven 仓库:
```xml
<dependency>
<groupId>org.lionsoul</groupId>
<artifactId>ip2region</artifactId>
<version>2.6.1</version>
</dependency>
```
### 完全基于文件的查询
```java
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.*;
import java.util.concurrent.TimeUnit;
public class SearcherTest {
public static void main(String[] args) {
// 1、创建 searcher 对象
String dbPath = "ip2region.xdb file path";
Searcher searcher = null;
try {
searcher = Searcher.newWithFileOnly(dbPath);
} catch (IOException e) {
System.out.printf("failed to create searcher with `%s`: %s\n", dbPath, e);
return;
}
// 2、查询
try {
String ip = "1.2.3.4";
long sTime = System.nanoTime();
String region = searcher.searchByStr(ip);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("failed to search(%s): %s\n", ip, e);
return;
}
// 3、备注并发使用每个线程需要创建一个独立的 searcher 对象单独使用。
}
}
```
### 缓存 `VectorIndex` 索引
我们可以提前从 xdb 文件中加载出来 VectorIndex 数据,然后全局缓存,每次创建 Searcher 对象的时候使用全局的 VectorIndex 缓存可以减少一次固定的 IO 操作,从而加速查询,减少 IO 压力。
```java
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.*;
import java.util.concurrent.TimeUnit;
public class SearcherTest {
public static void main(String[] args) {
String dbPath = "ip2region.xdb file path";
// 1、从 dbPath 中预先加载 VectorIndex 缓存,并且把这个得到的数据作为全局变量,后续反复使用。
byte[] vIndex;
try {
vIndex = Searcher.loadVectorIndexFromFile(dbPath);
} catch (Exception e) {
System.out.printf("failed to load vector index from `%s`: %s\n", dbPath, e);
return;
}
// 2、使用全局的 vIndex 创建带 VectorIndex 缓存的查询对象。
Searcher searcher;
try {
searcher = Searcher.newWithVectorIndex(dbPath, vIndex);
} catch (Exception e) {
System.out.printf("failed to create vectorIndex cached searcher with `%s`: %s\n", dbPath, e);
return;
}
// 3、查询
try {
String ip = "1.2.3.4";
long sTime = System.nanoTime();
String region = searcher.searchByStr(ip);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("failed to search(%s): %s\n", ip, e);
return;
}
// 备注:每个线程需要单独创建一个独立的 Searcher 对象,但是都共享全局的制度 vIndex 缓存。
}
}
```
### 缓存整个 xdb 数据
我们也可以预先加载整个 ip2region.xdb 的数据到内存,然后基于这个数据创建查询对象来实现完全基于文件的查询,类似之前的 memory search。
```java
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.*;
import java.util.concurrent.TimeUnit;
public class SearcherTest {
public static void main(String[] args) {
String dbPath = "ip2region.xdb file path";
// 1、从 dbPath 加载整个 xdb 到内存。
byte[] cBuff;
try {
cBuff = Searcher.loadContentFromFile(dbPath);
} catch (Exception e) {
System.out.printf("failed to load content from `%s`: %s\n", dbPath, e);
return;
}
// 2、使用上述的 cBuff 创建一个完全基于内存的查询对象。
Searcher searcher;
try {
searcher = Searcher.newWithBuffer(cBuff);
} catch (Exception e) {
System.out.printf("failed to create content cached searcher: %s\n", e);
return;
}
// 3、查询
try {
String ip = "1.2.3.4";
long sTime = System.nanoTime();
String region = searcher.searchByStr(ip);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("failed to search(%s): %s\n", ip, e);
return;
}
// 备注:并发使用,用整个 xdb 数据缓存创建的查询对象可以安全的用于并发,也就是你可以把这个 searcher 对象做成全局对象去垮线程访问。
}
}
```
# 编译测试程序
通过 maven 来编译测试程序。
```bash
# cd 到 java binding 的根目录
cd binding/java/
mvn compile package
```
然后会在当前目录的 target 目录下得到一个 ip2region-{version}.jar 的打包文件。
# 查询测试
可以通过 `java -jar ip2region-{version}.jar search` 命令来测试查询:
```bash
➜ java git:(v2.0_xdb) ✗ java -jar target/ip2region-2.6.0.jar search
java -jar ip2region-{version}.jar search [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如:使用默认的 data/ip2region.xdb 文件进行查询测试:
```bash
➜ java git:(v2.0_xdb) ✗ java -jar target/ip2region-2.6.0.jar search --db=../../data/ip2region.xdb
ip2region xdb searcher test program, cachePolicy: vectorIndex
type 'quit' to exit
ip2region>> 1.2.3.4
{region: 美国|0|华盛顿|0|谷歌, ioCount: 7, took: 82 μs}
ip2region>>
```
输入 ip 即可进行查询测试,也可以设置 `cache-policy` 来分别测试 file/vectorIndex/content 三种不同缓存方式的查询效果。
# bench 测试
可以通过 `java -jar ip2region-{version}.jar bench` 命令来进行 bench 测试,一方面确保 `xdb` 文件没有错误,一方面可以评估查询性能:
```bash
➜ java git:(v2.0_xdb) ✗ java -jar target/ip2region-2.6.0.jar bench
java -jar ip2region-{version}.jar bench [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如:通过默认的 data/ip2region.xdb 和 data/ip.merge.txt 文件进行 bench 测试:
```bash
➜ java git:(v2.0_xdb) ✗ java -jar target/ip2region-2.6.0.jar bench --db=../../data/ip2region.xdb --src=../../data/ip.merge.txt
Bench finished, {cachePolicy: vectorIndex, total: 3417955, took: 8s, cost: 2 μs/op}
```
可以通过设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 三种不同缓存方式的性能。
@Note: 注意 bench 使用的 src 文件要是生成对应 xdb 文件相同的源文件。

190
binding/java/pom.xml Normal file
View File

@ -0,0 +1,190 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.lionsoul</groupId>
<artifactId>ip2region</artifactId>
<version>2.6.1</version>
<packaging>jar</packaging>
<name>ip2region</name>
<url>https://github.com/lionsoul2014/ip2region</url>
<description>Open source offline internet address db manager framework and locator</description>
<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>
<scm>
<url>git@github.com:lionsoul2014/ip2region.git</url>
<connection>scm:git:git@github.com:lionsoul2014/ip2region.git</connection>
<developerConnection>scm:git:git@github.com:lionsoul2014/ip2region.git</developerConnection>
</scm>
<developers>
<developer>
<id>lionsoul</id>
<name>chenxin</name>
<email>chenxin619315@gmail.com</email>
</developer>
</developers>
<issueManagement>
<url>https://github.com/lionsoul2014/ip2region/issues</url>
<system>Github issues</system>
</issueManagement>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.1.2</version>
<executions>
<execution>
<id>attach-sources</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9</version>
<executions>
<execution>
<id>attach-javadocs</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
<configuration>
<additionalparam>${javadoc.opts}</additionalparam>
</configuration>
</execution>
</executions>
<configuration>
<failOnError>false</failOnError>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.lionsoul.ip2region.SearchTest</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>java8-doclint-disabled</id>
<activation>
<jdk>[1.8,)</jdk>
</activation>
<properties>
<javadoc.opts>-Xdoclint:none</javadoc.opts>
</properties>
</profile>
<profile>
<id>release</id>
<build>
<plugins>
<!-- Source -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>jar-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- Javadoc -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
<configuration>
<additionalparam>${javadoc.opts}</additionalparam>
</configuration>
</execution>
</executions>
</plugin>
<!-- GPG -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-gpg-plugin</artifactId>
<version>1.5</version>
<executions>
<execution>
<phase>verify</phase>
<goals>
<goal>sign</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<distributionManagement>
<snapshotRepository>
<id>oss</id>
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
</snapshotRepository>
<repository>
<id>oss</id>
<url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
</repository>
</distributionManagement>
</profile>
</profiles>
</project>

View File

@ -0,0 +1,225 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/23
package org.lionsoul.ip2region;
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.*;
import java.util.concurrent.TimeUnit;
public class SearchTest {
public static void printHelp(String[] args) {
System.out.print("ip2region xdb searcher\n");
System.out.print("java -jar ip2region-{version}.jar [command] [command options]\n");
System.out.print("Command: \n");
System.out.print(" search search input test\n");
System.out.print(" bench search bench test\n");
}
public static Searcher createSearcher(String dbPath, String cachePolicy) throws IOException {
if ("file".equals(cachePolicy)) {
return Searcher.newWithFileOnly(dbPath);
} else if ("vectorIndex".equals(cachePolicy)) {
byte[] vIndex = Searcher.loadVectorIndexFromFile(dbPath);
return Searcher.newWithVectorIndex(dbPath, vIndex);
} else if ("content".equals(cachePolicy)) {
byte[] cBuff = Searcher.loadContentFromFile(dbPath);
return Searcher.newWithBuffer(cBuff);
} else {
throw new IOException("invalid cache policy `" + cachePolicy + "`, options: file/vectorIndex/content");
}
}
public static void searchTest(String[] args) throws IOException {
String dbPath = "", cachePolicy = "vectorIndex";
for (final String r : args) {
if (r.length() < 5) {
continue;
}
if (r.indexOf("--") != 0) {
continue;
}
int sIdx = r.indexOf('=');
if (sIdx < 0) {
System.out.printf("missing = for args pair `%s`\n", r);
return;
}
String key = r.substring(2, sIdx);
String val = r.substring(sIdx + 1);
// System.out.printf("key=%s, val=%s\n", key, val);
if ("db".equals(key)) {
dbPath = val;
} else if ("cache-policy".equals(key)) {
cachePolicy = val;
} else {
System.out.printf("undefined option `%s`\n", r);
return;
}
}
if (dbPath.length() < 1) {
System.out.print("java -jar ip2region-{version}.jar search [command options]\n");
System.out.print("options:\n");
System.out.print(" --db string ip2region binary xdb file path\n");
System.out.print(" --cache-policy string cache policy: file/vectorIndex/content\n");
return;
}
Searcher searcher = createSearcher(dbPath, cachePolicy);
final BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));
System.out.printf("ip2region xdb searcher test program, cachePolicy: %s\ntype 'quit' to exit\n", cachePolicy);
while ( true ) {
System.out.print("ip2region>> ");
String line = reader.readLine().trim();
if ( line.length() < 2 ) {
continue;
}
if ( line.equalsIgnoreCase("quit") ) {
break;
}
try {
double sTime = System.nanoTime();
String region = searcher.searchByStr(line);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("{err: %s, ioCount: %d}\n", e, searcher.getIOCount());
}
}
reader.close();
searcher.close();
System.out.println("searcher test program exited, thanks for trying");
}
public static void benchTest(String[] args) throws IOException {
String dbPath = "", srcPath = "", cachePolicy = "vectorIndex";
for (final String r : args) {
if (r.length() < 5) {
continue;
}
if (r.indexOf("--") != 0) {
continue;
}
int sIdx = r.indexOf('=');
if (sIdx < 0) {
System.out.printf("missing = for args pair `%s`\n", r);
return;
}
String key = r.substring(2, sIdx);
String val = r.substring(sIdx + 1);
if ("db".equals(key)) {
dbPath = val;
} else if ("src".equals(key)) {
srcPath = val;
} else if ("cache-policy".equals(key)) {
cachePolicy = val;
} else {
System.out.printf("undefined option `%s`\n", r);
return;
}
}
if (dbPath.length() < 1 || srcPath.length() < 1) {
System.out.print("java -jar ip2region-{version}.jar bench [command options]\n");
System.out.print("options:\n");
System.out.print(" --db string ip2region binary xdb file path\n");
System.out.print(" --src string source ip text file path\n");
System.out.print(" --cache-policy string cache policy: file/vectorIndex/content\n");
return;
}
Searcher searcher = createSearcher(dbPath, cachePolicy);
long count = 0, costs = 0, tStart = System.nanoTime();
String line;
final BufferedReader reader = new BufferedReader(new FileReader(srcPath));
while ((line = reader.readLine()) != null) {
String l = line.trim();
String[] ps = l.split("\\|", 3);
if (ps.length != 3) {
System.out.printf("invalid ip segment `%s`\n", l);
return;
}
long sip;
try {
sip = Searcher.checkIpAddr(ps[0]);
} catch (Exception e) {
System.out.printf("check start ip `%s`: %s\n", ps[0], e);
return;
}
long eip;
try {
eip = Searcher.checkIpAddr(ps[1]);
} catch (Exception e) {
System.out.printf("check end ip `%s`: %s\n", ps[1], e);
return;
}
if (sip > eip) {
System.out.printf("start ip(%s) should not be greater than end ip(%s)\n", ps[0], ps[1]);
return;
}
long mip = (sip + eip) >> 1;
for (final long ip : new long[]{sip, (sip + mip) >> 1, mip, (mip + eip) >> 1, eip}) {
long sTime = System.nanoTime();
String region = searcher.search(ip);
costs += System.nanoTime() - sTime;
// check the region info
if (!ps[2].equals(region)) {
System.out.printf("failed search(%s) with (%s != %s)\n", Searcher.long2ip(ip), region, ps[2]);
return;
}
count++;
}
}
reader.close();
searcher.close();
long took = System.nanoTime() - tStart;
System.out.printf("Bench finished, {cachePolicy: %s, total: %d, took: %ds, cost: %d μs/op}\n",
cachePolicy, count, TimeUnit.NANOSECONDS.toSeconds(took),
count == 0 ? 0 : TimeUnit.NANOSECONDS.toMicros(costs/count));
}
public static void main(String[] args) {
if (args.length < 1) {
printHelp(args);
return;
}
if ("search".equals(args[0])) {
try {
searchTest(args);
} catch (IOException e) {
System.out.printf("failed running search test: %s\n", e);
}
} else if ("bench".equals(args[0])) {
try {
benchTest(args);
} catch (IOException e) {
System.out.printf("failed running bench test: %s\n", e);
}
} else {
printHelp(args);
}
}
}

View File

@ -0,0 +1,44 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/23
package org.lionsoul.ip2region;
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.IOException;
public class UtilTest {
public static void testIP2Long() {
String ip = "1.2.3.4";
long ipAddr = 0;
try {
ipAddr = Searcher.checkIpAddr(ip);
} catch (Exception e) {
System.out.printf("failed to check ip: %s\n", e);
return;
}
if (ipAddr != 16909060) {
System.out.print("failed ip2long\n");
return;
}
String ip2 = Searcher.long2ip(ipAddr);
if (!ip.equals(ip2)) {
System.out.print("failed long2ip\n");
return;
}
System.out.printf("passed: ip=%s, ipAddr=%d, ip2=%s\n", ip, ipAddr, ip2);
}
public static void main(String[] args) {
System.out.print("testing IP2Long ... \n");
testIP2Long();
}
}

View File

@ -0,0 +1,37 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/23
package org.lionsoul.ip2region.xdb;
import java.awt.image.SampleModel;
public class Header {
public final int version;
public final int indexPolicy;
public final int createdAt;
public final int startIndexPtr;
public final int endIndexPtr;
public Header(byte[] buff) {
assert buff.length >= 16;
version = Searcher.getInt2(buff, 0);
indexPolicy = Searcher.getInt2(buff, 2);
createdAt = Searcher.getInt(buff, 4);
startIndexPtr = Searcher.getInt(buff, 8);
endIndexPtr = Searcher.getInt(buff, 12);
}
@Override public String toString() {
return "{" +
"Version: " + version + ',' +
"IndexPolicy" + indexPolicy + ',' +
"CreatedAt" + createdAt + ',' +
"StartIndexPtr" + startIndexPtr + ',' +
"EndIndexPtr" + endIndexPtr +
'}';
}
}

View File

@ -0,0 +1,267 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package org.lionsoul.ip2region.xdb;
// xdb searcher (Not thread safe implementation)
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/23
import java.io.IOException;
import java.io.RandomAccessFile;
import java.util.Arrays;
public class Searcher {
// constant defined copied from the xdb maker
public static final int HeaderInfoLength = 256;
public static final int VectorIndexRows = 256;
public static final int VectorIndexCols = 256;
public static final int VectorIndexSize = 8;
public static final int SegmentIndexSize = 14;
// random access file handle for file based search
private final RandomAccessFile handle;
private int ioCount = 0;
// vector index.
// use the byte[] instead of VectorIndex entry array to keep
// the minimal memory allocation.
private final byte[] vectorIndex;
// xdb content buffer, used for in-memory search
private final byte[] contentBuff;
// --- static method to create searchers
public static Searcher newWithFileOnly(String dbPath) throws IOException {
return new Searcher(dbPath, null, null);
}
public static Searcher newWithVectorIndex(String dbPath, byte[] vectorIndex) throws IOException {
return new Searcher(dbPath, vectorIndex, null);
}
public static Searcher newWithBuffer(byte[] cBuff) throws IOException {
return new Searcher(null, null, cBuff);
}
// --- End of creator
public Searcher(String dbFile, byte[] vectorIndex, byte[] cBuff) throws IOException {
if (cBuff != null) {
this.handle = null;
this.vectorIndex = null;
this.contentBuff = cBuff;
} else {
this.handle = new RandomAccessFile(dbFile, "r");
this.vectorIndex = vectorIndex;
this.contentBuff = null;
}
}
public void close() throws IOException {
if (this.handle != null) {
this.handle.close();
}
}
public int getIOCount() {
return ioCount;
}
public String searchByStr(String ip) throws Exception {
long ipAddr = checkIpAddr(ip);
return search(ipAddr);
}
public String search(long ip) throws IOException {
// reset the global counter
this.ioCount = 0;
// locate the segment index block based on the vector index
int sPtr = 0, ePtr = 0;
int il0 = (int) ((ip >> 24) & 0xFF);
int il1 = (int) ((ip >> 16) & 0xFF);
int idx = il0 * VectorIndexCols * VectorIndexSize + il1 * VectorIndexSize;
// System.out.printf("il0: %d, il1: %d, idx: %d\n", il0, il1, idx);
if (vectorIndex != null) {
sPtr = getInt(vectorIndex, idx);
ePtr = getInt(vectorIndex, idx + 4);
} else if (contentBuff != null) {
sPtr = getInt(contentBuff, HeaderInfoLength + idx);
ePtr = getInt(contentBuff, HeaderInfoLength + idx + 4);
} else {
final byte[] buff = new byte[8];
read(HeaderInfoLength + idx, buff);
sPtr = getInt(buff, 0);
ePtr = getInt(buff, 4);
}
// System.out.printf("sPtr: %d, ePtr: %d\n", sPtr, ePtr);
// binary search the segment index block to get the region info
final byte[] buff = new byte[SegmentIndexSize];
int dataLen = -1, dataPtr = -1;
int l = 0, h = (ePtr - sPtr) / SegmentIndexSize;
while (l <= h) {
int m = (l + h) >> 1;
int p = sPtr + m * SegmentIndexSize;
// read the segment index
read(p, buff);
long sip = getIntLong(buff, 0);
if (ip < sip) {
h = m - 1;
} else {
long eip = getIntLong(buff, 4);
if (ip > eip) {
l = m + 1;
} else {
dataLen = getInt2(buff, 8);
dataPtr = getInt(buff, 10);
break;
}
}
}
// empty match interception
// System.out.printf("dataLen: %d, dataPtr: %d\n", dataLen, dataPtr);
if (dataPtr < 0) {
return null;
}
// load and return the region data
final byte[] regionBuff = new byte[dataLen];
read(dataPtr, regionBuff);
return new String(regionBuff);
}
protected void read(int offset, byte[] buffer) throws IOException {
// check the in-memory buffer first
if (contentBuff != null) {
// @TODO: reduce data copying, directly decode the data ?
System.arraycopy(contentBuff, offset, buffer, 0, buffer.length);
return;
}
// read from the file handle
assert handle != null;
handle.seek(offset);
this.ioCount++;
int rLen = handle.read(buffer);
if (rLen != buffer.length) {
throw new IOException("incomplete read: read bytes should be " + buffer.length);
}
}
// --- static cache util function
public static Header loadHeader(RandomAccessFile handle) throws IOException {
handle.seek(0);
final byte[] buff = new byte[HeaderInfoLength];
handle.read(buff);
return new Header(buff);
}
public static Header loadHeaderFromFile(String dbPath) throws IOException {
RandomAccessFile handle = new RandomAccessFile(dbPath, "r");
return loadHeader(handle);
}
public static byte[] loadVectorIndex(RandomAccessFile handle) throws IOException {
handle.seek(HeaderInfoLength);
int len = VectorIndexRows * VectorIndexCols * SegmentIndexSize;
final byte[] buff = new byte[len];
int rLen = handle.read(buff);
if (rLen != len) {
throw new IOException("incomplete read: read bytes should be " + len);
}
return buff;
}
public static byte[] loadVectorIndexFromFile(String dbPath) throws IOException {
RandomAccessFile handle = new RandomAccessFile(dbPath, "r");
return loadVectorIndex(handle);
}
public static byte[] loadContent(RandomAccessFile handle) throws IOException {
handle.seek(0);
final byte[] buff = new byte[(int) handle.length()];
int rLen = handle.read(buff);
if (rLen != buff.length) {
throw new IOException("incomplete read: read bytes should be " + buff.length);
}
return buff;
}
public static byte[] loadContentFromFile(String dbPath) throws IOException {
RandomAccessFile handle = new RandomAccessFile(dbPath, "r");
return loadContent(handle);
}
// --- End cache load util function
// --- static util method
/* get an int from a byte array start from the specified offset */
public static long getIntLong(byte[] b, int offset) {
return (
((b[offset++] & 0x000000FFL)) |
((b[offset++] << 8) & 0x0000FF00L) |
((b[offset++] << 16) & 0x00FF0000L) |
((b[offset ] << 24) & 0xFF000000L)
);
}
public static int getInt(byte[] b, int offset) {
return (
((b[offset++] & 0x000000FF)) |
((b[offset++] << 8) & 0x0000FF00) |
((b[offset++] << 16) & 0x00FF0000) |
((b[offset ] << 24) & 0xFF000000)
);
}
public static int getInt2(byte[] b, int offset) {
return (
(b[offset++] & 0x000000FF) |
(b[offset ] & 0x0000FF00)
);
}
/* long int to ip string */
public static String long2ip( long ip )
{
return String.valueOf((ip >> 24) & 0xFF) + '.' +
((ip >> 16) & 0xFF) + '.' + ((ip >> 8) & 0xFF) + '.' + ((ip) & 0xFF);
}
public static final byte[] shiftIndex = {24, 16, 8, 0};
/* check the specified ip address */
public static long checkIpAddr(String ip) throws Exception {
String[] ps = ip.split("\\.");
if (ps.length != 4) {
throw new Exception("invalid ip address `" + ip + "`");
}
long ipAddr = 0;
for (int i = 0; i < ps.length; i++) {
int val = Integer.parseInt(ps[i]);
if (val > 255) {
throw new Exception("ip part `"+ps[i]+"` should be less then 256");
}
ipAddr |= ((long) val << shiftIndex[i]);
}
return ipAddr & 0xFFFFFFFFL;
}
}

7
binding/lua/ReadMe.md Normal file
View File

@ -0,0 +1,7 @@
# ip2region lua 查询客户端实现
# 使用方式
# 查询测试
# bench 测试

7
binding/lua_c/ReadMe.md Normal file
View File

@ -0,0 +1,7 @@
# ip2region lua c 扩展查询客户端实现
# 使用方式
# 查询测试
# bench 测试

View File

@ -1,3 +0,0 @@
# nodejs 客户端
现已实现同步查询,异步查询未实现,用法参考 test.js

7
binding/nodejs/ReadMe.md Normal file
View File

@ -0,0 +1,7 @@
# ip2region nodejs 查询客户端实现
# 使用方式
# 查询测试
# bench 测试

View File

@ -1,277 +0,0 @@
/**
* ip2region client for nodejs
*
* project: https://github.com/lionsoul2014/ip2region
*
* @author dongyado<dongyado@gmail.com>
* */
var fs = require('fs');
var ipbase = [16777216, 65536, 256, 1]; // for ip2long
var ip2region = {};
ip2region.db_file_path = null;
ip2region.db_fd = null;
var totalBlocks = 0;
var firstIndexPtr = 0;
var lastIndexPtr = 0;
var superBlock = new Buffer(8);
var indexBlockLength = 12;
var totalHeaderLength = 4096;
/**
* binary search synchronized
* */
ip2region.binarySearchSync = function(ip)
{
var low = 0;
var mid = 0;
var high = totalBlocks;
var dataPos = 0;
var pos = 0;
var sip = 0;
var eip = 0;
var indexBuffer = new Buffer(12);
if( typeof(ip) == 'string' ) ip = ip2long(ip);
// binary search
while( low <= high ) {
mid = ((low + high) >> 1);
pos = firstIndexPtr + mid * indexBlockLength;
fs.readSync(this.db_fd, indexBuffer, 0, indexBlockLength, pos);
sip = getLong(indexBuffer, 0);
//console.log( ' sip : ' + sip + ' eip : ' + eip );
if ( ip < sip) {
high = mid - 1;
} else {
eip = getLong(indexBuffer, 4);
if ( ip > eip ) {
low = mid + 1;
} else {
dataPos = getLong(indexBuffer, 8);
break;
}
}
}
// read data
if (dataPos == 0) return null;
var dataLen = ((dataPos >> 24) & 0xFF);
var dataPos = (dataPos & 0x00FFFFFF);
var dataBuffer = new Buffer(dataLen);
fs.readSync(this.db_fd, dataBuffer, 0, dataLen, dataPos);
var city_id = getLong(dataBuffer, 0);
var data = dataBuffer.toString('utf8', 4, dataLen);
//console.log(city_id);
//console.log(data);
return { city: city_id, region: data };
}
var headerSip = null;
var headerPtr = 0;
var headerLen = 0;
/**
* btree search synchronized
* */
ip2region.btreeSearchSync = function(ip)
{
var indexBlockBuffer = new Buffer(indexBlockLength);
var headerIndexBuffer = new Buffer(totalHeaderLength);
if( typeof(ip) == 'string' ) ip = ip2long(ip);
var i = 0;
// header index handler
if (headerSip == null) {
fs.readSync(this.db_fd, headerIndexBuffer, 0, totalHeaderLength, 8);
headerSip = new Array();
headerPtr = new Array();
var startIp = 0;
var dataPtr = 0;
for ( i = 0; i < totalHeaderLength; i += 8) {
startIp = getLong(headerIndexBuffer, i);
dataPtr = getLong(headerIndexBuffer, i + 4);
if ( dataPtr == 0) break;
headerSip.push(startIp);
headerPtr.push(dataPtr);
headerLen++; // header index size count
}
}
// first search (in header index)
var low = 0;
var mid = 0;
var high = headerLen;
var sptr = 0;
var eptr = 0;
while(low <= high) {
mid = ((low + high) >> 1);
if (ip == headerSip[mid]) {
if ( m > 0) {
sptr = headerPtr[mid - 1];
eptr = headerPtr[mid];
} else {
sptr = headerPtr[mid];
eptr = headerPtr[mid + 1];
}
break;
}
if ( ip < headerSip[mid]) {
if (mid == 0) {
sptr = headerPtr[mid];
eptr = headerPtr[mid + 1];
break;
} else if ( ip > headerSip[mid - 1]) {
sptr = headerPtr[mid - 1];
eptr = headerPtr[mid];
break;
}
high = mid - 1;
} else {
if ( mid == headerLen - 1) {
sptr = headerPtr[mid - 1];
eptr = headerPtr[mid];
break;
} else if ( ip <= headerSip[mid + 1]) {
sptr = headerPtr[mid];
eptr = headerPtr[mid + 1];
break;
}
low = mid + 1;
}
}
// match nothing
if (sptr == 0) return null;
// second search (in index)
var blockLen = eptr - sptr;
var blockBuffer = new Buffer(blockLen + indexBlockLength);
fs.readSync(this.db_fd, blockBuffer, 0, blockLen + indexBlockLength, sptr);
low = 0;
high = blockLen / indexBlockLength;
var p = 0;
var sip = 0;
var eip = 0;
var dataPtr = 0;
while(low <= high) {
mid = ((low + high) >> 1);
p = mid * indexBlockLength;
sip = getLong(blockBuffer, p);
if (ip < sip) {
high = mid - 1;
} else {
eip = getLong(blockBuffer, p + 4);
if (ip > eip) {
low = mid + 1;
} else {
dataPtr = getLong(blockBuffer, p + 8);
break;
}
}
}
// read data
if (dataPtr == 0) return null;
var dataLen = ((dataPtr >> 24) & 0xFF);
var dataPtr = (dataPtr & 0x00FFFFFF);
var dataBuffer = new Buffer(dataLen);
fs.readSync(this.db_fd, dataBuffer, 0, dataLen, dataPtr);
var city_id = getLong(dataBuffer, 0);
var data = dataBuffer.toString('utf8', 4, dataLen);
//console.log(city_id);
//console.log(data);
return { city: city_id, region: data };
}
/**
* convert ip to long (xxx.xxx.xxx.xxx to a integer)
* */
function ip2long(ip)
{
var val = 0;
ip.split('.').forEach(function(ele, i){
val += ipbase[i] * ele;
});
return val;
}
/**
* get long value from buffer with specified offset
* */
function getLong(buffer, offset)
{
var val = (
(buffer[offset] & 0x000000FF) |
((buffer[offset + 1] << 8) & 0x0000FF00) |
((buffer[offset + 2] << 16) & 0x00FF0000) |
((buffer[offset + 3] << 24) & 0xFF000000)
);
// convert to unsigned int
if (val < 0) {
val = val >>> 0;
}
return val;
}
exports.create = function(db_path)
{
if (typeof(db_path) == "undefined" || fs.exists(db_path) ) {
throw("[ip2region] db file not exists : " + db_path);
}
ip2region.db_file_path = db_path;
try {
ip2region.db_fd = fs.openSync(ip2region.db_file_path, 'r');
} catch(e) {
throw("[ip2region] Can not open ip2region.db file , path : "
+ ip2region.db_file_path);
}
// init basic search environment
if (totalBlocks == 0) {
fs.readSync(ip2region.db_fd, superBlock, 0, 8, 0);
firstIndexPtr = getLong(superBlock, 0);
lastIndexPtr = getLong(superBlock, 4);
totalBlocks = (lastIndexPtr - firstIndexPtr)
/ indexBlockLength + 1;
}
return ip2region;
}
exports.destroy = function(ip2rObj)
{
ip2rObj.db_file_path = null;
fs.closeSync(ip2rObj.db_fd);
}

View File

@ -1,24 +0,0 @@
{
"name": "ip2region",
"version": "0.0.1",
"description": "ip database ",
"main": "ip2region.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": {
"type": "git",
"url": "https://github.com/lionsoul2016/ip2region"
},
"keywords": [
"ip2region",
"ip",
"region"
],
"author": "dongyado",
"license": "ISC",
"bugs": {
"url": "https://github.com/lionsoul2016/ip2region/issues"
},
"homepage": "https://github.com/lionsoul2016/ip2region"
}

View File

@ -1,15 +0,0 @@
/**
* test program
* */
var Ip2region = require('./ip2region');
// 1. create a ip2region object
var ipObj = Ip2region.create('../../data/ip2region.db');
// 2. use
//console.log(ipObj.binarySearchSync("120.24.78.68"));
console.log(ipObj.btreeSearchSync("120.24.78.68"));
// 3. destroy
Ip2region.destroy(ipObj);

135
binding/php/ReadMe.md Normal file
View File

@ -0,0 +1,135 @@
# ip2region xdb php 查询客户端实现
# 使用方式
### 完全基于文件的查询
```php
$dbFile = "ip2region.xdb file path";
try {
$searcher = XdbSearcher::newWithFileOnly($dbFile);
} catch (Exception $e) {
printf("failed to create searcher with '%s': %s\n", $dbFile, $e);
return;
}
$ip = '1.2.3.4';
$sTime = XdbSearcher::now();
$region = $searcher->search($ip);
if ($region === null) {
// something is wrong
printf("failed search(%s)\n", $ip);
return;
}
printf("{region: %s, took: %.5f ms}\n", $region, XdbSearcher::now() - $sTime);
// 备注:并发使用,每个线程或者协程需要创建一个独立的 searcher 对象。
```
### 缓存 `VectorIndex` 索引
如果你的 php 母环境支持,可以预先加载 vectorIndex 缓存,然后做成全局变量,每次创建 Searcher 的时候使用全局的 vectorIndex可以减少一次固定的 IO 操作从而加速查询,减少 io 压力。
```php
// 1、从 dbPath 加载 VectorIndex 缓存,把下述的 vIndex 变量缓存到内存里面。
$vIndex = XdbSearcher::loadVectorFromFile($dbPath);
if ($vIndex === null) {
printf("failed to load vector index from '%s'\n", $dbPath);
return;
}
// 2、使用全局的 vIndex 创建带 VectorIndex 缓存的查询对象。
try {
$searcher = XdbSearcher::newWithVectorIndex($dbFile, $vIndex);
} catch (Exception $e) {
printf("failed to create vectorIndex cached searcher with '%s': %s\n", $dbFile, $e);
return;
}
// 3、查询
$sTime = XdbSearcher::now();
$region = $searcher->search('1.2.3.4');
if ($region === null) {
printf("failed search(1.2.3.4)\n");
return;
}
printf("{region: %s, took: %.5f ms}\n", $region, XdbSearcher::now() - $sTime);
// 备注:并发使用,每个线程或者协程需要创建一个独立的 searcher 对象,但是都共享统一的只读 vectorIndex。
```
### 缓存整个 xdb 数据
如果你的 PHP 目环境支持,可以预先加载整个 `xdb` 的数据到内存,这样可以实现完全基于内存的查询,类似之前的 memory search 查询。
```php
// 1、从 dbPath 加载整个 xdb 到内存。
$cBuff = XdbSearcher::loadContentFromFile($dbPath);
if ($cBuff === null) {
printf("failed to load content buffer from '%s'\n", $dbPath);
return;
}
// 2、使用全局的 cBuff 创建带完全基于内存的查询对象。
try {
$searcher = XdbSearcher::newWithBuffer($cBuff);
} catch (Exception $e) {
printf("failed to create buffer cached searcher: %s\n", $dbFile, $e);
return;
}
// 3、查询
$sTime = XdbSearcher::now();
$region = $searcher->search('1.2.3.4');
if ($region === null) {
printf("failed search(1.2.3.4)\n");
return;
}
printf("{region: %s, took: %.5f ms}\n", $region, XdbSearcher::now() - $sTime);
// 备注:并发使用,用整个 xdb 缓存创建的 searcher 对象可以安全用于并发。
```
# 查询测试
通过 `search_test.php` 脚本来进行查询测试:
```bash
➜ php git:(v2.0_xdb) ✗ php ./search_test.php
php ./search_test.php [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如:使用默认的 data/ip2region.xdb 进行查询测试:
```bash
➜ php git:(v2.0_xdb) ✗ php ./search_test.php --db=../../data/ip2region.xdb --cache-policy=vectorIndex
ip2region xdb searcher test program, cachePolicy: vectorIndex
type 'quit' to exit
ip2region>> 1.2.3.4
{region: 美国|0|华盛顿|0|谷歌, ioCount: 7, took: 0.04492 ms}
ip2region>>
```
输入 ip 即可进行查询测试。也可以设置 `cache-policy` 来分别测试 file/vectorIndex/content 三种不同缓存类型的查询。
# bench 测试
通过 `bench_test.php` 脚本来进行自动 bench 测试,一方面确保 `xdb` 文件没有错误,一方面通过大量的查询测试平均查询性能:
```bash
➜ php git:(v2.0_xdb) ✗ php ./bench_test.php
php ./bench_test.php [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如:通过默认的 data/ip2region.xdb 和 data/ip.merge.txt 来进行 bench 测试:
```bash
➜ php git:(v2.0_xdb) ✗ php ./bench_test.php --db=../../data/ip2region.xdb --src=../../data/ip.merge.txt --cache-policy=vectorIndex
Bench finished, {cachePolicy: vectorIndex, total: 3417955, took: 15s, cost: 0.005 ms/op}
```
可以通过设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 三种不同的缓存类型的性能。
@Note:请注意 bench 使用的 src 文件需要是生成对应的 xdb 文件的相同的源文件。

View File

@ -0,0 +1,345 @@
<?php
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
//
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/21
class XdbSearcher
{
const HeaderInfoLength = 256;
const VectorIndexRows = 256;
const VectorIndexCols = 256;
const VectorIndexSize = 8;
const SegmentIndexSize = 14;
// xdb file handle
private $handle = null;
// header info
private $header = null;
private $ioCount = 0;
// vector index in binary string.
// string decode will be faster than the map based Array.
private $vectorIndex = null;
// xdb content buffer
private $contentBuff = null;
// ---
// static function to create searcher
/**
* @throws Exception
*/
public static function newWithFileOnly($dbFile) {
return new XdbSearcher($dbFile, null, null);
}
/**
* @throws Exception
*/
public static function newWithVectorIndex($dbFile, $vIndex) {
return new XdbSearcher($dbFile, $vIndex);
}
/**
* @throws Exception
*/
public static function newWithBuffer($cBuff) {
return new XdbSearcher(null, null, $cBuff);
}
// --- End of static creator
/**
* initialize the xdb searcher
* @throws Exception
*/
function __construct($dbFile, $vectorIndex=null, $cBuff=null) {
// check the content buffer first
if ($cBuff != null) {
$this->vectorIndex = null;
$this->contentBuff = $cBuff;
} else {
// open the xdb binary file
$this->handle = fopen($dbFile, "r");
if ($this->handle === false) {
throw new Exception("failed to open xdb file '%s'", $dbFile);
}
$this->vectorIndex = $vectorIndex;
}
}
function close() {
if ($this->handle != null) {
fclose($this->handle);
}
}
function getIOCount() {
return $this->ioCount;
}
/**
* find the region info for the specified ip address
* @throws Exception
*/
function search($ip) {
// check and convert the sting ip to a 4-bytes long
if (is_string($ip)) {
$t = self::ip2long($ip);
if ($t === null) {
throw new Exception("invalid ip address `$ip`");
}
$ip = $t;
}
// reset the global counter
$this->ioCount = 0;
// locate the segment index block based on the vector index
$il0 = ($ip >> 24) & 0xFF;
$il1 = ($ip >> 16) & 0xFF;
$idx = $il0 * self::VectorIndexCols * self::VectorIndexSize + $il1 * self::VectorIndexSize;
if ($this->vectorIndex != null) {
$sPtr = self::getLong($this->vectorIndex, $idx);
$ePtr = self::getLong($this->vectorIndex, $idx + 4);
} else if ($this->contentBuff != null) {
$sPtr = self::getLong($this->contentBuff, self::HeaderInfoLength + $idx);
$ePtr = self::getLong($this->contentBuff, self::HeaderInfoLength + $idx + 4);
} else {
// read the vector index block
$buff = $this->read(self::HeaderInfoLength + $idx, 8);
if ($buff === null) {
throw new Exception("failed to read vector index at ${idx}");
}
$sPtr = self::getLong($buff, 0);
$ePtr = self::getLong($buff, 4);
}
// printf("sPtr: %d, ePtr: %d\n", $sPtr, $ePtr);
// binary search the segment index to get the region info
$dataLen = 0;
$dataPtr = null;
$l = 0;
$h = ($ePtr - $sPtr) / self::SegmentIndexSize;
while ($l <= $h) {
$m = ($l + $h) >> 1;
$p = $sPtr + $m * self::SegmentIndexSize;
// read the segment index
$buff = $this->read($p, self::SegmentIndexSize);
if ($buff == null) {
throw new Exception("failed to read segment index at ${p}");
}
$sip = self::getLong($buff, 0);
if ($ip < $sip) {
$h = $m - 1;
} else {
$eip = self::getLong($buff, 4);
if ($ip > $eip) {
$l = $m + 1;
} else {
$dataLen = self::getShort($buff, 8);
$dataPtr = self::getLong($buff, 10);
break;
}
}
}
// match nothing interception.
// @TODO: could this even be a case ?
// printf("dataLen: %d, dataPtr: %d\n", $dataLen, $dataPtr);
if ($dataPtr == null) {
return null;
}
// load and return the region data
$buff = $this->read($dataPtr, $dataLen);
if ($buff == null) {
return null;
}
return $buff;
}
// read specified bytes from the specified index
private function read($offset, $len) {
// check the in-memory buffer first
if ($this->contentBuff != null) {
return substr($this->contentBuff, $offset, $len);
}
// read from the file
$r = fseek($this->handle, $offset);
if ($r == -1) {
return null;
}
$this->ioCount++;
$buff = fread($this->handle, $len);
if ($buff === false) {
return null;
}
if (strlen($buff) != $len) {
return null;
}
return $buff;
}
// --- static util functions ----
// convert a string ip to long
public static function ip2long($ip)
{
$ip = ip2long($ip);
if ($ip === false) {
return null;
}
// convert signed int to unsigned int if on 32 bit operating system
if ($ip < 0 && PHP_INT_SIZE == 4) {
$ip = sprintf("%u", $ip);
}
return $ip;
}
// read a 4bytes long from a byte buffer
public static function getLong($b, $idx)
{
$val = (ord($b[$idx])) | (ord($b[$idx+1]) << 8)
| (ord($b[$idx+2]) << 16) | (ord($b[$idx+3]) << 24);
// convert signed int to unsigned int if on 32 bit operating system
if ($val < 0 && PHP_INT_SIZE == 4) {
$val = sprintf("%u", $val);
}
return $val;
}
// read a 2bytes short from a byte buffer
public static function getShort($b, $idx)
{
return ((ord($b[$idx])) | (ord($b[$idx+1]) << 8));
}
// load header info from a specified file handle
public static function loadHeader($handle) {
if (fseek($handle, 0) == -1) {
return null;
}
$buff = fread($handle, self::HeaderInfoLength);
if ($buff === false) {
return null;
}
// read bytes length checking
if (strlen($buff) != self::HeaderInfoLength) {
return null;
}
// return the decoded header info
return array(
'version' => self::getShort($buff, 0),
'indexPolicy' => self::getShort($buff, 2),
'createdAt' => self::getLong($buff, 4),
'startIndexPtr' => self::getLong($buff, 8),
'endIndexPtr' => self::getLong($buff, 12)
);
}
// load header info from the specified xdb file path
public static function loadHeaderFromFile($dbFile) {
$handle = fopen($dbFile, 'r');
if ($handle === false) {
return null;
}
return self::loadHeader($handle);
}
// load vector index from a file handle
public static function loadVectorIndex($handle) {
if (fseek($handle, self::HeaderInfoLength) == -1) {
return null;
}
$rLen = self::VectorIndexRows * self::VectorIndexCols * self::SegmentIndexSize;
$buff = fread($handle, $rLen);
if ($buff === false) {
return null;
}
if (strlen($buff) != $rLen) {
return null;
}
return $buff;
}
// load vector index from a specified xdb file path
public static function loadVectorIndexFromFile($dbFile) {
$handle = fopen($dbFile, 'r');
if ($handle === false) {
return null;
}
return self::loadVectorIndex($handle);
}
// load the xdb content from a file handle
public static function loadContent($handle) {
if (fseek($handle, 0, SEEK_END) == -1) {
return null;
}
$size = ftell($handle);
if ($size === false) {
return null;
}
// seek to the head for reading
if (fseek($handle, 0) == -1) {
return null;
}
$buff = fread($handle, $size);
if ($buff === false) {
return null;
}
// read length checking
if (strlen($buff) != $size) {
return null;
}
return $buff;
}
// load the xdb content from a file path
public static function loadContentFromFile($dbFile) {
$str = file_get_contents($dbFile, false);
if ($str === false) {
return null;
} else {
return $str;
}
}
public static function now() {
return (microtime(true) * 1000);
}
}

176
binding/php/bench_test.php Normal file
View File

@ -0,0 +1,176 @@
<?php
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
//
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/22
require dirname(__FILE__) . '/XdbSearcher.class.php';
function printHelp($argv) {
printf("php %s [command options]\n", $argv[0]);
printf("options: \n");
printf(" --db string ip2region binary xdb file path\n");
printf(" --src string source ip text file path\n");
printf(" --cache-policy string cache policy: file/vectorIndex/content\n");
}
if($argc < 2) {
printHelp($argv);
return;
}
$dbFile = "";
$srcFile = "";
$cachePolicy = 'vectorIndex';
array_shift($argv);
foreach ($argv as $r) {
if (strlen($r) < 5) {
continue;
}
if (strpos($r, '--') != 0) {
continue;
}
$sIdx = strpos($r, "=");
if ($sIdx < 0) {
printf("missing = for args pair %s\n", $r);
return;
}
$key = substr($r, 2, $sIdx - 2);
$val = substr($r, $sIdx + 1);
if ($key == 'db') {
$dbFile = $val;
} else if ($key == 'src') {
$srcFile = $val;
} else if ($key == 'cache-policy') {
$cachePolicy = $val;
} else {
printf("undefined option `%s`\n", $r);
return;
}
}
if (strlen($dbFile) < 1 || strlen($srcFile) < 1) {
printHelp($argv);
return;
}
// printf("debug: dbFile: %s, cachePolicy: %s\n", $dbFile, $cachePolicy);
// create the xdb searcher by the cache-policy
switch ( $cachePolicy ) {
case 'file':
try {
$searcher = XdbSearcher::newWithFileOnly($dbFile);
} catch (Exception $e) {
printf("failed to create searcher with '%s': %s\n", $dbFile, $e);
return;
}
break;
case 'vectorIndex':
$vIndex = XdbSearcher::loadVectorIndexFromFile($dbFile);
if ($vIndex == null) {
printf("failed to load vector index from '%s'\n", $dbFile);
return;
}
try {
$searcher = XdbSearcher::newWithVectorIndex($dbFile, $vIndex);
} catch (Exception $e) {
printf("failed to create vector index cached searcher with '%s': %s\n", $dbFile, $e);
return;
}
break;
case 'content':
$cBuff = XdbSearcher::loadContentFromFile($dbFile);
if ($cBuff == null) {
printf("failed to load xdb content from '%s'\n", $dbFile);
return;
}
try {
$searcher = XdbSearcher::newWithBuffer($cBuff);
} catch (Exception $e) {
printf("failed to create content cached searcher: %s", $e);
return;
}
break;
default:
printf("undefined cache-policy `%s`\n", $cachePolicy);
return;
}
// do the bench test
$handle = fopen($srcFile, "r");
if ($handle === false) {
printf("failed to open source text file `%s`\n", $srcFile);
return null;
}
$count = 0;
$costs = 0;
$sTime = XdbSearcher::now();
while (!feof($handle)) {
$line = trim(fgets($handle, 1024));
if (strlen($line) < 1) {
continue;
}
$ps = explode('|', $line, 3);
if (count($ps) != 3) {
printf("invalid ip segment line `${line}`\n");
return;
}
$sip = XdbSearcher::ip2long($ps[0]);
if ($sip === null) {
printf("invalid start ip `%s`\n", $ps[0]);
return;
}
$eip = XdbSearcher::ip2long($ps[1]);
if ($eip === null) {
printf("invalid end ip `%s`\n", $ps[1]);
return;
}
if ($sip > $eip) {
printf("start ip(%s) should not be greater than end ip(%s)\n", $ps[0], $ps[1]);
return;
}
$mip = ($sip + $eip) >> 1;
foreach ([$sip, ($sip + $mip) >> 1, $mip, ($mip + $eip) >> 1, $eip] as $ip) {
try {
$cTime = XdbSearcher::now();
$region = $searcher->search($ip);
$costs += XdbSearcher::now() - $cTime;
} catch (Exception $e) {
printf("failed to search ip `%s`\n", long2ip($ip));
return;
}
if ($region == null) {
printf("failed to search ip `%s`\n", long2ip($ip));
return;
}
// check the region info
if ($region != $ps[2]) {
printf("failed search(%s) with (%s != %s)\n", long2ip($ip), $region, $ps[2]);
return;
}
$count++;
}
}
// close the searcher at last
fclose($handle);
$searcher->close();
printf("Bench finished, {cachePolicy: %s, total: %d, took: %ds, cost: %.3f ms/op}\n",
$cachePolicy, $count, (XdbSearcher::now() - $sTime)/1000, $count == 0 ? 0 : $costs/$count);

133
binding/php/search_test.php Normal file
View File

@ -0,0 +1,133 @@
<?php
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
//
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/21
require dirname(__FILE__) . '/XdbSearcher.class.php';
function printHelp($argv) {
printf("php %s [command options]\n", $argv[0]);
printf("options: \n");
printf(" --db string ip2region binary xdb file path\n");
printf(" --cache-policy string cache policy: file/vectorIndex/content\n");
}
if($argc < 2) {
printHelp($argv);
return;
}
$dbFile = "";
$cachePolicy = 'vectorIndex';
array_shift($argv);
foreach ($argv as $r) {
if (strlen($r) < 5) {
continue;
}
if (strpos($r, '--') != 0) {
continue;
}
$sIdx = strpos($r, "=");
if ($sIdx < 0) {
printf("missing = for args pair %s\n", $r);
return;
}
$key = substr($r, 2, $sIdx - 2);
$val = substr($r, $sIdx + 1);
if ($key == 'db') {
$dbFile = $val;
} else if ($key == 'cache-policy') {
$cachePolicy = $val;
} else {
printf("undefined option `%s`\n", $r);
return;
}
}
if (strlen($dbFile) < 1) {
printHelp($argv);
return;
}
// printf("debug: dbFile: %s, cachePolicy: %s\n", $dbFile, $cachePolicy);
// create the xdb searcher by the cache-policy
switch ( $cachePolicy ) {
case 'file':
try {
$searcher = XdbSearcher::newWithFileOnly($dbFile);
} catch (Exception $e) {
printf("failed to create searcher with '%s': %s\n", $dbFile, $e);
return;
}
break;
case 'vectorIndex':
$vIndex = XdbSearcher::loadVectorIndexFromFile($dbFile);
if ($vIndex == null) {
printf("failed to load vector index from '%s'\n", $dbFile);
return;
}
try {
$searcher = XdbSearcher::newWithVectorIndex($dbFile, $vIndex);
} catch (Exception $e) {
printf("failed to create vector index cached searcher with '%s': %s\n", $dbFile, $e);
return;
}
break;
case 'content':
$cBuff = XdbSearcher::loadContentFromFile($dbFile);
if ($cBuff == null) {
printf("failed to load xdb content from '%s'\n", $dbFile);
return;
}
try {
$searcher = XdbSearcher::newWithBuffer($cBuff);
} catch (Exception $e) {
printf("failed to create content cached searcher: %s", $e);
return;
}
break;
default:
printf("undefined cache-policy `%s`\n", $cachePolicy);
return;
}
printf("ip2region xdb searcher test program, cachePolicy: ${cachePolicy}\ntype 'quit' to exit\n");
while ( true ) {
echo "ip2region>> ";
$line = trim(fgets(STDIN));
if (strlen($line) < 2) {
continue;
}
if ($line == 'quit') {
break;
}
if (XdbSearcher::ip2long($line) === null) {
echo "Error: invalid ip address\n";
continue;
}
$sTime = XdbSearcher::now();
try {
$region = $searcher->search($line);
} catch (Exception $e) {
printf("search call failed: %s\n", $e);
continue;
}
printf("{region: %s, ioCount: %d, took: %.5f ms}\n",
$region, $searcher->getIOCount(), XdbSearcher::now() - $sTime);
}
// close the searcher at last
$searcher->close();
printf("searcher test program exited, thanks for trying\n");

57
binding/php/util_test.php Normal file
View File

@ -0,0 +1,57 @@
<?php
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
//
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/22
require dirname(__FILE__) . '/XdbSearcher.class.php';
function testLoadHeader() {
$header = XdbSearcher::loadHeaderFromFile('../../data/ip2region.xdb');
if ($header == null) {
printf("failed to load header from file\n");
return;
}
printf("header loaded: ");
print_r($header);
}
function testLoadVectorIndex() {
$vIndex = XdbSearcher::loadVectorIndexFromFile('../../data/ip2region.xdb');
if ($vIndex == null) {
printf("failed to load vector index from file\n");
return;
}
printf("vector index loaded: length=%d\n", strlen($vIndex));
}
function testLoadContent() {
$cBuff = XdbSearcher::loadContentFromFile('../../data/ip2region.xdb');
if ($cBuff == null) {
printf("failed to load content from file\n");
return;
}
printf("content loaded, length=%d\n", strlen($cBuff));
}
printf("testing loadHeader ... \n");
$now = XdbSearcher::now();
testLoadHeader();
printf("done, cost: %0.5f ms\n\n", XdbSearcher::now() - $now);
printf("testing loadVectorIndex ... \n");
$now = XdbSearcher::now();
testLoadVectorIndex();
printf("done, cost: %0.5f ms\n\n", XdbSearcher::now() - $now);
printf("testing loadContent ... \n");
$now = XdbSearcher::now();
testLoadContent();
printf("done, cost: %0.5f ms\n\n", XdbSearcher::now() - $now);

View File

@ -0,0 +1,7 @@
# ip2region php5 c 扩展查询客户端实现
# 使用方式
# 查询测试
# bench 测试

View File

@ -0,0 +1,7 @@
# ip2region php7 c 扩展查询客户端实现
# 使用方式
# 查询测试
# bench 测试

View File

@ -1,29 +0,0 @@
# ip2region 的PHP拓展(php5版本)
### 安装步骤
* git clone https://github.com/lionsoul2016/ip2region.git
* cd ip2region
* cp binding/php_extension/php5/ip2region 到 php source code 的ext目录下
* cp binding/c/下面所有的文件到 php source code 的ext/ip2region/lib 目录下
* 在ext/ip2region下运行
phpize
./configure
make && sudo make install
* 配置ip2region.ini 指定db_file路径(cli/fpm)
extension=ip2region.so
ip2region.db_file=/path/to/ip2region.db
ip2region.db 在项目根目录下的data文件夹下
* 测试
在ext/ip2region/下运行
php ip2region.php
### 使用
参考ip2region/ip2region.php

View File

@ -1,29 +0,0 @@
# ip2region 的PHP拓展(php7版本)
### 安装步骤
* git clone https://github.com/lionsoul2016/ip2region.git
* cd ip2region
* cp binding/php_extension/php5/ip2region 到 php source code 的ext目录下
* cp binding/c/下面所有的文件到 php source code 的ext/ip2region/lib 目录下
* 在ext/ip2region下运行
phpize
./configure
make && sudo make install
* 配置ip2region.ini 指定db_file路径(cli/fpm)
extension=ip2region.so
ip2region.db_file=/path/to/ip2region.db
ip2region.db 在项目根目录下的data文件夹下
* 测试
在ext/ip2region/下运行
php ip2region.php
### 使用
参考ip2region/ip2region.php

7
binding/python/ReadMe.md Normal file
View File

@ -0,0 +1,7 @@
# ip2region python 查询客户端实现
# 使用方式
# 查询测试
# bench 测试

View File

@ -1,240 +0,0 @@
#-*- coding:utf-8 -*-
"""
" ip2region python seacher client module
"
" Author: koma<komazhang@foxmail.com>
" Date : 2015-11-06
"""
import struct, io, socket, sys
class Ip2Region(object):
__headerSip = []
__headerPtr = []
__f = None
__sPtr = 0
__indexLen = 0
__dbBinStr = ''
def __init__(self, dbfile):
self.initDatabase(dbfile)
def memorySearch(self, ip):
"""
" memory search method
" param: ip
"""
if not ip.isdigit(): ip = self.ip2long(ip)
if self.__dbBinStr == '':
self.__dbBinStr = self.__f.read() #read all the contents in file
self.__sPtr = self.getLong(self.__dbBinStr, 0)
endPtr = self.getLong(self.__dbBinStr, 4)
self.__indexLen = endPtr - self.__sPtr
startPtr = self.__sPtr
indexLen = self.__indexLen
dbBinStr = self.__dbBinStr
l, h, mixPtr = (0, int(indexLen/12), 0)
while l <= h:
m = int((l+h)/2)
ptr = startPtr + m*12
sip = self.getLong(dbBinStr, ptr)
eip = self.getLong(dbBinStr, ptr+4)
if ip >= sip:
if ip > eip:
l = m + 1
else:
mixPtr = self.getLong(dbBinStr, ptr+8)
break;
else:
h = m - 1
if mixPtr == 0: return "N2"
return self.returnData(mixPtr)
def binarySearch(self, ip):
"""
" binary search method
" param: ip
"""
if not ip.isdigit(): ip = self.ip2long(ip)
if self.__indexLen < 1:
self.__f.seek(0)
b = self.__f.read(8)
self.__sPtr = self.getLong(b, 0)
endPtr = self.getLong(b, 4)
self.__indexLen = endPtr - self.__sPtr
startPtr = self.__sPtr
indexLen = self.__indexLen
self.__f.seek(startPtr)
b = self.__f.read(indexLen+12)
l, h, mixPtr = (0, int(indexLen/12), 0)
while l <= h:
m = int((l+h)/2)
ptr = startPtr + m*12
self.__f.seek(ptr)
b = self.__f.read(12)
sip = self.getLong(b, 0)
eip = self.getLong(b, 4)
if ip >= sip:
if ip > eip:
l = m + 1
else:
mixPtr = self.getLong(b, 8)
break;
else:
h = m - 1
if mixPtr == 0: return "N2"
return self.returnData(mixPtr)
def btreeSearch(self, ip):
"""
" b-tree search method
" param: ip
"""
if not ip.isdigit(): ip = self.ip2long(ip)
if len(self.__headerSip) < 1:
#pass the super block
self.__f.seek(8)
#read the header block
b = self.__f.read(4086)
#parse the header block
sip = None
ptr = None
for i in range(0, len(b)-1, 8):
sip = self.getLong(b, i)
ptr = self.getLong(b, i+4)
if ptr == 0:
break
self.__headerSip.append(sip)
self.__headerPtr.append(ptr)
headerLen = len(self.__headerSip) - 1
l, h, sptr, eptr = (0, headerLen, 0, 0)
while l <= h:
m = int((l+h)/2)
if ip == self.__headerSip[m]:
if m > 0:
sptr = self.__headerPtr[m-1]
eptr = self.__headerPtr[m]
break;
else:
sptr = self.__headerPtr[m]
eptr = self.__headerPtr[m+1]
break;
if ip > self.__headerSip[m]:
if m == headerLen:
sptr = self.__headerPtr[m-1]
eptr = self.__headerPtr[m]
break;
elif ip < self.__headerSip[m+1]:
sptr = self.__headerPtr[m]
eptr = self.__headerPtr[m+1]
break;
l = m + 1
else:
if m == 0:
sptr = self.__headerPtr[m]
eptr = self.__headerPtr[m+1]
break;
elif ip > self.__headerSip[m-1]:
sptr = self.__headerPtr[m-1]
eptr = self.__headerPtr[m]
break;
h = m - 1
if sptr == 0: return "N1"
indexLen = eptr - sptr
self.__f.seek(sptr)
b = self.__f.read(indexLen + 12)
l, h, mixPtr = (0, int(indexLen/12), 0)
while l <= h:
m = int((l+h)/2)
offset = m * 12
if ip >= self.getLong(b, offset):
if ip > self.getLong(b, offset+4):
l = m + 1
else:
mixPtr = self.getLong(b, offset+8)
break;
else:
h = m - 1
if mixPtr == 0: return "N2"
return self.returnData(mixPtr)
def initDatabase(self, dbfile):
"""
" initialize the database for search
" param: dbFile
"""
try:
self.__f = io.open(dbfile, "rb")
except IOError, e:
print "[Error]: ", e
sys.exit()
def returnData(self, dsptr):
"""
" get ip data from db file by data start ptr
" param: dsptr
"""
dataPtr = dsptr & 0x00FFFFFFL
dataLen = (dsptr >> 24) & 0xFF
self.__f.seek(dataPtr)
data = self.__f.read(dataLen)
return {
"city_id": self.getLong(data, 0),
"region" : data[4:]
}
def ip2long(self, ip):
_ip = socket.inet_aton(ip)
return struct.unpack("!L", _ip)[0]
def isip(self, ip):
p = ip.split(".")
if len(p) != 4 : return False
for pp in p:
if not pp.isdigit() : return False
if len(pp) > 3 : return False
if int(pp) > 255 : return False
return True
def getLong(self, b, offset):
if len( b[offset:offset+4] ) == 4:
return struct.unpack('I', b[offset:offset+4])[0]
return 0
def close(self):
self.__headerSip = None
self.__headerPtr = None
self.__f.close()
self.__f = None

View File

@ -1,79 +0,0 @@
#-*- coding:utf-8 -*-
"""
" ip2region python seacher client module
"
" Autho: koma<komazhang@foxmail.com>
" Date : 2015-11-06
"""
import struct, sys, os, time
from ip2Region import Ip2Region
def testSearch():
"""
" ip2region test function
"""
llen = len(sys.argv)
if llen < 2:
print "Usage: python ip2Region.py [ip2region db file] [alrogrithm]"
print "Algorithm: binary or b-tree"
return 0
dbFile = sys.argv[1]
method = 1
algorithm = "b-tree"
if (not os.path.isfile(dbFile)) or (not os.path.exists(dbFile)):
print "[Error]: Specified db file is not exists."
return 0
if llen > 2:
algorithm = sys.argv[2]
if algorithm == "binary":
method = 2
elif algorithm == "memory":
method = 3
print "initializing %s..." % (algorithm)
print "+----------------------------------+"
print "| ip2region test program |"
print "| Author: chenxin619315@gmail.com. |"
print "| Type 'quit' to exit program |"
print "+----------------------------------+"
searcher = Ip2Region(dbFile);
while True:
line = raw_input("ip2region>> ")
line = line.strip()
if line == "":
print "[Error]: Invalid ip address."
continue
if line == "quit":
print "[Info]: Thanks for your use, Bye."
break
if not searcher.isip(line):
print "[Error]: Invalid ip address."
continue
sTime = time.time() * 1000
if method == 1:
data = searcher.btreeSearch(line)
elif method == 2:
data = searcher.binarySearch(line)
else:
data = searcher.memorySearch(line)
eTime = time.time() * 1000
if isinstance(data, dict):
print "%s|%s in %f millseconds" % (data["city_id"], data["region"], eTime-sTime)
else:
print "[Error]: ", data
searcher.close()
if __name__ == "__main__":
testSearch()

7
binding/rust/ReadMe.md Normal file
View File

@ -0,0 +1,7 @@
# ip2region rust 查询客户端实现
# 使用方式
# 查询测试
# bench 测试

File diff suppressed because it is too large Load Diff

Binary file not shown.

BIN
data/ip2region.xdb Normal file

Binary file not shown.

7
maker/c/ReadMe.md Normal file
View File

@ -0,0 +1,7 @@
# ip2region xdb c语言生成实现
# 数据生成
# 数据查询
# bench 测试

10
maker/golang/Makefile Normal file
View File

@ -0,0 +1,10 @@
# ip2region golang maker makefile
all: build
.PHONY: all
build:
go build -o xdb_maker
test:
go test -v ./...
clean:
find ./ -name xdb_maker | xargs rm -f

86
maker/golang/ReadMe.md Normal file
View File

@ -0,0 +1,86 @@
# ip2region xdb golang 生成实现
# 程序编译
通过如下方式编译得到 xdb_maker 可执行程序:
```
# 切换到golang maker 根目录
make
```
编译成功后会在当前目录生成一个 xdb_maker 的可执行文件
# 数据生成
通过 `xdb_maker gen` 命令生成 ip2region.xdb 二进制文件:
```
➜ golang git:(v2.0_xdb) ✗ ./xdb_maker gen
./xdb_maker gen [command options]
options:
--src string source ip text file path
--dst string destination binary xdb file path
```
例如,使用默认的 data/ip.merge.txt 作为源数据,生成一个 ip2region.xdb 到当前目录:
```bash
➜ golang git:(v2.0_xdb) ✗ ./xdb_maker gen --src=../../data/ip.merge.txt --dst=./ip2region.xdb
# 会看到一堆输出,最终会看到类似如下输出表示运行结束
...
2022/06/16 16:38:48 maker.go:317: write done, with 13804 data blocks and (683591, 720221) index blocks
2022/06/16 16:38:48 main.go:89: Done, elapsed: 33.615278847s
```
# 数据查询
通过 `xdb_maker search` 命令来测试查询输入的 ip
```
➜ golang git:(v2.0_xdb) ✗ ./xdb_maker search
./xdb_maker search [command options]
options:
--db string ip2region binary xdb file path
```
例如,使用自带的 xdb 文件来运行查询测试:
```bash
➜ golang git:(v2.0_xdb) ✗ ./xdb_maker search --db=../../data/ip2region.xdb
ip2region xdb search test program, commands:
loadIndex : load the vector index for search speedup.
clearIndex: clear the vector index.
quit : exit the test program
ip2region>> 103.192.227.215
{region:中国|0|香港|0|0, iocount:8, took:87.151µs}
ip2region>> loadIndex
vector index cached
ip2region>> 39.114.2.16
{region:韩国|0|0|0|SK宽带, iocount:2, took:50.005µs}
ip2region>> 120.24.130.96
{region:中国|0|广东省|深圳市|阿里云, iocount:2, took:32.805µs}
ip2region>>
```
# bench 测试
如果你自主生成了 xdb 文件,请确保运行如下的 `xdb_maker bench` 命令来确保生成的的 xdb 文件的正确性:
```
➜ golang git:(v2.0_xdb) ✗ ./xdb_maker bench
./xdb_maker bench [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--ignore-error bool keep going if bench failed
```
例如:使用 data/ip.merge.txt 源文件来 bench 测试 data/ip2region.xdb 这个 xdb 文件:
```bash
➜ golang git:(v2.0_xdb) ✗ ./xdb_maker bench --db=../../data/ip2region.xdb --src=../../data/ip.merge.txt
# 会看到一堆输出,看到类似如下的数据表示 bench 测试通过了,否则就会报错
...
try to bench segment: `224.0.0.0|255.255.255.255|0|0|0|内网IP|内网IP`
|-try to bench ip '224.0.0.0' ... --[Ok]
|-try to bench ip '231.255.255.255' ... --[Ok]
|-try to bench ip '239.255.255.255' ... --[Ok]
|-try to bench ip '247.255.255.255' ... --[Ok]
|-try to bench ip '255.255.255.255' ... --[Ok]
Bench finished, {count: 3417955, failed: 0, took: 52.200116397s}
```
*请注意 bench 测试使用的 `src` 文件需要是对应的生成 ip2region.xdb 的源文件相同*。
如果运行过程中有错误会立马停止运行,也可以执行 --ignore-error=true 参数来忽略错误,在最后看 failed 的统计结果。

3
maker/golang/go.mod Normal file
View File

@ -0,0 +1,3 @@
module github.com/lionsoul2014/ip2region/maker/golang
go 1.17

0
maker/golang/go.sum Normal file
View File

331
maker/golang/main.go Normal file
View File

@ -0,0 +1,331 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package main
import (
"bufio"
"fmt"
"github.com/lionsoul2014/ip2region/maker/golang/xdb"
"log"
"os"
"strings"
"time"
)
func printHelp() {
fmt.Printf("ip2region xdb maker\n")
fmt.Printf("%s [command] [command options]\n", os.Args[0])
fmt.Printf("Command: \n")
fmt.Printf(" gen generate the binary db file\n")
fmt.Printf(" search binary xdb search test\n")
fmt.Printf(" bench binary xdb bench test\n")
}
func genDb() {
var err error
var srcFile, dstFile = "", ""
var indexPolicy = xdb.VectorIndexPolicy
for i := 2; i < len(os.Args); i++ {
r := os.Args[i]
if len(r) < 5 {
continue
}
if strings.Index(r, "--") != 0 {
continue
}
var sIdx = strings.Index(r, "=")
if sIdx < 0 {
fmt.Printf("missing = for args pair '%s'\n", r)
return
}
switch r[2:sIdx] {
case "src":
srcFile = r[sIdx+1:]
case "dst":
dstFile = r[sIdx+1:]
case "index":
indexPolicy, err = xdb.IndexPolicyFromString(r[sIdx+1:])
if err != nil {
fmt.Printf("parse policy: %s", err.Error())
return
}
default:
fmt.Printf("undefine option `%s`\n", r)
return
}
}
if srcFile == "" || dstFile == "" {
fmt.Printf("%s gen [command options]\n", os.Args[0])
fmt.Printf("options:\n")
fmt.Printf(" --src string source ip text file path\n")
fmt.Printf(" --dst string destination binary xdb file path\n")
return
}
// make the binary file
tStart := time.Now()
maker, err := xdb.NewMaker(indexPolicy, srcFile, dstFile)
if err != nil {
fmt.Printf("failed to create %s\n", err)
return
}
err = maker.Init()
if err != nil {
fmt.Printf("failed Init: %s\n", err)
return
}
err = maker.Start()
if err != nil {
fmt.Printf("failed Start: %s\n", err)
return
}
err = maker.End()
if err != nil {
fmt.Printf("failed End: %s\n", err)
}
log.Printf("Done, elapsed: %s\n", time.Since(tStart))
}
func testSearch() {
var err error
var dbFile = ""
for i := 2; i < len(os.Args); i++ {
r := os.Args[i]
if len(r) < 5 {
continue
}
if strings.Index(r, "--") != 0 {
continue
}
var eIdx = strings.Index(r, "=")
if eIdx < 0 {
fmt.Printf("missing = for args pair '%s'\n", r)
return
}
switch r[2:eIdx] {
case "db":
dbFile = r[eIdx+1:]
default:
fmt.Printf("undefined option '%s'\n", r)
return
}
}
if dbFile == "" {
fmt.Printf("%s search [command options]\n", os.Args[0])
fmt.Printf("options:\n")
fmt.Printf(" --db string ip2region binary xdb file path\n")
return
}
searcher, err := xdb.NewSearcher(dbFile)
if err != nil {
fmt.Printf("failed to create searcher with `%s`: %s\n", dbFile, err.Error())
return
}
defer func() {
searcher.Close()
fmt.Printf("test program exited, thanks for trying\n")
}()
fmt.Println(`ip2region xdb search test program, commands:
loadIndex : load the vector index for search speedup.
clearIndex: clear the vector index.
quit : exit the test program`)
reader := bufio.NewReader(os.Stdin)
for {
fmt.Print("ip2region>> ")
str, err := reader.ReadString('\n')
if err != nil {
log.Fatalf("failed to read string: %s", err)
}
line := strings.TrimSpace(strings.TrimSuffix(str, "\n"))
if len(line) == 0 {
continue
}
// command interception and execution
if line == "loadIndex" {
err = searcher.LoadVectorIndex()
if err != nil {
log.Fatalf("failed to load vector index: %s", err)
}
fmt.Printf("vector index cached\n")
continue
} else if line == "clearIndex" {
searcher.ClearVectorIndex()
fmt.Printf("vector index cleared\n")
continue
} else if line == "quit" {
break
}
ip, err := xdb.CheckIP(line)
if err != nil {
fmt.Printf("invalid ip address `%s`\n", line)
continue
}
tStart := time.Now()
region, ioCount, err := searcher.Search(ip)
if err != nil {
fmt.Printf("\x1b[0;31m{err:%s, iocount:%d}\x1b[0m\n", err.Error(), ioCount)
} else {
fmt.Printf("\x1b[0;32m{region:%s, iocount:%d, took:%s}\x1b[0m\n", region, ioCount, time.Since(tStart))
}
}
}
func testBench() {
var err error
var dbFile, srcFile = "", ""
var ignoreError = false
for i := 2; i < len(os.Args); i++ {
r := os.Args[i]
if len(r) < 5 {
continue
}
if strings.Index(r, "--") != 0 {
continue
}
var sIdx = strings.Index(r, "=")
if sIdx < 0 {
fmt.Printf("missing = for args pair '%s'\n", r)
return
}
switch r[2:sIdx] {
case "db":
dbFile = r[sIdx+1:]
case "src":
srcFile = r[sIdx+1:]
case "ignore-error":
v := r[sIdx+1:]
if v == "true" || v == "1" {
ignoreError = true
} else if v == "false" || v == "0" {
ignoreError = false
} else {
fmt.Printf("invalid value for ignore-error option, could be false/0 or true/1\n")
return
}
default:
fmt.Printf("undefined option '%s'\n", r)
return
}
}
if dbFile == "" || srcFile == "" {
fmt.Printf("%s bench [command options]\n", os.Args[0])
fmt.Printf("options:\n")
fmt.Printf(" --db string ip2region binary xdb file path\n")
fmt.Printf(" --src string source ip text file path\n")
fmt.Printf(" --ignore-error bool keep going if bench failed\n")
return
}
searcher, err := xdb.NewSearcher(dbFile)
if err != nil {
fmt.Printf("failed to create searcher with `%s`: %s\n", dbFile, err)
return
}
defer func() {
searcher.Close()
}()
handle, err := os.OpenFile(srcFile, os.O_RDONLY, 0600)
if err != nil {
fmt.Printf("failed to open source text file: %s\n", err)
return
}
var count, errCount, tStart = 0, 0, time.Now()
var scanner = bufio.NewScanner(handle)
scanner.Split(bufio.ScanLines)
for scanner.Scan() {
var l = strings.TrimSpace(strings.TrimSuffix(scanner.Text(), "\n"))
var ps = strings.SplitN(l, "|", 3)
if len(ps) != 3 {
fmt.Printf("invalid ip segment line `%s`\n", l)
return
}
sip, err := xdb.CheckIP(ps[0])
if err != nil {
fmt.Printf("check start ip `%s`: %s\n", ps[0], err)
return
}
eip, err := xdb.CheckIP(ps[1])
if err != nil {
fmt.Printf("check end ip `%s`: %s\n", ps[1], err)
return
}
if sip > eip {
fmt.Printf("start ip(%s) should not be greater than end ip(%s)\n", ps[0], ps[1])
return
}
fmt.Printf("try to bench segment: `%s`\n", l)
mip := xdb.MidIP(sip, eip)
for _, ip := range []uint32{sip, xdb.MidIP(sip, mip), mip, xdb.MidIP(mip, eip), eip} {
fmt.Printf("|-try to bench ip '%s' ... ", xdb.Long2IP(ip))
region, _, err := searcher.Search(ip)
if err != nil {
fmt.Printf("failed to search ip '%s': %s\n", xdb.Long2IP(ip), err)
return
}
// check the region info
count++
if region != ps[2] {
errCount++
fmt.Printf(" --[Failed] (%s != %s)\n", region, ps[2])
if ignoreError == false {
return
}
} else {
fmt.Printf(" --[Ok]\n")
}
}
}
fmt.Printf("Bench finished, {count: %d, failed: %d, took: %s}\n", count, errCount, time.Since(tStart))
}
func main() {
if len(os.Args) < 2 {
printHelp()
return
}
// set the log flag
log.SetFlags(log.Ldate | log.Ltime | log.Lshortfile)
switch strings.ToLower(os.Args[1]) {
case "gen":
genDb()
case "search":
testSearch()
case "bench":
testBench()
default:
printHelp()
}
}

93
maker/golang/xdb/index.go Normal file
View File

@ -0,0 +1,93 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package xdb
import (
"encoding/binary"
"fmt"
"strings"
)
type IndexPolicy int
const (
VectorIndexPolicy IndexPolicy = 1
BTreeIndexPolicy IndexPolicy = 2
)
func IndexPolicyFromString(str string) (IndexPolicy, error) {
switch strings.ToLower(str) {
case "vector":
return VectorIndexPolicy, nil
case "btree":
return BTreeIndexPolicy, nil
default:
return VectorIndexPolicy, fmt.Errorf("invalid policy '%s'", str)
}
}
const SegmentIndexBlockSize = 14
type SegmentIndexBlock struct {
StartIP uint32
EndIP uint32
DataLen uint16
DataPtr uint32
}
func SegmentIndexDecode(input []byte) (*SegmentIndexBlock, error) {
if len(input) < 14 {
return nil, fmt.Errorf("input is less than 14 bytes")
}
return &SegmentIndexBlock{
StartIP: binary.LittleEndian.Uint32(input),
EndIP: binary.LittleEndian.Uint32(input[4:]),
DataLen: binary.LittleEndian.Uint16(input[8:]),
DataPtr: binary.LittleEndian.Uint32(input[10:]),
}, nil
}
func (s *SegmentIndexBlock) Encode() []byte {
var buff = make([]byte, 14)
binary.LittleEndian.PutUint32(buff, s.StartIP)
binary.LittleEndian.PutUint32(buff[4:], s.EndIP)
binary.LittleEndian.PutUint16(buff[8:], s.DataLen)
binary.LittleEndian.PutUint32(buff[10:], s.DataPtr)
return buff
}
func (s *SegmentIndexBlock) String() string {
return fmt.Sprintf("{sip: %d, eip: %d, len: %d, ptr: %d}", s.StartIP, s.EndIP, s.DataLen, s.DataPtr)
}
// ------------
type VectorIndexBlock struct {
FirstPtr uint32
LastPtr uint32
}
func VectorIndexBlockDecode(input []byte) (*VectorIndexBlock, error) {
if len(input) < 8 {
return nil, fmt.Errorf("input should be not less then 8 bytes")
}
return &VectorIndexBlock{
FirstPtr: binary.LittleEndian.Uint32(input),
LastPtr: binary.LittleEndian.Uint32(input[4:]),
}, nil
}
func (v VectorIndexBlock) Encode() []byte {
var buff = make([]byte, 8)
binary.LittleEndian.PutUint32(buff, v.FirstPtr)
binary.LittleEndian.PutUint32(buff[4:], v.LastPtr)
return buff
}
func (v VectorIndexBlock) String() string {
return fmt.Sprintf("{FristPtr: %d, LastPtr: %d}", v.FirstPtr, v.LastPtr)
}

358
maker/golang/xdb/maker.go Normal file
View File

@ -0,0 +1,358 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ----
// ip2region database v2.0 structure
//
// +----------------+-------------------+---------------+--------------+
// | header space | speed up index | data payload | block index |
// +----------------+-------------------+---------------+--------------+
// | 256 bytes | 512 KiB (fixed) | dynamic size | dynamic size |
// +----------------+-------------------+---------------+--------------+
//
// 1. padding space : for header info like block index ptr, version, release date eg ... or any other temporary needs.
// -- 2bytes: version number, different version means structure update, it fixed to 2 for now
// -- 2bytes: index algorithm code.
// -- 4bytes: generate unix timestamp (version)
// -- 4bytes: index block start ptr
// -- 4bytes: index block end ptr
//
//
// 2. data block : region or whatever data info.
// 3. segment index block : binary index block.
// 4. vector index block : fixed index info for block index search speed up.
// space structure table:
// -- 0 -> | 1rt super block | 2nd super block | 3rd super block | ... | 255th super block
// -- 1 -> | 1rt super block | 2nd super block | 3rd super block | ... | 255th super block
// -- 2 -> | 1rt super block | 2nd super block | 3rd super block | ... | 255th super block
// -- ...
// -- 255 -> | 1rt super block | 2nd super block | 3rd super block | ... | 255th super block
//
//
// super block structure:
// +-----------------------+----------------------+
// | first index block ptr | last index block ptr |
// +-----------------------+----------------------+
//
// data entry structure:
// +--------------------+-----------------------+
// | 2bytes (for desc) | dynamic length |
// +--------------------+-----------------------+
// data length whatever in bytes
//
// index entry structure
// +------------+-----------+---------------+------------+
// | 4bytes | 4bytes | 2bytes | 4 bytes |
// +------------+-----------+---------------+------------+
// start ip end ip data length data ptr
package xdb
import (
"bufio"
"encoding/binary"
"fmt"
"log"
"os"
"strings"
"time"
)
const VersionNo = 2
const HeaderInfoLength = 256
const VectorIndexRows = 256
const VectorIndexCols = 256
const VectorIndexSize = 8
const VectorIndexLength = VectorIndexRows * VectorIndexCols * VectorIndexSize
type Maker struct {
srcHandle *os.File
dstHandle *os.File
indexPolicy IndexPolicy
segments []*Segment
regionPool map[string]uint32
vectorIndex [VectorIndexCols][VectorIndexRows]VectorIndexBlock
}
func NewMaker(policy IndexPolicy, srcFile string, dstFile string) (*Maker, error) {
// open the source file with READONLY mode
srcHandle, err := os.OpenFile(srcFile, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("open source file `%s`: %w", srcFile, err)
}
// open the destination file with Read/Write mode
dstHandle, err := os.OpenFile(dstFile, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666)
if err != nil {
return nil, fmt.Errorf("open target file `%s`: %w", dstFile, err)
}
return &Maker{
srcHandle: srcHandle,
dstHandle: dstHandle,
indexPolicy: policy,
segments: []*Segment{},
regionPool: map[string]uint32{},
vectorIndex: [VectorIndexCols][VectorIndexRows]VectorIndexBlock{},
}, nil
}
func (m *Maker) initDbHeader() error {
log.Printf("try to init the db header ... ")
_, err := m.dstHandle.Seek(0, 0)
if err != nil {
return err
}
// make and write the header space
var header = make([]byte, 256)
// 1, version number
binary.LittleEndian.PutUint16(header, uint16(VersionNo))
// 2, index policy code
binary.LittleEndian.PutUint16(header[2:], uint16(m.indexPolicy))
// 3, generate unix timestamp
binary.LittleEndian.PutUint32(header[4:], uint32(time.Now().Unix()))
// 4, index block start ptr
binary.LittleEndian.PutUint32(header[8:], uint32(0))
// 5, index block end ptr
binary.LittleEndian.PutUint32(header[12:], uint32(0))
_, err = m.dstHandle.Write(header)
if err != nil {
return err
}
return nil
}
func (m *Maker) loadSegments() error {
log.Printf("try to load the segments ... ")
var last *Segment = nil
var tStart = time.Now()
var scanner = bufio.NewScanner(m.srcHandle)
scanner.Split(bufio.ScanLines)
for scanner.Scan() {
var l = strings.TrimSpace(strings.TrimSuffix(scanner.Text(), "\n"))
log.Printf("load segment: `%s`", l)
var ps = strings.SplitN(l, "|", 3)
if len(ps) != 3 {
return fmt.Errorf("invalid ip segment line `%s`", l)
}
sip, err := CheckIP(ps[0])
if err != nil {
return fmt.Errorf("check start ip `%s`: %s", ps[0], err)
}
eip, err := CheckIP(ps[1])
if err != nil {
return fmt.Errorf("check end ip `%s`: %s", ps[1], err)
}
if sip > eip {
return fmt.Errorf("start ip(%s) should not be greater than end ip(%s)", ps[0], ps[1])
}
if len(ps[2]) < 1 {
return fmt.Errorf("empty region info in segment line `%s`", l)
}
var seg = &Segment{
StartIP: sip,
EndIP: eip,
Region: ps[2],
}
// check the continuity of the data segment
if last != nil {
if last.EndIP+1 != seg.StartIP {
return fmt.Errorf("discontinuous data segment: last.eip+1(%d) != seg.sip(%d, %s)", sip, eip, ps[0])
}
}
m.segments = append(m.segments, seg)
last = seg
}
log.Printf("all segments loaded, length: %d, elapsed: %s", len(m.segments), time.Since(tStart))
return nil
}
// Init the db binary file
func (m *Maker) Init() error {
// init the db header
err := m.initDbHeader()
if err != nil {
return fmt.Errorf("init db header: %w", err)
}
// load all the segments
err = m.loadSegments()
if err != nil {
return fmt.Errorf("load segments: %w", err)
}
return nil
}
// refresh the vector index of the specified ip
func (m *Maker) setVectorIndex(ip uint32, ptr uint32) {
var viBlock = &m.vectorIndex[(ip>>24)&0xFF][(ip>>16)&0xFF]
if viBlock.FirstPtr == 0 {
viBlock.FirstPtr = ptr
viBlock.LastPtr = ptr + SegmentIndexBlockSize
} else {
viBlock.LastPtr = ptr + SegmentIndexBlockSize
}
}
// Start to make the binary file
func (m *Maker) Start() error {
if len(m.segments) < 1 {
return fmt.Errorf("empty segment list")
}
// 1, write all the region/data to the binary file
_, err := m.dstHandle.Seek(int64(HeaderInfoLength+VectorIndexLength), 0)
if err != nil {
return fmt.Errorf("seek to data first ptr: %w", err)
}
log.Printf("try to write the data block ... ")
for _, seg := range m.segments {
log.Printf("try to write region '%s' ... ", seg.Region)
ptr, has := m.regionPool[seg.Region]
if has {
log.Printf(" --[Cached] with ptr=%d", ptr)
continue
}
var region = []byte(seg.Region)
if len(region) > 0xFFFF {
return fmt.Errorf("too long region info `%s`: shoul be less than %d bytes", seg.Region, 0xFFFF)
}
// get the first ptr of the next region
pos, err := m.dstHandle.Seek(0, 1)
if err != nil {
return fmt.Errorf("seek to current ptr: %w", err)
}
_, err = m.dstHandle.Write(region)
if err != nil {
return fmt.Errorf("write region '%s': %w", seg.Region, err)
}
m.regionPool[seg.Region] = uint32(pos)
log.Printf(" --[Added] with ptr=%d", pos)
}
// 2, write the index block and cache the super index block
log.Printf("try to write the segment index block ... ")
var counter, startIndexPtr, endIndexPtr = 0, int64(-1), int64(-1)
for _, seg := range m.segments {
dataPtr, has := m.regionPool[seg.Region]
if !has {
return fmt.Errorf("missing ptr cache for region `%s`", seg.Region)
}
var dataLen = len(seg.Region)
if dataLen < 1 {
// @TODO: could this even be a case ?
return fmt.Errorf("empty region info for segment '%s'", seg)
}
var segList = seg.Split()
log.Printf("try to index segment(%d splits) %s ...", len(segList), seg.String())
for _, s := range segList {
pos, err := m.dstHandle.Seek(0, 1)
if err != nil {
return fmt.Errorf("seek to segment index block: %w", err)
}
var sIndex = &SegmentIndexBlock{
StartIP: s.StartIP,
EndIP: s.EndIP,
DataLen: uint16(dataLen),
DataPtr: dataPtr,
}
_, err = m.dstHandle.Write(sIndex.Encode())
if err != nil {
return fmt.Errorf("write segment index for '%s': %w", s.String(), err)
}
log.Printf("|-segment index: %d, ptr: %d, segment: %s\n", counter, pos, s.String())
m.setVectorIndex(s.StartIP, uint32(pos))
counter++
// check and record the start index ptr
if startIndexPtr == -1 {
startIndexPtr = pos
}
endIndexPtr = pos
}
}
// synchronized the vector index block
log.Printf("try to write the vector index block ... ")
_, err = m.dstHandle.Seek(int64(HeaderInfoLength), 0)
if err != nil {
return fmt.Errorf("seek vector index first ptr: %w", err)
}
for i, l := range m.vectorIndex {
for j, c := range l {
_, err = m.dstHandle.Write(c.Encode())
if err != nil {
return fmt.Errorf("write vector index [%d][%d]: %w", i, j, err)
}
}
}
// synchronized the segment index info
log.Printf("try to write the segment index ptr ... ")
var buff = make([]byte, 8)
binary.LittleEndian.PutUint32(buff, uint32(startIndexPtr))
binary.LittleEndian.PutUint32(buff[4:], uint32(endIndexPtr))
_, err = m.dstHandle.Seek(8, 0)
if err != nil {
return fmt.Errorf("seek segment index ptr: %w", err)
}
_, err = m.dstHandle.Write(buff)
if err != nil {
return fmt.Errorf("write segment index ptr: %w", err)
}
log.Printf("write done, dataBlocks: %d, indexBlocks: (%d, %d), indexPtr: (%d, %d)",
len(m.regionPool), len(m.segments), counter, startIndexPtr, endIndexPtr)
return nil
}
func (m *Maker) End() error {
err := m.dstHandle.Close()
if err != nil {
return err
}
err = m.srcHandle.Close()
if err != nil {
return err
}
return nil
}

View File

@ -0,0 +1,199 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// ip2region database v2.0 searcher.
// this is part of the maker for testing and validate.
// please use the searcher in binding/golang for production use.
// And this is a Not thread safe implementation.
package xdb
import (
"encoding/binary"
"fmt"
"os"
)
type Searcher struct {
handle *os.File
// header info
header []byte
// use it only when this feature enabled.
// Preload the vector index will reduce the number of IO operations
// thus speedup the search process
vectorIndex [][]*VectorIndexBlock
}
func NewSearcher(dbFile string) (*Searcher, error) {
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, err
}
return &Searcher{
handle: handle,
header: nil,
vectorIndex: nil,
}, nil
}
func (s *Searcher) Close() {
if s.handle != nil {
err := s.handle.Close()
if err != nil {
return
}
}
}
// LoadVectorIndex load and cache the vector index for search speedup.
// this will take up VectorIndexRows x VectorIndexCols x VectorIndexSize bytes memory.
func (s *Searcher) LoadVectorIndex() error {
// loaded already
if s.vectorIndex != nil {
return nil
}
// load all the vector index block
_, err := s.handle.Seek(HeaderInfoLength, 0)
if err != nil {
return fmt.Errorf("seek to vector index: %w", err)
}
var buff = make([]byte, VectorIndexRows*VectorIndexCols*VectorIndexSize)
rLen, err := s.handle.Read(buff)
if err != nil {
return err
}
if rLen != len(buff) {
return fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
// decode the vector index blocks
var vectorIndex = make([][]*VectorIndexBlock, VectorIndexRows)
for r := 0; r < VectorIndexRows; r++ {
vectorIndex[r] = make([]*VectorIndexBlock, VectorIndexCols)
for c := 0; c < VectorIndexCols; c++ {
offset := r*VectorIndexCols*VectorIndexSize + c*VectorIndexSize
vectorIndex[r][c], err = VectorIndexBlockDecode(buff[offset:])
if err != nil {
return fmt.Errorf("decode vector index at [%d][%d]: %w", r, c, err)
}
}
}
s.vectorIndex = vectorIndex
return nil
}
// ClearVectorIndex clear preloaded vector index cache
func (s *Searcher) ClearVectorIndex() {
s.vectorIndex = nil
}
// Search find the region for the specified ip address
func (s *Searcher) Search(ip uint32) (string, int, error) {
// locate the segment index block based on the vector index
var ioCount = 0
var vIndex *VectorIndexBlock
if s.vectorIndex != nil {
vIndex = s.vectorIndex[(ip>>24)&0xFF][(ip>>16)&0xFF]
} else {
l0, l1 := (ip>>24)&0xFF, (ip>>16)&0xFF
offset := l0*VectorIndexCols*VectorIndexSize + l1*VectorIndexSize
pos, err := s.handle.Seek(int64(HeaderInfoLength+offset), 0)
if err != nil {
return "", ioCount, fmt.Errorf("seek to vector index[%d][%d]: %w", l0, l1, err)
}
ioCount++
var buff = make([]byte, 8)
rLen, err := s.handle.Read(buff)
if err != nil {
return "", ioCount, fmt.Errorf("read vector index at %d: %w", pos, err)
}
if rLen != len(buff) {
return "", ioCount, fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
vIndex, err = VectorIndexBlockDecode(buff)
if err != nil {
return "", ioCount, fmt.Errorf("invalid vector index block at %d: %w", pos, err)
}
}
//log.Printf("vIndex=%s", vIndex)
// binary search the segment index to get the region
var dataLen, dataPtr = 0, uint32(0)
var buff = make([]byte, SegmentIndexBlockSize)
var l, h = 0, int((vIndex.LastPtr - vIndex.FirstPtr) / SegmentIndexBlockSize)
for l <= h {
// log.Printf("l=%d, h=%d", l, h)
m := (l + h) >> 1
p := vIndex.FirstPtr + uint32(m*SegmentIndexBlockSize)
// log.Printf("m=%d, p=%d", m, p)
_, err := s.handle.Seek(int64(p), 0)
if err != nil {
return "", ioCount, fmt.Errorf("seek to segment block at %d: %w", p, err)
}
ioCount++
rLen, err := s.handle.Read(buff)
if err != nil {
return "", ioCount, fmt.Errorf("read segment index at %d: %w", p, err)
}
if rLen != len(buff) {
return "", ioCount, fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
// segIndex, err := SegmentIndexDecode(buff)
// if err != nil {
// return "", fmt.Errorf("invalid segment index block at %d: %w", p, err)
// }
// decode the data step by step to reduce the unnecessary calculations
sip := binary.LittleEndian.Uint32(buff)
if ip < sip {
h = m - 1
} else {
eip := binary.LittleEndian.Uint32(buff[4:])
if ip > eip {
l = m + 1
} else {
dataLen = int(binary.LittleEndian.Uint16(buff[8:]))
dataPtr = binary.LittleEndian.Uint32(buff[10:])
break
}
}
}
if dataLen == 0 {
return "", ioCount, nil
}
// load and return the region data
_, err := s.handle.Seek(int64(dataPtr), 0)
if err != nil {
return "", ioCount, fmt.Errorf("seek to data block at %d: %w", dataPtr, err)
}
ioCount++
var regionBuff = make([]byte, dataLen)
rLen, err := s.handle.Read(regionBuff)
if err != nil {
return "", ioCount, fmt.Errorf("read region data at %d: %w", dataPtr, err)
}
if rLen != dataLen {
return "", ioCount, fmt.Errorf("incomplete read: readed bytes should be %d", dataLen)
}
return string(regionBuff), ioCount, nil
}

View File

@ -0,0 +1,97 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package xdb
import (
"fmt"
"strings"
)
type Segment struct {
StartIP uint32
EndIP uint32
Region string
}
func SegmentFrom(seg string) (*Segment, error) {
var ps = strings.SplitN(seg, "|", 3)
if len(ps) != 3 {
return nil, fmt.Errorf("invalid ip segment `%s`", seg)
}
sip, err := CheckIP(ps[0])
if err != nil {
return nil, fmt.Errorf("check start ip `%s`: %s", ps[0], err)
}
eip, err := CheckIP(ps[1])
if err != nil {
return nil, fmt.Errorf("check end ip `%s`: %s", ps[1], err)
}
if sip > eip {
return nil, fmt.Errorf("start ip(%s) should not be greater than end ip(%s)", ps[0], ps[1])
}
return &Segment{
StartIP: sip,
EndIP: eip,
Region: ps[2],
}, nil
}
// Split the segment based on the pre-two bytes
func (s *Segment) Split() []*Segment {
// 1, split the segment with the first byte
var tList []*Segment
var sByte1, eByte1 = (s.StartIP >> 24) & 0xFF, (s.EndIP >> 24) & 0xFF
var nSip = s.StartIP
for i := sByte1; i <= eByte1; i++ {
sip := (i << 24) | (nSip & 0xFFFFFF)
eip := (i << 24) | 0xFFFFFF
if eip < s.EndIP {
nSip = (i + 1) << 24
} else {
eip = s.EndIP
}
// append the new segment (maybe)
tList = append(tList, &Segment{
StartIP: sip,
EndIP: eip,
// @Note: don't bother to copy the region
/// Region: s.Region,
})
}
// 2, split the segments with the second byte
var segList []*Segment
for _, seg := range tList {
base := seg.StartIP & 0xFF000000
nSip := seg.StartIP
sb2, eb2 := (seg.StartIP>>16)&0xFF, (seg.EndIP>>16)&0xFF
for i := sb2; i <= eb2; i++ {
sip := base | (i << 16) | (nSip & 0xFFFF)
eip := base | (i << 16) | 0xFFFF
if eip < seg.EndIP {
nSip = 0
} else {
eip = seg.EndIP
}
segList = append(segList, &Segment{
StartIP: sip,
EndIP: eip,
Region: s.Region,
})
}
}
return segList
}
func (s *Segment) String() string {
return Long2IP(s.StartIP) + "|" + Long2IP(s.EndIP) + "|" + s.Region
}

73
maker/golang/xdb/util.go Normal file
View File

@ -0,0 +1,73 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package xdb
import (
"encoding/binary"
"fmt"
"strconv"
"strings"
)
// Util function
func CheckIP(ip string) (uint32, error) {
var ps = strings.Split(ip, ".")
if len(ps) != 4 {
return 0, fmt.Errorf("invalid ip address `%s`", ip)
}
var buff = make([]byte, 4)
for i, s := range ps {
d, err := strconv.Atoi(s)
if err != nil {
return 0, fmt.Errorf("the %dth part `%s` is not an integer", i, s)
}
if d < 0 || d > 255 {
return 0, fmt.Errorf("the %dth part `%s` should be an integer bettween 0 and 255", i, s)
}
buff[i] = byte(d)
}
// convert the ip to integer
return binary.BigEndian.Uint32(buff), nil
}
func Long2IP(ip uint32) string {
var buff = make([]string, 4)
buff[0] = fmt.Sprintf("%d", (ip>>24)&0xFF)
buff[1] = fmt.Sprintf("%d", (ip>>16)&0xFF)
buff[2] = fmt.Sprintf("%d", (ip>>8)&0xFF)
buff[3] = fmt.Sprintf("%d", (ip>>0)&0xFF)
return strings.Join(buff, ".")
}
func MidIP(sip uint32, eip uint32) uint32 {
return uint32((uint64(sip) + uint64(eip)) >> 1)
}
func CheckSegments(segList []*Segment) error {
var last *Segment
for _, seg := range segList {
// sip must <= eip
if seg.StartIP > seg.EndIP {
return fmt.Errorf("segment `%s`: start ip should not be greater than end ip", seg.String())
}
// check the continuity of the data segment
if last != nil {
if last.EndIP+1 != seg.StartIP {
return fmt.Errorf("discontinuous segment `%s`: last.eip+1 != cur.sip", seg.String())
}
}
last = seg
}
return nil
}

View File

@ -0,0 +1,31 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package xdb
import (
"fmt"
"testing"
)
func TestSplitSegment(t *testing.T) {
// var str = "1.1.0.0|1.3.3.24|中国|广东|深圳|电信"
// var str = "0.0.0.0|1.255.225.254|0|0|0|内网IP|内网IP"
var str = "28.201.224.0|29.34.191.255|美国|0|0|0|0"
seg, err := SegmentFrom(str)
if err != nil {
t.Fatalf("failed to parser segment '%s': %s", str, err)
}
fmt.Printf("idx: src, seg: %s\n", seg.String())
var segList = seg.Split()
err = CheckSegments(segList)
if err != nil {
t.Fatalf("check segments: %s", err.Error())
}
for i, s := range segList {
fmt.Printf("idx: %3d, seg: %s\n", i, s.String())
}
}

7
maker/java/ReadMe.md Normal file
View File

@ -0,0 +1,7 @@
# ip2region xdb java 生成实现
# 数据生成
# 数据查询
# bench 测试

61
v1.0/CHANGES.md Normal file
View File

@ -0,0 +1,61 @@
### 1.8
1. 数据升级至2018/12/31的版本国外的也增加了大量的城市数据。
2. Java的binding的Searcher类增加了如下的接口和实现
```java
DbSearcher(DbConfig dbConfig, string dbBinStr);
```
maven坐标
```xml
<dependency>
<groupId>org.lionsoul</groupId>
<artifactId>ip2region</artifactId>
<version>1.7.2</version>
</dependency>
```
3. Javascript的pacakge上传到了npm。
4. 增加纯lua实现的lua binding (lua 5.2版本依赖bit模块对5.3的支持将使用lua自带的bit操作)。
5. 增加C实现的lua_c binding (和c的性能等同建议使用)。
6. 优化部分binding的demo运行和使用文档。
7. 修复python binding的benchmark的部分错误。
### 1.2.4
1, 花了近两周的时间重写了数据的升级算法,再次提高准确率,升级过程如下(此处应该有掌声):
```shell
1. ip预分段利于分布式同步升级
2. 分段同步升级,目前使用四个机器同时升级
3自动验证数据查缺补全
4数据合并
5数据格式标准化目前2个工作
1)ipip.net的数据和淘宝数据保持区域名称统一ipip.net的数据不带“省”和“市”关键字。
2)香港澳门台湾等国家信息修复淘宝原始数据有问题reported at http://git.oschina.net/lionsoul/ip2region/issues/21
6重复ip段的合并得到data目录下的ip.merge.txt
7生产data目录下的ip2region.db二进制数据库文件
```
2修复原始数据关于“香港台湾澳门”国家信息错误的bugreported at http://git.oschina.net/lionsoul/ip2region/issues/21
3将数据升级至2017/03/15同步版本
4增加c_mmap查询客户端C客户端查询文件读取使用内存映射加速感谢[Leo Ma](http://git.oschina.net/begeekmyfriend)的贡献
备注部分阿里云数据有问题缺失的部分使用免费开放的ipip.net的数据代替了。
### 1.2.3
1. 优化升级算法进行更深度的分裂可以使准确率再度提高同时数据的更新也花费了2天多的时间
2. 数据升级至2016/12/14版本的数据包括原始数据和binary数据库的数据
### 1.2.2
1. ip数据文件更新至2016/11/02版本
2. 增加ip2region.db数据库文件的java生成工具请参考README获取使用方式
3. 新增golang查询客户端的实现感谢 [@mohong122](https://github.com/mohong122) 的关注和贡献
### 1.2
1. 新增分列式升级算法大大的增加了数据的准确率基本避免之前各大网友反馈的些许ip定位错误
2. 数据升级是2016/06/30版
3. 优化数据文件生成算法ip2region.db文件由原来的3.5M降为1.5M42亿个IP地址皆大欢喜啊
4. C/PHP/JAVA客户端增加纯内存搜索模式python, php扩展nodejs会在后续版本加上
说明因为数据文件只有1.5M对于PHP,java这类IO优化类语言提升速度不大C有一个数量级的提升

2
v1.0/Cargo.toml Normal file
View File

@ -0,0 +1,2 @@
[workspace]
members = ["binding/rust", "binding/rust/example"]

175
v1.0/README.md Normal file
View File

@ -0,0 +1,175 @@
# Ip2region是什么
阅读其他语言介绍:[简体中文](README.md) | [English](README_en.md)
ip2region - 准确率99.9%的离线IP地址定位库0.0x毫秒级查询ip2region.db数据库只有数MB提供了java,php,c,python,nodejs,golang,c#等查询绑定和Binary,B树,内存三种查询算法。
# Ip2region特性
### 99.9%准确率
数据聚合了一些知名ip到地名查询提供商的数据这些是他们官方的的准确率经测试着实比经典的纯真IP定位准确一些。<br />
ip2region的数据聚合自以下服务商的开放API或者数据(升级程序每秒请求次数2到4次): <br />
01, &gt;80%, 淘宝IP地址库, [http://ip.taobao.com/](http://ip.taobao.com/) <br />
02, ≈10%, GeoIP, [https://geoip.com/](https://geoip.com/) <br />
03, ≈2%, 纯真IP库, [http://www.cz88.net/](http://www.cz88.net/) <br />
<b>备注:</b>如果上述开放API或者数据都不给开放数据时ip2region将停止数据的更新服务。
### 标准化的数据格式
每条ip数据段都固定了格式
```
_城市Id|国家|区域|省份|城市|ISP_
```
只有中国的数据精确到了城市其他国家有部分数据只能定位到国家后前的选项全部是0已经包含了全部你能查到的大大小小的国家请忽略前面的城市Id个人项目需求
### 体积小
包含了全部的IP生成的数据库文件ip2region.db只有几MB最小的版本只有1.5MB随着数据的详细度增加数据库的大小也慢慢增大目前还没超过8MB。
### 查询速度快
全部的查询客户端单次查询都在0.x毫秒级别内置了三种查询算法
1. memory算法整个数据库全部载入内存单次查询都在0.1x毫秒内C语言的客户端单次查询在0.00x毫秒级别。
2. binary算法基于二分查找基于ip2region.db文件不需要载入内存单次查询在0.x毫秒级别。
3. b-tree算法基于btree算法基于ip2region.db文件不需要载入内存单词查询在0.x毫秒级别比binary算法更快。
任何客户端b-tree都比binary算法快当然memory算法固然是最快的
### 多查询客户端的支持
已经集成的客户端有java、C#、php、c、python、nodejs、php扩展(php5和php7)、golang、rust、lua、lua_c, nginx。
binding | 描述 | 开发状态 | binary查询耗时 | b-tree查询耗时 | memory查询耗时
:-: | :-: | :-: | :-: | :-: | :-:
[c](binding/c) | ANSC c binding | 已完成 | 0.0x毫秒 | 0.0x毫秒 | 0.00x毫秒
[c#](binding/c#) | c# binding | 已完成 | 0.x毫秒 | 0.x毫秒 | 0.1x毫秒
[golang](binding/golang) | golang binding | 已完成 | 0.x毫秒 | 0.x毫秒 | 0.1x毫秒
[java](binding/java) | java binding | 已完成 | 0.x毫秒 | 0.x毫秒 | 0.1x毫秒
[lua](binding/lua) | lua实现的binding | 已完成 | 0.x毫秒 | 0.x毫秒 | 0.x毫秒
[lua_c](binding/lua_c) | lua的c扩展 | 已完成 | 0.0x毫秒 | 0.0x毫秒 | 0.00x毫秒
[nginx](binding/nginx) | nginx的c扩展 | 已完成 | 0.0x毫秒 | 0.0x毫秒 | 0.00x毫秒
[nodejs](binding/nodejs) | nodejs | 已完成 | 0.x毫秒 | 0.x毫秒 | 0.1x毫秒
[php](binding/php) | php实现的binding | 已完成 | 0.x毫秒 | 0.1x毫秒 | 0.1x毫秒
[php5_ext](binding/php5_ext) | php5的c扩展 | 已完成 | 0.0x毫秒 | 0.0x毫秒 | 0.00x毫秒
[php7_ext](binding/php7_ext) | php7的c扩展 | 已完成 | 0.0毫秒 | 0.0x毫秒 | 0.00x毫秒
[python](binding/python) | python bindng | 已完成 | 0.x毫秒 | 0.x毫秒 | 0.x毫秒
[rust](binding/rust) | rust binding | 已完成 | 0.x毫秒 | 0.x毫秒 | 0.x毫秒
# ip2region快速测试
请参考每个binding下的README说明去运行cli测试程序例如C语言的demo运行如下
```shell
cd binding/c/
gcc -g -O2 testSearcher.c ip2region.c
./a.out ../../data/ip2region.db
```
会看到如下cli界面
```shell
initializing B-tree ...
+----------------------------------+
| ip2region test script |
| Author: chenxin619315@gmail.com |
| Type 'quit' to exit program |
+----------------------------------+
p2region>> 101.105.35.57
2163|中国|华南|广东省|深圳市|鹏博士 in 0.02295 millseconds
```
输入IP地址开始测试第一次会稍微有点慢在运行命令后面接入binary,memory来尝试其他算法建议使用b-tree算法速度和并发需求的可以使用memory算法具体集成请参考不同binding下的测试源码。
# ip2region安装
具体请参考每个binding下的README文档和测试demo以下是一些可用的快捷安装方式
### maven仓库地址
```xml
<dependency>
<groupId>org.lionsoul</groupId>
<artifactId>ip2region</artifactId>
<version>1.7.2</version>
</dependency>
```
### nodejs
```
npm install node-ip2region --save
```
### nuget安装
```shell
Install-Package IP2Region
```
### php composer
```shell
# 插件来自https://github.com/zoujingli/ip2region
composer require zoujingli/ip2region
```
# ip2region 并发使用
1. 全部binding的各个search接口都<b>不是</b>线程安全的实现不同线程可以通过创建不同的查询对象来使用并发量很大的情况下binary和b-tree算法可能会打开文件数过多的错误请修改内核的最大允许打开文件数(fs.file-max=一个更高的值)或者使用持久化的memory算法。
2. memorySearch接口在发布对象前进行一次预查询(本质上是把ip2region.db文件加载到内存),可以安全用于多线程环境。
# ip2region.db的生成
从1.8版本开始ip2region开源了ip2region.db生成程序的java实现提供了ant编译支持编译后会得到以下提到的dbMaker-{version}.jar对于需要研究生成程序的或者更改自定义生成配置的请参考${ip2region_root}/maker/java内的java源码。
从ip2region 1.2.2版本开始里面提交了一个dbMaker-{version}.jar的可以执行jar文件用它来完成这个工作
1. 确保你安装好了java环境不玩Java的童鞋就自己谷歌找找拉临时用一用几分钟的事情
2. cd到${ip2region_root}/maker/java然后运行如下命令
```shell
java -jar dbMaker-{version}.jar -src 文本数据文件 -region 地域csv文件 [-dst 生成的ip2region.db文件的目录]
# 文本数据文件db文件的原始文本数据文件路径自带的ip2region.db文件就是/data/ip.merge.txt生成而来的你可以换成自己的或者更改/data/ip.merge.txt重新生成
# 地域csv文件该文件目的是方便配置ip2region进行数据关系的存储得到的数据包含一个city_id这个直接使用/data/origin/global_region.csv文件即可
# ip2region.db文件的目录是可选参数没有指定的话会在当前目录生成一份./data/ip2region.db文件
```
3. 获取生成的ip2region.db文件覆盖原来的ip2region.db文件即可
4. 默认的ip2region.db文件生成命令:
```shell
cd ${ip2region_root}/java/
java -jar dbMaker-1.2.2.jar -src ./data/ip.merge.txt -region ./data/global_region.csv
# 会看到一大片的输出
```
# 相关备注
### 声明
ip2region重点在于<b>研究IP数据的存储设计和各种语言的查询实现</b>并没有原始IP数据的支撑数据来源请参考上面的描述升级数据需要很多IP的支撑并且会对原始平台造成一定量的请求压力本项目不保证及时的数据更新没有也不会有商用版本你可以使用自定义的数据导入ip2region进行自定义查询的实现。
### 技术交流
1. 数据库文件的结构和原理请阅读 @冬芽 的blog[“ip2region数据库文件的结构和原理”](https://github.com/dongyado/dongyado.github.io/blob/master/_posts/2016-08-18-structure-of-ip2region-database-file.md)[ip2region数据结构设计和实现视频分享](https://www.bilibili.com/video/BV1wv4y1N7SD)
2. ip2region交流分享微信lionsoul2014(请备注ip2region)QQ1187582057(很少关注)
3. 基于检测算法的数据更新方式视频分享:[数据更新实现视频分享part1](https://www.bilibili.com/video/BV1934y1E7Q5/)[数据更新实现视频分享part2](https://www.bilibili.com/video/BV1pF411j7Aw/)

197
v1.0/README_en.md Normal file
View File

@ -0,0 +1,197 @@
# What is Ip2region?
Read this in other languages: [简体中文](README.md) | [English](README_en.md)
ip2region - offline IP address location library with 99.9% accuracy, 0.0x millisecond queries, ip2region.db database is only a few MB, provides query bindings for
```
java|php|c|python|nodejs|golang|c#
```
and three query algorithms for
```
Binary|B-tree|memory
```
# Ip2region features
### 99.9% accuracy rate
The data is aggregated from some well-known ip to place name lookup providers, these are their official accuracy rates, tested to be actually a bit more accurate than the classic pure IP location.
The data of ip2region is aggregated from the open API or data of the following service providers (2 to 4 requests per second for the upgrade program):
01. &gt;80% - [Taobao IP address database](http://ip.taobao.com/) <br />
02. ≈10% - [GeoIP](https://geoip.com/) <br />
03. ≈2% - [Pure IP Library](http://www.cz88.net/) <br />
<b>Remarks:</b> If none of the above open APIs or data give open data ip2region will stop the update service for the data.
### Standardized data format
Each ip data segment has a fixed format of.
```
_cityId|country|region|province|city|ISP_
```
Only the data for China is accurate to the city, some of the data for other countries can only be located to the country, and all the options before the latter are 0, which already contains all the big and small countries you can check (please ignore the city Id in front, personal project requirements).
### Small size
Contains all the IPs, the generated database file ip2region.db is only a few MB, the smallest version is only 1.5 MB, with the increase in the level of detail of the data the size of the database is slowly increasing, currently not more than 8 MB.
### Fast query speed
All of the query client single query are in the 0.x millisecond level, built-in three query algorithms
1. memory algorithm: the entire database is loaded into memory, single query are within 0.1x milliseconds, C language client single query in 0.00x milliseconds level.
2. binary algorithm: based on dichotomous lookup, based on ip2region.db file, no need to load memory, single query in 0.x milliseconds level.
3. b-tree algorithm: based on btree algorithm, based on ip2region.db file, no need to load memory, word query at 0.x milliseconds level, faster than binary algorithm.
Any client b-tree is faster than binary algorithm, of course memory algorithm is certainly the fastest!
### Multi-query client support
Clients already integrated are:
```
java, C#, php, c, python, nodejs, php extensions (php5 and php7), golang, rust, lua, lua_c, nginx.
```
binding | description | development status | binary query time consuming | b-tree query time consuming | memory query time consuming
:-: | :-: | :-: | :-: | :-: | :-:
[c](binding/c) | ANSC c binding | completed | 0.0x ms | 0.0x ms | 0.00x ms
[c#](binding/c#) | c# binding | completed | 0.x ms | 0.x ms | 0.1x ms
[golang](binding/golang) | golang binding | done | 0.x ms | 0.x ms | 0.1x ms
[java](binding/java) | java binding | completed | 0.x ms | 0.x ms | 0.1x ms
[lua](binding/lua) | lua implementation of binding | done | 0.x ms | 0.x ms | 0.x ms
[lua_c](binding/lua_c) | lua's c extension | done | 0.0x ms | 0.0x ms | 0.00x ms
[nginx](binding/nginx) | c extensions for nginx | done | 0.0x ms | 0.0x ms | 0.00x ms
[nodejs](binding/nodejs) | nodejs | completed | 0.x ms | 0.x ms | 0.1x ms
[php](binding/php) | php implementation of binding | done | 0.x ms | 0.1x ms | 0.1x ms
[php5_ext](binding/php5_ext) | c extensions for php5 | done | 0.0x ms | 0.0x ms | 0.00x ms
[php7_ext](binding/php7_ext) | c extensions for php7 | done | 0.0x ms | 0.0x ms | 0.00x ms
[python](binding/python) | python bindng | done | 0.x ms | 0.x ms | 0.x ms
[rust](binding/rust) | rust binding | done | 0.x ms | 0.x ms | 0.x ms
# ip2region quick test
Please refer to the README instructions under each binding to run the cli test program, for example the C demo runs as follows.
```shell
cd binding/c/
gcc -g -O2 testSearcher.c ip2region.c
. /a.out ... /... /data/ip2region.db
```
You will see the following cli interface.
```
initializing B-tree ...
+----------------------------------+
| ip2region test script |
| Author: chenxin619315@gmail.com |
| Type 'quit' to exit program |
+----------------------------------+
p2region>> 101.105.35.57
2163|China|South|Guangdong|Shenzhen|Dr. Peng in 0.02295 millseconds
```
Enter the IP address to start the test, the first time will be slightly slow, after the run command access binary,memory to try other algorithms, it is recommended to use the b-tree algorithm, speed and concurrency requirements can use the memory algorithm, please refer to the test source code under different binding for specific integration.
# ip2region installation
Refer to the README documentation and test demos under each binding for details, and here are some available shortcut installations.
### maven repository address
```xml
<dependency>
<groupId>org.lionsoul</groupId>
<artifactId>ip2region</artifactId>
<version>1.7.2</version>
</dependency>
```
### nodejs
```
npm install node-ip2region --save
```
### nuget install
```shell
NuGet install IP2Region
```
### php composer
```shell
# Plugin from: https://github.com/zoujingli/ip2region
composer require zoujingli/ip2region
```
# ip2region Concurrent use
1. each search interface of all binding is <b>not</b> a thread-safe implementation, different threads can use it by creating different query objects. with high concurrency, the binary and b-tree algorithms may have too many open files error, please modify the kernel's maximum number of open files allowed (fs.file-max=a higher value), or use a persistent memory algorithm.
2. memorySearch interface, which performs a pre-query before publishing the object (essentially loading the ip2region.db file into memory), can be safely used in a multi-threaded environment.
# ip2region.db generation
Starting from version 1.8, ip2region has open sourced the java implementation of the ip2region.db generator, providing ant compilation support, which will result in the dbMaker-{version}.jar mentioned below, for those who need to study the generator or change the custom generation configuration please refer to ${ip2region_ root}/maker/java for the java source code.
Starting with ip2region version 1.2.2 there is an executable jar file dbMaker-{version}.jar committed inside, which is used to do this.
1. make sure you have the java environment installed (kids who don't play with Java will Google themselves to find a pull, temporary use, a matter of minutes)
2. cd to `${ip2region_root}/maker/java` and run the following command.
```shell
java -jar dbMaker-{version}.jar -src text data file -region geographic csv file [-dst directory of the generated ip2region.db file]
# text data file: the original text data file path of the db file, the self-contained ip2region.db file is generated from /data/ip.merge.txt, you can replace it with your own or change /data/ip.merge.txt to regenerate it
# geographic csv file: the purpose of this file is to facilitate the configuration of ip2region for data relationship storage, the data obtained contains a city_id, this can be directly used /data/origin/global_region.csv file
# ip2region.db file directory: is optional, if not specified, a copy will be generated in the current directory. /data/ip2region.db file
```
3. get the generated ip2region.db file to overwrite the original ip2region.db file
4. The default ip2region.db file generation command:
```shell
cd ${ip2region_root}/java/
java -jar dbMaker-1.2.2.jar -src . /data/ip.merge.txt -region . /data/global_region.csv
# You'll see a big chunk of output
```
# Related remarks
### Declaration
ip2region focuses on <b>
researching IP data storage design and query implementation in various languages</b>, there is no original IP data support, please refer to the description above for data sources, upgrading data requires a lot of IP support and will cause a certain amount of request pressure on the original platform, this project does not guarantee timely data updates, there is no and will not be a commercial version, you can use You can use the custom data import ip2region for custom query implementation.
### Technical Communication
1. the structure and principle of the database file, please read @冬芽's blog:
- [ip2region Database File Structure and Principle](https://github.com/dongyado/dongyado.github.io/blob/master/_posts/2016-08-18-structure-of-ip2region-database-file.md)
- [ip2region Data Structure Design and Implementation Video share](https://www.bilibili.com/video/BV1wv4y1N7SD)
2. ip2region exchange and sharing
- WeChat: lionsoul2014 (please note ip2region)
- QQ: 1187582057 (little attention)
3. based on the detection algorithm of the data update way video sharing:
- [Data Update to Achieve Video Sharing Part1](https://www.bilibili.com/video/BV1934y1E7Q5/)
- [Data Update to Achieve Video Sharing part2](https://www.bilibili.com/video/BV1pF411j7Aw/)

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

View File

@ -0,0 +1,13 @@
<?xml version='1.0' encoding='UTF-8'?>
<Library LVVersion="19008000">
<Property Name="NI.Lib.Icon" Type="Bin">'1#!!!!!!!)!"1!&amp;!!!-!%!!!@````]!!!!"!!%!!!(]!!!*Q(C=\&gt;7R=2MR%!81N=?"5Q&lt;/07RB7W!,&lt;'&amp;&lt;9+K1,7Q,&lt;)%N&lt;!NMA3X)DW?-RJ(JQ"I\%%Z,(@`BA#==ZB3RN;]28_,V7@PWW`:R`&gt;HV*SU_WE@\N_XF[3:^^TX\+2YP)D7K6;G-RV3P)R`ZS%=_]J'XP/5N&lt;XH,7V\SEJ?]Z#5P?=J4HP+5JTTFWS%0?=B$DD1G(R/.1==!IT.+D)`B':\B'2Z@9XC':XC':XBUC?%:HO%:HO&amp;R7QT0]!T0]!S0I4&lt;*&lt;)?=:XA-(]X40-X40-VDSGC?"GC4N9(&lt;)"D2,L;4ZGG?ZH%;T&gt;-]T&gt;-]T?.S.%`T.%`T.)^&lt;NF8J4@-YZ$S'C?)JHO)JHO)R&gt;"20]220]230[;*YCK=ASI2F=)1I.Z5/Z5PR&amp;)^@54T&amp;5TT&amp;5TQO&lt;5_INJ6Z;"[(H#&gt;ZEC&gt;ZEC&gt;Z$"(*ETT*ETT*9^B)HO2*HO2*(F.&amp;]C20]C2)GN4UE1:,.[:/+5A?0^NOS?UJ^3&lt;*\9B9GT@7JISVW7*NIFC&lt;)^:$D`5Q9TWE7)M@;V&amp;D,6;M29DVR]6#R],%GC47T9_/=@&gt;Z5V&gt;V57&gt;V5E&gt;V5(OV?^T[FTP?\`?YX7ZRP6\D=LH%_8S/U_E5R_-R$I&gt;$\0@\W/VW&lt;[_"\Y[X&amp;],0^^+,]T_J&gt;`J@_B_]'_.T`$KO.@I"XC-_N!!!!!!</Property>
<Property Name="NI.Lib.SourceVersion" Type="Int">419463168</Property>
<Property Name="NI.Lib.Version" Type="Str">1.0.0.0</Property>
<Property Name="NI.LV.All.SourceOnly" Type="Bool">false</Property>
<Item Name="binarySearch.vi" Type="VI" URL="../binarySearch.vi"/>
<Item Name="Close.vi" Type="VI" URL="../Close.vi"/>
<Item Name="example.vi" Type="VI" URL="../example.vi"/>
<Item Name="Get data.vi" Type="VI" URL="../Get data.vi"/>
<Item Name="ip2region.ctl" Type="VI" URL="../ip2region.ctl"/>
<Item Name="Open.vi" Type="VI" URL="../Open.vi"/>
</Library>

View File

@ -0,0 +1,7 @@
ip2region的LabVIEW库<br>
使用方法1. 将该目录下的所有文件复制到你的项目文件夹下;<br>
2. 导入ip2region.lvlib库文件<br>
3. 查看Example.vi了解使用方法<br>
<br>
<br>
注意该库依赖ni_lib_unicode-2.0.1.5.vip请使用VIPM安装。

View File

@ -0,0 +1,54 @@
using BenchmarkDotNet.Attributes;
using IP2Region.Models;
using System.Threading.Tasks;
namespace IP2Region.Test.Benchmark
{
[RankColumn]
public class DbSearch_Test : TestBase
{
private string RandomIP = "";
[Benchmark]
public DataBlock MemorySearch()
{
RandomIP = GetRandomIP();
return _search.MemorySearch(RandomIP);
}
[Benchmark]
public async Task<DataBlock> MemorySearch_Async()
{
RandomIP = GetRandomIP();
return await _search.MemorySearchAsync(RandomIP);
}
[Benchmark]
public DataBlock BinarySearch()
{
RandomIP = GetRandomIP();
return _search.BinarySearch(RandomIP);
}
[Benchmark]
public async Task<DataBlock> BinarySearch_Async()
{
RandomIP = GetRandomIP();
return await _search.BinarySearchAsync(RandomIP);
}
[Benchmark]
public DataBlock BtreeSearch()
{
RandomIP = GetRandomIP();
return _search.BtreeSearch(RandomIP);
}
[Benchmark]
public async Task<DataBlock> BtreeSearch_Async()
{
RandomIP = GetRandomIP();
return await _search.BtreeSearchAsync(RandomIP);
}
}
}

View File

@ -0,0 +1,16 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp2.0</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.11.0" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\IP2Region\IP2Region.csproj" />
</ItemGroup>
</Project>

View File

@ -0,0 +1,15 @@
using BenchmarkDotNet.Running;
using System;
namespace IP2Region.Test.Benchmark
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Now starting benchmark test, please wait......");
var summary = BenchmarkRunner.Run<DbSearch_Test>();
Console.ReadLine();
}
}
}

View File

@ -0,0 +1,49 @@
using BenchmarkDotNet.Attributes;
using System;
namespace IP2Region.Test.Benchmark
{
public class TestBase
{
protected DbSearcher _search;
private readonly string _dBFilePath = "";
public TestBase()
{
}
public TestBase(String DBFilePath)
{
_dBFilePath = DBFilePath;
}
[GlobalSetup]
public void Init()
{
if (String.IsNullOrEmpty(_dBFilePath))
{
_search = new DbSearcher(AppContext.BaseDirectory + @"\DB\ip2region.db");
}
else
{
_search = new DbSearcher(_dBFilePath);
}
}
[GlobalCleanup]
public void Dispose()
{
_search.Dispose();
}
public String GetRandomIP()
{
return new Random(Guid.NewGuid().GetHashCode()).Next(0, 255).ToString() + "."
+ new Random(Guid.NewGuid().GetHashCode()).Next(0, 255).ToString() + "."
+ new Random(Guid.NewGuid().GetHashCode()).Next(0, 255).ToString() + "."
+ new Random(Guid.NewGuid().GetHashCode()).Next(0, 255).ToString();
}
}
}

Binary file not shown.

View File

@ -0,0 +1,26 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp2.0</TargetFramework>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.7.0" />
<PackageReference Include="xunit" Version="2.3.1" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.3.1" />
<DotNetCliToolReference Include="dotnet-xunit" Version="2.3.1" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\IP2Region\IP2Region.csproj" />
</ItemGroup>
<ItemGroup>
<None Update="DB\ip2region.db">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>

View File

@ -0,0 +1,74 @@
using System;
using System.Threading.Tasks;
using Xunit;
namespace IP2Region.Test.xUnit
{
public class SearchTest : IDisposable
{
private readonly DbSearcher _search;
public String GetRandomIP()
{
return new Random(Guid.NewGuid().GetHashCode()).Next(0, 255).ToString() + "."
+ new Random(Guid.NewGuid().GetHashCode()).Next(0, 255).ToString() + "."
+ new Random(Guid.NewGuid().GetHashCode()).Next(0, 255).ToString() + "."
+ new Random(Guid.NewGuid().GetHashCode()).Next(0, 255).ToString();
}
public SearchTest()
{
_search = new DbSearcher(Environment.CurrentDirectory + @"\DB\ip2region.db");
}
[Fact]
public void Search_Test()
{
string memResult = _search.MemorySearch("223.104.246.20").Region;
string binarySearchResult = _search.BinarySearch("223.104.246.20").Region;
string binaryTreeSearchResult = _search.BtreeSearch("223.104.246.20").Region;
Assert.NotNull(memResult);
Assert.NotNull(binarySearchResult);
Assert.NotNull(binaryTreeSearchResult);
Assert.Equal(memResult, binarySearchResult);
Assert.Equal(binaryTreeSearchResult, memResult);
}
[Fact]
public void Search_Correct_Test()
{
for (int i = 0; i < 1000000; i++)
{
var newIp=GetRandomIP();
string memResult = _search.MemorySearch(newIp).Region;
string binarySearchResult = _search.BinarySearch(newIp).Region;
string binaryTreeSearchResult = _search.BtreeSearch(newIp).Region;
Assert.True(memResult == binarySearchResult && memResult == binaryTreeSearchResult);
}
}
[Fact]
public async Task SearchAsync_Test()
{
// We don't need the synchronizeContext, so just set to false
var memResult = await _search.MemorySearchAsync("183.192.62.65").ConfigureAwait(false);
var binarySearchResult = await _search.BinarySearchAsync("183.192.62.65").ConfigureAwait(false);
var bTreeSearchResult = await _search.BtreeSearchAsync("183.192.62.65").ConfigureAwait(false);
Assert.NotNull(memResult.Region);
Assert.NotNull(binarySearchResult.Region);
Assert.NotNull(bTreeSearchResult.Region);
Assert.Equal(memResult.Region, binarySearchResult.Region);
Assert.Equal(bTreeSearchResult.Region, memResult.Region);
}
public void Dispose()
{
_search.Dispose();
}
}
}

Binary file not shown.

View File

@ -0,0 +1,412 @@
//*******************************
// Created By Rocher Kong
// Github https://github.com/RocherKong
// Date 2018.02.09
//*******************************
using IP2Region.Models;
using System;
using System.IO;
using System.Text;
using System.Threading.Tasks;
namespace IP2Region
{
public class DbSearcher : IDisposable
{
const int BTREE_ALGORITHM = 1;
const int BINARY_ALGORITHM = 2;
const int MEMORY_ALGORITYM = 3;
private DbConfig _dbConfig = null;
/// <summary>
/// db file access handler
/// </summary>
private Stream _raf = null;
/// <summary>
/// header blocks buffer
/// </summary>
private long[] _headerSip = null;
private int[] _headerPtr = null;
private int _headerLength;
/// <summary>
/// super blocks info
/// </summary>
private long _firstIndexPtr = 0;
private long _lastIndexPtr = 0;
private int _totalIndexBlocks = 0;
/// <summary>
/// for memory mode
/// the original db binary string
/// </summary>
private byte[] _dbBinStr = null;
/// <summary>
/// Get by index ptr.
/// </summary>
private DataBlock GetByIndexPtr(long ptr)
{
_raf.Seek(ptr, SeekOrigin.Begin);
byte[] buffer = new byte[12];
_raf.Read(buffer, 0, buffer.Length);
long extra = Utils.GetIntLong(buffer, 8);
int dataLen = (int)((extra >> 24) & 0xFF);
int dataPtr = (int)((extra & 0x00FFFFFF));
_raf.Seek(dataPtr, SeekOrigin.Begin);
byte[] data = new byte[dataLen];
_raf.Read(data, 0, data.Length);
int city_id = (int)Utils.GetIntLong(data, 0);
string region = Encoding.UTF8.GetString(data, 4, data.Length - 4);
return new DataBlock(city_id, region, dataPtr);
}
public DbSearcher(DbConfig dbConfig, string dbFile)
{
if (_dbConfig == null)
{
_dbConfig = dbConfig;
}
_raf = new FileStream(dbFile, FileMode.Open, FileAccess.Read, FileShare.Read);
}
public DbSearcher(string dbFile) : this(null, dbFile) { }
public DbSearcher(DbConfig dbConfig, Stream dbFileStream)
{
if (_dbConfig == null)
{
_dbConfig = dbConfig;
}
_raf = dbFileStream;
}
public DbSearcher(Stream dbFileStream) : this(null, dbFileStream) { }
#region Sync Methods
/// <summary>
/// Get the region with a int ip address with memory binary search algorithm.
/// </summary>
private DataBlock MemorySearch(long ip)
{
int blen = IndexBlock.LENGTH;
if (_dbBinStr == null)
{
_dbBinStr = new byte[(int)_raf.Length];
_raf.Seek(0L, SeekOrigin.Begin);
_raf.Read(_dbBinStr, 0, _dbBinStr.Length);
//initialize the global vars
_firstIndexPtr = Utils.GetIntLong(_dbBinStr, 0);
_lastIndexPtr = Utils.GetIntLong(_dbBinStr, 4);
_totalIndexBlocks = (int)((_lastIndexPtr - _firstIndexPtr) / blen) + 1;
}
//search the index blocks to define the data
int l = 0, h = _totalIndexBlocks;
long sip = 0;
while (l <= h)
{
int m = (l + h) >> 1;
int p = (int)(_firstIndexPtr + m * blen);
sip = Utils.GetIntLong(_dbBinStr, p);
if (ip < sip)
{
h = m - 1;
}
else
{
sip = Utils.GetIntLong(_dbBinStr, p + 4);
if (ip > sip)
{
l = m + 1;
}
else
{
sip = Utils.GetIntLong(_dbBinStr, p + 8);
break;
}
}
}
//not matched
if (sip == 0) return null;
//get the data
int dataLen = (int)((sip >> 24) & 0xFF);
int dataPtr = (int)((sip & 0x00FFFFFF));
int city_id = (int)Utils.GetIntLong(_dbBinStr, dataPtr);
string region = Encoding.UTF8.GetString(_dbBinStr, dataPtr + 4, dataLen - 4);//new String(dbBinStr, dataPtr + 4, dataLen - 4, Encoding.UTF8);
return new DataBlock(city_id, region, dataPtr);
}
/// <summary>
/// Get the region throught the ip address with memory binary search algorithm.
/// </summary>
public DataBlock MemorySearch(string ip)
{
return MemorySearch(Utils.Ip2long(ip));
}
/// <summary>
/// Get the region with a int ip address with b-tree algorithm.
/// </summary>
private DataBlock BtreeSearch(long ip)
{
//check and load the header
if (_headerSip == null)
{
_raf.Seek(8L, SeekOrigin.Begin); //pass the super block
byte[] b = new byte[4096];
_raf.Read(b, 0, b.Length);
//fill the header
int len = b.Length >> 3, idx = 0; //b.lenght / 8
_headerSip = new long[len];
_headerPtr = new int[len];
long startIp, dataPtrTemp;
for (int i = 0; i < b.Length; i += 8)
{
startIp = Utils.GetIntLong(b, i);
dataPtrTemp = Utils.GetIntLong(b, i + 4);
if (dataPtrTemp == 0) break;
_headerSip[idx] = startIp;
_headerPtr[idx] = (int)dataPtrTemp;
idx++;
}
_headerLength = idx;
}
//1. define the index block with the binary search
if (ip == _headerSip[0])
{
return GetByIndexPtr(_headerPtr[0]);
}
else if (ip == _headerPtr[_headerLength - 1])
{
return GetByIndexPtr(_headerPtr[_headerLength - 1]);
}
int l = 0, h = _headerLength, sptr = 0, eptr = 0;
int m = 0;
while (l <= h)
{
m = (l + h) >> 1;
//perfectly matched, just return it
if (ip == _headerSip[m])
{
if (m > 0)
{
sptr = _headerPtr[m - 1];
eptr = _headerPtr[m];
}
else
{
sptr = _headerPtr[m];
eptr = _headerPtr[m + 1];
}
}
//less then the middle value
else if (ip < _headerSip[m])
{
if (m == 0)
{
sptr = _headerPtr[m];
eptr = _headerPtr[m + 1];
break;
}
else if (ip > _headerSip[m - 1])
{
sptr = _headerPtr[m - 1];
eptr = _headerPtr[m];
break;
}
h = m - 1;
}
else
{
if (m == _headerLength - 1)
{
sptr = _headerPtr[m - 1];
eptr = _headerPtr[m];
break;
}
else if (ip <= _headerSip[m + 1])
{
sptr = _headerPtr[m];
eptr = _headerPtr[m + 1];
break;
}
l = m + 1;
}
}
//match nothing just stop it
if (sptr == 0) return null;
//2. search the index blocks to define the data
int blockLen = eptr - sptr, blen = IndexBlock.LENGTH;
byte[] iBuffer = new byte[blockLen + blen]; //include the right border block
_raf.Seek(sptr, SeekOrigin.Begin);
_raf.Read(iBuffer, 0, iBuffer.Length);
l = 0; h = blockLen / blen;
long sip = 0;
int p = 0;
while (l <= h)
{
m = (l + h) >> 1;
p = m * blen;
sip = Utils.GetIntLong(iBuffer, p);
if (ip < sip)
{
h = m - 1;
}
else
{
sip = Utils.GetIntLong(iBuffer, p + 4);
if (ip > sip)
{
l = m + 1;
}
else
{
sip = Utils.GetIntLong(iBuffer, p + 8);
break;
}
}
}
//not matched
if (sip == 0) return null;
//3. get the data
int dataLen = (int)((sip >> 24) & 0xFF);
int dataPtr = (int)((sip & 0x00FFFFFF));
_raf.Seek(dataPtr, SeekOrigin.Begin);
byte[] data = new byte[dataLen];
_raf.Read(data, 0, data.Length);
int city_id = (int)Utils.GetIntLong(data, 0);
String region = Encoding.UTF8.GetString(data, 4, data.Length - 4);// new String(data, 4, data.Length - 4, "UTF-8");
return new DataBlock(city_id, region, dataPtr);
}
/// <summary>
/// Get the region throught the ip address with b-tree search algorithm.
/// </summary>
public DataBlock BtreeSearch(string ip)
{
return BtreeSearch(Utils.Ip2long(ip));
}
/// <summary>
/// Get the region with a int ip address with binary search algorithm.
/// </summary>
private DataBlock BinarySearch(long ip)
{
int blen = IndexBlock.LENGTH;
if (_totalIndexBlocks == 0)
{
_raf.Seek(0L, SeekOrigin.Begin);
byte[] superBytes = new byte[8];
_raf.Read(superBytes, 0, superBytes.Length);
//initialize the global vars
_firstIndexPtr = Utils.GetIntLong(superBytes, 0);
_lastIndexPtr = Utils.GetIntLong(superBytes, 4);
_totalIndexBlocks = (int)((_lastIndexPtr - _firstIndexPtr) / blen) + 1;
}
//search the index blocks to define the data
int l = 0, h = _totalIndexBlocks;
byte[] buffer = new byte[blen];
long sip = 0;
while (l <= h)
{
int m = (l + h) >> 1;
_raf.Seek(_firstIndexPtr + m * blen, SeekOrigin.Begin); //set the file pointer
_raf.Read(buffer, 0, buffer.Length);
sip = Utils.GetIntLong(buffer, 0);
if (ip < sip)
{
h = m - 1;
}
else
{
sip = Utils.GetIntLong(buffer, 4);
if (ip > sip)
{
l = m + 1;
}
else
{
sip = Utils.GetIntLong(buffer, 8);
break;
}
}
}
//not matched
if (sip == 0) return null;
//get the data
int dataLen = (int)((sip >> 24) & 0xFF);
int dataPtr = (int)((sip & 0x00FFFFFF));
_raf.Seek(dataPtr, SeekOrigin.Begin);
byte[] data = new byte[dataLen];
_raf.Read(data, 0, data.Length);
int city_id = (int)Utils.GetIntLong(data, 0);
String region = Encoding.UTF8.GetString(data, 4, data.Length - 4);//new String(data, 4, data.Length - 4, "UTF-8");
return new DataBlock(city_id, region, dataPtr);
}
/// <summary>
/// Get the region throught the ip address with binary search algorithm.
/// </summary>
public DataBlock BinarySearch(String ip)
{
return BinarySearch(Utils.Ip2long(ip));
}
#endregion
#region Async Methods
/// <summary>
/// Get the region throught the ip address with memory binary search algorithm.
/// </summary>
public Task<DataBlock> MemorySearchAsync(string ip)
{
return Task.FromResult(MemorySearch(ip));
}
/// <summary>
/// Get the region throught the ip address with b-tree search algorithm.
/// </summary>
public Task<DataBlock> BtreeSearchAsync(string ip)
{
return Task.FromResult(BtreeSearch(ip));
}
/// <summary>
/// Get the region throught the ip address with binary search algorithm.
/// </summary>
public Task<DataBlock> BinarySearchAsync(string ip)
{
return Task.FromResult(BinarySearch(ip));
}
#endregion
/// <summary>
/// Close the db.
/// </summary>
public void Close()
{
_headerSip = null;
_headerPtr = null;
_dbBinStr = null;
_raf.Close();
}
public void Dispose()
{
Close();
}
}
}

View File

@ -0,0 +1,26 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFrameworks>netstandard2.0;net46;net45;net47</TargetFrameworks>
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
<Authors>Rocher</Authors>
<Company>Rocher</Company>
<Description>准确率99.9%的ip地址定位库0.0x毫秒级查询, 支持Binary,B树,内存三种查询算法妈妈再也不用担心我的ip地址定位</Description>
<PackageReleaseNotes>ip2region - 最自由的ip地址查询库ip到地区的映射库提供Binary,B树和纯内存三种查询算法妈妈再也不用担心我的ip地址定位。</PackageReleaseNotes>
<Copyright>Rocher</Copyright>
<PackageTags>ip2region c# IP</PackageTags>
<RepositoryUrl>https://gitee.com/rocherkong/IP2Region</RepositoryUrl>
<Version>1.2.0</Version>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(TargetFramework)|$(Platform)'=='Debug|netstandard2.0|AnyCPU'">
<LangVersion>7</LangVersion>
</PropertyGroup>
<ItemGroup>
<None Update="DB\ip2region.db">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>

View File

@ -0,0 +1,50 @@
//*******************************
// Created By Rocher Kong
// Github https://github.com/RocherKong
// Date 2018.02.09
//*******************************
namespace IP2Region.Models
{
public class DataBlock
{
#region Private Properties
public int CityID
{
get;
private set;
}
public string Region
{
get;
private set;
}
public int DataPtr
{
get;
private set;
}
#endregion
#region Constructor
public DataBlock(int city_id, string region, int dataPtr = 0)
{
CityID = city_id;
Region = region;
DataPtr = dataPtr;
}
public DataBlock(int city_id, string region):this(city_id,region,0)
{
}
#endregion
public override string ToString()
{
return $"{CityID}|{Region}|{DataPtr}";
}
}
}

View File

@ -0,0 +1,49 @@
//*******************************
// Created By Rocher Kong
// Github https://github.com/RocherKong
// Date 2018.02.09
//*******************************
using System;
namespace IP2Region.Models
{
public class DbMakerConfigException : Exception
{
public string ErrMsg { get; private set; }
public DbMakerConfigException(string errMsg)
{
ErrMsg = errMsg;
}
}
public class DbConfig
{
public int TotalHeaderSize
{
get;
private set;
}
public int indexBlockSize
{
get;
private set;
}
public DbConfig(int totalHeaderSize)
{
if ((totalHeaderSize % 8) != 0)
{
throw new DbMakerConfigException("totalHeaderSize must be times of 8");
}
TotalHeaderSize = totalHeaderSize;
//4 * 2048
indexBlockSize = 8192;
}
public DbConfig():this(8 * 2048)
{
}
}
}

View File

@ -0,0 +1,49 @@
//*******************************
// Created By Rocher Kong
// Github https://github.com/RocherKong
// Date 2018.02.09
//*******************************
namespace IP2Region.Models
{
internal class HeaderBlock
{
public long IndexStartIp
{
get;
private set;
}
public int IndexPtr
{
get;
private set;
}
public HeaderBlock(long indexStartIp, int indexPtr)
{
IndexStartIp = indexStartIp;
IndexPtr = indexPtr;
}
/// <summary>
/// Get the bytes for total storage
/// </summary>
/// <returns>
/// Bytes gotten.
/// </returns>
public byte[] GetBytes()
{
/*
* +------------+-----------+
* | 4bytes | 4bytes |
* +------------+-----------+
* start ip index ptr
*/
byte[] b = new byte[8];
Utils.WriteIntLong(b, 0, IndexStartIp);
Utils.WriteIntLong(b, 4, IndexPtr);
return b;
}
}
}

View File

@ -0,0 +1,65 @@
//*******************************
// Created By Rocher Kong
// Github https://github.com/RocherKong
// Date 2018.02.09
//*******************************
namespace IP2Region
{
internal class IndexBlock
{
public const int LENGTH = 12;
public long StartIP
{
get;
private set;
}
public long EndIp
{
get;
private set;
}
public uint DataPtr
{
get;
private set;
}
public int DataLen
{
get;
private set;
}
public IndexBlock(long startIp, long endIp, uint dataPtr, int dataLen)
{
StartIP = startIp;
EndIp = endIp;
DataPtr = dataPtr;
DataLen = dataLen;
}
public byte[] GetBytes()
{
/*
* +------------+-----------+-----------+
* | 4bytes | 4bytes | 4bytes |
* +------------+-----------+-----------+
* start ip end ip data ptr + len
*/
byte[] b = new byte[12];
Utils.WriteIntLong(b, 0, StartIP); //start ip
Utils.WriteIntLong(b, 4, EndIp); //end ip
//write the data ptr and the length
long mix = DataPtr | ((DataLen << 24) & 0xFF000000L);
Utils.WriteIntLong(b, 8, mix);
return b;
}
}
}

View File

@ -0,0 +1,122 @@
//*******************************
// Created By Rocher Kong
// Github https://github.com/RocherKong
// Date 2018.02.09
//*******************************
using System;
namespace IP2Region
{
public class IPInValidException : Exception
{
const string ERROR_MSG = "IP Illigel. Please input a valid IP.";
public IPInValidException() : base(ERROR_MSG) { }
}
internal static class Utils
{
/// <summary>
/// Write specfield bytes to a byte array start from offset.
/// </summary>
public static void Write(byte[] b, int offset, ulong v, int bytes)
{
for (int i = 0; i < bytes; i++)
{
b[offset++] = (byte)((v >> (8 * i)) & 0xFF);
}
}
/// <summary>
/// Write a int to a byte array.
/// </summary>
public static void WriteIntLong(byte[] b, int offset, long v)
{
b[offset++] = (byte)((v >> 0) & 0xFF);
b[offset++] = (byte)((v >> 8) & 0xFF);
b[offset++] = (byte)((v >> 16) & 0xFF);
b[offset] = (byte)((v >> 24) & 0xFF);
}
/// <summary>
/// Get a int from a byte array start from the specifiled offset.
/// </summary>
public static long GetIntLong(byte[] b, int offset)
{
return (
((b[offset++] & 0x000000FFL)) |
((b[offset++] << 8) & 0x0000FF00L) |
((b[offset++] << 16) & 0x00FF0000L) |
((b[offset] << 24) & 0xFF000000L)
);
}
/// <summary>
/// Get a int from a byte array start from the specifield offset.
/// </summary>
public static int GetInt3(byte[] b, int offset)
{
return (
(b[offset++] & 0x000000FF) |
(b[offset++] & 0x0000FF00) |
(b[offset] & 0x00FF0000)
);
}
public static int GetInt2(byte[] b, int offset)
{
return (
(b[offset++] & 0x000000FF) |
(b[offset] & 0x0000FF00)
);
}
public static int GetInt1(byte[] b, int offset)
{
return (
(b[offset] & 0x000000FF)
);
}
/// <summary>
/// String ip to long ip.
/// </summary>
public static long Ip2long(string ip)
{
string[] p = ip.Split('.');
if (p.Length != 4) throw new IPInValidException();
foreach (string pp in p)
{
if (pp.Length > 3) throw new IPInValidException();
if (!int.TryParse(pp, out int value) || value > 255)
{
throw new IPInValidException();
}
}
var bip1 = long.TryParse(p[0], out long ip1);
var bip2 = long.TryParse(p[1], out long ip2);
var bip3 = long.TryParse(p[2], out long ip3);
var bip4 = long.TryParse(p[3], out long ip4);
if (!bip1 || !bip2 || !bip3 || !bip4
|| ip4 > 255 || ip1 > 255 || ip2 > 255 || ip3 > 255
|| ip4 < 0 || ip1 < 0 || ip2 < 0 || ip3 < 0)
{
throw new IPInValidException();
}
long p1 = ((ip1 << 24) & 0xFF000000);
long p2 = ((ip2 << 16) & 0x00FF0000);
long p3 = ((ip3 << 8) & 0x0000FF00);
long p4 = ((ip4 << 0) & 0x000000FF);
return ((p1 | p2 | p3 | p4) & 0xFFFFFFFFL);
}
/// <summary>
/// Int to ip string.
/// </summary>
public static string Long2ip(long ip)
{
return $"{(ip >> 24) & 0xFF}.{(ip >> 16) & 0xFF}.{(ip >> 8) & 0xFF}.{ip & 0xFF}";
}
}
}

View File

@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
</startup>
</configuration>

Binary file not shown.

View File

@ -0,0 +1,60 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
<PropertyGroup>
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
<ProjectGuid>{2D73D7EB-AE36-4039-A839-DA40209052A4}</ProjectGuid>
<OutputType>Exe</OutputType>
<RootNamespace>IP2Region_NetFx_Test</RootNamespace>
<AssemblyName>IP2Region_NetFx_Test</AssemblyName>
<TargetFrameworkVersion>v4.5</TargetFrameworkVersion>
<FileAlignment>512</FileAlignment>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
<PlatformTarget>AnyCPU</PlatformTarget>
<DebugSymbols>true</DebugSymbols>
<DebugType>full</DebugType>
<Optimize>false</Optimize>
<OutputPath>bin\Debug\</OutputPath>
<DefineConstants>DEBUG;TRACE</DefineConstants>
<ErrorReport>prompt</ErrorReport>
<WarningLevel>4</WarningLevel>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
<PlatformTarget>AnyCPU</PlatformTarget>
<DebugType>pdbonly</DebugType>
<Optimize>true</Optimize>
<OutputPath>bin\Release\</OutputPath>
<DefineConstants>TRACE</DefineConstants>
<ErrorReport>prompt</ErrorReport>
<WarningLevel>4</WarningLevel>
</PropertyGroup>
<ItemGroup>
<Reference Include="System" />
<Reference Include="System.Core" />
<Reference Include="System.Xml.Linq" />
<Reference Include="System.Data.DataSetExtensions" />
<Reference Include="Microsoft.CSharp" />
<Reference Include="System.Data" />
<Reference Include="System.Net.Http" />
<Reference Include="System.Xml" />
</ItemGroup>
<ItemGroup>
<Compile Include="Program.cs" />
<Compile Include="Properties\AssemblyInfo.cs" />
</ItemGroup>
<ItemGroup>
<None Include="App.config" />
<None Include="DB\ip2region.db">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\IP2Region\IP2Region.csproj">
<Project>{a27d09f4-4da8-4925-90ec-43e5f5288630}</Project>
<Name>IP2Region</Name>
</ProjectReference>
</ItemGroup>
<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
</Project>

View File

@ -0,0 +1,17 @@
using IP2Region;
using System;
namespace IP2Region_NetFx_Test
{
class Program
{
static void Main(string[] args)
{
using (var _search = new DbSearcher(Environment.CurrentDirectory + @"\DB\ip2region.db"))
{
Console.WriteLine(_search.MemorySearch("183.192.62.65").Region);
Console.Read();
}
}
}
}

View File

@ -0,0 +1,35 @@
using System.Reflection;
using System.Runtime.InteropServices;
// 有关程序集的一般信息由以下
// 控制。更改这些特性值可修改
// 与程序集关联的信息。
[assembly: AssemblyTitle("IP2Region_NetFx_Test")]
[assembly: AssemblyDescription("")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("")]
[assembly: AssemblyProduct("IP2Region_NetFx_Test")]
[assembly: AssemblyCopyright("Copyright © 2018")]
[assembly: AssemblyTrademark("")]
[assembly: AssemblyCulture("")]
// 将 ComVisible 设置为 false 会使此程序集中的类型
//对 COM 组件不可见。如果需要从 COM 访问此程序集中的类型
//请将此类型的 ComVisible 特性设置为 true。
[assembly: ComVisible(false)]
// 如果此项目向 COM 公开,则下列 GUID 用于类型库的 ID
[assembly: Guid("2d73d7eb-ae36-4039-a839-da40209052a4")]
// 程序集的版本信息由下列四个值组成:
//
// 主版本
// 次版本
// 生成号
// 修订号
//
// 可以指定所有值,也可以使用以下所示的 "*" 预置版本号和修订号
// 方法是按如下所示使用“*”: :
// [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]

42
v1.0/binding/c#/README.md Normal file
View File

@ -0,0 +1,42 @@
# IP2Region C# Client
## How To Use
### 1.Install from Nuget. Support .Net Framework (>=4.5) And netstandard 2.0(.net core)
```powershell
Install-Package IP2Region
```
### 2.Init DbSearcher with [the newest DBFile](https://github.com/lionsoul2014/ip2region/blob/master/data/ip2region.db) Downloaded into your project, and invoke your search methods as below:
```csharp
using (var _search = new DbSearcher(Environment.CurrentDirectory + @"\DB\ip2region.db"))
{
_search.MemorySearch("183.192.62.65").Region;
_search.BinarySearch("183.192.62.65").Region;
_search.BtreeSearch("183.192.62.65").Region;
}
```
For async methods (Methods with the suffix "Async"), just put them with `async` and `await`.
If you don't need the `SynchronizedContext`, please use `ConfigureAwait(false)`.
For more about this, please read https://msdn.microsoft.com/en-us/magazine/jj991977.aspx.
```csharp
public async Task SearchAsync_Test()
{
// We don't need the synchronizeContext, so just set to false.
// So as for BinarySearchAsync and BtreeSearchAsync
var memResult = await _search.MemorySearchAsync("183.192.62.65").ConfigureAwait(false);
}
```
## Test Result(From /IP2Region.Test.Benchmark)
Method | Mean | Error | StdDev | Rank |
------------------- |---------: |----------: |----------: |-----: |
MemorySearch | 10.66 us | 0.1424 us | 0.1332 us | 1 |
MemorySearch_Async | 10.90 us | 0.2060 us | 0.2023 us | 2 |
BinarySearch | 52.22 us | 0.6403 us | 0.5347 us | 5 |
BinarySearch_Async | 53.03 us | 1.0271 us | 0.9608 us | 6 |
BtreeSearch | 19.05 us | 0.2464 us | 0.2305 us | 3 |
BtreeSearch_Async | 19.40 us | 0.3820 us | 0.6690 us | 4 |
Because the speed of the program lookup has been fast, the asynchronous method here is just the packaging of the synchronization method, the essence is the synchronization method, the asynchronous method is not recommended, and asynchronous will bring the performance loss, unless you need the asynchronous call format.

26
v1.0/binding/c/Makefile Normal file
View File

@ -0,0 +1,26 @@
A = @
CFLAGS= -g
CC= gcc
LIBS= -I./
LDFLAGS=
RM = rm -f
MV = mv
testSearcher: ip2region.o
$(A) $(CC) $(CFLAGS) $(LDFLAGS) $(LIBS) -o testSearcher testSearcher.c ip2region.c
$(A) $(RM) ip2region.o testSearcher.o
db:
$(A) cd ../../;\
export LANG=en_US.UTF-8;\
java -jar dbMaker-1.2.2.jar -src ./data/ip.merge.txt -region ./data/global_region.csv -dst .;\
$(MV) ./ip2region.db ./binding/c/;\
cd ./binding/c
all: db testSearcher
clean:
$(A) $(RM) testSearcher
.PHONY: all clean testSearcher db

View File

@ -1,8 +1,8 @@
/** /**
* default ip2region implementation * default ip2region implementation
* *
* @see #ip2region.h * @see #ip2region.h
* @author chenxin<chenxin619315@gmail.com> * @author chenxin<chenxin619315@gmail.com>
* @date 2015-10-30 * @date 2015-10-30
*/ */
@ -13,9 +13,9 @@
/** /**
* create a new ip2region object * create a new ip2region object
* *
* @param dbFile path * @param dbFile path
*/ */
IP2R_API uint_t ip2region_create(ip2region_t ip2rObj, char *dbFile) IP2R_API uint_t ip2region_create(ip2region_t ip2rObj, const char *dbFile)
{ {
memset(ip2rObj, 0x00, sizeof(ip2region_entry)); memset(ip2rObj, 0x00, sizeof(ip2region_entry));
ip2rObj->headerLen = 0; ip2rObj->headerLen = 0;
@ -51,22 +51,27 @@ IP2R_API uint_t ip2region_create(ip2region_t ip2rObj, char *dbFile)
/** /**
* destroy the specifield ip2region object * destroy the specifield ip2region object
* *
* @param ip2region_t * @param ip2region_t
*/ */
IP2R_API uint_t ip2region_destroy(ip2region_t ip2rObj) IP2R_API uint_t ip2region_destroy(ip2region_t ip2rObj)
{ {
IP2R_FREE(ip2rObj->HeaderSip); if ( ip2rObj->HeaderSip != NULL ) {
ip2rObj->HeaderSip = NULL; IP2R_FREE(ip2rObj->HeaderSip);
IP2R_FREE(ip2rObj->HeaderPtr); ip2rObj->HeaderSip = NULL;
ip2rObj->HeaderPtr = NULL; }
//close the db file resource if ( ip2rObj->HeaderPtr != NULL ) {
IP2R_FREE(ip2rObj->HeaderPtr);
ip2rObj->HeaderPtr = NULL;
}
// close the db file resource
if ( ip2rObj->dbHandler != NULL ) { if ( ip2rObj->dbHandler != NULL ) {
fclose(ip2rObj->dbHandler); fclose(ip2rObj->dbHandler);
ip2rObj->dbHandler = NULL; ip2rObj->dbHandler = NULL;
} }
//free the db binary string // free the db binary string
if ( ip2rObj->dbBinStr != NULL ) { if ( ip2rObj->dbBinStr != NULL ) {
IP2R_FREE(ip2rObj->dbBinStr); IP2R_FREE(ip2rObj->dbBinStr);
ip2rObj->dbBinStr = NULL; ip2rObj->dbBinStr = NULL;
@ -149,7 +154,7 @@ IP2R_API uint_t ip2region_memory_search(ip2region_t ip2rObj, uint_t ip, databloc
return 1; return 1;
} }
IP2R_API uint_t ip2region_memory_search_string(ip2region_t ip2rObj, char *ip, datablock_t datablock) IP2R_API uint_t ip2region_memory_search_string(ip2region_t ip2rObj, const char *ip, datablock_t datablock)
{ {
return ip2region_memory_search(ip2rObj, ip2long(ip), datablock); return ip2region_memory_search(ip2rObj, ip2long(ip), datablock);
} }
@ -157,10 +162,10 @@ IP2R_API uint_t ip2region_memory_search_string(ip2region_t ip2rObj, char *ip, da
/** /**
* get the region associated with the specifield ip address with binary search algorithm * get the region associated with the specifield ip address with binary search algorithm
* *
* @param ip2rObj * @param ip2rObj
* @param ip * @param ip
* @param datablock * @param datablock
* @return uint_t * @return uint_t
*/ */
IP2R_API uint_t ip2region_binary_search(ip2region_t ip2rObj, uint_t ip, datablock_t datablock) IP2R_API uint_t ip2region_binary_search(ip2region_t ip2rObj, uint_t ip, datablock_t datablock)
{ {
@ -226,7 +231,7 @@ IP2R_API uint_t ip2region_binary_search(ip2region_t ip2rObj, uint_t ip, databloc
return 1; return 1;
} }
IP2R_API uint_t ip2region_binary_search_string(ip2region_t ip2rObj, char *ip, datablock_t datablock) IP2R_API uint_t ip2region_binary_search_string(ip2region_t ip2rObj, const char *ip, datablock_t datablock)
{ {
return ip2region_binary_search(ip2rObj, ip2long(ip), datablock); return ip2region_binary_search(ip2rObj, ip2long(ip), datablock);
} }
@ -234,10 +239,10 @@ IP2R_API uint_t ip2region_binary_search_string(ip2region_t ip2rObj, char *ip, da
/** /**
* get the region associated with the specifield ip address with b-tree algorithm * get the region associated with the specifield ip address with b-tree algorithm
* *
* @param ip2rObj * @param ip2rObj
* @param ip * @param ip
* @param datablock * @param datablock
* @return uint_t * @return uint_t
*/ */
IP2R_API uint_t ip2region_btree_search(ip2region_t ip2rObj, uint_t ip, datablock_t datablock) IP2R_API uint_t ip2region_btree_search(ip2region_t ip2rObj, uint_t ip, datablock_t datablock)
{ {
@ -355,7 +360,7 @@ IP2R_API uint_t ip2region_btree_search(ip2region_t ip2rObj, uint_t ip, datablock
return 1; return 1;
} }
IP2R_API uint_t ip2region_btree_search_string(ip2region_t ip2rObj, char *ip, datablock_t datablock) IP2R_API uint_t ip2region_btree_search_string(ip2region_t ip2rObj, const char *ip, datablock_t datablock)
{ {
return ip2region_btree_search(ip2rObj, ip2long(ip), datablock); return ip2region_btree_search(ip2rObj, ip2long(ip), datablock);
} }
@ -363,11 +368,11 @@ IP2R_API uint_t ip2region_btree_search_string(ip2region_t ip2rObj, char *ip, dat
/** /**
* get a unsinged long(4bytes) from a specifield buffer start from the specifield offset * get a unsinged long(4bytes) from a specifield buffer start from the specifield offset
* *
* @param buffer * @param buffer
* @param offset * @param offset
* @return uint_t * @return uint_t
*/ */
IP2R_API uint_t getUnsignedInt(char *buffer, int offset) IP2R_API uint_t getUnsignedInt(const char *buffer, int offset)
{ {
return ( return (
((buffer[offset ]) & 0x000000FF) | ((buffer[offset ]) & 0x000000FF) |
@ -380,22 +385,28 @@ IP2R_API uint_t getUnsignedInt(char *buffer, int offset)
/** /**
* string ip to long * string ip to long
* *
* @param ip * @param ip
* @return uint_t * @return uint_t
*/ */
IP2R_API uint_t ip2long(char *ip) IP2R_API uint_t ip2long(const char *ip)
{ {
int i = 0, p = 24; int i = 0, p = 24;
char buffer[4], *cs = ip; char buffer[4];
const char *cs = ip;
uint_t ipval = 0; uint_t ipval = 0;
while ( *cs != '\0' ) { while ( *cs != '\0' ) {
if ( *cs == ' ' ) {
cs++;
continue;
}
if ( i > 3 ) {
ipval = 0;
break;
}
if ( *cs == '.' ) { if ( *cs == '.' ) {
//single part length limit //single part length limit
if ( i > 3 ) {
ipval = 0;
break;
}
if ( p < 0 ) break; if ( p < 0 ) break;
buffer[i] = '\0'; buffer[i] = '\0';
@ -420,9 +431,9 @@ IP2R_API uint_t ip2long(char *ip)
/** /**
* long to string ip * long to string ip
* *
* @param ip * @param ip
* @param buffer * @param buffer
* @return uint_t(1 for success and 0 for failed) * @return uint_t(1 for success and 0 for failed)
*/ */
IP2R_API uint_t long2ip(uint_t ip, char *buffer) IP2R_API uint_t long2ip(uint_t ip, char *buffer)
{ {

152
v1.0/binding/c/ip2region.h Normal file
View File

@ -0,0 +1,152 @@
/**
* ip2region header file
*
* @author chenxin<chenxin619315@gmail.com>
*/
#ifndef _IP2REGION_H
#define _IP2REGION_H
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
//yat, just take it as this way, 99 percent and they works find
#if ( defined(_WIN32) || defined(_WINDOWS_) || defined(__WINDOWS_) )
# define OS_WINDOW
#else
# define OS_LINUX
#endif
#ifdef OS_WINDOW
# define IP2R_API extern __declspec(dllexport)
#else
/*platform shared library statement :: unix*/
# define IP2R_API extern
#endif
#define print(str) printf("%s", str)
#define println(str) printf("%s\n", str)
/*
* memory allocation macro definition.
* cause we should use emalloc,ecalloc .ege. in php.
* so you could make it better apdat the php environment.
*/
#define IP2R_CALLOC(_bytes, _blocks) calloc(_bytes, _blocks)
#define IP2R_MALLOC(_bytes) malloc(_bytes)
#define IP2R_FREE(_ptr) free(_ptr)
typedef unsigned short ushort_t;
typedef unsigned char uchar_t;
typedef unsigned int uint_t;
typedef unsigned long ulong_t;
#define INDEX_BLOCK_LENGTH 12
//thats 8 * 1024
#define TOTAL_HEADER_LENGTH 8192
/*
* ip2region properties struct
*/
typedef struct {
uint_t *HeaderSip; //header start ip blocks
uint_t *HeaderPtr; //header ptr blocks
uint_t headerLen; //header block number
char *dbFile; //path of db file
FILE *dbHandler; //file handler
char *dbBinStr; //db binary string for memory search mode
uint_t firstIndexPtr; //first index ptr
uint_t lastIndexPtr; //last index ptr
uint_t totalBlocks; //total index blocks number
} ip2region_entry;
typedef ip2region_entry * ip2region_t;
/*
* data block
*/
typedef struct {
uint_t city_id;
char region[256];
} datablock_entry;
typedef datablock_entry * datablock_t;
/**
* create a new ip2region object
*
* @param ip2rObj
* @param dbFile path
*/
IP2R_API uint_t ip2region_create(ip2region_t, const char *);
/**
* destroy the specified ip2region object
*
* @param ip2region_t
*/
IP2R_API uint_t ip2region_destroy(ip2region_t);
/**
* get the region associated with the specified ip address with the memory binary search algorithm
*
* @param ip2region_t
* @param uint_t
* @param datablock_t
* @date 2016/06/30
*/
IP2R_API uint_t ip2region_memory_search(ip2region_t, uint_t, datablock_t);
IP2R_API uint_t ip2region_memory_search_string(ip2region_t, const char *, datablock_t);
/**
* get the region associated with the specified ip address with binary search algorithm
*
* @param ip2rObj
* @param ip
* @param datablock
* @return uint_t
*/
IP2R_API uint_t ip2region_binary_search(ip2region_t, uint_t, datablock_t);
IP2R_API uint_t ip2region_binary_search_string(ip2region_t, const char *, datablock_t);
/**
* get the region associated with the specified ip address with b-tree algorithm
*
* @param ip2rObj
* @param ip
* @param datablock
* @return uint_t
*/
IP2R_API uint_t ip2region_btree_search(ip2region_t, uint_t, datablock_t);
IP2R_API uint_t ip2region_btree_search_string(ip2region_t, const char *, datablock_t);
/**
* get a unsinged long(4bytes) from a specified buffer start from the specified offset
*
* @param buffer
* @param offset
* @return uint_t
*/
IP2R_API uint_t getUnsignedInt(const char *, int);
/**
* string ip to long
*
* @param ip
* @return uint_t
*/
IP2R_API uint_t ip2long(const char *);
/**
* long to string ip
*
* @param ip
* @param buffer
* @return uint_t(1 for success and 0 for failed)
*/
IP2R_API uint_t long2ip(uint_t, char *);
#endif /*end ifndef*/

View File

@ -1,7 +1,7 @@
/** /**
* test ip2region searcher program * test ip2region searcher program
* *
* @author chenxin<chenxin619315@gmail.com> * @author chenxin<chenxin619315@gmail.com>
* @date 2015-10-30 * @date 2015-10-30
*/ */
@ -49,7 +49,7 @@ int main( int argc, char **argv )
datablock_entry datablock; datablock_entry datablock;
char *dbFile = NULL, *algorithm = NULL; char *dbFile = NULL, *algorithm = NULL;
char line[256]; char line[256];
uint_t (*func_ptr)(ip2region_t, char *, datablock_t); uint_t (*func_ptr)(ip2region_t, const char *, datablock_t);
double s_time, c_time; double s_time, c_time;
memset(&datablock, 0x00, sizeof(datablock_entry)); memset(&datablock, 0x00, sizeof(datablock_entry));
@ -95,7 +95,7 @@ int main( int argc, char **argv )
printf("%d|%s in %.5f millseconds\n", datablock.city_id, datablock.region, c_time); printf("%d|%s in %.5f millseconds\n", datablock.city_id, datablock.region, c_time);
} }
//destory the ip2rObj // destory the ip2rObj
ip2region_destroy(&ip2rEntry); ip2region_destroy(&ip2rEntry);
return 0; return 0;

View File

@ -0,0 +1,328 @@
/**
* ip2region implementation with mmap support created by Leo Ma
*
* @see #ip2region.h
* @author chenxin<chenxin619315@gmail.com>
* @author Leo Ma<http://git.oschina.net/begeekmyfriend>
* @date 2017/01/12
*/
#include "ip2region.h"
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/mman.h>
/**
* create a new ip2region object
*
* @param dbFile path
*/
IP2R_API uint_t ip2region_create(ip2region_t ip2rObj, char *dbFile)
{
struct stat st;
memset(ip2rObj, 0x00, sizeof(ip2region_entry));
ip2rObj->dbFile = IP2R_MALLOC(1024);
strcpy(ip2rObj->dbFile, dbFile);
int fd = open(dbFile, O_RDONLY);
if ( fd < 0 ) {
//fprintf(stderr, "Fail to open the db file %s\n", ip2rObj->dbFile);
return 0;
}
if ( fstat(fd, &st) < 0 ) {
//fprintf(stderr, "Fail to stat the db file %s\n", ip2rObj->dbFile);
return 0;
}
ip2rObj->dbSize = st.st_size;
ip2rObj->dbBinStr = mmap(NULL, ip2rObj->dbSize, PROT_READ, MAP_PRIVATE, fd, 0);
if ( ip2rObj->dbBinStr == (void *) -1 ) {
//fprintf(stderr, "Fail to map the db file %s\n", ip2rObj->dbFile);
return 0;
}
// Close db file as soon as possible.
close(fd);
ip2rObj->headerLen = 0;
ip2rObj->HeaderSip = (uint_t *) IP2R_MALLOC(TOTAL_HEADER_LENGTH);
if ( ip2rObj->HeaderSip == NULL ) {
return 0;
}
ip2rObj->HeaderPtr = (uint_t *) IP2R_MALLOC(TOTAL_HEADER_LENGTH);
if ( ip2rObj->HeaderPtr == NULL ) {
IP2R_FREE(ip2rObj->HeaderSip);
return 0;
}
ip2rObj->firstIndexPtr = getUnsignedInt(ip2rObj->dbBinStr, 0);
ip2rObj->lastIndexPtr = getUnsignedInt(ip2rObj->dbBinStr, 4);
ip2rObj->totalBlocks = (ip2rObj->lastIndexPtr-ip2rObj->firstIndexPtr)/INDEX_BLOCK_LENGTH + 1;
return 1;
}
/**
* destroy the specifield ip2region object
*
* @param ip2region_t
*/
IP2R_API uint_t ip2region_destroy(ip2region_t ip2rObj)
{
IP2R_FREE(ip2rObj->dbFile);
IP2R_FREE(ip2rObj->HeaderSip);
IP2R_FREE(ip2rObj->HeaderPtr);
//free the db binary string
if ( ip2rObj->dbBinStr != NULL ) {
munmap(ip2rObj->dbBinStr, ip2rObj->dbSize);
}
memset(ip2rObj, 0x00, sizeof(ip2region_entry));
return 1;
}
/**
* get the region associated with the specified ip address with the memory binary search algorithm
*
* @param ip2rObj
* @param ip
* @param datablock
*/
IP2R_API uint_t ip2region_binary_search(ip2region_t ip2rObj, uint_t ip, datablock_t datablock)
{
int l, h, m, p;
uint_t sip, eip, dptr;
int dataLen, dataptr;
char *buffer;
l = 0; h = ip2rObj->totalBlocks; dptr = 0;
while ( l <= h ) {
m = (l + h) >> 1;
p = ip2rObj->firstIndexPtr + m * INDEX_BLOCK_LENGTH;
buffer = ip2rObj->dbBinStr + p;
sip = getUnsignedInt(buffer, 0);
if ( ip < sip ) {
h = m - 1;
} else {
eip = getUnsignedInt(buffer, 4);
if ( ip > eip ) {
l = m + 1;
} else {
dptr = getUnsignedInt(buffer, 8);
break;
}
}
}
if ( dptr == 0 ) return 0;
//get the data
dataLen = ((dptr >> 24) & 0xFF);
dataptr = (dptr & 0x00FFFFFF);
buffer = ip2rObj->dbBinStr + dataptr;
//fill the data to the datablock
datablock->city_id = getUnsignedInt(buffer, 0);
dataLen -= 4; //reduce the length of the city_id
memcpy(datablock->region, buffer + 4, dataLen);
datablock->region[dataLen] = '\0';
return 1;
}
IP2R_API uint_t ip2region_binary_search_string(ip2region_t ip2rObj, char *ip, datablock_t datablock)
{
return ip2region_binary_search(ip2rObj, ip2long(ip), datablock);
}
/**
* get the region associated with the specifield ip address with b-tree algorithm
*
* @param ip2rObj
* @param ip
* @param datablock
* @return uint_t
*/
IP2R_API uint_t ip2region_btree_search(ip2region_t ip2rObj, uint_t ip, datablock_t datablock)
{
int i, j;
int l, m, h, p, sptr, eptr, indexBlockLen, dataLen, dataptr;
uint_t sip, eip, idxptr, dptr;
char *buffer;
if ( ip2rObj->headerLen == 0 ) {
buffer = ip2rObj->dbBinStr + 8;
for ( i = j = 0; i < TOTAL_HEADER_LENGTH; i += 8, j++ ) {
sip = getUnsignedInt(buffer, i);
idxptr = getUnsignedInt(buffer, i + 4);
if ( idxptr == 0 ) break;
ip2rObj->HeaderSip[j] = sip;
ip2rObj->HeaderPtr[j] = idxptr;
}
ip2rObj->headerLen = j;
}
//search the header block to define the index block
l = 0; h = ip2rObj->headerLen; sptr = 0; eptr = 0;
while ( l <= h ) {
m = ((l + h) >> 1);
//perfetc matched, just return it
if ( ip == ip2rObj->HeaderSip[m] ) {
if ( m > 0 ) {
sptr = ip2rObj->HeaderPtr[m-1];
eptr = ip2rObj->HeaderPtr[m ];
} else {
sptr = ip2rObj->HeaderPtr[m ];
eptr = ip2rObj->HeaderPtr[m+1];
}
break;
}
//less then the middle value
if ( ip < ip2rObj->HeaderSip[m] ) {
if ( m == 0 ) {
sptr = ip2rObj->HeaderPtr[m ];
eptr = ip2rObj->HeaderPtr[m+1];
break;
} else if ( ip > ip2rObj->HeaderSip[m-1] ) {
sptr = ip2rObj->HeaderPtr[m-1];
eptr = ip2rObj->HeaderPtr[m ];
break;
}
h = m - 1;
} else {
if ( m == ip2rObj->headerLen - 1 ) {
sptr = ip2rObj->HeaderPtr[m-1];
eptr = ip2rObj->HeaderPtr[m ];
break;
} else if ( ip <= ip2rObj->HeaderSip[m+1] ) {
sptr = ip2rObj->HeaderPtr[m ];
eptr = ip2rObj->HeaderPtr[m+1];
break;
}
l = m + 1;
}
}
//not matched just stop it
if ( sptr == 0 ) return 0;
indexBlockLen = eptr - sptr;
buffer = ip2rObj->dbBinStr + sptr;
dptr = 0; l = 0; h = indexBlockLen / INDEX_BLOCK_LENGTH;
while ( l <= h ) {
m = ((l + h) >> 1);
p = m * INDEX_BLOCK_LENGTH;
sip = getUnsignedInt(buffer, p);
if ( ip < sip ) {
h = m - 1;
} else {
eip = getUnsignedInt(buffer, p + 4);
if ( ip > eip ) {
l = m + 1;
} else {
dptr = getUnsignedInt(buffer, p + 8);
break;
}
}
}
if ( dptr == 0 ) return 0;
dataLen = ((dptr >> 24) & 0xFF);
dataptr = (dptr & 0x00FFFFFF);
buffer = ip2rObj->dbBinStr + dataptr;
datablock->city_id = getUnsignedInt(buffer, 0);
dataLen -= 4;
memcpy(datablock->region, buffer + 4, dataLen);
datablock->region[dataLen] = '\0';
return 1;
}
IP2R_API uint_t ip2region_btree_search_string(ip2region_t ip2rObj, char *ip, datablock_t datablock)
{
return ip2region_btree_search(ip2rObj, ip2long(ip), datablock);
}
/**
* get a unsinged long(4bytes) from a specifield buffer start from the specifield offset
*
* @param buffer
* @param offset
* @return uint_t
*/
IP2R_API uint_t getUnsignedInt(char *buffer, int offset)
{
return (
((buffer[offset ]) & 0x000000FF) |
((buffer[offset+1] << 8) & 0x0000FF00) |
((buffer[offset+2] << 16) & 0x00FF0000) |
((buffer[offset+3] << 24) & 0xFF000000)
);
}
/**
* string ip to long
*
* @param ip
* @return uint_t
*/
IP2R_API uint_t ip2long(char *ip)
{
int i = 0, p = 24;
char buffer[4], *cs = ip;
uint_t ipval = 0;
while ( *cs != '\0' ) {
if ( *cs == '.' ) {
//single part length limit
if ( i > 3 ) {
ipval = 0;
break;
}
if ( p < 0 ) break;
buffer[i] = '\0';
ipval |= (atoi(buffer) << p);
p -= 8;
i = 0;
} else {
buffer[i++] = *cs;
}
cs++;
}
//append the rest parts
if ( i > 3 ) return 0;
buffer[i] = '\0';
ipval |= atoi(buffer);
return ipval;
}
/**
* long to string ip
*
* @param ip
* @param buffer
* @return uint_t(1 for success and 0 for failed)
*/
IP2R_API uint_t long2ip(uint_t ip, char *buffer)
{
return 0;
}

View File

@ -2,6 +2,7 @@
* ip2region header file * ip2region header file
* *
* @author chenxin<chenxin619315@gmail.com> * @author chenxin<chenxin619315@gmail.com>
* @author Leo Ma<http://git.oschina.net/begeekmyfriend>
*/ */
#ifndef _IP2REGION_H #ifndef _IP2REGION_H
@ -43,7 +44,7 @@ typedef unsigned int uint_t;
typedef unsigned long ulong_t; typedef unsigned long ulong_t;
#define INDEX_BLOCK_LENGTH 12 #define INDEX_BLOCK_LENGTH 12
#define TOTAL_HEADER_LENGTH 4096 #define TOTAL_HEADER_LENGTH 8192
/* /*
* ip2region properties struct * ip2region properties struct
@ -53,9 +54,8 @@ typedef struct {
uint_t *HeaderPtr; //header ptr blocks uint_t *HeaderPtr; //header ptr blocks
uint_t headerLen; //header block number uint_t headerLen; //header block number
char *dbFile; //path of db file char *dbFile; //path of db file
FILE *dbHandler; //file handler
char *dbBinStr; //db binary string for memory search mode char *dbBinStr; //db binary string for memory search mode
uint_t dbSize;
uint_t firstIndexPtr; //first index ptr uint_t firstIndexPtr; //first index ptr
uint_t lastIndexPtr; //last index ptr uint_t lastIndexPtr; //last index ptr
uint_t totalBlocks; //total index blocks number uint_t totalBlocks; //total index blocks number
@ -88,25 +88,13 @@ IP2R_API uint_t ip2region_create(ip2region_t, char *);
IP2R_API uint_t ip2region_destroy(ip2region_t); IP2R_API uint_t ip2region_destroy(ip2region_t);
/** /**
* get the region associated with the specified ip address with the memory binary search algorithm * get the region associated with the specified ip address with memory binary search algorithm
* *
* @param ip2region_t * @param ip2region_t
* @param uint_t * @param uint_t
* @param datablock_t * @param datablock_t
* @date 2016/06/30 * @date 2016/06/30
*/ */
IP2R_API uint_t ip2region_memory_search(ip2region_t, uint_t, datablock_t);
IP2R_API uint_t ip2region_memory_search_string(ip2region_t, char *, datablock_t);
/**
* get the region associated with the specified ip address with binary search algorithm
*
* @param ip2rObj
* @param ip
* @param datablock
* @return uint_t
*/
IP2R_API uint_t ip2region_binary_search(ip2region_t, uint_t, datablock_t); IP2R_API uint_t ip2region_binary_search(ip2region_t, uint_t, datablock_t);
IP2R_API uint_t ip2region_binary_search_string(ip2region_t, char *, datablock_t); IP2R_API uint_t ip2region_binary_search_string(ip2region_t, char *, datablock_t);

View File

@ -0,0 +1,99 @@
/**
* test ip2region searcher program
*
* @author chenxin<chenxin619315@gmail.com>
* @date 2015-10-30
*/
#include "ip2region.h"
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/time.h>
#define __PRINT_ABOUT__ \
println("+-------------------------------------+"); \
println("| ip2region test program |"); \
println("| Author: chenxin619315@gmail.com. |"); \
println("| Type 'quit' to exit the program. |"); \
println("+-------------------------------------+");
//read a line from a command line.
static char *getLine( FILE *fp, char *__dst )
{
register int c;
register char *cs;
cs = __dst;
while ( ( c = getc( fp ) ) != EOF ) {
if ( c == '\n' ) break;
*cs++ = c;
}
*cs = '\0';
return ( c == EOF && cs == __dst ) ? NULL : __dst;
}
static double getTime()
{
struct timeval tv;
struct timezone tz;
gettimeofday(&tv, &tz);
return (tv.tv_sec * 1000 + ((double)tv.tv_usec)/1000);
}
int main( int argc, char **argv )
{
ip2region_entry ip2rEntry;
datablock_entry datablock;
char *dbFile = NULL, *algorithm = NULL;
char line[256];
uint_t (*func_ptr)(ip2region_t, char *, datablock_t);
double s_time, c_time;
memset(&datablock, 0x00, sizeof(datablock_entry));
if ( argc < 2 ) {
printf("Usage: a.out [ip2region db file path] [algorithm]\n");
return 0;
}
dbFile = argv[1];
algorithm = "B-tree";
func_ptr = ip2region_btree_search_string;
if ( argc >= 3 ) {
if ( strcmp(argv[2], "binary") == 0 ) {
algorithm = "File";
func_ptr = ip2region_binary_search_string;
}
}
//create a new ip2rObj
printf("+--initializing %s ... \n", algorithm);
if ( ip2region_create(&ip2rEntry, dbFile) == 0 ) {
println("Error: Fail to create the ip2region object\n");
return 0;
}
__PRINT_ABOUT__;
while ( 1 ) {
print("ip2region>> ");
getLine( stdin, line );
if ( strlen(line) < 2 ) continue;
if ( strcasecmp( line, "quit" ) == 0 ) {
println("+--Bye!");
break;
}
s_time = getTime();
func_ptr(&ip2rEntry, line, &datablock);
c_time = getTime() - s_time;
printf("%d|%s in %.5f millseconds\n", datablock.city_id, datablock.region, c_time);
}
//destory the ip2rObj
ip2region_destroy(&ip2rEntry);
return 0;
}

View File

@ -1,15 +1,15 @@
###golang 实现ip地址查询 ### golang 实现ip地址查询
####获取 #### 获取
``` ```
go get github.com/mohong122/ip2region/binding/golang go get github.com/lionsoul2014/ip2region/binding/golang
``` ```
####使用 #### 使用
```golang ```golang
@ -17,7 +17,7 @@ package main
import ( import (
"fmt" "fmt"
"github.com/mohong122/ip2region/binding/golang" "github.com/lionsoul2014/ip2region/binding/golang/ip2region"
) )
func main() { func main() {
@ -38,7 +38,7 @@ func main() {
``` ```
####返回对象 #### 返回对象
```golang ```golang
type IpInfo struct { type IpInfo struct {
CityId int64 CityId int64
@ -64,7 +64,7 @@ BenchmarkBinarySearch-4| 30000 | 42680 ns/op
``` ```
cd /binging/golang cd /binging/golang
go run main.go ../../data/ip2Region.db go run main.go ../../data/ip2region.db
Or Or

Some files were not shown because too many files have changed in this diff Show More