您的位置: 首页  >  文章  >  机器学习实战之决策树

机器学习实战之决策树

分类: 文章 • 2022-10-28 21:16:24

A simple connection pool based PyMySQL. Mainly focus on multi threads or async mode when use pymysql, but also compatible with single thread mode for convenience when you need to use these two mode together. Within multi threads mode support the multiplexing similar feature(when use connection with Context Manager Protocol).

Problem: When use pymysql with python multi threads, generally we will face the questions:

1、It can't share a connection created by main thread with all sub-threads. It will result error like this: pymysql.err.InternalError: Packet sequence number wrong - got 0 expected 1

2、If we make every sub-thread to create a connection and close it when this sub-thread end, that's workable but obviously lead to high cost on establish connections with MySQL.

So I implement this module aimed at create as least connections as possible with MySQL in multi-threads programing.

This module contain two class:

  • Connection is a subclass of pymysql.connections.Connection, it can use with or without connectionpool, It's usage is all the same with pymysql. The detail(when with connectionpool, it should take additional action to maintain the pool) implement about connection pool is hiddened. This class provide a wrapped execute_query() method for convenience, which take several parameters.


  • ConnectionPool's instance represent the real connection_pool.


Use example

multi-threads mode:

The mainly difference with single-thread mode is that we should maintain the status of the pool. Such as 'get connection from pool' or 'put connection back to pool', in which case there are also some case to deal, such as:

  • when get connection from a pool: we should deal with the timeout and retry parameters

  • when put connection back to pool: if we executed queries without exceptions, this connection can go back to pool directly; but if exception occurred, we should decided whether this connection should go back to pool depend on if it is reusable(base on the exception type). If the connection shouldn't bo back to pool, we close it and recreate a new connection then put it to the pool.

Luckily, this module will take care of these complicated details for you automatic.

There also can create more than one connection_pool(with distinct ConnectionPool.name attribute) to associate with different databases.

In the example below, we will see how it work within connection_pool feature:



    

  1. >>> import pymysql_pool
    2. 

  2. >>> pymysql_pool.logger.setLevel('DEBUG')
    3. 

  3. >>> config={'host':'xxxx', 'user':'xxx', 'password':'xxx', 'database':'xxx', 'antocomit':True}
    4. 

    5. 

  5. ### Create a connection pool with 2 connection in it
    6. 

  6. >>> pool1 = pymysql_pool.ConnectionPool(size=2, name='pool1', **config)
    7. 

  7. >>> pool1.size()
    8. 

  8. 2
    9. 

  9. >>> con1 = pool1.get_connection()
    10. 

  10. 2017-12-25 21:38:48    DEBUG: Get connection from pool(pool1)
    11. 

  11. >>> con2 = pool1.get_connection()
    12. 

  12. 2017-12-25 21:38:51    DEBUG: Get connection from pool(pool1)
    13. 

  13. >>> pool1.size()
    14. 

  14. 0
    15. 

    16. 

  16. ### We can prophesy that here will occur some exception, because the pool1 is empty
    17. 

  17. >>> con3 = pool1.get_connection(timeout=0, retry_num=0)
    18. 

  18. Traceback (most recent call last):
    19. 

  19.   File "e:\github\pymysql-connpool\pymysql_pool.py", line 115, in get_connection
    20. 

  20.     conn = self._pool.get(timeout=timeout) if timeout > 0 else self._pool.get_nowait()
    21. 

  21. queue.Empty
    22. 

    23. 

  23. During handling of the above exception, another exception occurred:
    24. 

    25. 

  25. Traceback (most recent call last):
    26. 

  26.   File "<pyshell#37>", line 1, in <module>
    27. 

  27.     con3 = pool1.get_connection(timeout=0, retry_num=0)
    28. 

  28.   File "e:\github\pymysql-connpool\pymysql_pool.py", line 128, in get_connection
    29. 

  29.     self.name, timeout, total_times))
    30. 

  30. pymysql_pool.GetConnectionFromPoolError: can't get connection from pool(pool1) within 0*1 second(s)
    31. 

    32. 

  32. ### Now let's see the connection's behavior when call close() method and use with Context Manager Protocol
    33. 

  33. >>> con1.close()
    34. 

  34. 2017-12-25 21:39:56    DEBUG: Put connection back to pool(pool1)
    35. 

  35. >>> with con1 as cur:
    36. 

  36.     cur.execute('select 1+1')
    37. 

    38. 

  38. 1
    39. 

  39. 2017-12-25 21:40:25    DEBUG: Put connection back to pool(pool1)
    40. 

  40. ### We can see that the module maintain the pool appropriate when(and only when) we call the close() method or use the Context Manager Protocol of connection object.
    41. 



NOTE 1: We should always use one of the close() method or Context Manager Protocol of connection object, otherwise the pool will exhaust soon. 


NOTE 2: The Context Manager Protocol is preferred, it can achieve the "multiplexing" similar effect. 


NOTE 3: When use close() method, take care never use a connection object's close() method more than one time(you know why~).





作者：jkklee，6年运维老司机一枚，擅长高并发及复杂场景下的故障排查和性能优化。目前比较侧重于将自己的运维积累转化成通用易用的各种工具，希望能帮到更多的运维同胞。


GitHub：https://github.com/jkklee/pymysql-connpool 


机器学习实战之决策树



Python中文社区作为一个去中心化的全球技术社区，以成为全球20万Python中文开发者的精神部落为愿景，目前覆盖各大主流媒体和协作平台，与阿里、腾讯、百度、微软、亚马逊、开源中国、CSDN等业界知名公司和技术社区建立了广泛的联系，拥有来自十多个国家和地区数万名登记会员，会员来自以*部、工信部、清华大学、北京大学、北京邮电大学、中国人民银行、中科院、中金、华为、BAT、谷歌、微软等为代表的*机关、科研单位、金融机构以及海内外知名公司，全平台近20万开发者关注。


为促进Python中文开发者的开源项目的发展，Python中文社区全平台将定期报导华人Python开发者开源项目的发展情况，让更多的开发者关注和参与您的开源项目，并且我们将凭借在业界的影响力努力开拓更多资源，为开源项目开发者提供更多支持，欢迎开源项目开发者踊跃报名，也欢迎您推荐优秀的开源项目，请点击阅读原文报名。




机器学习实战之决策树




往期开源项目介绍


Python开源项目介绍：用zmail简单地发邮件


Python开源项目介绍：网站日志分析工具


Python中文社区开源项目计划：ImagePy





▼点击下方阅读原文参与报名或推荐


															

																					

								

									相关推荐