Mesut's Python problems and solutions
I have or had problems while coding python, you might find some fixes or some tricks
- Permissions and working with raw sockets using pycharm
- Sharing a dictionary between processes on python
- Connecting to MS SQL Server with python
Permissions and working with raw sockets using pycharm
The problem is that raw_sockets require root privileges to initiate. This is not good and creates a lot of fuzz when you do runs on pycharm, you need to go terminal execute the code using sudo and stopping it creates a lot of additional fuzz if you are using threads or processes libraries. And this solution is only for developers that uses linux environments
To overcome this problem;
You can setcap the interpreter but you must be carefull because this is dangerous. After this point all the code you run with this feture will be commited with root privileges so if you are doing packet generation or some crazy stuff you might create network problems. Watch it !!!
sudo setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /usr/bin/python3.10
Or run with sudo in pycharm, use the script below as your interpreter.
There is a catch, you should also require to be a nopassword user in your sudoers config.
-
Get your user name in to sudoers
sudo echo "your_user_name ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/your_user_name
-
Change your pycharm config to use the python-sudo
Copy this and don't forget to change the "home/rammses/PycharmProjects/mikrohelper" section before creating the python-sudo.sh file
#!/bin/bash sudo /home/rammses/PycharmProjects/mikrohelper/venv/bin/python "$@"
- put it into venv/bin folder
- name it python-sudo.sh
- set it executable chmod +x ./python-sudo.sh
- set your project interpreter as shown below
Sharing a dictionary between processes on python
As you might know Python has a problem called
as GIL which means global interpreter lock. This lock prevents sharing variables between processes. Basically process creates another interpreted processs which also means double the memory, and process power.
To overcome this problem, guys at python created a solution called Manager() not only you can share data between processes, you can share data between computers, nice isn't it ?
Quoted from python3 documentation;
Server process managers are more flexible than using shared memory objects because they can be made to support arbitrary object types. Also, a single manager can be shared by processes on different computers over a network. They are, however, slower than using shared memory.
On my case, i needed to update a dictionary on one process and a flask api was going to serve requests based on this dictionary
However there was a problem, when i needed to update nested dictionaries on process i saw that dict was never updated, and look's like the Manager() class has a bug which doesn't update the values on dictionaries. Quoting from python "https://docs.python.org/3/library/multiprocessing.html#proxy-objects"
I did use manager within a context manager, it was initiated like this ;
here is an example to update the manager() owned dictionary ;
class Flow():
def __init__(self):
print('init method called')
self.sample_array = {}
def __enter__(self):
print('enter method called')
self.sample_array['detail']=0
return self
def increase_detail(self,count):
self.sample_array['detail'] += count
def __exit__(self, exc_type, exc_val, exc_tb):
print("exited")
print(self.sample_array)
# def __exit__(self, exc_type, exc_value, exc_traceback):
# print('exit method called')
the increase_details() function represents a stream to update the sample_array['detail'] by 1 on every interval, how ever this wasn't happening and i couldn't find any legitimate solution to this.
What i did was to create a copy of the array, do the nested updates on new array and copy the whole array back to manager. Like this;
def increase_detail(self,count):
# self.sample_array['detail'] += count
_sa_array= self.sample_array
_sa_array['detail'] += count
self.sample_array=_sa_array
Lame but solved my problem, i'd like to know possible solutions to this problem, if you have one mail me mailto mesut
Connecting to MS SQL Server with python
The problem
Sql server connection cant find the sql engine
The last line looks for a sql server engine named SQL SERVER however we don't use it anymore.
Collecting pypyodbc
Using cached https://files.pythonhosted.org/packages/62/94/a5bb72a83366c3249d60c7c465b25cb4252b6be4cc2b13eef048e8e73085/pypyodbc-1.3.6.tar.gz
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from pypyodbc)
Building wheels for collected packages: pypyodbc
Running setup.py bdist_wheel for pypyodbc ... done
Stored in directory: /root/.cache/pip/wheels/bd/68/0f/b23f408ec9ad90e883abc69ae16bf87ac822259de735c9f77c
Successfully built pypyodbc
Installing collected packages: pypyodbc
Successfully installed pypyodbc-1.3.6
root@miharbines:/home/miharbines/Test# python3 dbtest1.py
Traceback (most recent call last):
File "dbtest1.py", line 3, in <module>
connection = pypyodbc.connect('Driver={SQL Server};Server=x.y.z.d;DATABASE=trade;UID=sa;PWD=x.x.x.x.x.x')
File "/usr/local/lib/python3.6/dist-packages/pypyodbc.py", line 2454, in _init_
self.connect(connectString, autocommit, ansi, timeout, unicode_results, readonly)
File "/usr/local/lib/python3.6/dist-packages/pypyodbc.py", line 2507, in connect
check_success(self, ret)
File "/usr/local/lib/python3.6/dist-packages/pypyodbc.py", line 1009, in check_success
ctrl_err(SQL_HANDLE_DBC, ODBC_obj.dbc_h, ret, ODBC_obj.ansi)
File "/usr/local/lib/python3.6/dist-packages/pypyodbc.py", line 983, in ctrl_err
raise OperationalError(state,err_text)
pypyodbc.OperationalError: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'SQL Server' : file not found")
Solution
İnstall MS SQL version 17 db engine from ms repos
sudo -i
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list > /etc/apt/sources.list.d/mssql-release.list
sudo apt update
sudo ACCEPT_EULA=Y apt-get install -y msodbcsql17
sudo apt-get install -y unixodbc-dev
Switch the driver
From
conn = pyodbc.connect('DRIVER={SQL Server};SERVER=server_name;DATABASE=database_name;UID=user;PWD=password')
to
conn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER=server_name;DATABASE=database_name;UID=user;PWD=password')
Output
2nd Problem
My friend was using MS SQL 2014 default install and TLS 1.0 was the only option from produced by MS SQL 2014, we had to install the latest service packs so it can support newer TLS version such as tls 1.2 or 1.3
here is the link for turkish one
After patching the server 2014, the connection got established and code worked as expected.