UWSGI cannot write Unicode data in a log file that is redirected from the Stdout log file from Python

I am using uWSGI (2.0.11.2) and Python (3.4.3) to serve my pyramid (1.5.7) application on Ubuntu 14.04. I noticed that I was Error encountered in uWSGI logs related to unicode decoding:

#
# one of the situations when exception is raised is
# when SQLAlchemy (which has set INFO logging level)
# tries to write an SQL statement containing unicode charater
# into log file
#
2016-02-26 16:01:38,734 INFO [sqlalchemy.engine.base.Engine][b'uWSGIWorker5Core0'] BEGIN (implicit)
2016-02-26 16:01:38,735 INFO [sqlalchemy.engine.base.Engine][b'uWSGIWorker5Core0'] SELECT * FROM staging WHERE company_name = %(company_name_1)s AND time = %(time_1)s AND ship_name = %(ship_name_1)s
# exact place (missing line) where SQLAlchemy is trying to print out
# query parameters, which in this case include unicode character
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.4/ logging/__init__.py", line 980, in emit
stream.write(msg)
Unicod eEncodeError:'ascii' codec can't encode character'ú' in position 132: ordinal not in range(128)
Call stack:
File "/home/mk/.virtualenvs/api/lib /python3.4/site-packages/sqltap/wsgi.py", line 42, in __call__
return self.app(environ, start_response)
File "/home/mk/.virtualenvs/api/ lib/python3.4/site-packages/pyramid/router.py", line 242, in __call__
response = self.invoke_subrequest(request, use_tweens=True)
#
# the stack continues...
# full stack here -> https://bpaste.net/show/8e12af790372
#
File "/home/mk/.virtualenvs/api/lib/python3. 4/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
compiled_sql, distilled_params
File "/home/mk/.virtualenvs/api/lib/python3.4/site -packages/sqlalchemy/engine/base.py", line 1100, in _execute_context
sql_util._repr_params(parameters, batches=10)
Unable to print the message and arguments-possible formatting error.
Use the traceback above to help find the error.

I also noticed that writing the same line to the log file generated by Pyramid (uWSGI is not involved) works very well without any errors, and it is inserted correctly Unicode characters.

I use this command to run uWSGI:

/usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals< /pre> 

In the vassals folder, I added the symbolic link uWSGI configuration from my Pyramid application, it looks like this:

[uwsgi] 

host = %h
username = mk
project_name = api
project_root = /shared/projects/python/%(project_name)

env = PYTHONIOENCODING=UTF-8

; this env var is generated based on host name
env = APP_INI_FILE=develop.ini

; folders config
home_folder = /home/%(username)
virtualenv_folder = %(home_folder)/.virtualenvs/%(project_name)
logs_folder = %(home_folder)/logs/%(project_name)
chdir = %(project_root)
socket = %(project_root)/%(project_name).sock
pidfile = %(project_root)/%(project_name).pid
virtualenv = %(virtualenv_folder)
daemonize = %(logs_folder)/uwsgi.log

; core stuff
mas ter = true
vacuum = true
processes = 5
enable-threads = true

; socket conf
chmod-socket = 666 # invoking the One
chown-socket = %(username)
uid = %(username)
gid = %(username)

; log conf
log-reopen = true
logfile-chown = %(username)
logfile-chmod = 644

; app conf
module = wsgi:application
harakiri = 120
max-requests = 500
post-buffering = 1
paste = config:%p
paste-logger = $p

Pyramid's configuration file, All log records are defined as follows:

###
# app configuration
# http://docs.pylonsproject.org/ projects/pyramid/en/1.5-branch/narr/environment.html
###

[DEFAULT]
home_dir = /home/mk

[app:main]
use = egg:api

pyramid.reload_templates = false
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en

sqlalchemy.url = postgre sql://XXX:[email protected]:5432/ze_database?client_encoding=utf8

[server:main]
use = egg:waitress#main
host = 0.0 .0.0
port = 6543

###
# logging configuration
# http://docs.pylonsproject.org/projects/pyramid/en/1.5- branch/narr/logging.html
###

[loggers]
keys = root, sqlalchemy

[handlers]
keys = console, debuglog

[formatters]
keys = generic, short

[logger_root]
level = DEBUG
handlers = console, debuglog

[logger_sqlalchemy]
level = INFO
handlers =
qualname = sqlalchemy.engine
# "level = INFO" logs SQL queries.
# "level = DEBUG" logs SQL queries and results.
# "level = WARN" logs neither. (Recommended for production systems.)

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = DEBUG
formatter = generic

[handler_debuglog]
class = handlers.RotatingFileHandler
args = ('%(home_dir)s/logs/api/pyramid_debug.log','a', 1024000000, 10)
level = DEBUG
formatter = generic

[formatter_generic ]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s

[formatter_short]< br />format = %(asctime)s %(message)s

Finally, my Pyramid wsgi.py file is very simple:

import os
from pyramid.paster import get_app, setup_logging

here = os.path.dirname(os.path.abspath(__file__))
conf = os.path.join( here, os.environ.get('APP_INI_FILE')) # APP_INI_FILE variable is set in uwsgi.ini
setup_logging(conf)

application = get_app(conf,'main')

Essentially, I redirect the application's log to stderr (or stdout, as far as I have noticed, it's the same), and I also put it in a separate file (pyramid_debug. log). In my case, stderr is the log file of the uWSGI daemon, and this is where the error occurs.

Although LC_ALL and related variables are set to en_EN.UTF-8 on the system , But I also tried to use various localization-related environment variables and set them explicitly in my Pyramid's wsgi application, but bad luck-for example, only setting PYTHONIOENCODING = UTF-8 variable in uWSGI configuration solved The problem on my local computer, but it was not resolved on the server after deployment.

The obvious problem here is-how to handle uWSGI properly in this case Write unicode characters in the log file?

Edit the file "/usr/lib/python3.4/logging/__init__.py", 980 lines and change

stream.write(msg)

to

stream .write(msg.encode('utf-8'))

It is very likely that the way the stream type changes should not break your UTF-8 encoding function, but it is actually because of Pythonics. (It seems that regardless of the world Scream "UTF-8" loudly, and Python ignores everyone.)

For example, if you are processing a file, it will ignore all environment variables you try:

< /p>

# test.py
import sys
sys.stdout.write(u'ö ')

Test:

max% python test.py
ö
max% python test.py> f
Traceback (most recent call last):
File "test. py", line 2, in
sys.stdout.write(u'ö ')
UnicodeEncodeError:'ascii' codec can't encode character u'ö' in position 0: ordinal not in range(128)

I am using uWSGI (2.0.11.2) and Python (3.4.3) for my pyramid (1.5 .7) The application provides services. I noticed that I encountered an error in the uWSGI log related to unicode decoding:

#
# one of the situations when exception is raised is
# when SQLAlchemy (which has set INFO logging level)
# tries to write an SQL statement containing unicode charater
# into log file
#
2016-02-26 16:01:38,734 INFO [sqlalchemy.engine .base.Engine][b'uWSGIWorker5Core0'] BEGIN (implicit)
2016-02-26 16:01:38,735 INFO [sqlalchemy.engine.base.Engine][b'uWSGIWorker5Core0'] SELECT * FROM staging WHERE company_name = %(company_name_1)s AND time = %(time_1)s AND ship_name = %(ship_name_1)s
# exact place (missing line) where SQLAlchemy is trying to print out
# query parameters, which in this case include unicode character
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.4/logging/__init__.py ", line 980, in emit
stream.write(msg)
UnicodeEncodeError:'ascii' codec can't encode character'ú' in position 132: ordinal not in range(128)
Call stack:
File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/sqltap/wsgi.py", line 42, in __call__
return self.app (envir on, start_response)
File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/pyramid/router.py", line 242, in __call__
response = self. invoke_subrequest(request, use_tweens=True)
#
# the stack continues...
# full stack here -> https://bpaste.net/show/8e12af790372
#
File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
compiled_sql, distilled_params
File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1100, in _execute_context
sql_util._repr_params(parameters, batches=10 )
Unable to print the message and arguments-possible formatting error.
Use the traceback above to help find the error.

I also noticed that the same line is written to Pyramid The generated log file (not involving uWSGI) works very well without any errors, and the unicode characters are inserted correctly.

I use this command to run uWSGI:

/usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals

In the vassals folder, I started from my Pyramid application Added symbolic link uWSGI configuration, it looks like this:

[uwsgi]

host = %h
username = mk< br />project_name = api
project_root = /shared/projects/python/%(project_name)

env = PYTHONIOENCODING=UTF-8

; this env var is generated based on host name
env = APP_INI_FILE=develop.ini

; folders config
home_folder = /home/%(username)
virtualenv_folder = %(home_folder )/.virtualenvs/%(project_name)
logs_folder = %(home_folder)/logs/%(project_name)
chdir = %(project_root)
socket = %(project_root)/%(project_name ).sock
pidfile = %(project_root)/%(project_name).pid
virtualenv = %(virtualenv_folder)
daemonize = %(logs_folder)/uwsgi.log

; core stuff
master = true
vacuum = true
processes = 5
enable-threads = true

; socket conf
chmod-socket = 666 # invoking the One
chown-socket = %(username)
uid = %(username)
gid = %(username)

; log conf
log-reopen = true
logfile-chown = %(username)
logfile-chmod = 644

; app conf
module = wsgi:application
harakiri = 120
max-requests = 500
post-buffering = 1
paste = config:%p
paste-logger = $p

Pyramid's configuration file, All log records are defined as follows:

###
# app configuration
# http://docs.pylonsproject.org/ projects/pyramid/en/1.5-branch/narr/environment.html
###

[DEFAULT]
home_dir = /home/mk

[app:main]
use = egg:api

pyramid.reload_templates = false
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en

sqlalchemy.url = postgresql://XXX:[email protected]:5432/ze_database?client_encoding=utf8
< br />[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543

###
# logging configuration
# http://docs.pylonsproject.org/projects/pyramid/en/1.5-bra nch/narr/logging.html
###

[loggers]
keys = root, sqlalchemy

[handlers]
keys = console, debuglog

[formatters]
keys = generic, short

[logger_root]
level = DEBUG
handlers = console, debuglog

[logger_sqlalchemy]
level = INFO
handlers =
qualname = sqlalchemy.engine
# "level = INFO" logs SQL queries.
# "level = DEBUG" logs SQL queries and results.
# "level = WARN" logs neither. (Recommended for production systems.)

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = DEBUG
formatter = generic

[handler_debuglog]
class = handlers.RotatingFileHandler
args = ('%(home_dir)s/logs/api/pyramid_debug.log','a', 1024000000, 10)
level = DEBUG
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s

[ formatter_short]
form at = %(asctime)s %(message)s

Finally, my Pyramid wsgi.py file is very simple:

import os
from pyramid.paster import get_app, setup_logging

here = os.path.dirname(os.path.abspath(__file__))
conf = os.path.join(here, os .environ.get('APP_INI_FILE')) # APP_INI_FILE variable is set in uwsgi.ini
setup_logging(conf)

application = get_app(conf,'main')

< p> Essentially, I redirect the application log to stderr (or stdout, as far as I have noticed, it is the same), and I am also in a separate file (pyramid_debug.log) at the same time Write it. In my case, stderr is the log file of the uWSGI daemon, and this is where the error occurs.

Although LC_ALL and related variables are set to en_EN.UTF-8 on the system, I Also tried to use various localization-related environment variables and set them explicitly in my Pyramid's wsgi application, but bad luck-for example, setting only PYTHONIOENCODING=UTF-8 variable in uWSGI configuration solved my local computer The problem above, but it was not resolved on the server after deployment.

The obvious problem here is-how to correctly handle uWSGI writing unicode characters in the log file in this case?

Edit the file "/usr/lib/python3.4/logging/__init__.py", line 980 and change

< p>

stream.write(msg)

to

stream.write(msg.encode('utf-8' ))

It is very likely that the way the stream type changes should not break your UTF-8 encoding function, but it is actually because of Pythonics. (It seems that no matter the world screams "UTF-8" loudly, Python ignores it Everyone.)

For example, if you are processing a file, all environment variables you try will be ignored:

# test.py
import sys
sys.stdout.write(u'ö ')

Test:

max% python test. py
ö
max% python test.py> f
Traceback (most recent call last):
File "test.py", line 2, in
sys.stdout.write(u'ö
')
UnicodeEncodeError:'ascii' codec can't encode character u'ö' in position 0: ordinal not in range(128)

Leave a Comment

Your email address will not be published.