UWSGI cannot write Unicode data in a log file that is redirected from the Stdout log file from Python

#
# one of the situations when exception is raised is
# when SQLAlchemy (which has set INFO logging level)
# tries to write an SQL statement containing unicode charater
# into log file
#
2016-02-26 16:01:38,734 INFO [sqlalchemy.engine.base.Engine][b'uWSGIWorker5Core0'] BEGIN (implicit)
2016-02-26 16:01:38,735 INFO [sqlalchemy.engine.base.Engine][b'uWSGIWorker5Core0'] SELECT * FROM staging WHERE company_name = %(company_name_1)s AND time = %(time_1)s AND ship_name = %(ship_name_1)s
# exact place (missing line) where SQLAlchemy is trying to print out
 # query parameters, which in this case include unicode character
--- Logging error ---
Traceback (most recent call last):
 File "/usr/lib/python3.4/ logging/__init__.py", line 980, in emit
 stream.write(msg)
Unicod eEncodeError:'ascii' codec can't encode character'ú' in position 132: ordinal not in range(128)
Call stack:
 File "/home/mk/.virtualenvs/api/lib /python3.4/site-packages/sqltap/wsgi.py", line 42, in __call__
 return self.app(environ, start_response)
 File "/home/mk/.virtualenvs/api/ lib/python3.4/site-packages/pyramid/router.py", line 242, in __call__
 response = self.invoke_subrequest(request, use_tweens=True)
 #
 # the stack continues...
 # full stack here -> https://bpaste.net/show/8e12af790372
 #
 File "/home/mk/.virtualenvs/api/lib/python3. 4/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
 compiled_sql, distilled_params
 File "/home/mk/.virtualenvs/api/lib/python3.4/site -packages/sqlalchemy/engine/base.py", line 1100, in _execute_context
 sql_util._repr_params(parameters, batches=10)
Unable to print the message and arguments-possible formatting error.
 Use the traceback above to help find the error.

/usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals< /pre> 
In the vassals folder, I added the symbolic link uWSGI configuration from my Pyramid application, it looks like this:
 
 
[uwsgi] 

host = %h
username = mk
project_name = api
project_root = /shared/projects/python/%(project_name)

env = PYTHONIOENCODING=UTF-8

; this env var is generated based on host name
env = APP_INI_FILE=develop.ini

; folders config
home_folder = /home/%(username)
virtualenv_folder = %(home_folder)/.virtualenvs/%(project_name)
logs_folder = %(home_folder)/logs/%(project_name)
 chdir = %(project_root)
socket = %(project_root)/%(project_name).sock
pidfile = %(project_root)/%(project_name).pid
virtualenv = %(virtualenv_folder) 
daemonize = %(logs_folder)/uwsgi.log

; core stuff
mas ter = true
vacuum = true
processes = 5
enable-threads = true

; socket conf
chmod-socket = 666 # invoking the One 
chown-socket = %(username)
uid = %(username)
gid = %(username)

; log conf
log-reopen = true
logfile-chown = %(username)
logfile-chmod = 644

; app conf
module = wsgi:application
harakiri = 120 
max-requests = 500
post-buffering = 1
paste = config:%p
paste-logger = $p
Pyramid's configuration file, All log records are defined as follows:

###
# app configuration
# http://docs.pylonsproject.org/ projects/pyramid/en/1.5-branch/narr/environment.html
###

[DEFAULT]
home_dir = /home/mk

[app:main]
use = egg:api

pyramid.reload_templates = false
pyramid.debug_authorization = false
pyramid.debug_notfound = false
 pyramid.debug_routematch = false
pyramid.default_locale_name = en

sqlalchemy.url = postgre sql://XXX:[email protected]:5432/ze_database?client_encoding=utf8

[server:main]
use = egg:waitress#main
host = 0.0 .0.0
port = 6543

###
# logging configuration
# http://docs.pylonsproject.org/projects/pyramid/en/1.5- branch/narr/logging.html
###

[loggers]
keys = root, sqlalchemy

[handlers]
keys = console, debuglog

[formatters]
keys = generic, short

[logger_root]
level = DEBUG
handlers = console, debuglog

[logger_sqlalchemy]
level = INFO
handlers =
qualname = sqlalchemy.engine
# "level = INFO" logs SQL queries.
# "level = DEBUG" logs SQL queries and results.
# "level = WARN" logs neither. (Recommended for production systems.)

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = DEBUG
formatter = generic

[handler_debuglog]
class = handlers.RotatingFileHandler
args = ('%(home_dir)s/logs/api/pyramid_debug.log','a', 1024000000, 10)
level = DEBUG
formatter = generic

[formatter_generic ]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s

[formatter_short]< br />format = %(asctime)s %(message)s
Finally, my Pyramid wsgi.py file is very simple:

import os
from pyramid.paster import get_app, setup_logging

here = os.path.dirname(os.path.abspath(__file__))
conf = os.path.join( here, os.environ.get('APP_INI_FILE')) # APP_INI_FILE variable is set in uwsgi.ini
setup_logging(conf)

application = get_app(conf,'main') 
 Essentially, I redirect the application's log to stderr (or stdout, as far as I have noticed, it's the same), and I also put it in a separate file (pyramid_debug. log). In my case, stderr is the log file of the uWSGI daemon, and this is where the error occurs.
 
Although LC_ALL and related variables are set to en_EN.UTF-8 on the system , But I also tried to use various localization-related environment variables and set them explicitly in my Pyramid's wsgi application, but bad luck-for example, only setting PYTHONIOENCODING = UTF-8 variable in uWSGI configuration solved The problem on my local computer, but it was not resolved on the server after deployment.
 
The obvious problem here is-how to handle uWSGI properly in this case Write unicode characters in the log file? 
 

stream.write(msg)

stream .write(msg.encode('utf-8'))

# test.py
import sys
sys.stdout.write(u'ö
')

max% python test.py
ö
max% python test.py> f
Traceback (most recent call last):
 File "test. py", line 2, in 
 sys.stdout.write(u'ö
')
UnicodeEncodeError:'ascii' codec can't encode character u'ö' in position 0: ordinal not in range(128)

I am using uWSGI (2.0.11.2) and Python (3.4.3) for my pyramid (1.5 .7) The application provides services. I noticed that I encountered an error in the uWSGI log related to unicode decoding:

#
# one of the situations when exception is raised is
# when SQLAlchemy (which has set INFO logging level)
# tries to write an SQL statement containing unicode charater
# into log file
#
2016-02-26 16:01:38,734 INFO [sqlalchemy.engine .base.Engine][b'uWSGIWorker5Core0'] BEGIN (implicit)
2016-02-26 16:01:38,735 INFO [sqlalchemy.engine.base.Engine][b'uWSGIWorker5Core0'] SELECT * FROM staging WHERE company_name = %(company_name_1)s AND time = %(time_1)s AND ship_name = %(ship_name_1)s
# exact place (missing line) where SQLAlchemy is trying to print out
# query parameters, which in this case include unicode character
--- Logging error ---
Traceback (most recent call last):
 File "/usr/lib/python3.4/logging/__init__.py ", line 980, in emit
 stream.write(msg)
UnicodeEncodeError:'ascii' codec can't encode character'ú' in position 132: ordinal not in range(128)
Call stack:
 File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/sqltap/wsgi.py", line 42, in __call__
 return self.app (envir on, start_response)
 File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/pyramid/router.py", line 242, in __call__
 response = self. invoke_subrequest(request, use_tweens=True)
 #
 # the stack continues...
 # full stack here -> https://bpaste.net/show/8e12af790372
 # 
 File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
 compiled_sql, distilled_params
 File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1100, in _execute_context
 sql_util._repr_params(parameters, batches=10 )
Unable to print the message and arguments-possible formatting error.
Use the traceback above to help find the error.

I also noticed that the same line is written to Pyramid The generated log file (not involving uWSGI) works very well without any errors, and the unicode characters are inserted correctly.

I use this command to run uWSGI:

/usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals

In the vassals folder, I started from my Pyramid application Added symbolic link uWSGI configuration, it looks like this:

[uwsgi]

host = %h
username = mk< br />project_name = api
project_root = /shared/projects/python/%(project_name)

env = PYTHONIOENCODING=UTF-8

; this env var is generated based on host name
env = APP_INI_FILE=develop.ini

; folders config
home_folder = /home/%(username)
virtualenv_folder = %(home_folder )/.virtualenvs/%(project_name)
logs_folder = %(home_folder)/logs/%(project_name)
chdir = %(project_root)
socket = %(project_root)/%(project_name ).sock
pidfile = %(project_root)/%(project_name).pid
virtualenv = %(virtualenv_folder)
daemonize = %(logs_folder)/uwsgi.log

; core stuff
master = true
vacuum = true
processes = 5
enable-threads = true

; socket conf
 chmod-socket = 666 # invoking the One
chown-socket = %(username)
uid = %(username)
gid = %(username)

; log conf
log-reopen = true
logfile-chown = %(username)
logfile-chmod = 644

; app conf
module = wsgi:application
harakiri = 120 
max-requests = 500
post-buffering = 1
paste = config:%p
paste-logger = $p

Pyramid's configuration file, All log records are defined as follows:

###
# app configuration
# http://docs.pylonsproject.org/ projects/pyramid/en/1.5-branch/narr/environment.html
###

[DEFAULT]
home_dir = /home/mk

[app:main]
use = egg:api

pyramid.reload_templates = false
pyramid.debug_authorization = false
pyramid.debug_notfound = false
 pyramid.debug_routematch = false
pyramid.default_locale_name = en

sqlalchemy.url = postgresql://XXX:[email protected]:5432/ze_database?client_encoding=utf8
< br />[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543

###
 # logging configuration
# http://docs.pylonsproject.org/projects/pyramid/en/1.5-bra nch/narr/logging.html
###

[loggers]
keys = root, sqlalchemy

[handlers]
keys = console, debuglog

[formatters]
keys = generic, short

[logger_root]
level = DEBUG
handlers = console, debuglog

[logger_sqlalchemy]
level = INFO
handlers =
qualname = sqlalchemy.engine
# "level = INFO" logs SQL queries.
# "level = DEBUG" logs SQL queries and results.
# "level = WARN" logs neither. (Recommended for production systems.)

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = DEBUG
formatter = generic

[handler_debuglog]
class = handlers.RotatingFileHandler
args = ('%(home_dir)s/logs/api/pyramid_debug.log','a', 1024000000, 10)
level = DEBUG
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s

[ formatter_short]
form at = %(asctime)s %(message)s

Finally, my Pyramid wsgi.py file is very simple:

import os
from pyramid.paster import get_app, setup_logging

here = os.path.dirname(os.path.abspath(__file__))
conf = os.path.join(here, os .environ.get('APP_INI_FILE')) # APP_INI_FILE variable is set in uwsgi.ini
setup_logging(conf)

application = get_app(conf,'main')

< p> Essentially, I redirect the application log to stderr (or stdout, as far as I have noticed, it is the same), and I am also in a separate file (pyramid_debug.log) at the same time Write it. In my case, stderr is the log file of the uWSGI daemon, and this is where the error occurs.

Although LC_ALL and related variables are set to en_EN.UTF-8 on the system, I Also tried to use various localization-related environment variables and set them explicitly in my Pyramid's wsgi application, but bad luck-for example, setting only PYTHONIOENCODING=UTF-8 variable in uWSGI configuration solved my local computer The problem above, but it was not resolved on the server after deployment.

The obvious problem here is-how to correctly handle uWSGI writing unicode characters in the log file in this case?

Edit the file "/usr/lib/python3.4/logging/__init__.py", line 980 and change

< p>

stream.write(msg)

to

stream.write(msg.encode('utf-8' ))

It is very likely that the way the stream type changes should not break your UTF-8 encoding function, but it is actually because of Pythonics. (It seems that no matter the world screams "UTF-8" loudly, Python ignores it Everyone.)

For example, if you are processing a file, all environment variables you try will be ignored:

# test.py
import sys
sys.stdout.write(u'ö
')

Test:

max% python test. py
ö
max% python test.py> f
Traceback (most recent call last):
 File "test.py", line 2, in 
 sys.stdout.write(u'ö

')
UnicodeEncodeError:'ascii' codec can't encode character u'ö' in position 0: ordinal not in range(128)