[Airflow] UnicodeEncodeError: 'charmap' codec can't encode characters 오류 원인과 회피방법

airflow 2.x 버전을 설치하여 관리하고 있었는데, 다음과 같은 오류가 발생했다.

UnicodeEncodeError: 'charmap' codec can't encode characters in position 217-220: character maps to <undefined>

실행되기전에 오류가 난것이기 때문에, airflow scheduler 로그에 해당 로그가 남아있었고, 한참 원인을 찾아 헤맸는데 원인은 한글 주석이 영향을 주는것이었다.

[2023-02-01 18:23:52,459] {logging_mixin.py:115} INFO - [2023-02-01 18:23:52,459] {dag.py:2968} INFO - Setting next_dagrun for my_test_dag to 2023-02-01T09:22:00+00:00, run_after=2023-02-01T09:23:00+00:00
[2023-02-01 18:23:52,475] {processor.py:164} ERROR - Got an exception! Propagating...
Traceback (most recent call last):
  File "/home/airflow/py3/lib/python3.9/site-packages/airflow/dag_processing/processor.py", line 155, in _run_file_processor
    result: Tuple[int, int] = dag_file_processor.process_file(
  File "/home/airflow/py3/lib/python3.9/site-packages/airflow/utils/session.py", line 71, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/py3/lib/python3.9/site-packages/airflow/dag_processing/processor.py", line 660, in process_file
    dagbag.sync_to_db()
  File "/home/airflow/py3/lib/python3.9/site-packages/airflow/utils/session.py", line 71, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/py3/lib/python3.9/site-packages/airflow/models/dagbag.py", line 616, in sync_to_db
    for attempt in run_with_db_retries(logger=self.log):
  File "/home/airflow/py3/lib/python3.9/site-packages/tenacity/__init__.py", line 382, in __iter__
    do = self.iter(retry_state=retry_state)
  File "/home/airflow/py3/lib/python3.9/site-packages/tenacity/__init__.py", line 349, in iter
    return fut.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/home/airflow/py3/lib/python3.9/site-packages/airflow/models/dagbag.py", line 630, in sync_to_db
    DAG.bulk_write_to_db(self.dags.values(), session=session)
  File "/home/airflow/py3/lib/python3.9/site-packages/airflow/utils/session.py", line 68, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/py3/lib/python3.9/site-packages/airflow/models/dag.py", line 2515, in bulk_write_to_db
    session.flush()
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3345, in flush
    self._flush(objects)
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3485, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 3445, in _flush
    flush_context.execute()
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
    rec.execute(self)
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 244, in save_obj
    _emit_insert_statements(
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1096, in _emit_insert_statements
    c = connection._execute_20(
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
    self._handle_dbapi_exception(
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2030, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "/home/airflow/py3/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 719, in do_execute
    cursor.execute(statement, parameters)
  File "/home/airflow/py3/lib/python3.9/site-packages/MySQLdb/cursors.py", line 199, in execute
    args = tuple(map(db.literal, args))
  File "/home/airflow/py3/lib/python3.9/site-packages/MySQLdb/connections.py", line 275, in literal
    s = self.string_literal(o.encode(self.encoding))
  File "/usr/local/lib/python3.9/encodings/cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 217-220: character maps to <undefined>

회피방법

결과적으로는 작성했던 DAG 의 모든 한글을 모두 제거후 동작됨을 확인했다. 특이한건 주석에 있는 한글이 존재해도 이런 오류가 발생했는데, 근본적인 해결방법은 찾지 못했고 일단 한글 주석을 모두 제거하고 실행하는걸 성공했다.

 

주석을 한글로 못달면 불편한게 많기 때문에, 근본적인 해결방법을 찾게되면 다시 알려주도록 하겠다.