Attempt to retrieve data via JDBC fails with error Transliteration failed, reason: invalid UTF8 data

Problem retrieving character fields using SQL on when data contains character from a code page that is not defined for the database. Attempts to retrieve data via JDBC fail with error Transliteration failed, reason: invalid UTF8 data, but data is retrieved without error using ABL or ODBC.

Feb 8, 2019
Knowledge

Title

URL Name

000052030

Article Number

000166889

Environment

Product: OpenEdge Version: All supported versions OS: All supported platforms Other: JDBC, SQL-92

Question/Problem Description

Problem retrieving character fields using SQL on when data contains character from a code page that is not defined for the database.

Attempt to retrieve data via JDBC fails with error Transliteration failed, reason: invalid UTF8 data.

Retrieving the same data using ABL or via ODBC completes without error.

Despite completing without error, ABL and ODBC do not show the correct value for the character in question.

ABL session:
ABL wrong character

Microsoft ODBCTest utility (Unicode version):
Microsoft ODBCTest wrong character

Data in question contains extended characters, for example:

Letter	ISO8859-1 / 1252 value	UTF-8 value
Capital N with tilde (Ñ)	D1	c3,91
Capital U with umlaut (Ü)	DC	c3,9c

Exporting .d file seems to show that the data is correct:

"123456789" "" "" "" "" "" "" "" "" 11/11/11 "ABCDEFG" 3000 "AB080" "RÜCKSTELLUNGSKONTEN"
.
PSC
filename=LONGDESC
records=0000000000001
ldbname=testutf
timestamp=YYYY/MM/DD-HH:MM:SS
numformat=44,46
dateformat=mdy-1950
map=NO-MAP
cpstream=utf-8
.
0000000217

Viewing the Hex value of the .d file shows that the extended character has a single-byte value (DC for Ü in the sample above).

Steps to Reproduce

Clarifying Information

Database internal code page is UTF-8.

ABL client session is running with -cpinternal UTF-8 -cpstream UTF-8.

Accessing data with a Java application calling the JDBC.

The same error can be reproduced using the OpenEdge SQLEXP utility and reading the same data.

Error Message

[DataDirect][OpenEdge JDBC Driver]Transliteration failed, reason: invalid UTF8 data

Defect Number

Enhancement Number

Cause

This is expected behavior. The character in the field was from a different code page than that defined for the database.

For example, the .d dumped using UTF-8 shows that a single-byte value has been stored in a database that uses a multi-byte code page. In fact, it should show something similar to the following:

"987654321" "" "" "" "" "" "" "" "" 12/12/12 "gfedcba" 2000 "CD080" "RÃœCKSTELLUNGSKONTEN"

The extended character has been split into 2 bytes (Ã = C3, oe = 9c), correctly representing a conversion to UTF-8.
As a result of the invalid character, the conversion fails and an error or incorrect data is returned.

Resolution

Remove the invalid character and replace it with the correct, corresponding character from the proper code page defined for the database.

For example,

Option 1: Update the field with the correct value via a UTF-8 session (-cpinternal UTF-8 -cpstream UTF-8):

FIND longdesc WHERE <condition>.
ASSIGN LD-DESCEDIT = "RÜCKSTELLUNGSKONTEN".

Option 2: Export, Delete and Import

a. EXPORT the problem record to file and review the character field content, e.g.

OUTPUT TO tabout.d
FIND longdesc WHERE <condition>.
EXPORT longdesc.
OUTPUT CLOSE.

b. Delete the record from the table.

c. With a UTF-8 session (-cpinternal UTF-8 -cpstream UTF-8), reload the record from the Data Dictionary / Administration.

Data Dictionary / Administration should display the message "There is no Code Page defined in this import file".
When asked to supply a code page, enter the relevant single-byte code page that corresponds with the originating locale.
This will result in a conversion from the single-byte code page to UTF-8 during load, correcting the problem.
For example, 1252 or ISO8859-1 would be appropriate for the examples above.

Additionally, to assist in finding other instances where the character in the field does not conform to the code page of the database / client, refer to the code example provided in Article

i18N: ABL code to find field data that does not match the database codepage

Workaround

Notes

References to Other Documentation:

OpenEdge® Development Internationalizing Applications
https://documentation.progress.com/output/OpenEdge117/openedge117/#page/dvint%2Funderstanding-code-pages.html%23

Progress Articles:

Why is there a difference of behavior in handling a foreign language character using SQL client versus using ABL client?
Why does the Progress database allow invalid UTF-8 characters to be inserted?

Keyword Phrase

Last Modified Date

9/19/2025 3:32 PM

Attempt to retrieve data via JDBC fails with error Transliteration failed, reason: invalid UTF8 data

Files(0)

Was this article helpful?

Was this article helpful?

Attempt to retrieve data via JDBC fails with error Transliteration failed, reason: invalid UTF8 data

Information

Related Articles